How to trace a Python application with eBPF/BCC

Intro

  1. Update an external endpoint every time the application enters a new function.
  2. Dump logs for each transaction and interprets them.
  3. Make all the connections of the micro-services passing through a proxy which can keep of every single process.

Tutorial (25 mins ~):

Requirements

  1. A GCP Account, if you don’t have an account, you can register here, and you will get 300USD as free credits.
  2. Git installed.
  3. Terraform 0.12.* installed in your machine (If you don’t have it, you can download it from here)
  4. I also suggest a general understanding of Terraform and eBPF. However, it is not essential. More information about eBPF can be found here: eBPF intro & eBPF verifier

Tutorial description

Create the infrastructure

NOTE: To run Terraform on GCP, you will need a service account and the related key-file. You can find more information here: https://cloud.google.com/iam/docs/creating-managing-service-accounts https://www.terraform.io/docs/providers/google/index.html
git clone https://github.com/ish-xyz/bcc-python3-profiler-demo
.
├── app
│ └── app.py
├── main.tf
├── provider.tf
└── templates
└── bootstrap.sh.tpl
vi keyfile.json   # Insert the GCP key-file here to set up the authentication

terraform init
terraform plan # NOTE: You should see 0 destroy, if not check your infrastructure
terraform apply -auto-approve
ssh -i demo-key.pub demo-user@{server_ip}

Install BCC and other required packages

sudo -i
cd /home/demo-user/ish-ar.io-demo
yum install bcc-tools wget make -y
export PATH=${PATH}:/usr/share/bcc/tools/
[root@ish-ar-demo-bcc ~]# execsnoopPCOMM PID PPID RET ARGS

Understanding BCC/Uflow

tplist -l $(which python3) # Empty output

Compile Python3 with DTrace

yum install systemtap-sdt-devel yum-utils -y
wget https://www.python.org/ftp/python/3.6.8/Python-3.6.8.tar.xztar -xvf Python-3.6.8.tar.xz rm -rf Python-3.6.8.tar.xz cd Python-3.6.8
./configure --with-dtracemake # You might have some warnings, don't worry about them make install
tplist -l ./python   # You should have an output like this:[root@ish-ar-demo-bcc Python-3.6.8]# tplist -l ./python
b'./python' b'python':b'gc__start'
b'./python' b'python':b'gc__done'
b'./python' b'python':b'line'
b'./python' b'python':b'function__entry'
b'./python' b'python':b'function__return'

Run & Trace your application

[root@ish-ar-demo-bcc Python-3.6.8]# ./python ../app.py 
{'followers': 700, 'likes': 50000, 'pictures': 100, 'engagement': 0.7142857142857143}
unlink /lib/modules/4.18.0-147.3.1.el8_1.x86_64/buildln -s /usr/src/kernels/4.18.0-147.5.1.el8_1.x86_64 /lib/modules/4.18.0-147.3.1.el8_1.x86_64/build
./python ../app.py > /dev/null & 
pythonflow $! # Output

CPU PID TID TIME(us) METHOD
0 21541 21541 0.749 <- ../app.py.login
0 21541 21541 0.749 -> ../app.py.get_likes
0 21541 21541 3.752 <- ../app.py.get_likes
0 21541 21541 3.752 -> ../app.py.get_followers
0 21541 21541 6.755 <- ../app.py.get_followers
0 21541 21541 11.762 <- ../app.py.run_analytics
0 21541 21541 11.762 <- ../app.py.<module>

Considerations

Overhead

Do I need it?

--

--

--

Senior SRE @ Just Eat

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Deploying WordPress application on Kubernetes with AWS RDS using terraform

Live video streaming in Elixir made simple with Membrane

Retry & Circuit Breaker Patterns in C# with Polly

At the eye of the storm: how I helped save people during the disastrous Kerala floods

Harmony OS — A Capable ‘Android Rival’?

Cosets for Finite Groups in Sympy Combinatorics Module

DevOps: All You Need to Know

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Isham Araia

Isham Araia

Senior SRE @ Just Eat

More from Medium

Python “multiprocessing” “Can’t pickle…”

Accessing S3 Buckets from Python | Crimson Macaw

Wrapping your code into a Docker container

Self-harm: install DB2 on a Docker container

IBM Sample Database GSDB download