Ubuntu 20 with kernel version 5.4.0 does not emit application layer metric like http metric

[sumanshil]

kernel version

:~$ uname -r
5.4.0-1063-aws

OS version

$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

However it works with the following kernel version

$ uname -r
6.8.0-1021-aws

[def]

Could you please share the steps to reproduce this, including a sample app and how to run it?

[sumanshil]

To recreate the issue.

1. We can run a container which is making a remote http call.
2. Run coroot agent with the following command.
   sudo docker run --detach --name coroot-node-agent --pull=always --privileged --pid host --network host -v /sys/kernel/debug:/sys/kernel/debug:rw -v /sys/fs/cgroup:/host/sys/fs/cgroup:ro -p 9091:80 -e TRACES_ENDPOINT=http://localhost:5000/traces --network=host ghcr.io/coroot/coroot-node-agent:latest --cgroupfs-root=/host/sys/fs/cgroup
3. Check http://localhost:80/metrics to access the metrics.

[def]

I can't reproduce this:

docker run --rm --name load-generator --net host cylab/wrk2 -d300s -R1 --u_latency http://127.0.0.1

curl http://127.0.0.1/metrics 2>&1 |grep container_http_requests_total
container_http_requests_total{actual_destination="127.0.0.1:80",app_id="",container_id="/docker/load-generator",destination="127.0.0.1:80",machine_id="ab246e0ed33743a9be45ce592a43eed6",status="4xx",system_uuid="ab246e0e-d337-43a9-be45-ce592a43eed6"} 140

[sumanshil]

I can't reproduce this:

docker run --rm --name load-generator --net host cylab/wrk2 -d300s -R1 --u_latency http://127.0.0.1

curl http://127.0.0.1/metrics 2>&1 |grep container_http_requests_total
container_http_requests_total{actual_destination="127.0.0.1:80",app_id="",container_id="/docker/load-generator",destination="127.0.0.1:80",machine_id="ab246e0ed33743a9be45ce592a43eed6",status="4xx",system_uuid="ab246e0e-d337-43a9-be45-ce592a43eed6"} 140

Can you please share the kernel version uname -r and os version cat /etc/os-release of the host where you are running the containers?

[def]

I was using Ubuntu 20.04 with kernel 5.4.0-something-generic. I’ve already shut down the VM, so I can’t check.

[sumanshil]

One more difference between U20 and U24 is that U20 uses cgroupv1 and U24 uses cgourpv2. I am not sure if that makes any difference.