[docker] container & volume metrics (wraps #2740)

[hkaj]

Based on the great work of @parkr in #2740

and

• fix tests & conflicts
• change how we name and tag the metrics a little
• kill some dead code

[hkaj]

test failure unrelated

[parkr]

Looks good! Thanks for picking this up.

If there is no way to test this on Travis, were you able to get tests for the daemon working locally?

[hkaj]

@parkr yes i tested it locally. Testing it on travis would require sudo and docker, but to speed up tests we're using their container infrastructure which doesn't support sudo and DinD (wise decision, but painful here).

[parkr]

@hkaj Hey! Anything I can do to help get this past the finish line?

[parkr]

Tried this on a host with 5.10.1 installed:

2017-01-09 10:50:12 PST | ERROR | dd.collector | checks.docker_daemon(__init__.py:784) | Check 'docker_daemon' instance #0 failed
Traceback (most recent call last):
  File "/opt/datadog-agent/agent/checks/__init__.py", line 767, in run
    self.check(copy.deepcopy(instance))
  File "/etc/dd-agent/checks.d/docker_daemon.py", line 255, in check
    self._report_performance_metrics(containers_by_id)
  File "/etc/dd-agent/checks.d/docker_daemon.py", line 580, in _report_performance_metrics
    self._report_cgroup_metrics(container, tags)
  File "/etc/dd-agent/checks.d/docker_daemon.py", line 598, in _report_cgroup_metrics
    stat_file = self._get_cgroup_from_proc(cgroup["cgroup"], container['_pid'], cgroup['file'])
  File "/etc/dd-agent/checks.d/docker_daemon.py", line 861, in _get_cgroup_from_proc
    return DockerUtil.find_cgroup_from_proc(self._mountpoints, pid, cgroup, self.docker_util._docker_root) % (params)
  File "/opt/datadog-agent/agent/utils/dockerutil.py", line 311, in find_cgroup_from_proc
    with open(proc_path, 'r') as fp:
IOError: [Errno 2] No such file or directory: '/proc/24207/cgroup'

[therc]

@parkr: did you set docker_root: /host in the init_config section of your docker_daemon.yaml? I just wasted a whole hour on this because I had started from a more recent template and erroneously assumed that it was the default setting.

[parkr]

@parkr: did you set docker_root: /host in the init_config section of your docker_daemon.yaml? I just wasted a whole hour on this because I had started from a more recent template and erroneously assumed that it was the default setting.

No, I never set docker_root. In general, /proc/$PID/cgroup is the correct place to look, it's just that it doesn't always contain this file (depending on the process). Sometimes the proc quits before the cgroup info can be read. I think it's just a race condition.

[hkaj]

@hush-hush i rebased it, can you have a look please?

[hush-hush]

This should be on 2 line (that why the CI flake8 tests are falling). End I'm not sure we want to keep it, looks a lot like some debug lines.

[hush-hush]

why do we set values to 0 ? Doesn't that broke test_devicemapper_invalid_values test ?

[parkr]

🎉 Thank you, all!

[hkaj]

Thank you @parkr 🍰

[parkr]

@hkaj Thank you for following up and writing the tests. And for ensuring it got reviewed and merged eventually. 😄 Will this make it into a release in the coming week(s)?

[hkaj]

5.11 is dedicated to releasing the new APM agent so we won't release any change from dd-agent then, but 5.12 will follow soon after.