[docker_daemon] Add 5 opt-in checks for container & volume counts

[parkr]

What does this PR do?

This pull request adds 5 new metrics:

• docker.containers.count – equivalent to docker ps -a | wc -l
• docker.containers.state_dead.count – count of all dead containers
• docker.containers.state_exited.count – count of all exited containers
• docker.volumes.count – count of all volumes
• docker.volumes.dangling.count – count of all dangling volumes

Each metric has its own toggle in the settings so you can opt in to each on its own.

Motivation

We've been battling a bit of weirdness with containers and volumes lying around. We use ephemeral docker containers for some operations but found that volumes would be lying around in the "dangling" state, and that some containers weren't properly cleaned up.

Testing Guidelines

An overview on testing
is available in our contribution guidelines.

Additional Notes

I went with one monolithic commit here because the code was too close together for Git to play nicely with cherry-picking the commits. Let me know if that wasn't the right way to go and I can make 5 distinct commits.

[parkr]

/cc @ross

[olivielpeau]

Thanks @parkr! We'll review your changes in details soon.

No problem at all about the monolithic commit, it looks clean enough as it is 😃

We might want to add some tests on this in test_docker_daemon.py, we'll keep you updated

[parkr]

@olivielpeau Thanks! It's been a while since I have done any Python work so pardon the lack of idiomatic Python. I tried to fix it up by using filter and so forth where I can, but still wearing my Ruby hat. 😄 I wasn't able to get the tests fixed up yet, but will try to take a look soon. I'll be on vacation next week so it may have to wait until after that.

[remh]

any reason not to use a tag instead for the state ?

[parkr]

Hey @remh! No particular reason! I am used to a much more basic version of graphite and tags are new. How would you like to proceed here?

[hkaj]

Hi @parkr
The way we usually do this is to report docker.containers.count several times, each time with a different state tag. This could look like:

# const
STATES = ['created', 'restarting', 'running', 'paused', 'exited', 'dead']
...
def _report_container_count_by_state(self, containers_by_id):
    tags = self._get_tags(filtered[0], PERFORMANCE)
    m_func = FUNC_MAP[GAUGE][self.use_histogram]
    for state in STATES:
        # the filter changes for each state
        filterlambda = lambda x: not self._is_container_excluded(x) and container["State"] is state
        filtered = list(filter(filterlambda, containers_by_id))
        # add the state tag
        tags.append('state:{}'.format(state)
        m_func(self, 'docker.container.count'.format(suffix), len(filtered), tags=tags)

This way you can sum the reported values of docker.container.count on a graph to get the total count, or split the graph over the state tag and have the detailed count.

Let me know if it's still unclear.

(this snippet is not tested and could be optimized)

[remh]

Thanks a lot @parkr ! That's pretty cool! Could you also add tests for the metrics you are collecting please ?

[parkr]

Could you also add tests for the metrics you are collecting please ?

I just went looking through other PR's to see how this is best accomplished and didn't find recent examples of PR's which add functionality to this check and add tests. I'll take a stab but it might be terrible.

[hkaj]

looks like it's not used, you can remove it safely

[hkaj]

Hi @parkr
Do you have the bandwidth to finish this up? Otherwise I can take over. Not rushing you, I just want to make sure this PR eventually gets all the ❤️ it deserves (these are some interesting metrics!).

[parkr]

Do you have the bandwidth to finish this up?

I got stuck trying to write tests for this. Couldn't seem to mock the response from the docker API.

[parkr]

#3077, which supersedes this PR, was merged.