Skip to content

[docker_daemon] Add 5 opt-in checks for container & volume counts #2740

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
parkr wants to merge 3 commits into from
Closed

[docker_daemon] Add 5 opt-in checks for container & volume counts #2740

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b45ab16
Add 5 opt-in checks to docker_daemon
parkr Aug 9, 2016
ae21c95
Beginning work on docker_daemon tests.
parkr Aug 11, 2016
7c4716b
test_docker_daemon.py: fix syntax errors
parkr Sep 2, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 41 additions & 1 deletion checks.d/docker_daemon.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,11 @@ def init(self):
# Other options
self.collect_image_stats = _is_affirmative(instance.get('collect_images_stats', False))
self.collect_container_size = _is_affirmative(instance.get('collect_container_size', False))
self.collect_container_count = _is_affirmative(instance.get('collect_container_count', False))
self.collect_dead_container_count = _is_affirmative(instance.get('collect_dead_container_count', False))
self.collect_exited_container_count = _is_affirmative(instance.get('collect_exited_container_count', False))
self.collect_volume_count = _is_affirmative(instance.get('collect_volume_count', False))
self.collect_dangling_volume_count = _is_affirmative(instance.get('collect_dangling_volume_count', False))
self.collect_events = _is_affirmative(instance.get('collect_events', True))
self.collect_image_size = _is_affirmative(instance.get('collect_image_size', False))
self.collect_disk_stats = _is_affirmative(instance.get('collect_disk_stats', False))
Expand Down Expand Up @@ -261,6 +266,21 @@ def check(self, instance):
if self.collect_container_size:
self._report_container_size(containers_by_id)

if self.collect_container_count:
self._report_container_count_by_state(containers_by_id)

if self.collect_dead_container_count:
Copy link

@remh remh Aug 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be better to just report the number of containers by state for all states ? It could even be enabled by default as:

  • I believe the number of different possible states is bounded and small
  • We've already collected all the needed information (in containers_by_id)

self._report_container_count_by_state(containers_by_id, state="Dead")

if self.collect_exited_container_count:
self._report_container_count_by_state(containers_by_id, state="Exited")

if self.collect_volume_count:
self._report_volume_count()

if self.collect_dangling_volume_count:
self._report_volume_count(filters={'dangling': True})
Copy link

@remh remh Aug 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like, dangling volumes will already be collected when you call collect_volume_count line 266.

We should likely reuse that so we don't do 2 API calls to the docker API.

Copy link
Contributor Author

@parkr parkr Sep 2, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAICT, the resulting JSON we receive back does not include information about whether the volume is dangling. We cannot therefore filter based on the response of the first function call.


# Collect disk stats from Docker info command
if self.collect_disk_stats:
self._report_disk_stats()
Expand Down Expand Up @@ -491,14 +511,34 @@ def _report_container_size(self, containers_by_id):
tags = self._get_tags(container, PERFORMANCE)
m_func = FUNC_MAP[GAUGE][self.use_histogram]
if "SizeRw" in container:

m_func(self, 'docker.container.size_rw', container['SizeRw'],
tags=tags)
if "SizeRootFs" in container:
m_func(
self, 'docker.container.size_rootfs', container['SizeRootFs'],
tags=tags)

def _report_container_count_by_state(self, containers_by_id, state="Any"):
count = 0
Copy link
Member

@hkaj hkaj Sep 2, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like it's not used, you can remove it safely

tags = {}
filterlambda = lambda x: not self._is_container_excluded(x) and state is "Any" or container["State"] is state
Copy link
Member

@hkaj hkaj Sep 2, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

container["State"] --> x["State"], or rename x to container and self._is_container_excluded(x) --> self._is_container_excluded(container)

filtered = list(filter(filterlambda, containers_by_id))
tags = self._get_tags(filtered[0], PERFORMANCE)

m_func = FUNC_MAP[GAUGE][self.use_histogram]
# Report docker.container.count if state is "Any", otherwise
# report docker.container.state_STATE.count
suffix = ".state_{}".format(state.lower()) if state is not "Any" else ""
Copy link

@remh remh Aug 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any reason not to use a tag instead for the state ?

Copy link
Contributor Author

@parkr parkr Sep 2, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @remh! No particular reason! I am used to a much more basic version of graphite and tags are new. How would you like to proceed here?

Copy link
Member

@hkaj hkaj Sep 2, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @parkr
The way we usually do this is to report docker.containers.count several times, each time with a different state tag. This could look like:

# const
STATES = ['created', 'restarting', 'running', 'paused', 'exited', 'dead']
...
def _report_container_count_by_state(self, containers_by_id):
    tags = self._get_tags(filtered[0], PERFORMANCE)
    m_func = FUNC_MAP[GAUGE][self.use_histogram]
    for state in STATES:
        # the filter changes for each state
        filterlambda = lambda x: not self._is_container_excluded(x) and container["State"] is state
        filtered = list(filter(filterlambda, containers_by_id))
        # add the state tag
        tags.append('state:{}'.format(state)
        m_func(self, 'docker.container.count'.format(suffix), len(filtered), tags=tags)

This way you can sum the reported values of docker.container.count on a graph to get the total count, or split the graph over the state tag and have the detailed count.

Let me know if it's still unclear.

(this snippet is not tested and could be optimized)

m_func(self, 'docker.container{}.count'.format(suffix), len(filtered), tags=tags)

def _report_volume_count(self, filters={}):
volumes = self.docker_client.volumes(filters=filters)
count = len(volumes['Volumes'])

m_func = FUNC_MAP[GAUGE][self.use_histogram]
suffix = '.' + '-'.join(sorted(filters.keys())) if len(filters) is not 0 else ''
m_func(self, 'docker.volumes{}.count'.format(suffix), count)
Copy link

@remh remh Aug 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same thing here, should we use a tag instead ?


def _report_image_size(self, images):
for image in images:
tags = self._get_tags(image, IMAGE)
Expand Down
25 changes: 25 additions & 0 deletions conf.d/docker_daemon.yaml.example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,31 @@ instances:
#
# collect_container_size: false

# Collect the total container count with the docker.containers.count metric.
# Defaults to false.
#
# collect_container_count: false

# Collect the count of all containers in Dead state with the docker.container.state_dead.count metric.
# Defaults to false.
#
# collect_dead_container_count: false

# Collect the count of all containers in Exited state with the docker.container.state_exited.count metric.
# Defaults to false.
#
# collect_exited_container_count: false

# Collect the total volume count with the docker.volumes.count metric.
# Defaults to false.
#
# collect_volume_count: false

# Collect the count of all dangling volumes with the docker.volumes.dangling.count metric.
# Defaults to false.
#
# collect_dangling_volume_count: false

# Collect images stats
# Number of available active images and intermediate images as gauges.
# Defaults to false.
Expand Down
5 changes: 5 additions & 0 deletions tests/checks/integration/test_docker_daemon.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -466,6 +466,11 @@ def test_labels_collection(self):
"collect_labels_as_tags": ["label1"],
"collect_image_size": True,
"collect_images_stats": True,
"collect_container_count": True,
"collect_dead_container_count": True,
"collect_exited_container_count": True,
"collect_volume_count": True,
"collect_dangling_volume_count": True,
},
],
}
Expand Down