output: add metrics for number of writing events in secondary

[Watson1978]

Which issue(s) this PR fixes:
Fixes #

What this PR does / why we need it:
Add metrics for number of writing events in secondary to Output plugins
This metrics can be used to determine whether recovery is required from secondary if failure occurs.

Docs Changes:
fluent/fluentd-docs-gitbook#582

Release Note:
Same as the title.

[kenhys]

LGTM.

[daipom]

Hmm, we may need to work out the specifications in more detail.
For example...

• What this metrics mean exactly:
  • Is it really Number of stored chunks in secondary?
  • Maybe we can express it less misleadingly?
• The value increases in commit_write, but is it correct?
  • I'm concerned that the backup process calls commit_write as well.
  • What happens if the secondary uses delayed_commit?

For example, if a backup occurs in the secondary and it is also counted up as this metrics, it could be OK as a specification of this metrics.
What we need would be to sort out those specifications.

[Watson1978]

I implemented it thinking that the more accurate the value, it is the better.
but if we could realize the number of trying to write secondary, we can realize that restoration is necessary.
It is enough to know that the value is not accurate.

OK, I will improve this PR.

[daipom]

Should we add these getters?
I'm concerned that we don't have some already, though...

fluentd/lib/fluent/plugin/output.rb

Lines 174 to 196 in 0b72a77

	def num_errors
	@num_errors_metrics.get
	end
	def emit_count
	@emit_count_metrics.get
	end
	def emit_size
	@emit_size_metrics.get
	end
	def emit_records
	@emit_records_metrics.get
	end
	def write_count
	@write_count_metrics.get
	end
	def rollback_count
	@rollback_count_metrics.get
	end

[daipom]

Should we add these getters? I'm concerned that we don't have some already, though...

fluentd/lib/fluent/plugin/output.rb

Lines 174 to 196 in 0b72a77

	def num_errors
	@num_errors_metrics.get
	end
	def emit_count
	@emit_count_metrics.get
	end
	def emit_size
	@emit_size_metrics.get
	end
	def emit_records
	@emit_records_metrics.get
	end
	def write_count
	@write_count_metrics.get
	end
	def rollback_count
	@rollback_count_metrics.get
	end

It seems that we should add the getter because it is unnatural that write_second_count does not have a getter although write_count has a getter.

[Watson1978]

Should we add these getters?

I think we can remove these getter methods.
If it defined a getter method, I think we should use appropriately in the following section as well.

fluentd/lib/fluent/plugin/output.rb

Lines 1590 to 1601 in d066438

	stats = {
	'emit_records' => @emit_records_metrics.get,
	'emit_size' => @emit_size_metrics.get,
	# Respect original name
	# https://github.com/fluent/fluentd/blob/45c7b75ba77763eaf87136864d4942c4e0c5bfcd/lib/fluent/plugin/in_monitor_agent.rb#L284
	'retry_count' => @num_errors_metrics.get,
	'emit_count' => @emit_count_metrics.get,
	'write_count' => @write_count_metrics.get,
	'write_secondary_count' => @write_secondary_count_metrics.get,
	'rollback_count' => @rollback_count_metrics.get,
	'slow_flush_count' => @slow_flush_count_metrics.get,
	'flush_time_count' => @flush_time_count_metrics.get,

Look like we have use these getter methods only used in test codes.
Is it really necessary to define it?

[Watson1978]

Since it is easy to forget to add this manually, it would be better if the getter were generated automatically.

I will try that with another PR.
#4978

[daipom]

LGTM. Thanks!

[daipom]

@Watson1978 Could you update https://docs.fluentd.org/input/monitor_agent#output-example?
It looks very old 😢

[Watson1978]

@Watson1978 Could you update https://docs.fluentd.org/input/monitor_agent#output-example? It looks very old 😢

OK, I see :)

[Watson1978]

I will update the doc after #4981 .
Because its PR add the another metrics too into output.

[daipom]

I see!

[Watson1978]

I created PR to update the doc
fluent/fluentd-docs-gitbook#582