✨Adds metrics regarding new style dynamic services timings #6347

GitHK · 2024-09-11T08:18:05Z

What do these changes do?

Adding metrics regarding the times it takes to finish certain operations. The following labels are used for all the metrics (user_id, wallet_id, service_key and service_version).

Metrics based on times:

start_time_duration_seconds
stop_time_duration_seconds
pull_user_services_images_duration_seconds

Metrics based on rates:

output_ports_pull_rate_bps
input_ports_pull_rate_bps
pull_service_state_rate_bps
push_service_state_rate_bps

Below, truncated output of curl director-v2:8000/metrics after starting and stopping a service.

# HELP simcore_service_director_v2_dynamic_services_start_time_duration_seconds time to start dynamic-sidecar
# TYPE simcore_service_director_v2_dynamic_services_start_time_duration_seconds histogram
simcore_service_director_v2_dynamic_services_start_time_duration_seconds_bucket{le="10.0",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 0.0
simcore_service_director_v2_dynamic_services_start_time_duration_seconds_bucket{le="30.0",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_start_time_duration_seconds_bucket{le="60.0",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_start_time_duration_seconds_bucket{le="120.0",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_start_time_duration_seconds_bucket{le="180.0",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_start_time_duration_seconds_bucket{le="300.0",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_start_time_duration_seconds_bucket{le="420.0",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_start_time_duration_seconds_bucket{le="600.0",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_start_time_duration_seconds_bucket{le="900.0",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_start_time_duration_seconds_bucket{le="1200.0",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_start_time_duration_seconds_bucket{le="+Inf",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_start_time_duration_seconds_count{service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_start_time_duration_seconds_sum{service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 14.980425
# HELP simcore_service_director_v2_dynamic_services_start_time_duration_seconds_created time to start dynamic-sidecar
# TYPE simcore_service_director_v2_dynamic_services_start_time_duration_seconds_created gauge
simcore_service_director_v2_dynamic_services_start_time_duration_seconds_created{service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.7262094274215252e+09
# HELP simcore_service_director_v2_dynamic_services_stop_time_duration_seconds time to stop dynamic-sidecar
# TYPE simcore_service_director_v2_dynamic_services_stop_time_duration_seconds histogram
simcore_service_director_v2_dynamic_services_stop_time_duration_seconds_bucket{le="10.0",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 0.0
simcore_service_director_v2_dynamic_services_stop_time_duration_seconds_bucket{le="30.0",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_stop_time_duration_seconds_bucket{le="60.0",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_stop_time_duration_seconds_bucket{le="120.0",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_stop_time_duration_seconds_bucket{le="180.0",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_stop_time_duration_seconds_bucket{le="300.0",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_stop_time_duration_seconds_bucket{le="420.0",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_stop_time_duration_seconds_bucket{le="600.0",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_stop_time_duration_seconds_bucket{le="900.0",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_stop_time_duration_seconds_bucket{le="1200.0",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_stop_time_duration_seconds_bucket{le="+Inf",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_stop_time_duration_seconds_count{service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_stop_time_duration_seconds_sum{service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 21.994429
# HELP simcore_service_director_v2_dynamic_services_stop_time_duration_seconds_created time to stop dynamic-sidecar
# TYPE simcore_service_director_v2_dynamic_services_stop_time_duration_seconds_created gauge
simcore_service_director_v2_dynamic_services_stop_time_duration_seconds_created{service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.7262094669022238e+09
# HELP simcore_service_director_v2_dynamic_services_pull_user_services_images_duration_seconds time to pull docker images
# TYPE simcore_service_director_v2_dynamic_services_pull_user_services_images_duration_seconds histogram
simcore_service_director_v2_dynamic_services_pull_user_services_images_duration_seconds_bucket{le="1.048576e+06",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_pull_user_services_images_duration_seconds_bucket{le="3.145728e+07",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_pull_user_services_images_duration_seconds_bucket{le="6.291456e+07",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_pull_user_services_images_duration_seconds_bucket{le="9.437184e+07",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_pull_user_services_images_duration_seconds_bucket{le="1.2582912e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_pull_user_services_images_duration_seconds_bucket{le="1.572864e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_pull_user_services_images_duration_seconds_bucket{le="2.097152e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_pull_user_services_images_duration_seconds_bucket{le="3.145728e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_pull_user_services_images_duration_seconds_bucket{le="4.194304e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_pull_user_services_images_duration_seconds_bucket{le="5.24288e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_pull_user_services_images_duration_seconds_bucket{le="6.291456e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_pull_user_services_images_duration_seconds_bucket{le="+Inf",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_pull_user_services_images_duration_seconds_count{service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_pull_user_services_images_duration_seconds_sum{service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.1153693199157715
# HELP simcore_service_director_v2_dynamic_services_pull_user_services_images_duration_seconds_created time to pull docker images
# TYPE simcore_service_director_v2_dynamic_services_pull_user_services_images_duration_seconds_created gauge
simcore_service_director_v2_dynamic_services_pull_user_services_images_duration_seconds_created{service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.7262094253042605e+09
# HELP simcore_service_director_v2_dynamic_services_output_ports_pull_rate_bps rate at which output ports were pulled
# TYPE simcore_service_director_v2_dynamic_services_output_ports_pull_rate_bps histogram
simcore_service_director_v2_dynamic_services_output_ports_pull_rate_bps_bucket{le="1.048576e+06",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_output_ports_pull_rate_bps_bucket{le="3.145728e+07",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_output_ports_pull_rate_bps_bucket{le="6.291456e+07",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_output_ports_pull_rate_bps_bucket{le="9.437184e+07",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_output_ports_pull_rate_bps_bucket{le="1.2582912e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_output_ports_pull_rate_bps_bucket{le="1.572864e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_output_ports_pull_rate_bps_bucket{le="2.097152e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_output_ports_pull_rate_bps_bucket{le="3.145728e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_output_ports_pull_rate_bps_bucket{le="4.194304e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_output_ports_pull_rate_bps_bucket{le="5.24288e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_output_ports_pull_rate_bps_bucket{le="6.291456e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_output_ports_pull_rate_bps_bucket{le="+Inf",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_output_ports_pull_rate_bps_count{service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_output_ports_pull_rate_bps_sum{service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 0.0
# HELP simcore_service_director_v2_dynamic_services_output_ports_pull_rate_bps_created rate at which output ports were pulled
# TYPE simcore_service_director_v2_dynamic_services_output_ports_pull_rate_bps_created gauge
simcore_service_director_v2_dynamic_services_output_ports_pull_rate_bps_created{service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.7262094253008142e+09
# HELP simcore_service_director_v2_dynamic_services_input_ports_pull_rate_bps rate at which input ports were pulled
# TYPE simcore_service_director_v2_dynamic_services_input_ports_pull_rate_bps histogram
simcore_service_director_v2_dynamic_services_input_ports_pull_rate_bps_bucket{le="1.048576e+06",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_input_ports_pull_rate_bps_bucket{le="3.145728e+07",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_input_ports_pull_rate_bps_bucket{le="6.291456e+07",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_input_ports_pull_rate_bps_bucket{le="9.437184e+07",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_input_ports_pull_rate_bps_bucket{le="1.2582912e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_input_ports_pull_rate_bps_bucket{le="1.572864e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_input_ports_pull_rate_bps_bucket{le="2.097152e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_input_ports_pull_rate_bps_bucket{le="3.145728e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_input_ports_pull_rate_bps_bucket{le="4.194304e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_input_ports_pull_rate_bps_bucket{le="5.24288e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_input_ports_pull_rate_bps_bucket{le="6.291456e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_input_ports_pull_rate_bps_bucket{le="+Inf",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_input_ports_pull_rate_bps_count{service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_input_ports_pull_rate_bps_sum{service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 0.0
# HELP simcore_service_director_v2_dynamic_services_input_ports_pull_rate_bps_created rate at which input ports were pulled
# TYPE simcore_service_director_v2_dynamic_services_input_ports_pull_rate_bps_created gauge
simcore_service_director_v2_dynamic_services_input_ports_pull_rate_bps_created{service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.7262094344193647e+09
# HELP simcore_service_director_v2_dynamic_services_restore_service_state_rate_bps rate at which service states were recovered
# TYPE simcore_service_director_v2_dynamic_services_restore_service_state_rate_bps histogram
simcore_service_director_v2_dynamic_services_restore_service_state_rate_bps_bucket{le="1.048576e+06",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_restore_service_state_rate_bps_bucket{le="3.145728e+07",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_restore_service_state_rate_bps_bucket{le="6.291456e+07",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_restore_service_state_rate_bps_bucket{le="9.437184e+07",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_restore_service_state_rate_bps_bucket{le="1.2582912e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_restore_service_state_rate_bps_bucket{le="1.572864e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_restore_service_state_rate_bps_bucket{le="2.097152e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_restore_service_state_rate_bps_bucket{le="3.145728e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_restore_service_state_rate_bps_bucket{le="4.194304e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_restore_service_state_rate_bps_bucket{le="5.24288e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_restore_service_state_rate_bps_bucket{le="6.291456e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_restore_service_state_rate_bps_bucket{le="+Inf",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_restore_service_state_rate_bps_count{service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_restore_service_state_rate_bps_sum{service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 4798.210483805624
# HELP simcore_service_director_v2_dynamic_services_restore_service_state_rate_bps_created rate at which service states were recovered
# TYPE simcore_service_director_v2_dynamic_services_restore_service_state_rate_bps_created gauge
simcore_service_director_v2_dynamic_services_restore_service_state_rate_bps_created{service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.7262094253050907e+09
# HELP simcore_service_director_v2_dynamic_services_save_service_state_rate_bps rate at which service states were saved
# TYPE simcore_service_director_v2_dynamic_services_save_service_state_rate_bps histogram
simcore_service_director_v2_dynamic_services_save_service_state_rate_bps_bucket{le="1.048576e+06",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_save_service_state_rate_bps_bucket{le="3.145728e+07",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_save_service_state_rate_bps_bucket{le="6.291456e+07",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_save_service_state_rate_bps_bucket{le="9.437184e+07",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_save_service_state_rate_bps_bucket{le="1.2582912e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_save_service_state_rate_bps_bucket{le="1.572864e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_save_service_state_rate_bps_bucket{le="2.097152e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_save_service_state_rate_bps_bucket{le="3.145728e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_save_service_state_rate_bps_bucket{le="4.194304e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_save_service_state_rate_bps_bucket{le="5.24288e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_save_service_state_rate_bps_bucket{le="6.291456e+08",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_save_service_state_rate_bps_bucket{le="+Inf",service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_save_service_state_rate_bps_count{service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.0
simcore_service_director_v2_dynamic_services_save_service_state_rate_bps_sum{service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 2608.3767898529113
# HELP simcore_service_director_v2_dynamic_services_save_service_state_rate_bps_created rate at which service states were saved
# TYPE simcore_service_director_v2_dynamic_services_save_service_state_rate_bps_created gauge
simcore_service_director_v2_dynamic_services_save_service_state_rate_bps_created{service_key="simcore/services/dynamic/jupyter-math",service_version="3.0.3",user_id="1",wallet_id=""} 1.7262094582022214e+09

Related issue/s

closes Metrics for startup and shutdown of services #6095

How to test

Dev-ops checklist

No ENV changes or I properly updated ENV (read the instruction)

…rics-for-start-and-stop

codecov · 2024-09-11T08:37:27Z

Codecov Report

Attention: Patch coverage is 74.50980% with 39 lines in your changes missing coverage. Please review.

Project coverage is 85.9%. Comparing base (cafbf96) to head (0d2e9ab).
Report is 555 commits behind head on master.

Files with missing lines	Patch %	Lines
...s/dynamic_sidecar/scheduler/_core/_events_utils.py	25.0%	15 Missing ⚠️
...r_v2/modules/dynamic_sidecar/api_client/_public.py	10.0%	9 Missing ⚠️
...vice_director_v2/modules/instrumentation/_utils.py	84.8%	5 Missing ⚠️
...e_director_v2/models/dynamic_services_scheduler.py	71.4%	4 Missing ⚠️
...c_sidecar/scheduler/_core/_events_user_services.py	25.0%	3 Missing ⚠️
.../service-library/src/servicelib/instrumentation.py	0.0%	2 Missing ⚠️
...vice_director_v2/modules/instrumentation/_setup.py	92.8%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##           master   #6347      +/-   ##
=========================================
+ Coverage    84.5%   85.9%    +1.3%     
=========================================
  Files          10    1133    +1123     
  Lines         214   49444   +49230     
  Branches       25     930     +905     
=========================================
+ Hits          181   42508   +42327     
- Misses         23    6775    +6752     
- Partials       10     161     +151

Flag	Coverage Δ
integrationtests	`58.2% <49.6%> (?)`
unittests	`85.0% <74.5%> (+0.4%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
.../service-library/src/servicelib/rabbitmq/_utils.py	`90.6% <ø> (ø)`
...service_director_v2/api/routes/dynamic_services.py	`90.1% <ø> (ø)`
...rc/simcore_service_director_v2/core/application.py	`96.4% <100.0%> (ø)`
...ules/dynamic_sidecar/scheduler/_core/_scheduler.py	`79.2% <100.0%> (ø)`
...ce_director_v2/modules/instrumentation/__init__.py	`100.0% <100.0%> (ø)`
...ice_director_v2/modules/instrumentation/_models.py	`100.0% <100.0%> (ø)`
...vice_dynamic_sidecar/modules/long_running_tasks.py	`94.4% <100.0%> (ø)`
...vice_director_v2/modules/instrumentation/_setup.py	`92.8% <92.8%> (ø)`
.../service-library/src/servicelib/instrumentation.py	`0.0% <0.0%> (ø)`
...c_sidecar/scheduler/_core/_events_user_services.py	`33.3% <25.0%> (ø)`
... and 4 more

... and 1129 files with indirect coverage changes

…rics-for-start-and-stop

sanderegg

Thanks for the first draft. I added a few comments that we can over together.

We cannot have size as a label as this is unbound.
Also the bucket size must be reduced until we gather more experience at least, no need to go so fine grained, otherwise prometheus will blow up.

Let's discuss later.

services/director-v2/src/simcore_service_director_v2/core/application.py

services/director-v2/src/simcore_service_director_v2/models/dynamic_services_scheduler.py

services/director-v2/src/simcore_service_director_v2/modules/instrumentation/_models.py

services/director-v2/src/simcore_service_director_v2/modules/instrumentation/_utils.py

services/dynamic-sidecar/src/simcore_service_dynamic_sidecar/modules/long_running_tasks.py

…rics-for-start-and-stop

services/director-v2/src/simcore_service_director_v2/core/application.py

...simcore_service_director_v2/modules/dynamic_sidecar/scheduler/_core/_events_user_services.py

...-v2/src/simcore_service_director_v2/modules/dynamic_sidecar/scheduler/_core/_events_utils.py

services/director-v2/src/simcore_service_director_v2/modules/instrumentation/_models.py

...-v2/src/simcore_service_director_v2/modules/dynamic_sidecar/scheduler/_core/_events_utils.py

services/director-v2/src/simcore_service_director_v2/modules/instrumentation/_utils.py

services/director-v2/src/simcore_service_director_v2/modules/instrumentation/_models.py

pcrespov

Nice!

…rics-for-start-and-stop

sonarqubecloud · 2024-09-18T08:06:33Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

Andrei Neagu added 2 commits September 11, 2024 10:16

added instrumentation basis

8bb01e9

Merge remote-tracking branch 'upstream/master' into pr-osparc-add-met…

3abec0b

…rics-for-start-and-stop

GitHK self-assigned this Sep 11, 2024

GitHK added the a:director-v2 issue related with the director-v2 service label Sep 11, 2024

GitHK added this to the Eisbock milestone Sep 11, 2024

Andrei Neagu added 7 commits September 11, 2024 13:07

adding metrics for starting and stopping

3baaa63

Merge remote-tracking branch 'upstream/master' into pr-osparc-add-met…

e4e41e3

…rics-for-start-and-stop

pull user service images and output ports with metrics

c05eab2

Merge remote-tracking branch 'upstream/master' into pr-osparc-add-met…

ab1c162

…rics-for-start-and-stop

added metrics for pulling inputs

a064f84

added metrics for restoring service state

cb7584d

Merge remote-tracking branch 'upstream/master' into pr-osparc-add-met…

1f58de7

…rics-for-start-and-stop

sanderegg reviewed Sep 12, 2024

View reviewed changes

Andrei Neagu added 13 commits September 13, 2024 06:59

Merge remote-tracking branch 'upstream/master' into pr-osparc-add-met…

00ff066

…rics-for-start-and-stop

using if in application

9907398

renaming

704c191

remove casting

361b028

refactor code safety

9d37764

refactor to store rate

50d985a

using rate buckets

87a9bbd

refactor bucket names and time buckets

71b6f07

rename subsystem

1303c91

total duration is required not the rate

b0d6e51

using same function to create namespaces

dee080c

adding metric for savign service states

e8f4ea3

Merge remote-tracking branch 'upstream/master' into pr-osparc-add-met…

7abd932

…rics-for-start-and-stop

GitHK commented Sep 13, 2024

View reviewed changes

Andrei Neagu added 2 commits September 13, 2024 08:33

renaming

819d2c6

refactor

de5e046

GitHK marked this pull request as ready for review September 13, 2024 06:49

GitHK requested a review from pcrespov as a code owner September 13, 2024 06:49

GitHK requested review from giancarloromeo and matusdrobuliak66 September 13, 2024 06:49

pcrespov approved these changes Sep 13, 2024

View reviewed changes

Andrei Neagu added 14 commits September 13, 2024 11:38

Merge remote-tracking branch 'upstream/master' into pr-osparc-add-met…

a6f131a

…rics-for-start-and-stop

fixed broken test

26b258a

fixed failing tests

9a7165f

enabling instrumentation

6d25e44

Merge remote-tracking branch 'upstream/master' into pr-osparc-add-met…

fc9fde8

…rics-for-start-and-stop

using epsilon to avoid issues

74e261e

refactred test

70362cf

Merge remote-tracking branch 'upstream/master' into pr-osparc-add-met…

7db3cc8

…rics-for-start-and-stop

fixed broken import

785a7ff

fixed broken test

7571f7d

Merge remote-tracking branch 'upstream/master' into pr-osparc-add-met…

7854bf7

…rics-for-start-and-stop

Merge remote-tracking branch 'upstream/master' into pr-osparc-add-met…

395eacb

…rics-for-start-and-stop

Merge remote-tracking branch 'upstream/master' into pr-osparc-add-met…

6975378

…rics-for-start-and-stop

update docs

0d2e9ab

GitHK modified the milestones: Eisbock, MartinKippenberger Sep 18, 2024

GitHK enabled auto-merge (squash) September 18, 2024 09:23

sanderegg approved these changes Sep 18, 2024

View reviewed changes

GitHK merged commit 9057d01 into ITISFoundation:master Sep 18, 2024
57 checks passed

GitHK deleted the pr-osparc-add-metrics-for-start-and-stop branch September 18, 2024 09:30

matusdrobuliak66 mentioned this pull request Sep 18, 2024

🚀 Release v1.77.0 / v1.77.1 / v1.77.2 #6212

Closed

29 tasks

matusdrobuliak66 mentioned this pull request Oct 30, 2024

🚀 Release v1.78.0 #6441

Closed

57 tasks

✨Adds metrics regarding new style dynamic services timings #6347

✨Adds metrics regarding new style dynamic services timings #6347

Uh oh!

Conversation

GitHK commented Sep 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What do these changes do?

Related issue/s

How to test

Dev-ops checklist

Uh oh!

codecov bot commented Sep 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sanderegg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pcrespov left a comment

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Sep 18, 2024

Quality Gate passed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

GitHK commented Sep 11, 2024 •

edited

Loading

codecov bot commented Sep 11, 2024 •

edited

Loading