How to configure pyroscope to receive more than 1 sample every 10 second?

[ogxd]

Hello,

I have set up a Pyroscope server 1.2.1 (target -all) and I am sending samples from two services with distinct versions (I have added a dynamic tag version). I am using pyroscope-dotnet.

I am noticing on the Pyroscope UI that I either have a sample from the first service or the second, but never both at the same time, as if Pyroscope had a rate limit to receive no more than 1 sample every 10s. Is there such a thing? Can be it modified?
My goal is to be able to "accurately" compare two versions of a service.

Here is my current Pyroscope configuration from /api/v1/status/config (sorry it's a little lengthy):

target: all
api:
    base-url: ""
server:
    http_listen_network: tcp
    http_listen_address: ""
    http_listen_port: 4040
    http_listen_conn_limit: 0
    grpc_listen_network: tcp
    grpc_listen_address: ""
    grpc_listen_port: 9095
    grpc_listen_conn_limit: 0
    tls_cipher_suites: ""
    tls_min_version: ""
    http_tls_config:
        cert: ""
        key: null
        client_ca: ""
        cert_file: ""
        key_file: ""
        client_auth_type: ""
        client_ca_file: ""
    grpc_tls_config:
        cert: ""
        key: null
        client_ca: ""
        cert_file: ""
        key_file: ""
        client_auth_type: ""
        client_ca_file: ""
    register_instrumentation: true
    graceful_shutdown_timeout: 30s
    http_server_read_timeout: 1m0s
    http_server_write_timeout: 1m0s
    http_server_idle_timeout: 2m0s
    grpc_server_max_recv_msg_size: 4194304
    grpc_server_max_send_msg_size: 4194304
    grpc_server_max_concurrent_streams: 100
    grpc_server_max_connection_idle: 2562047h47m16.854775807s
    grpc_server_max_connection_age: 2562047h47m16.854775807s
    grpc_server_max_connection_age_grace: 2562047h47m16.854775807s
    grpc_server_keepalive_time: 2h0m0s
    grpc_server_keepalive_timeout: 20s
    grpc_server_min_time_between_pings: 5m0s
    grpc_server_ping_without_stream_allowed: false
    log_format: logfmt
    log_level: debug
    log_source_ips_enabled: false
    log_source_ips_header: ""
    log_source_ips_regex: ""
    log_request_headers: false
    log_request_at_info_level_enabled: false
    log_request_exclude_headers_list: ""
    http_path_prefix: ""
distributor:
    pushtimeout: 5s
    pool_config:
        client_cleanup_period: 15s
        health_check_ingesters: true
        remote_timeout: 5s
    ring:
        kvstore:
            store: memberlist
            prefix: collectors/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
            etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        heartbeat_period: 15s
        heartbeat_timeout: 1m0s
        instance_id: be-pyroscope-0
        instance_interface_names:
            - eth0
        instance_port: 0
        instance_addr: ""
        instance_enable_ipv6: false
querier:
    pool_config:
        client_cleanup_period: 15s
        health_check_ingesters: true
        remote_timeout: 5s
    query_store_after: 4h0m0s
frontend:
    scheduler_address: ""
    scheduler_dns_lookup_period: 0s
    scheduler_worker_concurrency: 5
    grpc_client_config:
        max_recv_msg_size: 104857600
        max_send_msg_size: 104857600
        grpc_compression: ""
        rate_limit: 0
        rate_limit_burst: 0
        backoff_on_ratelimits: false
        backoff_config:
            min_period: 100ms
            max_period: 10s
            max_retries: 10
        initial_stream_window_size: 63KiB1023B
        initial_connection_window_size: 63KiB1023B
        tls_enabled: false
        tls_cert_path: ""
        tls_key_path: ""
        tls_ca_path: ""
        tls_server_name: ""
        tls_insecure_skip_verify: false
        tls_cipher_suites: ""
        tls_min_version: ""
        connect_timeout: 5s
        connect_backoff_base_delay: 1s
        connect_backoff_max_delay: 5s
    instance_interface_names:
        - eth0
    address: 10.42.133.175
frontend_worker:
    scheduler_address: ""
    dns_lookup_duration: 0s
    id: ""
    grpc_client_config:
        max_recv_msg_size: 104857600
        max_send_msg_size: 104857600
        grpc_compression: ""
        rate_limit: 0
        rate_limit_burst: 0
        backoff_on_ratelimits: false
        backoff_config:
            min_period: 100ms
            max_period: 10s
            max_retries: 10
        initial_stream_window_size: 63KiB1023B
        initial_connection_window_size: 63KiB1023B
        tls_enabled: false
        tls_cert_path: ""
        tls_key_path: ""
        tls_ca_path: ""
        tls_server_name: ""
        tls_insecure_skip_verify: false
        tls_cipher_suites: ""
        tls_min_version: ""
        connect_timeout: 5s
        connect_backoff_base_delay: 1s
        connect_backoff_max_delay: 5s
    max_concurrent: 4
limits:
    ingestion_rate_mb: 4
    ingestion_burst_size_mb: 2
    max_label_name_length: 1024
    max_label_value_length: 2048
    max_label_names_per_series: 30
    max_sessions_per_series: 0
    max_profile_size_bytes: 4194304
    max_profile_stacktrace_samples: 16000
    max_profile_stacktrace_sample_labels: 100
    max_profile_stacktrace_depth: 1000
    max_profile_symbol_value_length: 65535
    distributor_aggregation_window: 0s
    distributor_aggregation_period: 0s
    ingestion_tenant_shard_size: 0
    max_local_series_per_tenant: 0
    max_global_series_per_tenant: 5000
    max_query_lookback: 1w
    max_query_length: 1d
    max_query_parallelism: 0
    max_flamegraph_nodes_default: 8192
    max_flamegraph_nodes_max: 0
    store_gateway_tenant_shard_size: 0
    split_queries_by_interval: 0s
    compactor_blocks_retention_period: 0s
    compactor_split_and_merge_shards: 0
    compactor_split_and_merge_stage_size: 0
    compactor_split_groups: 1
    compactor_tenant_shard_size: 0
    compactor_partial_block_deletion_delay: 1d
    s3_sse_type: ""
    s3_sse_kms_key_id: ""
    s3_sse_kms_encryption_context: ""
    reject_older_than: 1h
    reject_newer_than: 10m
query_scheduler:
    max_outstanding_requests_per_tenant: 100
    querier_forget_delay: 0s
    grpc_client_config:
        max_recv_msg_size: 104857600
        max_send_msg_size: 104857600
        grpc_compression: ""
        rate_limit: 0
        rate_limit_burst: 0
        backoff_on_ratelimits: false
        backoff_config:
            min_period: 100ms
            max_period: 10s
            max_retries: 10
        initial_stream_window_size: 63KiB1023B
        initial_connection_window_size: 63KiB1023B
        tls_enabled: false
        tls_cert_path: ""
        tls_key_path: ""
        tls_ca_path: ""
        tls_server_name: ""
        tls_insecure_skip_verify: false
        tls_cipher_suites: ""
        tls_min_version: ""
        connect_timeout: 5s
        connect_backoff_base_delay: 1s
        connect_backoff_max_delay: 5s
    service_discovery_mode: ring
    ring:
        kvstore:
            store: memberlist
            prefix: collectors/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
            etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        heartbeat_period: 15s
        heartbeat_timeout: 1m0s
        instance_id: be-pyroscope-0
        instance_interface_names:
            - eth0
        instance_port: 0
        instance_addr: ""
        instance_enable_ipv6: false
    max_used_instances: 0
ingester:
    lifecycler:
        ring:
            kvstore:
                store: memberlist
                prefix: collectors/
                consul:
                    host: localhost:8500
                    acl_token: ""
                    http_client_timeout: 20s
                    consistent_reads: false
                    watch_rate_limit: 1
                    watch_burst_size: 1
                    cas_retry_delay: 1s
                etcd:
                    endpoints: []
                    dial_timeout: 10s
                    max_retries: 10
                    tls_enabled: false
                    tls_cert_path: ""
                    tls_key_path: ""
                    tls_ca_path: ""
                    tls_server_name: ""
                    tls_insecure_skip_verify: false
                    tls_cipher_suites: ""
                    tls_min_version: ""
                    username: ""
                    password: ""
                multi:
                    primary: ""
                    secondary: ""
                    mirror_enabled: false
                    mirror_timeout: 2s
            heartbeat_timeout: 1m0s
            replication_factor: 1
            zone_awareness_enabled: false
            excluded_zones: ""
        num_tokens: 128
        heartbeat_period: 5s
        heartbeat_timeout: 1m0s
        observe_period: 0s
        join_after: 0s
        min_ready_duration: 15s
        interface_names:
            - eth0
        enable_inet6: false
        final_sleep: 0s
        tokens_file_path: ""
        availability_zone: ""
        unregister_on_shutdown: true
        readiness_check_ring_health: true
        address: ""
        port: 0
        id: be-pyroscope-0
store_gateway:
    sharding_ring:
        kvstore:
            store: memberlist
            prefix: collectors/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
            etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        heartbeat_period: 15s
        heartbeat_timeout: 1m0s
        instance_id: be-pyroscope-0
        instance_interface_names:
            - eth0
        instance_port: 0
        instance_addr: ""
        instance_enable_ipv6: false
        replication_factor: 1
        tokens_file_path: ""
        zone_awareness_enabled: false
        wait_stability_min_duration: 0s
        wait_stability_max_duration: 5m0s
        instance_availability_zone: ""
        unregister_on_shutdown: true
    bucket_store:
        sync_dir: ./data/pyroscope-sync/
        sync_interval: 15m0s
        tenant_sync_concurrency: 10
        ignore_blocks_within: 3h0m0s
        meta_sync_concurrency: 20
        ignore_deletion_mark_delay: 30m0s
memberlist:
    node_name: ""
    randomize_node_name: true
    stream_timeout: 10s
    retransmit_factor: 4
    pull_push_interval: 30s
    gossip_interval: 200ms
    gossip_nodes: 3
    gossip_to_dead_nodes_time: 30s
    dead_node_reclaim_time: 0s
    compression_enabled: true
    advertise_addr: ""
    advertise_port: 7946
    cluster_label: be-pyroscope-dev-be-pyroscope
    cluster_label_verification_disabled: false
    join_members:
        - dns+be-pyroscope-memberlist.be-pyroscope-dev.svc.cluster.local.:7946
    min_join_backoff: 1s
    max_join_backoff: 1m0s
    max_join_retries: 10
    abort_if_cluster_join_fails: false
    rejoin_interval: 0s
    left_ingesters_timeout: 5m0s
    leave_timeout: 20s
    message_history_buffer_bytes: 0
    bind_addr: []
    bind_port: 7946
    packet_dial_timeout: 2s
    packet_write_timeout: 5s
    tls_enabled: false
    tls_cert_path: ""
    tls_key_path: ""
    tls_ca_path: ""
    tls_server_name: ""
    tls_insecure_skip_verify: false
    tls_cipher_suites: ""
    tls_min_version: ""
pyroscopedb:
    data_path: ./data
    max_block_duration: 1h0m0s
    row_group_target_size: 1342177280
    min_free_disk_gb: 10
    min_disk_available_percentage: 0.05
    enforcement_interval: 5m0s
    disable_enforcement: false
tracing:
    enabled: true
    profiling_enabled: false
overrides_exporter:
    ring:
        kvstore:
            store: memberlist
            prefix: collectors/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
            etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        heartbeat_period: 15s
        heartbeat_timeout: 1m0s
        instance_id: be-pyroscope-0
        instance_interface_names:
            - eth0
        instance_port: 0
        instance_addr: ""
        instance_enable_ipv6: false
        wait_stability_min_duration: 0s
        wait_stability_max_duration: 5m0s
runtime_config:
    period: 10s
    file: /etc/pyroscope/overrides/overrides.yaml
compactor:
    block_ranges:
        - 1h0m0s
        - 2h0m0s
        - 8h0m0s
    block_sync_concurrency: 8
    meta_sync_concurrency: 20
    data_dir: ./data-compactor
    compaction_interval: 30m0s
    compaction_retries: 3
    compaction_concurrency: 1
    first_level_compaction_wait_period: 25m0s
    cleanup_interval: 15m0s
    cleanup_concurrency: 20
    deletion_delay: 12h0m0s
    tenant_cleanup_delay: 0s
    max_compaction_time: 1h0m0s
    no_blocks_file_cleanup_enabled: false
    max_opening_blocks_concurrency: 16
    enabled_tenants: ""
    disabled_tenants: ""
    sharding_ring:
        kvstore:
            store: memberlist
            prefix: collectors/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
            etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        heartbeat_period: 15s
        heartbeat_timeout: 1m0s
        instance_id: be-pyroscope-0
        instance_interface_names:
            - eth0
        instance_port: 0
        instance_addr: ""
        instance_enable_ipv6: false
        wait_stability_min_duration: 0s
        wait_stability_max_duration: 5m0s
        wait_active_instance_timeout: 10m0s
    compaction_jobs_order: smallest-range-oldest-blocks-first
    compaction_split_by: fingerprint
storage:
    backend: ""
    s3:
        endpoint: ""
        region: ""
        bucket_name: ""
        secret_access_key: ""
        access_key_id: ""
        insecure: false
        signature_version: v4
        sse:
            type: ""
            kms_key_id: ""
            kms_encryption_context: ""
        http:
            idle_conn_timeout: 1m30s
            response_header_timeout: 2m0s
            insecure_skip_verify: false
            tls_handshake_timeout: 10s
            expect_continue_timeout: 1s
            max_idle_connections: 100
            max_idle_connections_per_host: 100
            max_connections_per_host: 0
    gcs:
        bucket_name: ""
        service_account: ""
    azure:
        account_name: ""
        account_key: ""
        connection_string: ""
        container_name: ""
        endpoint_suffix: ""
        max_retries: 3
        user_assigned_id: ""
    swift:
        auth_version: 0
        auth_url: ""
        username: ""
        user_domain_name: ""
        user_domain_id: ""
        user_id: ""
        password: ""
        domain_id: ""
        domain_name: ""
        project_id: ""
        project_name: ""
        project_domain_id: ""
        project_domain_name: ""
        region_name: ""
        container_name: ""
        max_retries: 3
        connect_timeout: 10s
        request_timeout: 5s
    cos:
        bucket: ""
        region: ""
        app_id: ""
        endpoint: ""
        secret_key: ""
        secret_id: ""
        http:
            idle_conn_timeout: 1m30s
            response_header_timeout: 2m0s
            insecure_skip_verify: false
            tls_handshake_timeout: 10s
            expect_continue_timeout: 1s
            max_idle_connections: 100
            max_idle_connections_per_host: 100
            max_connections_per_host: 0
    filesystem:
        dir: ./data-shared
    storage_prefix: ""
self_profiling:
    disable_push: true
    mutex_profile_fraction: 5
    block_profile_rate: 5
analytics:
    reporting_enabled: true

Any help would be greatly appreciated! 😄

[ogxd]

Is it multitenancy that I am looking for https://grafana.com/docs/pyroscope/latest/configure-server/about-tenant-ids/ ?

[ogxd]

It does not seem like I can use multitenancy_enabled in this chart. Maybe it is enabled by default, it is not specified.

[ogxd]

Ok I managed to enable multi-tenancy, but this is not really what I expected as it seems I cannot compare two tenants.

[ogxd]

ts=2024-01-18T10:22:00.10813305Z caller=ingest_handler.go:56 level=error msg="pyroscope ingest" err="pushing IngestInput-pprof failed resource_exhausted: push rate limit (4.0 MiB) exceeded while adding 2.3 MiB" orgID=anonymous
I might have found something

[ogxd]

Fixed it by changing these configs:

    limits:
      ingestion_rate_mb: 4 → 64
      ingestion_burst_size_mb: 2 → 32
      max_profile_stacktrace_samples: 16000 → 65536

Of course values depend on usecase, but this is what I needed.