Skip to main content

Enable Redis Stream Monitors

Overview​

To set up end-to-end monitoring for Redis, you need to gather metrics from two sources: the redis-exporter and redis-stream-monitor. The redis-exporter provides basic functional metrics, while the stream-monitor publishes more descriptive metrics as a separate service.

This guide focuses on deploying the redis-stream-monitor to capture detailed event framework metrics that are useful for monitoring entire events activity under Harness.

Part 1: Deploy Redis Stream Monitor Chart​

The redis stream monitor chart provides descriptive and meaningful metrics about Redis streams used by the Harness event framework.

Pull the Chart​

helm pull oci://us-west1-docker.pkg.dev/gcr-prod/harness-helm-artifacts/redis-stream-monitor --version 1.4.0 --untar

Configure Override Values​

Create a file named override.yaml with the following content:

global:
  monitoring:
    enabled: true
    port: 8889
    path: /metrics
    managedPlatform: "oss"
resources:
  limits:
    memory: 4096Mi
  requests:
    cpu: 1
    memory: 4096Mi
config:
  EVENTS_FRAMEWORK_ENV_NAMESPACE: ""
  STACK_DRIVER_METRICS_PUSH_ENABLED: "true"
  ENV: "SMP"

Install the Chart​

helm install redis-stream-monitor redis-stream-monitor -n <namespace> -f override.yaml

Available Metrics​

The following metrics are captured and made available through this service:

  • redis_streams_length - Number of messages in the stream
  • redis_streams_memory_usage - Memory used by the stream
  • redis_streams_average_message_size - Average size of messages in the stream
  • redis_streams_events_framework_deadletter_queue_size - Size of the dead letter queue
  • redis_streams_consumer_group_pending_count - Number of pending messages in consumer group
  • redis_streams_consumer_group_behind_by_count - How far behind the consumer group is

PromQL Dashboard Queries​

Single Stream Dashboards​

Use these queries to monitor a specific use case by setting the $usecase variable:

avg by(usecaseName) (redis_streams_memory_usage{usecaseName="$usecase"})
avg by(usecaseName) (redis_streams_length{usecaseName="$usecase"})
avg by(usecaseName) (redis_streams_average_message_size{usecaseName="$usecase"})
avg by(usecaseName, consumergroupName) (redis_streams_consumer_group_behind_by_count{usecaseName="$usecase"})
avg by(usecaseName, consumergroupName) (redis_streams_consumer_group_pending_count{usecaseName="$usecase"})
avg by(exported_namespace, usecaseName) (redis_streams_events_framework_deadletter_queue_size{usecaseName="$usecase"})

Multiple Stream Dashboards​

Use these queries to monitor all streams across all use cases:

avg by(usecaseName) (redis_streams_memory_usage{})
avg by(usecaseName) (redis_streams_length{})
avg by(usecaseName) (redis_streams_average_message_size{})
avg by(usecaseName, consumergroupName) (redis_streams_consumer_group_behind_by_count{})
avg by(usecaseName, consumergroupName) (redis_streams_consumer_group_pending_count{})
avg by(exported_namespace, usecaseName) (redis_streams_events_framework_deadletter_queue_size{})

Available Use Cases​

The usecaseName label can be one of the following values:

Expand to view all use cases
  • DEBEZIUM_gitOpsMongo.harness-gitops.applications
  • DEBEZIUM_gitOpsMongo.harness-gitops.utilization_snapshot
  • DEBEZIUM_harnessMongo.harness.applications
  • DEBEZIUM_ng-harness.ng-harness.moduleLicenses
  • DEBEZIUM_ngMongo.ng-harness.moduleLicenses
  • DEBEZIUM_pms-harness.pms-harness.planExecutionsSummary
  • DEBEZIUM_pmsMongo.pms-harness.planExecutionsSummary
  • DEBEZIUM_sscaMongo.ng-harness.instanceNG
  • LICENSES_USAGE_REDIS_EVENT_CONSUMER
  • async_filter_creation
  • cache_refresh
  • cd_deployment_event
  • cf_archive_ff_activation_audit
  • cf_archive_ff_audit
  • cf_create_env
  • cf_create_ff_activation_audit
  • cf_create_ff_audit
  • cf_create_seg_audit
  • cf_delete_env
  • cf_delete_ff_activation_audit
  • cf_delete_ff_audit
  • cf_delete_seg_audit
  • cf_dismiss_anomaly
  • cf_feature_metrics_data
  • cf_git_sync
  • cf_git_sync_now_events
  • cf_patch_ff_activation_audit
  • cf_patch_ff_audit
  • cf_patch_seg_audit
  • cf_proxy_key
  • cf_restore_ff_activation_audit
  • cf_restore_ff_audit
  • cf_svc_updates
  • cf_target_metrics
  • cg_general_event
  • cg_notify_event
  • chaos_change_events
  • ci_orchestration_notify_event
  • entity_activity
  • entity_crud
  • full_sync_stream
  • git_branch_hook_event_stream
  • git_config_stream
  • git_pr_event_stream
  • git_push_event_stream
  • iacm_orchestration_notify_event
  • instance_stats
  • ldap_group_sync
  • modulelicense
  • observer_event_channel
  • orchestration_log
  • pipeline_initiate_node
  • pipeline_interrupt
  • pipeline_interrupt_cd
  • pipeline_interrupt_ci
  • pipeline_interrupt_cv
  • pipeline_interrupt_iacm
  • pipeline_interrupt_pms
  • pipeline_interrupt_sto
  • pipeline_node_advise
  • pipeline_node_advise_cd
  • pipeline_node_advise_ci
  • pipeline_node_advise_cv
  • pipeline_node_advise_iacm
  • pipeline_node_advise_pms
  • pipeline_node_advise_sto
  • pipeline_node_facilitation
  • pipeline_node_facilitation_cd
  • pipeline_node_facilitation_ci
  • pipeline_node_facilitation_cv
  • pipeline_node_facilitation_iacm
  • pipeline_node_facilitation_pms
  • pipeline_node_facilitation_sto
  • pipeline_node_progress
  • pipeline_node_progress_cd
  • pipeline_node_progress_ci
  • pipeline_node_progress_cv
  • pipeline_node_progress_iacm
  • pipeline_node_progress_pms
  • pipeline_node_progress_sto
  • pipeline_node_resume
  • pipeline_node_resume_cd
  • pipeline_node_resume_ci
  • pipeline_node_resume_cv
  • pipeline_node_resume_iacm
  • pipeline_node_resume_pms
  • pipeline_node_resume_sto
  • pipeline_node_start
  • pipeline_node_start_cd
  • pipeline_node_start_ci
  • pipeline_node_start_cv
  • pipeline_node_start_iacm
  • pipeline_node_start_pms
  • pipeline_node_start_sto
  • pipeline_orchestration
  • pipeline_partial_plan_response
  • pipeline_sdk_response
  • pipeline_sdk_spawn
  • pipeline_sdk_step_response
  • pipeline_start_plan
  • plan_notify_event
  • pms_orchestration_notify_event
  • polling_events_stream
  • saml_authorization_assertion
  • setup_usage
  • srm_custom_change
  • srm_statemachine_event
  • sto_orchestration_notify_event
  • trigger_execution_events_stream
  • usermembership
  • webhook_events_stream
  • webhook_request_payload_data

Part 2: Configure Metric Scraping​

Option 1: Using Prometheus Operator (monitoring.coreos.com/v1)​

If you are using the Prometheus Operator with monitoring.coreos.com/v1 CRDs, create a PodMonitor resource to scrape metrics from the redis-stream-monitor pods.

note

If you encounter an error that the CRD is not present, ensure that you install the necessary CRDs when deploying your Prometheus instance. This resource is critical for Prometheus to scrape the metrics.

Create a file named podmonitor.yaml:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: redis-stream-monitor
  namespace: <namespace>
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: redis-stream-monitor
  podMetricsEndpoints:
    - port: "8889"
      interval: 120s
      path: "/metrics"

Apply the PodMonitor:

helm install redis-stream-monitor redis-stream-monitor -n <namespace> -f override.yaml
note

The PodMonitor resource will be automatically created by the Helm chart when managedPlatform: "oss" is configured.

Option 2: Using Other Observability Tools​

For other observability platforms, configure your scraper to:

  • Target port: 8889
  • Metrics path: /metrics
  • Pod selector: app.kubernetes.io/name: redis-stream-monitor

Summary​

You have now deployed the redis-stream-monitor to capture detailed Redis stream metrics. These metrics provide valuable insights into the Harness event framework, including stream length, memory usage, consumer lag, and dead letter queue sizes, enabling proactive monitoring and troubleshooting of event processing in your Harness environment.