List of Metrics#

This has the list of metrics which are available for usage. In many systems, only a subset of metrics get stored based on how they would be used.

Metric Tiers#

Metrics are grouped by “metricImportance” levels (mzmon-specific). These levels guide which metrics are prioritized in metric stores which have limited capacity.

The essential metrics are the set of metrics that are critical and you would always want to have available. These are used in alerting.

The recommended metrics are the set of metrics that are used in dashboards and are generally desirable for troubleshooting.

The extended set of metrics are used for optional/experimental dashboards.

The diagnostic set of metrics are used for in-depth troubleshooting and analysis.

In our materialize-monitoring configuration, we also provide an all min-importance for including absolutely everything. This is recommended if you have cheaper metric storage like our bundled Thanos provider.

Essential Metrics#

FIXME: Some links for alerts mistakenly point at the common-queries page.

• certmanager_certificate_ready_status
  Used labels: condition
  Example queries:

  • materialize.environmentd.certificate_not_ready
• cilium_bpf_map_pressure
  
  Example queries:

  • infra.cilium.bpf_map_pressure
• cilium_drop_count_total
  
  Example queries:

  • infra.cilium.drop_rate_elevated
• container_cpu_cfs_periods_total
  Used labels: container
  Example queries:

  • materialize.environmentd.cpu_throttled
• container_cpu_cfs_throttled_periods_total
  Used labels: container
  Example queries:

  • materialize.environmentd.cpu_throttled
• container_cpu_usage_seconds_total
  Used labels: container, namespace, pod
  Example queries:

  • infra.kubernetes.pods_high_cpu_ratio
  • infra.kubernetes.vector_high_cpu_ratio
  • materialize.environmentd.high_cpu
  • materialize.kubernetes.cpu.usage.absolute
  • materialize.kubernetes.cpu.usage.percent
  • materialize.kubernetes.pods.cpu_usage
• container_file_descriptors
  Used labels: container
  Example queries:

  • infra.kubernetes.file_descriptors_critical
  • infra.kubernetes.file_descriptors_elevated
  • infra.kubernetes.file_descriptors_warning
• container_fs_limit_bytes
  
  Example queries:

  • infra.kubernetes.container_disk_usage
• container_fs_usage_bytes
  
  Example queries:

  • infra.kubernetes.container_disk_usage
• container_last_seen
  Used labels: materialize_cloud_swap
  Example queries:

  • materialize.clusterd.swap_cluster_oom
• container_memory_swap
  
  Example queries:

  • materialize.clusterd.swap_cluster_oom
• container_memory_working_set_bytes
  Used labels: container, namespace, pod
  Example queries:

  • infra.kubernetes.memory_elevated
  • infra.kubernetes.memory_high
  • materialize.environmentd.high_memory
  • materialize.environmentd.memory_elevated
  • materialize.kubernetes.memory.usage.absolute
  • materialize.kubernetes.memory.usage.percent
  • materialize.kubernetes.pods.memory_usage
• container_network_receive_bytes_total
  Used labels: namespace, pod, workload
  Example queries:

  • infra.egress_gateway.excessive_traffic
  • infra.egress_gateway.high_traffic
  • infra.egress_gateway.low_traffic
  • infra.egress_gateway.traffic_missing_metrics
  • materialize.kubernetes.pods.network_rx
• container_network_receive_errors_total
  Used labels: namespace, pod
  Example queries:

  • materialize.kubernetes.pods.network_errors
• container_network_receive_packets_dropped_total
  Used labels: namespace, pod
  Example queries:

  • materialize.kubernetes.pods.network_drops
• container_network_transmit_bytes_total
  Used labels: namespace, pod
  Example queries:

  • materialize.kubernetes.pods.network_tx
• container_network_transmit_errors_total
  Used labels: namespace, pod
  Example queries:

  • materialize.kubernetes.pods.network_errors
• container_network_transmit_packets_dropped_total
  Used labels: namespace, pod
  Example queries:

  • materialize.kubernetes.pods.network_drops
• container_spec_cpu_period
  Used labels: container, namespace, pod
  Example queries:

  • materialize.environmentd.high_cpu
  • materialize.kubernetes.cpu.capacity
• container_spec_cpu_quota
  Used labels: container, namespace, pod
  Example queries:

  • materialize.environmentd.high_cpu
  • materialize.kubernetes.cpu.capacity
• container_spec_memory_limit_bytes
  Used labels: container, namespace, pod
  Example queries:

  • infra.kubernetes.memory_elevated
  • infra.kubernetes.memory_high
  • materialize.environmentd.high_memory
  • materialize.environmentd.memory_elevated
  • materialize.kubernetes.memory.capacity
  • materialize.kubernetes.memory.usage.percent
  • materialize.kubernetes.pods.memory_usage
• container_spec_memory_swap_limit_bytes
  
  Example queries:

  • materialize.clusterd.swap_cluster_oom
• container_start_time_seconds
  Used labels: container, namespace
  Example queries:

  • materialize.kubernetes.last_restart
• container_ulimits_soft
  Used labels: ulimit
  Example queries:

  • infra.kubernetes.file_descriptors_critical
  • infra.kubernetes.file_descriptors_elevated
  • infra.kubernetes.file_descriptors_warning
• coredns_dns_request_duration_seconds_bucket
  Used labels: zone
  Example queries:

  • infra.coredns.slow_queries
• crdb_dedicated_capacity
  
  Example queries:

  • infra.cockroachdb.disk_usage_critical
  • infra.cockroachdb.disk_usage_elevated
  • infra.cockroachdb.disk_usage_high
• crdb_dedicated_capacity_used
  
  Example queries:

  • infra.cockroachdb.disk_usage_critical
  • infra.cockroachdb.disk_usage_elevated
  • infra.cockroachdb.disk_usage_high
• crdb_dedicated_intentcount
  
  Example queries:

  • infra.cockroachdb.write_intent_accumulation_critical
  • infra.cockroachdb.write_intent_accumulation_high
• crdb_dedicated_ranges_unavailable
  
  Example queries:

  • infra.cockroachdb.ranges_unavailable
• crdb_dedicated_ranges_underreplicated
  
  Example queries:

  • infra.cockroachdb.ranges_underreplicated
• crdb_dedicated_rocksdb_read_amplification
  
  Example queries:

  • infra.cockroachdb.lsm_read_amplification_critical
  • infra.cockroachdb.lsm_read_amplification_high
• crdb_dedicated_schedules_backup_last_completed_time
  
  Example queries:

  • infra.cockroachdb.backup_missing
• crdb_dedicated_sql_mem_distsql_current
  
  Example queries:

  • infra.cockroachdb.sql_memory_pressure_high
  • infra.cockroachdb.sql_memory_rapid_growth
• crdb_dedicated_sql_service_latency_bucket
  
  Example queries:

  • infra.cockroachdb.query_latency
• crdb_dedicated_sys_cpu_combined_percent_normalized
  
  Example queries:

  • infra.cockroachdb.cpu_usage_critical
  • infra.cockroachdb.cpu_usage_high
• crdb_dedicated_sys_totalmem
  
  Example queries:

  • infra.cockroachdb.sql_memory_pressure_high
• kube_deployment_status_condition
  Used labels: condition, deployment, status
  Example queries:

  • infra.kubernetes.deployment_unavailable_critical
  • infra.kubernetes.deployment_unavailable_notice
  • infra.kubernetes.deployment_unavailable_warning
• kube_deployment_status_replicas_ready
  Used labels: namespace
  Example queries:

  • materialize.kubernetes.deployments.readiness
• kube_deployment_status_replicas_unavailable
  Used labels: namespace
  Example queries:

  • materialize.kubernetes.deployments.readiness
• kube_horizontalpodautoscaler_spec_max_replicas
  Used labels: namespace
  Example queries:

  • infra.kubernetes.hpa_replicas_critical
  • infra.kubernetes.hpa_replicas_high
• kube_horizontalpodautoscaler_status_current_replicas
  Used labels: namespace
  Example queries:

  • infra.kubernetes.hpa_replicas_critical
  • infra.kubernetes.hpa_replicas_high
• kube_node_spec_taint
  Used labels: key
  Example queries:

  • infra.kubernetes.node_unreachable
• kube_node_status_condition
  Used labels: condition, status
  Example queries:

  • infra.kubernetes.node_disk_pressure
• kube_pod_container_resource_limits
  Used labels: namespace, pod, resource
  Example queries:

  • materialize.kubernetes.cpu.usage.percent
  • materialize.kubernetes.pods.cpu_usage
• kube_pod_container_resource_requests
  Used labels: container, namespace, pod, resource, unit
  Example queries:

  • infra.kubernetes.daemonset_high_cpu
  • infra.kubernetes.daemonset_saturating_cpu
  • infra.kubernetes.daemonset_saturating_mem
  • infra.kubernetes.pods_high_cpu_ratio
  • infra.kubernetes.vector_high_cpu_ratio
• kube_pod_container_status_last_terminated_exitcode
  Used labels: container, pod
  Example queries:

  • materialize.clusterd.error_kill
  • materialize.clusterd.system_cluster_terminated
  • materialize.environmentd.terminated
• kube_pod_container_status_last_terminated_reason
  Used labels: container, namespace, reason
  Example queries:

  • infra.kubernetes.oomkill_core_systems
  • infra.kubernetes.oomkill_important_systems
  • infra.kubernetes.oomkill_nonessential_systems
  • materialize.clusterd.swap_cluster_oom
• kube_pod_container_status_restarts_total
  Used labels: container, namespace
  Example queries:

  • infra.kubernetes.oomkill_core_systems
  • infra.kubernetes.oomkill_important_systems
  • infra.kubernetes.oomkill_nonessential_systems
  • infra.kubernetes.pod_restart_rate_high
  • infra.kubernetes.pod_restart_rate_high_nonessential
  • materialize.clusterd.error_kill
  • materialize.clusterd.new_restarts_during_release
  • materialize.clusterd.swap_cluster_oom
• kube_pod_container_status_waiting
  
  Example queries:

  • infra.kubernetes.pods_stuck_in_waiting
• kube_pod_created
  
  Example queries:

  • materialize.clusterd.new_restarts_during_release
• kube_pod_start_time
  
  Example queries:

  • infra.kubernetes.pods_stuck_in_waiting
  • materialize.clusterd.not_receiving_commands
  • materialize.environmentd.uptime_sla
  • materialize.environmentd.uptime_slo
• kube_pod_status_phase
  Used labels: namespace, phase
  Example queries:

  • infra.kubernetes.infra_pod_pending
  • materialize.environmentd.pod_pending
  • materialize.environmentd.pod_pending_critical
  • materialize.kubernetes.pods.readiness
• kube_statefulset_status_replicas_ready
  Used labels: namespace
  Example queries:

  • materialize.kubernetes.statefulsets.ready
• kubelet_node_name
  Used labels: workload
  Example queries:

  • infra.egress_gateway.no_traffic
• kubelet_volume_stats_capacity_bytes
  Used labels: persistentvolumeclaim
  Example queries:

  • infra.kubernetes.volume_usage
• kubelet_volume_stats_used_bytes
  Used labels: persistentvolumeclaim
  Example queries:

  • infra.kubernetes.volume_usage
• loki_distributor_bytes_received_total
  Used labels: namespace
  Example queries:

  • infra.monitoring.logging_collection_down
• loki_panic_total
  
  Example queries:

  • infra.loki.panics
• loki_request_duration_seconds_bucket
  Used labels: route
  Example queries:

  • infra.loki.req_duration_high
• loki_request_duration_seconds_count
  Used labels: route, status_code
  Example queries:

  • infra.loki.push_err_high
  • infra.loki.req_err_high
  • infra.loki.writer_err_high
• loki_write_dropped_entries_total
  
  Example queries:

  • infra.loki.alloy_log_drops
• mz_adapter_commands
  Used labels: application_name, materialize_cloud_organization_name, status
  Example queries:

  • materialize.connections.adapter.command_rate
  • materialize.connections.adapter.commands_by_application
  • materialize.console.errors
• mz_auth_request_count
  Used labels: path, status
  Example queries:

  • materialize.auth.errors
  • materialize.auth.refresh_failures
• mz_cluster_server_last_command_received
  Used labels: pod, server_name
  Example queries:

  • materialize.clusterd.not_receiving_commands
• mz_dataflow_replica_expiration_remaining_seconds
  Used labels: pod
  Example queries:

  • materialize.clusterd.expiration_7d
• mz_external_calls_count
  Used labels: status
  Example queries:

  • materialize.external_uptime.checker_not_calling
  • materialize.external_uptime.connections_failing
• mz_external_envd_up
  
  Example queries:

  • materialize.external_uptime.envd_unreachable
• mz_memory_limiter_memory_limit_bytes
  Used labels: pod
  Example queries:

  • materialize.clusterd.system_cluster_high_memory
• mz_memory_limiter_memory_usage_bytes
  Used labels: pod
  Example queries:

  • materialize.clusterd.system_cluster_high_memory
• mz_parameter_frontend_last_cse_time_seconds
  
  Example queries:

  • materialize.launchdarkly.stale_cse
• mz_parameter_frontend_last_sse_time_seconds
  
  Example queries:

  • materialize.launchdarkly.stale_sse
• mz_persist_blob_failures
  
  Example queries:

  • materialize.persist.failures
• mz_persist_cmd_failed_count
  
  Example queries:

  • materialize.persist.failures
• mz_persist_columnar_op_count
  Used labels: op, result
  Example queries:

  • materialize.persist.failures
• mz_persist_columnar_validation_count
  Used labels: result
  Example queries:

  • materialize.persist.failures
• mz_persist_compaction_dropped
  
  Example queries:

  • materialize.persist.failures
• mz_persist_compaction_failed
  
  Example queries:

  • materialize.persist.failures
• mz_persist_compaction_noop
  
  Example queries:

  • materialize.persist.failures
• mz_persist_consensus_failures
  
  Example queries:

  • materialize.persist.failures
• mz_persist_external_blob_delete_noop_count
  
  Example queries:

  • materialize.persist.failures
• mz_persist_external_failed_count
  
  Example queries:

  • materialize.persist.failures
• mz_persist_lease_timeout_read
  
  Example queries:

  • materialize.persist.failures
• mz_persist_pushdown_parts_mismatched_stats_count
  
  Example queries:

  • materialize.persist.failures
• mz_persist_schema_cache_fetch_state_count
  
  Example queries:

  • materialize.persist.failures
• mz_persist_shard_unconsolidated_snapshot
  
  Example queries:

  • materialize.persist.failures
• mz_persist_state_update_state_slow_path
  
  Example queries:

  • materialize.persist.failures
• mz_shard_finalization_outstanding
  
  Example queries:

  • materialize.storage.collection_finalization_stuck
• mz_time_to_first_row_seconds_bucket
  Used labels: application_name, instance_id
  Example queries:

  • materialize.console.query_latency
• mz_txn_placeholder_schema_apply
  
  Example queries:

  • materialize.persist.failures
• node_network_receive_bytes_total
  Used labels: device
  Example queries:

  • infra.egress_gateway.excessive_traffic
  • infra.egress_gateway.high_traffic
  • infra.egress_gateway.low_traffic
  • infra.egress_gateway.no_traffic
  • infra.egress_gateway.traffic_missing_metrics
• node_network_transmit_bytes_total
  Used labels: device
  Example queries:

  • infra.egress_gateway.no_traffic
• up
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, job, namespace
  Example queries:

  • infra.monitoring.clusterd_metrics_missing
  • infra.monitoring.critical_metrics_missing
  • materialize.scraper.mzmon.clusterd
  • materialize.scraper.mzmon.environmentd
  • materialize.scraper.mzmon.orchestratord
• v2_mz_can_connect
  
  Example queries:

  • materialize.environmentd.uptime_sla
  • materialize.environmentd.uptime_slo
• v2_mz_envd_up
  
  Example queries:

  • materialize.environmentd.simplest_query
• v2_mz_views_query_successful
  
  Example queries:

  • materialize.environmentd.query_views_critical
  • materialize.environmentd.query_views_warning

Recommended Metrics#

• mz_active_sessions
  Used labels: materialize_cloud_organization_name
  Example queries:

  • materialize.connections.sessions.active
• mz_active_subscribes
  Used labels: materialize_cloud_organization_name
  Example queries:

  • materialize.compute.subscribes.active
• mz_arrangement_maintenance_seconds_total
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.compute.arrangements.maintenance_rate
  • materialize.compute.arrangements.maintenance_rate_by_worker
• mz_arrangement_record_count
  Used labels: collection_id, instance_id, materialize_cloud_organization_name, replica_id
  Example queries:

  • materialize.compute.arrangements.records.system
  • materialize.compute.arrangements.records.transient
  • materialize.compute.arrangements.records.user
• mz_compute_cluster_status
  Used labels: compute_cluster_id, compute_replica_id, compute_replica_name, materialize_cloud_organization_name
  Example queries:

  • materialize.clusters.count
  • materialize.clusters.info
  • materialize.clusters.replicas.count
  • materialize.clusters.replicas.sizes
  • materialize.health.clusters.status.percentage
  • materialize.health.environment.availability.percentage
  • materialize.info.version
• mz_compute_controller_hydration_queue_size
  Used labels: instance_id, materialize_cloud_organization_name, replica_id
  Example queries:

  • materialize.compute.hydration.queue_size
• mz_compute_hydration_time_seconds
  Used labels: hydrated, instance_id, materialize_cloud_organization_name, replica_id
  Example queries:

  • materialize.compute.hydration.slowest_collections
• mz_compute_peek_duration_seconds_bucket
  Used labels: instance_id, materialize_cloud_organization_name
  Example queries:

  • materialize.connections.peek_latency.p50
  • materialize.connections.peek_latency.p90
  • materialize.connections.peek_latency.p99
• mz_compute_replica_history_dataflow_count
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.compute.dataflows.count
  • materialize.compute.dataflows.count_by_worker
• mz_dataflow_elapsed_seconds_total
  Used labels: instance_id, materialize_cloud_organization_name, replica_id
  Example queries:

  • materialize.compute.dataflows.elapsed_rate
• mz_dataflow_wallclock_lag_seconds
  Used labels: instance_id, materialize_cloud_organization_name, quantile, replica_id
  Example queries:

  • materialize.compute.freshness.lag_by_cluster
  • materialize.compute.freshness.top_collections
  • materialize.compute.hydration.currently_hydrating
  • materialize.info.max_lag
• mz_indexes_count
  Used labels: materialize_cloud_organization_name
  Example queries:

  • materialize.compute.indexes.by_type
  • materialize.compute.indexes.count
• mz_mzd_views_count
  Used labels: materialize_cloud_organization_name
  Example queries:

  • materialize.compute.materialized_views.count
• mz_query_total
  Used labels: materialize_cloud_organization_name
  Example queries:

  • materialize.connections.queries.distribution
  • materialize.connections.queries.rate
  • materialize.connections.queries.rate_by_statement
• mz_sink_bytes_committed
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sinks.lag
  • materialize.storage.sinks.throughput
• mz_sink_bytes_staged
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sinks.lag
• mz_sink_iceberg_commit_conflicts
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sinks.iceberg.commit_failures
• mz_sink_iceberg_commit_duration_seconds_bucket
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sinks.iceberg.commit_latency
• mz_sink_iceberg_commit_failures
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sinks.iceberg.commit_failures
• mz_sink_iceberg_data_files_written
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sinks.iceberg.file_rate
• mz_sink_iceberg_delete_files_written
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sinks.iceberg.file_rate
• mz_sink_iceberg_snapshots_committed
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sinks.iceberg.file_rate
• mz_sink_rdkafka_connects
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sinks.kafka.connect_rate
• mz_sink_rdkafka_disconnects
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sinks.kafka.connect_rate
• mz_sink_rdkafka_outbuf_msg_cnt
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sinks.kafka.output_buffer
• mz_sink_rdkafka_txerrs
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sinks.kafka.tx_errors
• mz_source_bytes_received
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sources.bytes_received
• mz_source_messages_received
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sources.ingestion_by_replica
• mz_source_offset_commit_failures
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sources.upstream_errors
• mz_source_offset_committed
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sources.upstream_errors
• mz_source_offset_known
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sources.upstream_errors
• mz_storage_objects
  Used labels: materialize_cloud_organization_name, type
  Example queries:

  • materialize.storage.sinks.by_type
  • materialize.storage.sinks.count
  • materialize.storage.sources.by_type
  • materialize.storage.sources.catalog
  • materialize.storage.sources.count
• mz_tables_count
  Used labels: materialize_cloud_organization_name
  Example queries:

  • materialize.storage.tables.count
• mz_views_count
  Used labels: materialize_cloud_organization_name
  Example queries:

  • materialize.compute.views.count