List of Metrics#

This has the list of metrics which are available for usage. In many systems, only a subset of metrics get stored based on how they would be used.

Metric Tiers#

Importance is about storage, not stability: it says which metrics are worth keeping when capacity is limited, not what we promise about their names. For that, see Stability Guarantees and Deprecation Policy — in short, mz_* names come from Materialize itself and we disclose rather than freeze them.

Metrics are grouped by “metricImportance” levels (mzmon-specific). These levels guide which metrics are prioritized in metric stores which have limited capacity.

The essential metrics are the set of metrics that are critical and you would always want to have available. These are used in alerting.

The recommended metrics are the set of metrics that are used in dashboards and are generally desirable for troubleshooting.

The extended set of metrics are used for optional/experimental dashboards.

The diagnostic set of metrics are used for in-depth troubleshooting and analysis.

In our materialize-monitoring configuration, we also provide an all min-importance for including absolutely everything. This is recommended if you have cheaper metric storage like our bundled Thanos provider.

Essential Metrics#

FIXME: Some links for alerts mistakenly point at the common-queries page.

• certmanager_certificate_ready_status
  Used labels: condition
  Example queries:

  • materialize.environmentd.certificate_not_ready
• cilium_bpf_map_pressure
  
  Example queries:

  • infra.cilium.bpf_map_pressure
• cilium_drop_count_total
  
  Example queries:

  • infra.cilium.drop_rate_elevated
• container_cpu_cfs_periods_total
  Used labels: container
  Example queries:

  • materialize.environmentd.cpu_throttled
• container_cpu_cfs_throttled_periods_total
  Used labels: container
  Example queries:

  • materialize.environmentd.cpu_throttled
• container_cpu_usage_seconds_total
  Used labels: container, namespace, pod
  Example queries:

  • infra.kubernetes.pods_high_cpu_ratio
  • infra.kubernetes.vector_high_cpu_ratio
  • materialize.environmentd.high_cpu
  • materialize.generations.cpu
  • materialize.kubernetes.cpu.usage.absolute
  • materialize.kubernetes.cpu.usage.percent
  • materialize.kubernetes.pods.cpu_usage
• container_file_descriptors
  Used labels: container
  Example queries:

  • infra.kubernetes.file_descriptors_critical
  • infra.kubernetes.file_descriptors_elevated
  • infra.kubernetes.file_descriptors_warning
• container_fs_limit_bytes
  
  Example queries:

  • infra.kubernetes.container_disk_usage
• container_fs_usage_bytes
  
  Example queries:

  • infra.kubernetes.container_disk_usage
• container_last_seen
  Used labels: materialize_cloud_swap
  Example queries:

  • materialize.clusterd.swap_cluster_oom
• container_memory_swap
  
  Example queries:

  • materialize.clusterd.swap_cluster_oom
• container_memory_working_set_bytes
  Used labels: container, namespace, pod
  Example queries:

  • infra.kubernetes.memory_elevated
  • infra.kubernetes.memory_high
  • materialize.environmentd.high_memory
  • materialize.environmentd.memory_elevated
  • materialize.generations.memory
  • materialize.generations.pods
  • materialize.kubernetes.memory.usage.absolute
  • materialize.kubernetes.memory.usage.percent
  • materialize.kubernetes.pods.memory_usage
• container_network_receive_bytes_total
  Used labels: namespace, pod, workload
  Example queries:

  • infra.egress_gateway.excessive_traffic
  • infra.egress_gateway.high_traffic
  • infra.egress_gateway.low_traffic
  • infra.egress_gateway.traffic_missing_metrics
  • materialize.kubernetes.pods.network_rx
• container_network_receive_errors_total
  Used labels: namespace, pod
  Example queries:

  • materialize.kubernetes.pods.network_errors
• container_network_receive_packets_dropped_total
  Used labels: namespace, pod
  Example queries:

  • materialize.kubernetes.pods.network_drops
• container_network_transmit_bytes_total
  Used labels: namespace, pod
  Example queries:

  • materialize.kubernetes.pods.network_tx
• container_network_transmit_errors_total
  Used labels: namespace, pod
  Example queries:

  • materialize.kubernetes.pods.network_errors
• container_network_transmit_packets_dropped_total
  Used labels: namespace, pod
  Example queries:

  • materialize.kubernetes.pods.network_drops
• container_spec_cpu_period
  Used labels: container, namespace, pod
  Example queries:

  • materialize.environmentd.high_cpu
  • materialize.kubernetes.cpu.capacity
  • materialize.kubernetes.cpu.capacity.all_containers
• container_spec_cpu_quota
  Used labels: container, namespace, pod
  Example queries:

  • materialize.environmentd.high_cpu
  • materialize.kubernetes.cpu.capacity
  • materialize.kubernetes.cpu.capacity.all_containers
• container_spec_memory_limit_bytes
  Used labels: container, namespace, pod
  Example queries:

  • infra.kubernetes.memory_elevated
  • infra.kubernetes.memory_high
  • materialize.environmentd.high_memory
  • materialize.environmentd.memory_elevated
  • materialize.kubernetes.memory.capacity
  • materialize.kubernetes.memory.capacity.all_containers
  • materialize.kubernetes.memory.usage.percent
  • materialize.kubernetes.pods.memory_usage
• container_spec_memory_swap_limit_bytes
  
  Example queries:

  • materialize.clusterd.swap_cluster_oom
• container_start_time_seconds
  Used labels: container, namespace
  Example queries:

  • materialize.kubernetes.last_restart
• container_ulimits_soft
  Used labels: ulimit
  Example queries:

  • infra.kubernetes.file_descriptors_critical
  • infra.kubernetes.file_descriptors_elevated
  • infra.kubernetes.file_descriptors_warning
• coredns_dns_request_duration_seconds_bucket
  Used labels: zone
  Example queries:

  • infra.coredns.slow_queries
• crdb_dedicated_capacity
  
  Example queries:

  • infra.cockroachdb.disk_usage_critical
  • infra.cockroachdb.disk_usage_elevated
  • infra.cockroachdb.disk_usage_high
• crdb_dedicated_capacity_used
  
  Example queries:

  • infra.cockroachdb.disk_usage_critical
  • infra.cockroachdb.disk_usage_elevated
  • infra.cockroachdb.disk_usage_high
• crdb_dedicated_intentcount
  
  Example queries:

  • infra.cockroachdb.write_intent_accumulation_critical
  • infra.cockroachdb.write_intent_accumulation_high
• crdb_dedicated_ranges_unavailable
  
  Example queries:

  • infra.cockroachdb.ranges_unavailable
• crdb_dedicated_ranges_underreplicated
  
  Example queries:

  • infra.cockroachdb.ranges_underreplicated
• crdb_dedicated_rocksdb_read_amplification
  
  Example queries:

  • infra.cockroachdb.lsm_read_amplification_critical
  • infra.cockroachdb.lsm_read_amplification_high
• crdb_dedicated_schedules_backup_last_completed_time
  
  Example queries:

  • infra.cockroachdb.backup_missing
• crdb_dedicated_sql_mem_distsql_current
  
  Example queries:

  • infra.cockroachdb.sql_memory_pressure_high
  • infra.cockroachdb.sql_memory_rapid_growth
• crdb_dedicated_sql_service_latency_bucket
  
  Example queries:

  • infra.cockroachdb.query_latency
• crdb_dedicated_sys_cpu_combined_percent_normalized
  
  Example queries:

  • infra.cockroachdb.cpu_usage_critical
  • infra.cockroachdb.cpu_usage_high
• crdb_dedicated_sys_totalmem
  
  Example queries:

  • infra.cockroachdb.sql_memory_pressure_high
• kube_deployment_status_condition
  Used labels: condition, deployment, status
  Example queries:

  • infra.kubernetes.deployment_unavailable_critical
  • infra.kubernetes.deployment_unavailable_notice
  • infra.kubernetes.deployment_unavailable_warning
• kube_deployment_status_replicas_ready
  Used labels: namespace
  Example queries:

  • materialize.kubernetes.deployments.readiness
• kube_deployment_status_replicas_unavailable
  Used labels: namespace
  Example queries:

  • materialize.kubernetes.deployments.readiness
• kube_horizontalpodautoscaler_spec_max_replicas
  Used labels: namespace
  Example queries:

  • infra.kubernetes.hpa_replicas_critical
  • infra.kubernetes.hpa_replicas_high
• kube_horizontalpodautoscaler_status_current_replicas
  Used labels: namespace
  Example queries:

  • infra.kubernetes.hpa_replicas_critical
  • infra.kubernetes.hpa_replicas_high
• kube_node_created
  Used labels: node
  Example queries:

  • infra.nodes.created
• kube_node_info
  Used labels: node
  Example queries:

  • infra.nodes.info.address
  • infra.nodes.info.kernel
  • infra.nodes.info.kubelet
  • infra.nodes.info.os
  • infra.nodes.info.runtime
• kube_node_spec_taint
  Used labels: key, node
  Example queries:

  • infra.kubernetes.node_unreachable
  • infra.nodes.taints
• kube_node_spec_unschedulable
  Used labels: node
  Example queries:

  • infra.nodes.unschedulable
• kube_node_status_allocatable
  Used labels: node, resource
  Example queries:

  • infra.nodes.allocation.cpu
  • infra.nodes.allocation.memory
  • infra.nodes.allocation.pods
• kube_node_status_capacity
  Used labels: node, resource
  Example queries:

  • infra.nodes.capacity.cpu
  • infra.nodes.capacity.ephemeral_storage
  • infra.nodes.capacity.memory
  • infra.nodes.capacity.pods
• kube_node_status_condition
  Used labels: condition, node, status
  Example queries:

  • infra.kubernetes.node_disk_pressure
  • infra.nodes.condition.ready
  • infra.nodes.conditions
• kube_pod_container_resource_limits
  Used labels: namespace, node, pod, resource
  Example queries:

  • infra.nodes.pods.budgets
  • materialize.kubernetes.cpu.usage.percent
  • materialize.kubernetes.pods.cpu_usage
• kube_pod_container_resource_requests
  Used labels: container, namespace, node, pod, resource, unit
  Example queries:

  • infra.kubernetes.daemonset_high_cpu
  • infra.kubernetes.daemonset_saturating_cpu
  • infra.kubernetes.daemonset_saturating_mem
  • infra.kubernetes.pods_high_cpu_ratio
  • infra.kubernetes.vector_high_cpu_ratio
  • infra.nodes.allocation.cpu
  • infra.nodes.allocation.memory
  • infra.nodes.pods.budgets
• kube_pod_container_status_last_terminated_exitcode
  Used labels: container, pod
  Example queries:

  • materialize.clusterd.error_kill
  • materialize.clusterd.system_cluster_terminated
  • materialize.environmentd.terminated
• kube_pod_container_status_last_terminated_reason
  Used labels: container, namespace, reason
  Example queries:

  • infra.kubernetes.oomkill_core_systems
  • infra.kubernetes.oomkill_important_systems
  • infra.kubernetes.oomkill_nonessential_systems
  • materialize.clusterd.swap_cluster_oom
• kube_pod_container_status_restarts_total
  Used labels: container, namespace
  Example queries:

  • infra.kubernetes.oomkill_core_systems
  • infra.kubernetes.oomkill_important_systems
  • infra.kubernetes.oomkill_nonessential_systems
  • infra.kubernetes.pod_restart_rate_high
  • infra.kubernetes.pod_restart_rate_high_nonessential
  • infra.nodes.pods.restarts
  • materialize.clusterd.error_kill
  • materialize.clusterd.new_restarts_during_release
  • materialize.clusterd.swap_cluster_oom
• kube_pod_container_status_waiting
  
  Example queries:

  • infra.kubernetes.pods_stuck_in_waiting
• kube_pod_created
  
  Example queries:

  • materialize.clusterd.new_restarts_during_release
• kube_pod_info
  Used labels: node
  Example queries:

  • infra.nodes.allocation.pods
  • infra.nodes.pods.by_namespace
  • infra.nodes.pods.by_phase
  • infra.nodes.pods.not_ready
  • infra.nodes.pods.restarts
• kube_pod_start_time
  
  Example queries:

  • infra.kubernetes.pods_stuck_in_waiting
  • materialize.clusterd.not_receiving_commands
  • materialize.environmentd.uptime_sla
  • materialize.environmentd.uptime_slo
• kube_pod_status_phase
  Used labels: namespace, phase
  Example queries:

  • infra.kubernetes.infra_pod_pending
  • infra.nodes.allocation.pods
  • infra.nodes.pods.by_namespace
  • infra.nodes.pods.by_phase
  • infra.nodes.pods.not_ready
  • materialize.environmentd.pod_pending
  • materialize.environmentd.pod_pending_critical
  • materialize.kubernetes.pods.readiness
• kube_pod_status_ready
  Used labels: condition
  Example queries:

  • infra.nodes.pods.not_ready
• kube_statefulset_status_replicas_ready
  Used labels: namespace
  Example queries:

  • materialize.kubernetes.statefulsets.ready
• kubelet_node_name
  Used labels: workload
  Example queries:

  • infra.egress_gateway.no_traffic
• kubelet_volume_stats_capacity_bytes
  Used labels: persistentvolumeclaim
  Example queries:

  • infra.kubernetes.volume_usage
• kubelet_volume_stats_used_bytes
  Used labels: persistentvolumeclaim
  Example queries:

  • infra.kubernetes.volume_usage
• loki_distributor_bytes_received_total
  Used labels: namespace
  Example queries:

  • infra.monitoring.logging_collection_down
• loki_panic_total
  
  Example queries:

  • infra.loki.panics
• loki_request_duration_seconds_bucket
  Used labels: route
  Example queries:

  • infra.loki.req_duration_high
• loki_request_duration_seconds_count
  Used labels: route, status_code
  Example queries:

  • infra.loki.push_err_high
  • infra.loki.req_err_high
  • infra.loki.writer_err_high
• loki_write_dropped_entries_total
  
  Example queries:

  • infra.loki.alloy_log_drops
• mz_adapter_commands
  Used labels: application_name, materialize_cloud_organization_name, status
  Example queries:

  • materialize.connections.adapter.command_rate
  • materialize.connections.adapter.commands_by_application
  • materialize.console.errors
• mz_auth_request_count
  Used labels: path, status
  Example queries:

  • materialize.auth.errors
  • materialize.auth.refresh_failures
• mz_cluster_server_last_command_received
  Used labels: pod, server_name
  Example queries:

  • materialize.clusterd.not_receiving_commands
• mz_dataflow_replica_expiration_remaining_seconds
  Used labels: pod
  Example queries:

  • materialize.clusterd.expiration_7d
• mz_external_calls_count
  Used labels: status
  Example queries:

  • materialize.external_uptime.checker_not_calling
  • materialize.external_uptime.connections_failing
• mz_external_envd_up
  
  Example queries:

  • materialize.external_uptime.envd_unreachable
• mz_memory_limiter_memory_limit_bytes
  Used labels: pod
  Example queries:

  • materialize.clusterd.system_cluster_high_memory
• mz_memory_limiter_memory_usage_bytes
  Used labels: pod
  Example queries:

  • materialize.clusterd.system_cluster_high_memory
• mz_parameter_frontend_last_cse_time_seconds
  
  Example queries:

  • materialize.launchdarkly.stale_cse
• mz_parameter_frontend_last_sse_time_seconds
  
  Example queries:

  • materialize.launchdarkly.stale_sse
• mz_persist_blob_failures
  
  Example queries:

  • materialize.persist.failures
• mz_persist_cmd_failed_count
  
  Example queries:

  • materialize.persist.failures
• mz_persist_columnar_op_count
  Used labels: op, result
  Example queries:

  • materialize.persist.failures
• mz_persist_columnar_validation_count
  Used labels: result
  Example queries:

  • materialize.persist.failures
• mz_persist_compaction_dropped
  
  Example queries:

  • materialize.persist.failures
• mz_persist_compaction_failed
  
  Example queries:

  • materialize.persist.failures
• mz_persist_compaction_noop
  
  Example queries:

  • materialize.persist.failures
• mz_persist_consensus_failures
  
  Example queries:

  • materialize.persist.failures
• mz_persist_external_blob_delete_noop_count
  
  Example queries:

  • materialize.persist.failures
• mz_persist_external_failed_count
  
  Example queries:

  • materialize.persist.failures
• mz_persist_lease_timeout_read
  
  Example queries:

  • materialize.persist.failures
• mz_persist_pushdown_parts_mismatched_stats_count
  
  Example queries:

  • materialize.persist.failures
• mz_persist_schema_cache_fetch_state_count
  
  Example queries:

  • materialize.persist.failures
• mz_persist_shard_unconsolidated_snapshot
  
  Example queries:

  • materialize.persist.failures
• mz_persist_state_update_state_slow_path
  
  Example queries:

  • materialize.persist.failures
• mz_shard_finalization_outstanding
  
  Example queries:

  • materialize.storage.collection_finalization_stuck
• mz_time_to_first_row_seconds_bucket
  Used labels: application_name, instance_id
  Example queries:

  • materialize.console.query_latency
• mz_txn_placeholder_schema_apply
  
  Example queries:

  • materialize.persist.failures
• node_boot_time_seconds
  Used labels: instance
  Example queries:

  • node.uptime
• node_cpu_seconds_total
  Used labels: instance, mode
  Example queries:

  • node.cpu.utilization
  • node.debug.cpu.by_mode
  • node.debug.cpu.per_core
  • node.load.normalized
• node_disk_io_time_seconds_total
  Used labels: instance
  Example queries:

  • node.disk.io_utilization
• node_filefd_allocated
  Used labels: instance
  Example queries:

  • node.filefd.utilization
• node_filefd_maximum
  Used labels: instance
  Example queries:

  • node.filefd.utilization
• node_filesystem_avail_bytes
  Used labels: fstype, instance
  Example queries:

  • node.filesystem.available.ratio
• node_filesystem_readonly
  Used labels: fstype, instance
  Example queries:

  • node.filesystem.readonly
• node_filesystem_size_bytes
  Used labels: fstype, instance
  Example queries:

  • node.filesystem.available.ratio
• node_load1
  Used labels: instance
  Example queries:

  • node.load.normalized
• node_memory_MemAvailable_bytes
  Used labels: instance
  Example queries:

  • node.memory.available.ratio
• node_memory_MemTotal_bytes
  Used labels: instance
  Example queries:

  • node.debug.memory.breakdown
  • node.memory.available.ratio
• node_memory_SwapFree_bytes
  Used labels: instance
  Example queries:

  • node.swap.used.ratio
• node_memory_SwapTotal_bytes
  Used labels: instance
  Example queries:

  • node.swap.used.ratio
• node_network_receive_bytes_total
  Used labels: device, instance
  Example queries:

  • infra.egress_gateway.excessive_traffic
  • infra.egress_gateway.high_traffic
  • infra.egress_gateway.low_traffic
  • infra.egress_gateway.no_traffic
  • infra.egress_gateway.traffic_missing_metrics
  • node.debug.network.saturation
  • node.debug.network.throughput
  • node.network.rx.total
• node_network_receive_drop_total
  Used labels: instance
  Example queries:

  • node.network.drops
• node_network_receive_errs_total
  Used labels: instance
  Example queries:

  • node.network.errors
• node_network_transmit_bytes_total
  Used labels: device, instance
  Example queries:

  • infra.egress_gateway.no_traffic
  • node.debug.network.saturation
  • node.debug.network.throughput
  • node.network.tx.total
• node_network_transmit_drop_total
  Used labels: instance
  Example queries:

  • node.network.drops
• node_network_transmit_errs_total
  Used labels: instance
  Example queries:

  • node.network.errors
• node_nf_conntrack_entries
  Used labels: instance
  Example queries:

  • node.conntrack.utilization
• node_nf_conntrack_entries_limit
  Used labels: instance
  Example queries:

  • node.conntrack.utilization
• node_pressure_cpu_waiting_seconds_total
  Used labels: instance
  Example queries:

  • node.cpu.pressure
• node_pressure_memory_stalled_seconds_total
  Used labels: instance
  Example queries:

  • node.memory.pressure
• node_pressure_memory_waiting_seconds_total
  Used labels: instance
  Example queries:

  • node.memory.pressure
• node_scrape_collector_success
  Used labels: instance
  Example queries:

  • node.collector.success
• node_time_seconds
  Used labels: instance
  Example queries:

  • node.uptime
• node_vmstat_oom_kill
  Used labels: instance
  Example queries:

  • node.memory.oom_kills
• node_vmstat_pswpin
  Used labels: instance
  Example queries:

  • node.swap.activity
• node_vmstat_pswpout
  Used labels: instance
  Example queries:

  • node.swap.activity
• up
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, job, namespace
  Example queries:

  • infra.monitoring.clusterd_metrics_missing
  • infra.monitoring.critical_metrics_missing
  • materialize.scraper.mzmon.clusterd
  • materialize.scraper.mzmon.environmentd
  • materialize.scraper.mzmon.orchestratord
• v2_mz_can_connect
  
  Example queries:

  • materialize.environmentd.uptime_sla
  • materialize.environmentd.uptime_slo
• v2_mz_envd_up
  
  Example queries:

  • materialize.environmentd.simplest_query
• v2_mz_views_query_successful
  
  Example queries:

  • materialize.environmentd.query_views_critical
  • materialize.environmentd.query_views_warning

Recommended Metrics#

• environmentd_needs_update
  Used labels: namespace
  Example queries:

  • materialize.operator.environments.needing_update
• mz_active_sessions
  Used labels: materialize_cloud_organization_name
  Example queries:

  • materialize.connections.sessions.active
• mz_active_subscribes
  Used labels: materialize_cloud_organization_name
  Example queries:

  • materialize.compute.subscribes.active
• mz_arrangement_maintenance_seconds_total
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.compute.arrangements.maintenance_rate
  • materialize.compute.arrangements.maintenance_rate_by_worker
• mz_arrangement_record_count
  Used labels: collection_id, instance_id, materialize_cloud_organization_name, replica_id
  Example queries:

  • materialize.compute.arrangements.records.system
  • materialize.compute.arrangements.records.transient
  • materialize.compute.arrangements.records.user
• mz_compute_cluster_status
  Used labels: compute_cluster_id, compute_replica_id, compute_replica_name, materialize_cloud_organization_name, pod
  Example queries:

  • materialize.clusters.count
  • materialize.clusters.info
  • materialize.clusters.replicas.count
  • materialize.clusters.replicas.sizes
  • materialize.generations.version
  • materialize.health.clusters.status.percentage
  • materialize.health.environment.availability.percentage
  • materialize.info.version
• mz_compute_commands_total
  Used labels: materialize_cloud_organization_name, pod
  Example queries:

  • materialize.generations.active
• mz_compute_controller_hydration_queue_size
  Used labels: instance_id, materialize_cloud_organization_name, replica_id
  Example queries:

  • materialize.compute.hydration.queue_size
• mz_compute_hydration_time_seconds
  Used labels: hydrated, instance_id, materialize_cloud_organization_name, replica_id
  Example queries:

  • materialize.compute.hydration.slowest_collections
• mz_compute_peek_duration_seconds_bucket
  Used labels: instance_id, materialize_cloud_organization_name
  Example queries:

  • materialize.connections.peek_latency.p50
  • materialize.connections.peek_latency.p90
  • materialize.connections.peek_latency.p99
• mz_compute_replica_history_dataflow_count
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.compute.dataflows.count
  • materialize.compute.dataflows.count_by_worker
• mz_dataflow_elapsed_seconds_total
  Used labels: instance_id, materialize_cloud_organization_name, replica_id
  Example queries:

  • materialize.compute.dataflows.elapsed_rate
• mz_dataflow_wallclock_lag_seconds
  Used labels: instance_id, materialize_cloud_organization_name, pod, quantile, replica_id
  Example queries:

  • materialize.compute.freshness.lag_by_cluster
  • materialize.compute.freshness.lag_total_by_cluster
  • materialize.compute.freshness.top_collections
  • materialize.compute.hydration.currently_hydrating
  • materialize.generations.collections
  • materialize.generations.hydrating
  • materialize.generations.lag.max
  • materialize.generations.lag.total
  • materialize.generations.lag.total_by_cluster
  • materialize.info.max_lag
• mz_indexes_count
  Used labels: materialize_cloud_organization_name
  Example queries:

  • materialize.compute.indexes.by_type
  • materialize.compute.indexes.count
• mz_mzd_views_count
  Used labels: materialize_cloud_organization_name
  Example queries:

  • materialize.compute.materialized_views.count
• mz_query_total
  Used labels: materialize_cloud_organization_name
  Example queries:

  • materialize.connections.queries.distribution
  • materialize.connections.queries.rate
  • materialize.connections.queries.rate_by_statement
• mz_sink_bytes_committed
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sinks.lag
  • materialize.storage.sinks.throughput
• mz_sink_bytes_staged
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sinks.lag
• mz_sink_iceberg_commit_conflicts
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sinks.iceberg.commit_failures
• mz_sink_iceberg_commit_duration_seconds_bucket
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sinks.iceberg.commit_latency
• mz_sink_iceberg_commit_failures
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sinks.iceberg.commit_failures
• mz_sink_iceberg_data_files_written
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sinks.iceberg.file_rate
• mz_sink_iceberg_delete_files_written
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sinks.iceberg.file_rate
• mz_sink_iceberg_snapshots_committed
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sinks.iceberg.file_rate
• mz_sink_rdkafka_connects
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sinks.kafka.connect_rate
• mz_sink_rdkafka_disconnects
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sinks.kafka.connect_rate
• mz_sink_rdkafka_outbuf_msg_cnt
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sinks.kafka.output_buffer
• mz_sink_rdkafka_txerrs
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sinks.kafka.tx_errors
• mz_source_bytes_received
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sources.bytes_received
• mz_source_messages_received
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sources.ingestion_by_replica
• mz_source_offset_commit_failures
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sources.upstream_errors
• mz_source_offset_committed
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sources.upstream_errors
• mz_source_offset_known
  Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
  Example queries:

  • materialize.storage.sources.upstream_errors
• mz_storage_objects
  Used labels: materialize_cloud_organization_name, type
  Example queries:

  • materialize.storage.sinks.by_type
  • materialize.storage.sinks.count
  • materialize.storage.sources.by_type
  • materialize.storage.sources.catalog
  • materialize.storage.sources.count
• mz_tables_count
  Used labels: materialize_cloud_organization_name
  Example queries:

  • materialize.storage.tables.count
• mz_views_count
  Used labels: materialize_cloud_organization_name
  Example queries:

  • materialize.compute.views.count
• node_arp_entries
  Used labels: instance
  Example queries:

  • node.debug.arp.entries
• node_context_switches_total
  Used labels: instance
  Example queries:

  • node.debug.context_switches
• node_disk_io_time_weighted_seconds_total
  Used labels: instance
  Example queries:

  • node.debug.disk.queue_depth
• node_disk_read_bytes_total
  Used labels: instance
  Example queries:

  • node.debug.disk.throughput
• node_disk_read_time_seconds_total
  Used labels: instance
  Example queries:

  • node.debug.disk.latency
• node_disk_reads_completed_total
  Used labels: instance
  Example queries:

  • node.debug.disk.iops
  • node.debug.disk.latency
• node_disk_write_time_seconds_total
  Used labels: instance
  Example queries:

  • node.debug.disk.latency
• node_disk_writes_completed_total
  Used labels: instance
  Example queries:

  • node.debug.disk.iops
  • node.debug.disk.latency
• node_disk_written_bytes_total
  Used labels: instance
  Example queries:

  • node.debug.disk.throughput
• node_entropy_available_bits
  Used labels: instance
  Example queries:

  • node.debug.entropy.available
• node_entropy_pool_size_bits
  Used labels: instance
  Example queries:

  • node.debug.entropy.available
• node_filesystem_files
  Used labels: fstype, instance
  Example queries:

  • node.debug.filesystem.inodes.available.ratio
• node_filesystem_files_free
  Used labels: fstype, instance
  Example queries:

  • node.debug.filesystem.inodes.available.ratio
• node_intr_total
  Used labels: instance
  Example queries:

  • node.debug.context_switches
• node_memory_Buffers_bytes
  Used labels: instance
  Example queries:

  • node.debug.memory.breakdown
• node_memory_Cached_bytes
  Used labels: instance
  Example queries:

  • node.debug.memory.breakdown
• node_memory_Committed_AS_bytes
  Used labels: instance
  Example queries:

  • node.debug.memory.kernel
• node_memory_MemFree_bytes
  Used labels: instance
  Example queries:

  • node.debug.memory.breakdown
• node_memory_SReclaimable_bytes
  Used labels: instance
  Example queries:

  • node.debug.memory.breakdown
  • node.debug.memory.kernel
• node_memory_SUnreclaim_bytes
  Used labels: instance
  Example queries:

  • node.debug.memory.kernel
• node_memory_Slab_bytes
  Used labels: instance
  Example queries:

  • node.debug.memory.kernel
• node_netstat_TcpExt_ListenDrops
  Used labels: instance
  Example queries:

  • node.debug.tcp.errors
• node_netstat_TcpExt_ListenOverflows
  Used labels: instance
  Example queries:

  • node.debug.tcp.errors
• node_netstat_TcpExt_TCPRcvQDrop
  Used labels: instance
  Example queries:

  • node.debug.tcp.errors
• node_netstat_TcpExt_TCPSynRetrans
  Used labels: instance
  Example queries:

  • node.debug.tcp.retransmits
• node_netstat_TcpExt_TCPTimeouts
  Used labels: instance
  Example queries:

  • node.debug.tcp.errors
• node_netstat_Tcp_OutSegs
  Used labels: instance
  Example queries:

  • node.debug.tcp.retransmits
• node_netstat_Tcp_RetransSegs
  Used labels: instance
  Example queries:

  • node.debug.tcp.retransmits
• node_netstat_Udp_InErrors
  Used labels: instance
  Example queries:

  • node.debug.udp.errors
• node_netstat_Udp_NoPorts
  Used labels: instance
  Example queries:

  • node.debug.udp.errors
• node_netstat_Udp_RcvbufErrors
  Used labels: instance
  Example queries:

  • node.debug.udp.errors
• node_network_carrier
  Used labels: instance
  Example queries:

  • node.debug.network.operstate
• node_network_speed_bytes
  Used labels: instance
  Example queries:

  • node.debug.network.saturation
• node_network_up
  Used labels: instance
  Example queries:

  • node.debug.network.operstate
• node_schedstat_waiting_seconds_total
  Used labels: instance
  Example queries:

  • node.debug.schedstat.waiting
• node_scrape_collector_duration_seconds
  Used labels: instance
  Example queries:

  • node.debug.exporter.scrape_duration
• node_sockstat_TCP_alloc
  Used labels: instance
  Example queries:

  • node.debug.sockets.tcp
• node_sockstat_TCP_inuse
  Used labels: instance
  Example queries:

  • node.debug.sockets.tcp
• node_sockstat_TCP_mem_bytes
  Used labels: instance
  Example queries:

  • node.debug.sockets.memory
• node_sockstat_TCP_orphan
  Used labels: instance
  Example queries:

  • node.debug.sockets.tcp
• node_sockstat_TCP_tw
  Used labels: instance
  Example queries:

  • node.debug.sockets.tcp
• node_sockstat_UDP_mem_bytes
  Used labels: instance
  Example queries:

  • node.debug.sockets.memory
• node_softnet_dropped_total
  Used labels: instance
  Example queries:

  • node.debug.softnet.dropped
• node_softnet_processed_total
  Used labels: instance
  Example queries:

  • node.debug.softnet.processed
• node_softnet_times_squeezed_total
  Used labels: instance
  Example queries:

  • node.debug.softnet.squeezed
• node_timex_estimated_error_seconds
  Used labels: instance
  Example queries:

  • node.debug.time.drift
• node_timex_maxerror_seconds
  Used labels: instance
  Example queries:

  • node.debug.time.drift
• node_timex_offset_seconds
  Used labels: instance
  Example queries:

  • node.debug.time.drift
• node_timex_sync_status
  Used labels: instance
  Example queries:

  • node.debug.time.sync_status
• node_udp_queues
  Used labels: instance, ip, queue
  Example queries:

  • node.debug.udp.queues
• node_vmstat_pgfault
  Used labels: instance
  Example queries:

  • node.debug.memory.page_faults
• node_vmstat_pgmajfault
  Used labels: instance
  Example queries:

  • node.debug.memory.page_faults
• node_vmstat_pgscan_direct
  Used labels: instance
  Example queries:

  • node.debug.memory.reclaim
• node_vmstat_pgscan_kswapd
  Used labels: instance
  Example queries:

  • node.debug.memory.reclaim
• node_vmstat_pgsteal_direct
  Used labels: instance
  Example queries:

  • node.debug.memory.reclaim
• node_vmstat_pgsteal_kswapd
  Used labels: instance
  Example queries:

  • node.debug.memory.reclaim
• orchestratord_is_leader
  Used labels: namespace
  Example queries:

  • materialize.operator.reconciling.replicas
• orchestratord_reconciliation_duration_seconds_bucket
  Used labels: namespace
  Example queries:

  • materialize.operator.reconciliation.duration
• orchestratord_reconciliation_step_duration_seconds_bucket
  Used labels: namespace
  Example queries:

  • materialize.operator.reconciliation.step.duration.p99
• orchestratord_reconciliation_steps_total
  Used labels: namespace, outcome
  Example queries:

  • materialize.operator.reconciliation.steps.incomplete
  • materialize.operator.reconciliation.steps.rate
• orchestratord_reconciliations_total
  Used labels: namespace, outcome
  Example queries:

  • materialize.operator.reconciliation.failures.by_controller
  • materialize.operator.reconciliation.failures.total
  • materialize.operator.reconciliation.outcomes
  • materialize.operator.reconciliation.rate