# List of Metrics
# List of Metrics
This has the list of metrics which are available for usage.
In many systems, only a subset of metrics get stored based
on how they would be used.
## Metric Tiers
Importance is about **storage**, not stability: it says which metrics are worth keeping when capacity is limited, not what we promise about their names.
For that, see [Stability Guarantees and Deprecation Policy](../../stability/) — in short, `mz_*` names come from Materialize itself and we disclose rather than freeze them.
Metrics are grouped by "metricImportance" levels (mzmon-specific).
These levels guide which metrics are prioritized in
metric stores which have limited capacity.
The **essential** metrics are the set of metrics that
are critical and you would always want to have available.
These are used in alerting.
The **recommended** metrics are the set of metrics that
are used in dashboards and are generally desirable for
troubleshooting.
The **extended** set of metrics are used for optional/experimental
dashboards.
The **diagnostic** set of metrics are used for in-depth
troubleshooting and analysis.
In our `materialize-monitoring` configuration, we also
provide an **all** min-importance for including
absolutely everything.
This is recommended if you have cheaper metric storage
like our bundled Thanos provider.
## Essential Metrics
> [!WARNING]
> FIXME: Some links for alerts mistakenly point at the common-queries page.
- certmanager_certificate_ready_status
Used labels: condition
Example queries:
- cilium_bpf_map_pressure
Example queries:
- cilium_drop_count_total
Example queries:
- container_cpu_cfs_periods_total
Used labels: container
Example queries:
- container_cpu_cfs_throttled_periods_total
Used labels: container
Example queries:
- container_cpu_usage_seconds_total
Used labels: container, namespace, pod
Example queries:
- container_file_descriptors
Used labels: container
Example queries:
- container_fs_limit_bytes
Example queries:
- container_fs_usage_bytes
Example queries:
- container_last_seen
Used labels: materialize_cloud_swap
Example queries:
- container_memory_swap
Example queries:
- container_memory_working_set_bytes
Used labels: container, namespace, pod
Example queries:
- container_network_receive_bytes_total
Used labels: namespace, pod, workload
Example queries:
- container_network_receive_errors_total
Used labels: namespace, pod
Example queries:
- container_network_receive_packets_dropped_total
Used labels: namespace, pod
Example queries:
- container_network_transmit_bytes_total
Used labels: namespace, pod
Example queries:
- container_network_transmit_errors_total
Used labels: namespace, pod
Example queries:
- container_network_transmit_packets_dropped_total
Used labels: namespace, pod
Example queries:
- container_spec_cpu_period
Used labels: container, namespace, pod
Example queries:
- container_spec_cpu_quota
Used labels: container, namespace, pod
Example queries:
- container_spec_memory_limit_bytes
Used labels: container, namespace, pod
Example queries:
- container_spec_memory_swap_limit_bytes
Example queries:
- container_start_time_seconds
Used labels: container, namespace
Example queries:
- container_ulimits_soft
Used labels: ulimit
Example queries:
- coredns_dns_request_duration_seconds_bucket
Used labels: zone
Example queries:
- crdb_dedicated_capacity
Example queries:
- crdb_dedicated_capacity_used
Example queries:
- crdb_dedicated_intentcount
Example queries:
- crdb_dedicated_ranges_unavailable
Example queries:
- crdb_dedicated_ranges_underreplicated
Example queries:
- crdb_dedicated_rocksdb_read_amplification
Example queries:
- crdb_dedicated_schedules_backup_last_completed_time
Example queries:
- crdb_dedicated_sql_mem_distsql_current
Example queries:
- crdb_dedicated_sql_service_latency_bucket
Example queries:
- crdb_dedicated_sys_cpu_combined_percent_normalized
Example queries:
- crdb_dedicated_sys_totalmem
Example queries:
- kube_deployment_status_condition
Used labels: condition, deployment, status
Example queries:
- kube_deployment_status_replicas_ready
Used labels: namespace
Example queries:
- kube_deployment_status_replicas_unavailable
Used labels: namespace
Example queries:
- kube_horizontalpodautoscaler_spec_max_replicas
Used labels: namespace
Example queries:
- kube_horizontalpodautoscaler_status_current_replicas
Used labels: namespace
Example queries:
- kube_node_created
Used labels: node
Example queries:
- kube_node_info
Used labels: node
Example queries:
- kube_node_spec_taint
Used labels: key, node
Example queries:
- kube_node_spec_unschedulable
Used labels: node
Example queries:
- kube_node_status_allocatable
Used labels: node, resource
Example queries:
- kube_node_status_capacity
Used labels: node, resource
Example queries:
- kube_node_status_condition
Used labels: condition, node, status
Example queries:
- kube_pod_container_resource_limits
Used labels: namespace, node, pod, resource
Example queries:
- kube_pod_container_resource_requests
Used labels: container, namespace, node, pod, resource, unit
Example queries:
- kube_pod_container_status_last_terminated_exitcode
Used labels: container, pod
Example queries:
- kube_pod_container_status_last_terminated_reason
Used labels: container, namespace, reason
Example queries:
- kube_pod_container_status_restarts_total
Used labels: container, namespace
Example queries:
- kube_pod_container_status_waiting
Example queries:
- kube_pod_created
Example queries:
- kube_pod_info
Used labels: node
Example queries:
- kube_pod_start_time
Example queries:
- kube_pod_status_phase
Used labels: namespace, phase
Example queries:
- kube_pod_status_ready
Used labels: condition
Example queries:
- kube_statefulset_status_replicas_ready
Used labels: namespace
Example queries:
- kubelet_node_name
Used labels: workload
Example queries:
- kubelet_volume_stats_capacity_bytes
Used labels: persistentvolumeclaim
Example queries:
- kubelet_volume_stats_used_bytes
Used labels: persistentvolumeclaim
Example queries:
- loki_distributor_bytes_received_total
Used labels: namespace
Example queries:
- loki_panic_total
Example queries:
- loki_request_duration_seconds_bucket
Used labels: route
Example queries:
- loki_request_duration_seconds_count
Used labels: route, status_code
Example queries:
- loki_write_dropped_entries_total
Example queries:
- mz_adapter_commands
Used labels: application_name, materialize_cloud_organization_name, status
Example queries:
- mz_auth_request_count
Used labels: path, status
Example queries:
- mz_cluster_server_last_command_received
Used labels: pod, server_name
Example queries:
- mz_dataflow_replica_expiration_remaining_seconds
Used labels: pod
Example queries:
- mz_external_calls_count
Used labels: status
Example queries:
- mz_external_envd_up
Example queries:
- mz_memory_limiter_memory_limit_bytes
Used labels: pod
Example queries:
- mz_memory_limiter_memory_usage_bytes
Used labels: pod
Example queries:
- mz_parameter_frontend_last_cse_time_seconds
Example queries:
- mz_parameter_frontend_last_sse_time_seconds
Example queries:
- mz_persist_blob_failures
Example queries:
- mz_persist_cmd_failed_count
Example queries:
- mz_persist_columnar_op_count
Used labels: op, result
Example queries:
- mz_persist_columnar_validation_count
Used labels: result
Example queries:
- mz_persist_compaction_dropped
Example queries:
- mz_persist_compaction_failed
Example queries:
- mz_persist_compaction_noop
Example queries:
- mz_persist_consensus_failures
Example queries:
- mz_persist_external_blob_delete_noop_count
Example queries:
- mz_persist_external_failed_count
Example queries:
- mz_persist_lease_timeout_read
Example queries:
- mz_persist_pushdown_parts_mismatched_stats_count
Example queries:
- mz_persist_schema_cache_fetch_state_count
Example queries:
- mz_persist_shard_unconsolidated_snapshot
Example queries:
- mz_persist_state_update_state_slow_path
Example queries:
- mz_shard_finalization_outstanding
Example queries:
- mz_time_to_first_row_seconds_bucket
Used labels: application_name, instance_id
Example queries:
- mz_txn_placeholder_schema_apply
Example queries:
- node_boot_time_seconds
Used labels: instance
Example queries:
- node_cpu_seconds_total
Used labels: instance, mode
Example queries:
- node_disk_io_time_seconds_total
Used labels: instance
Example queries:
- node_filefd_allocated
Used labels: instance
Example queries:
- node_filefd_maximum
Used labels: instance
Example queries:
- node_filesystem_avail_bytes
Used labels: fstype, instance
Example queries:
- node_filesystem_readonly
Used labels: fstype, instance
Example queries:
- node_filesystem_size_bytes
Used labels: fstype, instance
Example queries:
- node_load1
Used labels: instance
Example queries:
- node_memory_MemAvailable_bytes
Used labels: instance
Example queries:
- node_memory_MemTotal_bytes
Used labels: instance
Example queries:
- node_memory_SwapFree_bytes
Used labels: instance
Example queries:
- node_memory_SwapTotal_bytes
Used labels: instance
Example queries:
- node_network_receive_bytes_total
Used labels: device, instance
Example queries:
- node_network_receive_drop_total
Used labels: instance
Example queries:
- node_network_receive_errs_total
Used labels: instance
Example queries:
- node_network_transmit_bytes_total
Used labels: device, instance
Example queries:
- node_network_transmit_drop_total
Used labels: instance
Example queries:
- node_network_transmit_errs_total
Used labels: instance
Example queries:
- node_nf_conntrack_entries
Used labels: instance
Example queries:
- node_nf_conntrack_entries_limit
Used labels: instance
Example queries:
- node_pressure_cpu_waiting_seconds_total
Used labels: instance
Example queries:
- node_pressure_memory_stalled_seconds_total
Used labels: instance
Example queries:
- node_pressure_memory_waiting_seconds_total
Used labels: instance
Example queries:
- node_scrape_collector_success
Used labels: instance
Example queries:
- node_time_seconds
Used labels: instance
Example queries:
- node_vmstat_oom_kill
Used labels: instance
Example queries:
- node_vmstat_pswpin
Used labels: instance
Example queries:
- node_vmstat_pswpout
Used labels: instance
Example queries:
- up
Used labels: cluster_environmentd_materialize_cloud_cluster_id, job, namespace
Example queries:
- v2_mz_can_connect
Example queries:
- v2_mz_envd_up
Example queries:
- v2_mz_views_query_successful
Example queries:
## Recommended Metrics
- environmentd_needs_update
Used labels: namespace
Example queries:
- mz_active_sessions
Used labels: materialize_cloud_organization_name
Example queries:
- mz_active_subscribes
Used labels: materialize_cloud_organization_name
Example queries:
- mz_arrangement_maintenance_seconds_total
Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
Example queries:
- mz_arrangement_record_count
Used labels: collection_id, instance_id, materialize_cloud_organization_name, replica_id
Example queries:
- mz_compute_cluster_status
Used labels: compute_cluster_id, compute_replica_id, compute_replica_name, materialize_cloud_organization_name, pod
Example queries:
- mz_compute_commands_total
Used labels: materialize_cloud_organization_name, pod
Example queries:
- mz_compute_controller_hydration_queue_size
Used labels: instance_id, materialize_cloud_organization_name, replica_id
Example queries:
- mz_compute_hydration_time_seconds
Used labels: hydrated, instance_id, materialize_cloud_organization_name, replica_id
Example queries:
- mz_compute_peek_duration_seconds_bucket
Used labels: instance_id, materialize_cloud_organization_name
Example queries:
- mz_compute_replica_history_dataflow_count
Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
Example queries:
- mz_dataflow_elapsed_seconds_total
Used labels: instance_id, materialize_cloud_organization_name, replica_id
Example queries:
- mz_dataflow_wallclock_lag_seconds
Used labels: instance_id, materialize_cloud_organization_name, pod, quantile, replica_id
Example queries:
- mz_indexes_count
Used labels: materialize_cloud_organization_name
Example queries:
- mz_mzd_views_count
Used labels: materialize_cloud_organization_name
Example queries:
- mz_query_total
Used labels: materialize_cloud_organization_name
Example queries:
- mz_sink_bytes_committed
Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
Example queries:
- mz_sink_bytes_staged
Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
Example queries:
- mz_sink_iceberg_commit_conflicts
Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
Example queries:
- mz_sink_iceberg_commit_duration_seconds_bucket
Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
Example queries:
- mz_sink_iceberg_commit_failures
Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
Example queries:
- mz_sink_iceberg_data_files_written
Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
Example queries:
- mz_sink_iceberg_delete_files_written
Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
Example queries:
- mz_sink_iceberg_snapshots_committed
Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
Example queries:
- mz_sink_rdkafka_connects
Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
Example queries:
- mz_sink_rdkafka_disconnects
Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
Example queries:
- mz_sink_rdkafka_outbuf_msg_cnt
Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
Example queries:
- mz_sink_rdkafka_txerrs
Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
Example queries:
- mz_source_bytes_received
Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
Example queries:
- mz_source_messages_received
Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
Example queries:
- mz_source_offset_commit_failures
Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
Example queries:
- mz_source_offset_committed
Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
Example queries:
- mz_source_offset_known
Used labels: cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, materialize_cloud_organization_name
Example queries:
- mz_storage_objects
Used labels: materialize_cloud_organization_name, type
Example queries:
- mz_tables_count
Used labels: materialize_cloud_organization_name
Example queries:
- mz_views_count
Used labels: materialize_cloud_organization_name
Example queries:
- node_arp_entries
Used labels: instance
Example queries:
- node_context_switches_total
Used labels: instance
Example queries:
- node_disk_io_time_weighted_seconds_total
Used labels: instance
Example queries:
- node_disk_read_bytes_total
Used labels: instance
Example queries:
- node_disk_read_time_seconds_total
Used labels: instance
Example queries:
- node_disk_reads_completed_total
Used labels: instance
Example queries:
- node_disk_write_time_seconds_total
Used labels: instance
Example queries:
- node_disk_writes_completed_total
Used labels: instance
Example queries:
- node_disk_written_bytes_total
Used labels: instance
Example queries:
- node_entropy_available_bits
Used labels: instance
Example queries:
- node_entropy_pool_size_bits
Used labels: instance
Example queries:
- node_filesystem_files
Used labels: fstype, instance
Example queries:
- node_filesystem_files_free
Used labels: fstype, instance
Example queries:
- node_intr_total
Used labels: instance
Example queries:
- node_memory_Buffers_bytes
Used labels: instance
Example queries:
- node_memory_Cached_bytes
Used labels: instance
Example queries:
- node_memory_Committed_AS_bytes
Used labels: instance
Example queries:
- node_memory_MemFree_bytes
Used labels: instance
Example queries:
- node_memory_SReclaimable_bytes
Used labels: instance
Example queries:
- node_memory_SUnreclaim_bytes
Used labels: instance
Example queries:
- node_memory_Slab_bytes
Used labels: instance
Example queries:
- node_netstat_TcpExt_ListenDrops
Used labels: instance
Example queries:
- node_netstat_TcpExt_ListenOverflows
Used labels: instance
Example queries:
- node_netstat_TcpExt_TCPRcvQDrop
Used labels: instance
Example queries:
- node_netstat_TcpExt_TCPSynRetrans
Used labels: instance
Example queries:
- node_netstat_TcpExt_TCPTimeouts
Used labels: instance
Example queries:
- node_netstat_Tcp_OutSegs
Used labels: instance
Example queries:
- node_netstat_Tcp_RetransSegs
Used labels: instance
Example queries:
- node_netstat_Udp_InErrors
Used labels: instance
Example queries:
- node_netstat_Udp_NoPorts
Used labels: instance
Example queries:
- node_netstat_Udp_RcvbufErrors
Used labels: instance
Example queries:
- node_network_carrier
Used labels: instance
Example queries:
- node_network_speed_bytes
Used labels: instance
Example queries:
- node_network_up
Used labels: instance
Example queries:
- node_schedstat_waiting_seconds_total
Used labels: instance
Example queries:
- node_scrape_collector_duration_seconds
Used labels: instance
Example queries:
- node_sockstat_TCP_alloc
Used labels: instance
Example queries:
- node_sockstat_TCP_inuse
Used labels: instance
Example queries:
- node_sockstat_TCP_mem_bytes
Used labels: instance
Example queries:
- node_sockstat_TCP_orphan
Used labels: instance
Example queries:
- node_sockstat_TCP_tw
Used labels: instance
Example queries:
- node_sockstat_UDP_mem_bytes
Used labels: instance
Example queries:
- node_softnet_dropped_total
Used labels: instance
Example queries:
- node_softnet_processed_total
Used labels: instance
Example queries:
- node_softnet_times_squeezed_total
Used labels: instance
Example queries:
- node_timex_estimated_error_seconds
Used labels: instance
Example queries:
- node_timex_maxerror_seconds
Used labels: instance
Example queries:
- node_timex_offset_seconds
Used labels: instance
Example queries:
- node_timex_sync_status
Used labels: instance
Example queries:
- node_udp_queues
Used labels: instance, ip, queue
Example queries:
- node_vmstat_pgfault
Used labels: instance
Example queries:
- node_vmstat_pgmajfault
Used labels: instance
Example queries:
- node_vmstat_pgscan_direct
Used labels: instance
Example queries:
- node_vmstat_pgscan_kswapd
Used labels: instance
Example queries:
- node_vmstat_pgsteal_direct
Used labels: instance
Example queries:
- node_vmstat_pgsteal_kswapd
Used labels: instance
Example queries:
- orchestratord_is_leader
Used labels: namespace
Example queries:
- orchestratord_reconciliation_duration_seconds_bucket
Used labels: namespace
Example queries:
- orchestratord_reconciliation_step_duration_seconds_bucket
Used labels: namespace
Example queries:
- orchestratord_reconciliation_steps_total
Used labels: namespace, outcome
Example queries:
- orchestratord_reconciliations_total
Used labels: namespace, outcome
Example queries: