Backend APIs#

This page is for operators with Kubernetes access to the monitoring namespace. Thanos Query, Loki, and Alertmanager each serve an HTTP API, and you can reach each one with kubectl without going through Grafana. Use this path when Grafana is down, when the stack itself is what you’re debugging, or when a script needs the raw API.

Before you start#

The backends have no authentication of their own, so anyone who can reach one reads everything it serves. Reach them through kubectl, and never through an ingress. For more information, see Securing.

To port-forward, your Kubernetes identity needs get and list on services and pods, and create on pods/portforward, in the backend’s namespace. The amtool examples need create on pods/exec instead.

In each kubectl snippet, set KUBECONTEXT to your cluster’s context and NAMESPACE to the backend’s namespace. Every backend runs in monitoring by default. Under the split-namespace profile, each runs in its own namespace: thanos, loki, and alertmanager.

Query metrics from Thanos Query#

Thanos Query serves the Prometheus HTTP API, so anything that speaks PromQL works against it.

To query metrics, follow these steps:

1. Forward a local port to Thanos Query:

   KUBECONTEXT=my-cluster
   NAMESPACE=monitoring
   
   kubectl --context "$KUBECONTEXT" -n "$NAMESPACE" port-forward svc/thanos-query 9090:9090

2. In another terminal, run a query with promtool or curl:

   promtool query instant http://localhost:9090 'up{namespace="materialize-environment"}'

   curl -s http://localhost:9090/api/v1/query --data-urlencode 'query=up' | jq '.data.result[0]'

Thanos Query also serves its own query UI at http://localhost:9090 while the port-forward runs.

For queries worth running, see Common queries. For what Thanos Query reads from, see Querying metrics.

Query logs from Loki#

The Loki gateway serves the Loki HTTP API. The bundled Loki runs with auth_enabled: true, so every request needs a tenant in the X-Scope-OrgID header. The pipeline writes to the loki tenant unless pipeline.logging.tenancy.staticTenant says otherwise.

To query logs, follow these steps:

1. Forward a local port to the Loki gateway:

   KUBECONTEXT=my-cluster
   NAMESPACE=monitoring
   
   kubectl --context "$KUBECONTEXT" -n "$NAMESPACE" port-forward svc/loki-gateway 3100:8080

2. In another terminal, run a query with logcli or curl:

   logcli query --addr=http://localhost:3100 --org-id=loki --since=1h --limit=50 \
     '{namespace="materialize-environment", app="environmentd"}'

   curl -s -H 'X-Scope-OrgID: loki' http://localhost:3100/loki/api/v1/query_range \
     --data-urlencode 'query={namespace="materialize-environment"}' --data-urlencode 'limit=20'

A request without the header fails with no org id. For the labels a log stream carries, and the attributes that are structured metadata instead, see Querying logs.

Read alerts and silences from Alertmanager#

amtool ships in the Alertmanager image, with a configuration that points it at the local Alertmanager. Run it in the pod rather than through a port-forward:

KUBECONTEXT=my-cluster
NAMESPACE=monitoring

kubectl --context "$KUBECONTEXT" -n "$NAMESPACE" exec alertmanager-0 -c alertmanager -- amtool alert query
kubectl --context "$KUBECONTEXT" -n "$NAMESPACE" exec alertmanager-0 -c alertmanager -- amtool silence query

To create and expire silences, see Maintenance windows.