Azure Monitor's default dashboards are a reasonable starting point and a poor destination. The platform's real strength — Kusto Query Language (KQL) against Log Analytics, paired with Application Insights for distributed tracing — only shows up once someone writes queries that answer actual incident questions instead of staring at CPU graphs.

Alert on symptoms, not raw resource metrics

The instinct is to alarm on CPU, memory, and disk directly. The better pattern alarms on what exhausting those resources actually causes — elevated latency, rising error rate, growing queue depth:

requests
| where timestamp > ago(5m)
| summarize
    total = count(),
    failed = countif(success == false)
    by bin(timestamp, 1m)
| extend errorRate = todouble(failed) / total * 100
| where errorRate > 5

Wire that query into an Azure Monitor scheduled alert rule rather than checking it manually:

az monitor scheduled-query create \
  --name "api-error-rate-high" \
  --resource-group prod-rg \
  --scopes /subscriptions/<sub-id>/resourceGroups/prod-rg/providers/microsoft.insights/components/api-insights \
  --condition "count 'requests' > 0" \
  --condition-query "requests | where success == false | summarize count() by bin(timestamp, 5m)" \
  --window-size 5m \
  --evaluation-frequency 5m \
  --severity 2

Require the condition to hold across the full evaluation window rather than firing on a single data point — that one setting eliminates most of the single-blip pages that make on-call rotations miserable without ever indicating a real problem.

Application Insights: distributed tracing with almost no code

For AKS-hosted services, Application Insights auto-instrumentation attaches without touching application code for most common runtimes:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    monitor.azure.com/instrumentation-language: "java" # or python, nodejs, dotnet
spec:
  template:
    metadata:
      annotations:
        monitor.azure.com/type: "opentelemetry"

Teams already instrumented with OpenTelemetry for a different backend can point the same SDK directly at Application Insights's collector endpoint instead — no rewrite of existing instrumentation required, just a different exporter configuration.

A KQL query that answers the actual incident question

The dashboards worth building aren't the ones with the most panels — they answer one question fast: is this us, or is this upstream?

dependencies
| where timestamp > ago(30m)
| summarize
    avgDuration = avg(duration),
    failureRate = countif(success == false) * 100.0 / count()
    by target, bin(timestamp, 5m)
| where failureRate > 10 or avgDuration > 1000
| order by timestamp desc

This surfaces which downstream dependency — a database, a third-party API, a cache — is actually degraded, rather than leaving the on-call engineer to guess from an aggregate latency number what's actually slow underneath the request.

Deploy markers on the same timeline as metrics

The single highest-value addition to any dashboard, and the one teams skip most often, is annotating deploys directly on the metrics timeline:

az monitor app-insights annotation create \
  --app api-insights \
  --resource-group prod-rg \
  --annotation-name "deploy-$(git rev-parse --short HEAD)" \
  --time (date -u +%Y-%m-%dT%H:%M:%SZ) \
  --category Deployment

Wire this into the CI/CD pipeline (see the AKS deployment pipeline I've written up separately) so it fires automatically on every production deploy. During an incident, "did this start right after a deploy" becomes a glance at the dashboard instead of a Slack archaeology exercise.

The alerting discipline that actually reduces fatigue

Every alert rule needs an owner and a runbook link attached before it ships — not retrofitted after the third time it pages someone with no context on what it means or what to do about it. If a rule has fired five times in a month and nobody has adjusted its threshold or fixed the underlying cause, that's the signal to either fix the root issue or delete the rule outright. A noisy alert that people learn to acknowledge-and-ignore is worse than no alert at all — it trains the team to stop trusting the signal right when a real incident needs it.

Cost control on Log Analytics ingestion

One Azure-specific gotcha worth flagging: Log Analytics bills per GB ingested, and verbose application logging can turn observability into a surprisingly large line item. Set a daily ingestion cap on the workspace during initial rollout, and use sampling in Application Insights for high-volume traces rather than capturing every single request at full fidelity.