Azure Observability with OpenTelemetry: A Production Guide

Observability is the ability to explain a system’s internal state from evidence it emits. Its purpose is not to collect everything. It should help a team detect customer harm, locate the responsible dependency, understand what changed, recover safely and learn before the next incident. Azure Monitor, Application Insights and Log Analytics provide capabilities; OpenTelemetry provides portable conventions and instrumentation. Neither replaces an operating model.

Begin with critical user and business journeys, service objectives, failure modes and owners. Define what an operator must know when login slows, an order disappears, a queue backs up or a regional dependency fails. Inventory existing signals and gaps, then design telemetry, access, retention, alert routing and runbooks together. Success is shorter detection and recovery with reliable evidence at an acceptable cost.

Start with Journeys, Then Choose Logs, Metrics and Traces

Metrics describe trends and thresholds efficiently: request rate, error ratio, latency distributions, saturation, queue age and business throughput. Logs preserve discrete facts needed for investigation and audit. Traces connect work across services and reveal where time and failure accumulated. Events, profiles and browser telemetry may add context, but each signal needs a decision it supports.

Start from a journey and work backwards. A checkout objective may require completion rate, p95 latency and payment failures; traces through API, queue and provider; and structured logs for reconciliation. Infrastructure CPU alone cannot tell whether customers succeeded. Conversely, recording every payload adds privacy and cost risk without improving diagnosis.

Define semantic names, units, required attributes, cardinality limits, ownership and validation. Test signal loss, exporter failure and clock skew. Telemetry must never become a synchronous dependency of the business request. Instrument both healthy and failed paths, and verify that an engineer unfamiliar with the service can follow the evidence.

Design OpenTelemetry as a Stable Instrumentation Layer

Use OpenTelemetry SDKs and automatic instrumentation where they produce correct, bounded signals; add manual spans and metrics around domain operations the framework cannot understand. Set service name, version, environment, cloud region and deployment identifiers consistently. Adopt semantic conventions deliberately and pin package and collector versions so an upgrade does not silently change names or volume.

An OpenTelemetry Collector can receive, enrich, filter, sample and export telemetry outside the application. Keep processors simple and observable. Decide whether collectors run per host, cluster or centrally from availability, network and ownership needs. Buffer within safe limits, protect endpoints and measure dropped spans, queue pressure, export latency and configuration version.

Sampling is a product and incident decision. Head sampling is cheap but can discard rare failures; tail sampling can retain errors and slow traces but needs more state and capacity. Preserve enough traffic to understand normal baselines, always retain material failures where feasible and document which conclusions sampled data cannot support. Test instrumentation overhead under production-shaped load.

Use Application Insights for Application Behavior

Application Insights can provide request and dependency telemetry, exceptions, availability results, browser performance and distributed traces within Azure Monitor. Define the resource topology before connecting services: environment and ownership boundaries, regions, workspace use, access, retention and whether several services should be queried together. Avoid a single undifferentiated resource that makes permissions, cost and incident scope ambiguous.

Standardize operation names and exclude identifiers from names and metric dimensions. Enrich telemetry with safe service, deployment and tenant class attributes, never raw secrets, tokens, request bodies or personal data. Treat browser collection separately: consent, source maps, IP handling and sampling need explicit privacy and security choices.

Validate automatic dependency detection and manual spans against the actual path. Check health endpoints, background work, queues, retries and external APIs. Synthetic availability tests should represent a useful journey and run from relevant locations, but they do not replace real-user or server evidence. Use release annotations or deployment attributes to compare behavior before and after change.

Use Log Analytics with a Query and Retention Purpose

Send structured records with consistent fields rather than sentences that require parsing. Define timestamp, severity, service, environment, operation, trace and span identifiers, safe tenant context, deployment version and event code. Put variable details in fields and keep message templates stable. Log decisions and material state transitions, not every method entry.

Design Log Analytics tables, access and retention from investigation, audit, residency and cost requirements. Separate hot operational queries from long-term compliance archives where appropriate. Use role-based access and private connectivity according to the threat model. Redact at the source because downstream filters cannot reliably undo sensitive data already emitted.

Maintain a small library of tested KQL queries for critical journeys, release comparison, failed dependencies, queue delay, authorization anomalies and trace lookup. Store queries and alert rules as code with owners and review dates. Measure ingestion delay, missing fields and schema drift. During an incident, operators need predictable queries and timestamps they can trust.

Preserve Correlation Across Every Asynchronous Boundary

Adopt W3C trace context at every supported boundary and propagate it deliberately through HTTP, messaging and scheduled work. A producer span should connect to consumer processing without pretending that time in a queue is server execution. Preserve message and business identifiers for reconciliation, but keep high-cardinality values out of metric dimensions.

Retries, batches and fan-out need explicit modeling. Record attempt number, retry reason and final outcome; link work when a strict parent-child tree would be misleading. Instrument database and external calls with safe operation names, not raw statements or URLs. Ensure asynchronous frameworks and SDK upgrades do not silently drop context.

Prove correlation with a representative journey from browser or edge through API, queue, worker, database and third party. Inject latency and failure, then ask an on-call engineer to find the limiting span and related logs. Check sampled and unsampled behavior, clock synchronization and correlation after redelivery. A trace is valuable when it shortens an actual investigation.

Build Alerts and Dashboards Around Decisions

Start alerts from service objectives and customer harm. Page for urgent conditions that require immediate human action: sustained error-budget burn, failed critical journeys, uncontrolled queue age or loss of a required dependency. Route lower-urgency capacity, cost and data-quality signals to owned work queues. Static infrastructure thresholds are supporting evidence, not the definition of an outage.

Every alert needs an owner, severity, destination, response expectation, diagnostic links and a runbook with safe first actions. Group related symptoms, suppress known maintenance carefully and use multiple windows to detect both fast and slow burn. Test the route outside office hours and review false positives, missed incidents and time to useful action.

Create dashboards for audiences and decisions. Operators need current journey health, dependencies, saturation, changes and drill-down. Product owners need completion and failure by meaningful segment. Executives need objective trends and material risk, not thousands of charts. Version dashboards and alert rules, annotate releases and incidents, and retire panels that no longer influence a decision.

Control Telemetry Cost and Connect It to Incident Work

Model telemetry cost as ingestion, processing, queries, retention, archive, export and operational time. Attribute it by service, environment, signal and owner. Find unbounded labels, duplicate ingestion, debug logs, verbose health checks and long retention before indiscriminately dropping data. Set budgets and anomaly alerts, but preserve the evidence required for service objectives, incidents and obligations.

Apply a hierarchy: prevent unnecessary emission, aggregate metrics, use structured events, sample traces by risk, filter known noise and choose retention by purpose. Debug verbosity should be time-bounded and approved. Rehydrating an archive takes time, so match storage tier to the incident response expectation. Measure the cost per active customer, transaction or another useful unit alongside reliability.

During an incident, record a timeline, hypotheses, queries, trace identifiers, decisions and changes. Preserve evidence with controlled access. After recovery, link customer impact to detection, diagnosis and repair gaps; improve instrumentation, alerts and runbooks with owners and deadlines. Observability earns its cost when later incidents become easier to detect, explain and prevent.

When Independent Observability Expertise Is Useful

Independent review is useful when teams cannot follow one critical journey across services, alerts are noisy, incidents depend on individual memory, telemetry cost grows unpredictably, sensitive data may be emitted or an OpenTelemetry migration crosses many runtimes. Review real traces, logs, metrics, dashboards, alerts, access, retention, incidents and bills rather than scoring a generic tool checklist.

A useful result is a prioritized signal and operating plan: missing evidence, unsafe fields, correlation breaks, cardinality risks, service objectives, sampling and retention decisions, alert owners, runbook gaps, expected cost and acceptance tests. Prove the design on one representative journey before expanding it.

GARNO.TECH can assess the current estate, design OpenTelemetry conventions, implement an end-to-end slice or improve Azure Monitor operations through Azure cloud development services. Configuration, dashboards, queries and runbooks remain client-controlled, with knowledge transfer included.

• Journeys: name critical customer outcomes, service objectives, failure modes and accountable owners before choosing signals.
• Conventions: standardize service, operation, version and safe context attributes with explicit cardinality and privacy limits.
• Correlation: propagate trace context through HTTP, queues and jobs; model retries, batches and fan-out without losing business identity.
• Signal quality: validate missing fields, dropped spans, ingestion delay, schema drift and instrumentation overhead under realistic load.
• Action: route objective-based alerts to named responders with severity, diagnostic links, runbooks and tested escalation.
• Economics: budget ingestion, queries, retention and export; remove waste by purpose while preserving incident and compliance evidence.