SigNoz and OpenTelemetry Python 2026: Self-Host Distributed Tracing and Logs

Modern applications are composed of many services talking to databases, queues, and external APIs. When a request is slow or failing, raw metrics tell you that something is wrong — but not where or why. Distributed tracing closes that gap by recording the exact path a request took through your system, with latency broken down span-by-span.

This opentelemetry python tutorial walks you through installing SigNoz, auto-instrumenting a FastAPI service in minutes, viewing traces and logs in one UI, and setting up alerts — all without paying Datadog prices.

1. The Observability Gap: Metrics Alone Don't Tell You Why

A CPU spike alert fires at 3 a.m. Your dashboard shows elevated request latency across the board. But which endpoint is responsible? Which downstream service? Which specific database query is the bottleneck? Metrics aggregate data across all requests, so you lose the per-request story entirely.

That is the observability gap:

• Metrics tell you how many and how much — perfect for alerting and capacity planning
• Logs tell you what happened on a single node — useful for debugging individual events
• Traces tell you why a specific request was slow, following it across every service it touched

Without distributed traces, a three-service latency regression can take hours to diagnose. With traces, you open the flamechart, click the slow span, and read the database statement — in minutes.

OpenTelemetry (OTel) is the CNCF standard that instruments your code once and exports all three signals to any backend. SigNoz is built natively on OTel and stores metrics, traces, and logs in a single ClickHouse database — no glue code, no separate query languages.

2. SigNoz vs Datadog: Cost Comparison

Datadog charges $15–23 per host per month for infrastructure monitoring, plus separate SKUs for APM, log management, and synthetics. At scale, those numbers compound quickly:

SigNoz is the most widely adopted datadog alternative open source project in the OTel ecosystem. You own your data, control retention, and pay zero per-host or per-seat fees. The ClickHouse backend compresses trace data aggressively, making 90-day retention practical on a single mid-range server.

3. Install SigNoz with Docker Compose

You need Docker Engine 20.10+ and Docker Compose v2. The entire stack — OTel Collector, ClickHouse, query service, and the React frontend — ships in one repository.

git clone -b main https://github.com/SigNoz/signoz.git
cd signoz/deploy
docker compose up -d

The first pull downloads roughly 2 GB of images. Once all containers are healthy, open http://localhost:3301 in your browser. The onboarding wizard prompts you to create an admin account.

Verify the stack is running:

docker compose ps

You should see clickhouse, otel-collector, query-service, frontend, and alertmanager all in the running state.

Tip: On a production server, place SigNoz behind an Nginx reverse proxy with TLS. The docker-compose-minimal.yaml variant drops the load-generator demo service and is recommended for production use.

4. Instrument a FastAPI App with OpenTelemetry Auto-Instrumentation

Auto-instrumentation injects tracing into popular frameworks (FastAPI, SQLAlchemy, httpx, Redis, etc.) without changing a single line of application code. Start with a minimal FastAPI app:

# main.py
from fastapi import FastAPI

app = FastAPI()

@app.get("/items/{item_id}")
async def read_item(item_id: int):
    return {"item_id": item_id, "status": "ok"}

Install OTel packages

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install

opentelemetry-bootstrap scans your installed packages and automatically installs the matching instrumentation libraries — for example opentelemetry-instrumentation-fastapi and opentelemetry-instrumentation-sqlalchemy if those packages are present.

Set environment variables

export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_SERVICE_NAME="items-service"
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment=production"

OTEL_EXPORTER_OTLP_ENDPOINT points at the SigNoz OTel Collector. Port 4317 is gRPC; port 4318 is HTTP/protobuf. Both are exposed by the Docker Compose stack by default.

Run with the instrumentation wrapper

opentelemetry-instrument uvicorn main:app --host 0.0.0.0 --port 8000

That single command wraps your process, monkey-patches all detected libraries, and starts exporting spans to SigNoz. Send a few requests:

curl http://localhost:8000/items/42

Within seconds you will see items-service appear in the SigNoz Services list.

5. View Traces in the SigNoz UI

Navigate to Traces in the left sidebar. You will see a list of recent requests with columns for duration, status code, and service name. Click any row to open the flamechart (waterfall view):

• Each horizontal bar is a span — a single unit of work with a start time and duration
• Spans nest to show parent-child relationships (HTTP handler spans contain DB query spans)
• Hovering a span reveals its attributes: HTTP method, DB statement, error messages

The Service Map tab (under APM) renders your architecture as a directed graph. Edge labels show request rate and error rate between services, making it immediately obvious which dependency is responsible for elevated latency.

6. Add Custom Spans with the Python SDK

Auto-instrumentation covers framework-level spans. For business logic — an expensive pricing calculation, a third-party API call, a cache lookup — add custom spans manually:

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

@app.get("/price/{sku}")
async def get_price(sku: str):
    with tracer.start_as_current_span("db_query") as span:
        span.set_attribute("db.sku", sku)
        price = await db.fetch_price(sku)
        span.set_attribute("price.value", price)
    return {"sku": sku, "price": price}

start_as_current_span is a context manager that closes the span automatically — and records any exception as an error — when the block exits. Child spans created inside the block (including those from auto-instrumented libraries) nest under this span in the flamechart.

7. Log Correlation: Link Log Lines to Trace IDs

Correlating log lines to the trace that generated them lets you jump from a noisy log stream directly to the offending trace. The OTel Python SDK injects the current trace_id and span_id into your log records automatically.

import logging
from opentelemetry.instrumentation.logging import LoggingInstrumentation

LoggingInstrumentation().instrument()

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

@app.get("/checkout")
async def checkout():
    logger.info("Processing checkout")   # trace_id is injected automatically
    ...

Configure your logger to emit JSON and forward logs to SigNoz via the OTel Collector's filelog receiver or by exporting directly over OTLP. In the SigNoz Logs tab, filter by trace_id to see every log line produced during a single request — regardless of which replica or pod emitted it.

8. SigNoz Alerts: P95 Latency Alert Setup

SigNoz ships with a built-in alerting engine backed by Alertmanager. To create a P95 latency alert:

1. Open Alerts in the left sidebar and click New Alert Rule
2. Choose Trace as the data source
3. Set the query: p95(duration_nano) > 500000000 (500 ms expressed in nanoseconds), with a filter on service.name = items-service
4. Set the evaluation window to 5 minutes and severity to warning
5. Route notifications to Slack, PagerDuty, or a webhook

Alerts fire based on the same ClickHouse data that powers the dashboards — there is no sampling gap between what you see and what triggers the alert.

9. SigNoz vs the Grafana Stack

Many teams already run Prometheus + Grafana. Adding tracing usually means bolting on Grafana Tempo, and log aggregation via Loki — three separate systems to configure, scale, and maintain.

The Grafana stack is mature and flexible, but each component has its own retention policy, query language, and scaling story. SigNoz trades some flexibility for significantly lower operational overhead: one data store, one query engine, one UI.

FAQ

Can I run SigNoz without Docker? Yes. A Helm chart is available for Kubernetes (helm install signoz signoz/signoz). A bare-metal install guide using the install.sh script is also documented in the SigNoz docs.

Does SigNoz support Python 2? No. The OpenTelemetry Python SDK requires Python 3.8+. Python 2 reached end-of-life in January 2020.

What if my service already sends traces to Jaeger or Zipkin? The SigNoz OTel Collector accepts Jaeger Thrift (port 14268), Zipkin (port 9411), and OTLP. Point your existing exporters at SigNoz and you are done — no application code changes required.

How do I handle high-cardinality trace data at scale? Use OTel's tail-sampling processor in the Collector to drop healthy traces and keep only slow or error traces. The SigNoz documentation includes a reference tail-sampling configuration.

Is SigNoz production-ready in 2026? Yes. SigNoz hit GA in 2023 and is used in production by hundreds of companies. The ClickHouse backend has been battle-tested at petabyte scale, and the project ships monthly releases with an active community.

Next Steps

• Add opentelemetry-instrumentation-sqlalchemy and opentelemetry-instrumentation-redis to capture database and cache spans automatically
• Explore the Dashboards tab to build custom ClickHouse-powered charts for your business metrics
• Use the Exceptions page to track unhandled Python exceptions with full stack traces linked to their parent traces
• Try the SigNoz Cloud free tier if you want managed hosting without running ClickHouse yourself

SigNoz gives you Datadog-class observability at a fraction of the cost. With OpenTelemetry as the instrumentation layer, you are never locked in — switching backends requires only a Collector config change, not an application redeployment.