Django and Celery structured event monitoring

Test the integration

Send and verify events with Django and Celery structured event monitoring

Observe a Celery task attempt, preserve its original result, and verify the recorded state before building charts.

1
Choose the attempt
Celery worker outcomes
2
Define the contract
job_name, job_id, queue_name, observed_state, and status
3
Log the attempt state
This example records SUCCESS, FAILURE, and RETRY from task_postrun as attempt states; a retry is not a final failure.
4
Check the stored event
Run a known task and inspect celery_task_attempt_finished for the observed attempt state; a retry row is not a final result.

Before you start

A server-side TELEMETRY_API_KEY
Celery installed in the worker process
Import this module when your Celery worker starts

Delivery setup

Install and initialize server-side

This Celery worker module sends events with Python standard-library HTTP; no Telemetry SDK installation or initialization is needed. Keep ingestion credentials out of browser bundles, client-visible environment variables, source control, logs, and exception messages.

1Load the signal module once per worker and keep export errors separate from task outcomes.
2Log an event when the operation succeeds, fails, retries, or times out.
3Send test events with known results and inspect the stored rows before enabling an alert.

Snippet

Start with one structured event

Add this shape where the workflow completes, fails, or retries. Then build the dashboard from real fields.

django-celery

Django and Celery structured event monitoring event

python

import json
import logging
import os
from urllib.request import Request, urlopen
from celery.signals import task_postrun


def report_export_failure():
    try:
        logging.getLogger(__name__).warning("Telemetry export unavailable")
    except Exception:
        pass


@task_postrun.connect(weak=False)
def record_task(sender=None, task_id=None, state=None, **kwargs):
    try:
        status = {"SUCCESS": "success", "FAILURE": "failed",
                  "RETRY": "retry_scheduled"}.get(state)
        if status is None:
            return  # Other states need their own observed lifecycle handlers.
        key = os.getenv("TELEMETRY_API_KEY")
        if not key:
            report_export_failure()
            return
        delivery = sender.request.delivery_info or {}
        payload = {
            "table": "celery_task_attempt_finished",
            "data": {
                "job_name": sender.name,
                "job_id": task_id,
                "queue_name": delivery.get("routing_key"),
                "observed_state": state,
                "status": status,
                "attempt": sender.request.retries + 1,
            },
        }
        request = Request("https://api.telemetry.sh/log",
                          data=json.dumps(payload).encode("utf-8"),
                          headers={"Authorization": key,
                                   "Content-Type": "application/json",
                                   "User-Agent": "telemetry-agent/1.0"},
                          method="POST")
        # Socket timeout, not a hard end-to-end deadline. Export can be lost.
        with urlopen(request, timeout=1.0) as response:
            if not 200 <= response.status < 300:
                report_export_failure()
    except Exception:
        # Do not let export or metadata errors replace the task outcome.
        report_export_failure()

Event schema

job_name, job_id, queue_name, observed_state, and status

attempt is retries + 1 for the observed execution

Only worker metadata; review task and queue identifiers before export

Check your setup

Checkpoint 1

This example records SUCCESS, FAILURE, and RETRY from task_postrun as attempt states; a retry is not a final failure.

Checkpoint 2

Preserve the logical task ID across retries and record retries + 1 as the attempt number.

Checkpoint 3

Import this module once in each worker process. It does not instrument Django requests or measure task duration.

Checkpoint 4

Export is best effort with a one-second socket timeout; it adds worker latency and does not implement a hard wall-clock deadline, durable delivery, or exporter retries.

Checkpoint 5

Keep task arguments, results, exception text, URLs, and credentials out of the event. Worker loss or termination can prevent task_postrun, so missing events are unknown.

Verification

Prove the event arrived

Run this after exercising known success and failure cases. Replace the fallback table name if your final event contract differs from the snippet.

django-celery-verification

Django and Celery structured event monitoring verification query

sql

SELECT *
FROM celery_task_attempt_finished
ORDER BY timestamp_utc DESC
LIMIT 20;

Confirm one terminal row per logical outcome, with the expected status, identifiers, units, and UTC time.

Inspect the inferred schema and verify that retries do not change field types or generate a new logical event ID.

Search the stored fields for credentials, raw payloads, prompts, private content, and unbounded error messages.

Exercise a provider timeout, ingestion rejection, and process shutdown before treating the dashboard as complete.

Implementation references

Review the event contract, data-safety guidance, and upstream primary documentation before enabling a new production path.

Python SDK Queue worker guide Celery task signals Django logging

Where to log

Keep the outcome event small and recoverable

This pattern provides

Record the outcome as an event you can query with SQL.
Stable fields for dashboards, alerts, and cross-event correlation.
Test events for checking success, failure, retries, and timeouts.

This pattern does not provide

An OTLP exporter, automatic collection pipeline, or replacement for detailed traces and diagnostic logs.
Exactly-once delivery merely because the payload contains an event ID.
Permission to collect raw provider payloads, user content, credentials, or regulated data.

Delivery and idempotency Batching and shutdown Troubleshoot ingestion

Example event schemas

Check what each event records, when to send it, and which field types it needs. Review the example payload and privacy checklist before using it in production.

Async reliability11 fields