Background Job Failure Monitor: implementation to decision
Treat the prompt as an implementation brief. The useful artifact is not copied code alone, but a reviewed event contract that produces a trustworthy answer.
- 1
Select the boundary
Instrument the point where job_started becomes final.
- 2
Create the contract
Start with job_started, job_completed, job_failed and keep every field typed, bounded, and privacy-reviewed.
- 3
Run a fixture
Exercise known success, failure, retry, and empty-result cases before relying on aggregate results.
- 4
Answer the question
Which jobs are failing repeatedly?
Template versus use case
This page is the implementation brief
Copy this template when the measurement goal is already clear. Use the matching use-case guide to review event boundaries, success definitions, and the decisions the resulting SQL should support.
Template
Paste this into your coding agent
Replace YOUR_API_KEY, run the flow locally, then verify the generated events and dashboards.
Background Job Failure Monitor
Instrument background jobs with Telemetry.
Use /skill.md and this Telemetry API key: YOUR_API_KEY
Log job_started, job_completed, job_failed, job_retried, and dead_letter_created with:
job_name, queue_name, attempt, status, duration_ms, scheduled_at, started_at, completed_at, item_count, retry_count, worker_name, and error_type.
Create queries and a dashboard for throughput by job, failures by job, p95 duration, retry volume, stalled jobs, and newest dead-letter events.
Do not log raw job payloads, credentials, webhook bodies, or customer content.Events to capture
Verification checklist
What a complete instrumentation pass leaves behind
Events
Synthetic events reach the intended table with stable names and field types.
Queries
The first SQL queries return plausible rows with an explicit time window.
Views
A dashboard uses the real fields and includes enough context to explain a change.
Safety
Prompts, bodies, credentials, signatures, and private content were checked for redaction.
Event schema starting points
Event contracts for this workflow
Review the row grain, emit boundary, required types, privacy classes, example payload, and validation checklist before adapting a query or snippet to production.
Related product capability
Continue this workflow in Alerts
Promote the reviewed reliability query into an owned threshold and response workflow.
Related SQL recipes
Answer the next question with SQL
Run the query against the structured fields from this workflow, inspect the example result, and turn a useful answer into a dashboard or alert.
Measure Background Job Retry and Failure Rate
Which background jobs consume the most retries or still fail?
Open recipeFind Stalled Background Jobs With SQL
Which jobs started but never produced a terminal event?
Open recipeMeasure Dead-Letter Queue Growth
Which queues are adding dead-letter jobs faster than they are resolving them?
Open recipeDetect Missed Cron Schedules
Which scheduled jobs ran later than their documented interval?
Open recipeDetect Background-Job Retry Storms
Which job types are spending the most work on retries right now?
Open recipeMore templates