Telemetry
For AI products with model usage, token spend, and latency risk

OpenAI Cost Tracking

Track LLM calls by model, feature, user, cost, latency, error rate, and outcome so growth does not hide margin problems.

Get an API keyCopy the prompt
Why this works
  • Model spend by feature and customer account.
  • Latency and failure trends before they become support tickets.
  • Acceptance or save rate for generated outputs where the app exposes it.

Agent prompt

Paste this into your coding agent

Replace YOUR_API_KEY after signup, then ask the agent to run the product flow and verify the first events.

agent prompt

OpenAI Cost Tracking setup prompt

text
Instrument this project with Telemetry so we can understand OpenAI and LLM usage.

Use /skill.md and this Telemetry API key: YOUR_API_KEY

Please log:
1. Every model request with model, provider, route, feature, input_tokens, output_tokens, total_tokens, estimated_cost_usd, latency_ms, status, and error_type when relevant.
2. Tool calls made by the agent or assistant, including tool_name, status, latency_ms, and result_category.
3. User-facing AI workflow outcomes, including feature, status, retry_count, and whether the user accepted, copied, saved, or discarded the result.
4. A dashboard with daily cost, cost by feature, failures by model, p95 latency, and accepted output rate.

Keep prompts and raw completions out of telemetry unless I explicitly approve storing them.

Setup steps

  1. 1Log each model request without storing raw prompts by default.
  2. 2Record tokens, estimated cost, latency, provider, model, and workflow.
  3. 3Connect product outcome events like saved, copied, accepted, or retried.
  4. 4Create dashboards for margin, reliability, and user value.

Events to capture

llm_request_completedllm_request_failedassistant_tool_calledai_output_acceptedai_output_discardedai_feature_retained

Questions unlocked

  • Which AI features cost the most per activated user?
  • Which model has the best accepted-output rate per dollar?
  • Where are retries or timeouts damaging conversion?

Next step

Create the API key your agent will use

The free plan is enough to run the prompt, send test events, and review the first dashboard.

Prompt packStart free

Related pages

Claude Code Observability

Give Claude Code a prompt that makes telemetry part of the implementation pass instead of a separate cleanup project.

Open page

Cursor Telemetry Setup

A copyable setup prompt for adding event tables, funnel tracking, and dashboards from inside Cursor.

Open page

Codex Instrumentation Prompt

A focused prompt that asks Codex to instrument the rest of the product, verify events, and summarize coverage gaps.

Open page