The LLM Cost Tracker: Know Exactly What Your AI Features Are Costing, Per Request
Per-model pricing, token-level accounting, monthly totals per product — FlowSquad's built-in cost tracker turns your AI spend from a mystery into a line item.
- FlowSquad's LLM Cost Tracker records input tokens, output tokens, model name, feature name, product, and user for every AI request — giving FinOps teams per-request granularity, not just monthly totals.
- The per-model pricing table is admin-configurable, so cost tracking stays accurate even when providers change their prices — update the rate once and all historical calculations refresh automatically based on stored token counts, not stored dollar amounts.
- Switching a high-volume product from GPT-4 Turbo to Claude 3.5 Sonnet reduces typical monthly AI spend by 65–70% for equivalent feature usage, based on FlowSquad customer telemetry across 12 enterprise deployments.
Why Is AI Spend So Hard to Track?
Most teams discover their LLM spend problem when they get the first invoice. OpenAI, Anthropic, and Azure provide usage dashboards, but these show aggregate spend for the entire API key — not broken down by which application, which team, which feature, or which use case drove the cost. A single API key shared across multiple products or environments makes attribution impossible. You know you spent $4,200 on GPT-4 last month; you have no idea whether that was Epic generation, AskProduct conversations, test case generation, or a runaway background job.
The problem worsens with multi-product platforms. An enterprise running 15 FlowSquad products across 6 business units cannot allocate AI costs to cost centers without per-product token accounting. Finance needs a number they can put in a budget line; engineering needs a number they can optimize; product managers need a number they can justify in ROI calculations. Generic provider dashboards deliver none of this granularity.
FlowSquad's LLM Cost Tracker was built to make AI spend as observable as any other infrastructure line item. Every LLM call goes through the cost tracking middleware before the request is dispatched to the provider, ensuring 100% coverage — no requests escape accounting, including background jobs, regeneration attempts, and failed calls that still incur tokens.
How Does FlowSquad Track Tokens Per Request?
The LLM cost tracking middleware intercepts every call to FlowSquad's provider registry. Before the call, it records the request context: product ID, user ID, feature name (e.g., epic_generation, story_generation, askproduct_query, hld_generation, test_case_generation), and model name. After the call, it extracts token counts from the provider response metadata — OpenAI, Anthropic, and Azure all return input and output token counts in their API responses. For Ollama, token counts are estimated from the response using the same tiktoken tokenizer to maintain accounting consistency.
Token counts, model name, feature name, timestamp, product ID, and user ID are written as a single row to the llm_usage_log table. Dollar costs are not stored — instead, they are computed dynamically at query time by joining the token log with the current pricing table. This means updating a model's price rate retroactively recalculates all historical cost figures without any data migration.
What Does the Per-Model Pricing Table Look Like?
The LLM Pricing Table is maintained in Admin → Cost Tracker → Pricing Configuration. Prices are stored as USD per 1,000 tokens (input and output separately) and are updated manually by the Global Admin when providers change their rates. The table ships with default rates that reflect pricing as of Q1 2026, but these are fully editable. Monthly cost estimates below assume a typical medium-usage FlowSquad product: 100 Epic/Story generation runs, 500 AskProduct queries, and 200 test case generation runs per month.
| Model | Input $/1k tokens | Output $/1k tokens | Est. Monthly Cost (medium usage) | On-Premise |
|---|---|---|---|---|
| GPT-4 Turbo | $0.010 | $0.030 | ~$420/month | No |
| GPT-4o | $0.005 | $0.015 | ~$210/month | No |
| Claude 3.5 Sonnet | $0.003 | $0.015 | ~$145/month | No |
| Claude 3 Haiku | $0.00025 | $0.00125 | ~$12/month | No |
| Azure OpenAI GPT-4o | $0.005 | $0.015 | ~$210/month | Azure VNet |
| Ollama Mistral 7B | $0.000 | $0.000 | $0/month | Full air-gap |
| Ollama Llama2 13B | $0.000 | $0.000 | $0/month | Full air-gap |
FinOps insight: The delta between GPT-4 Turbo (~$420/month per product) and Claude 3 Haiku (~$12/month per product) for the same medium-usage workload is $408/month per product. For an enterprise running 15 products, that's $6,120/month ($73,440/year) in savings by switching to a lighter model for routine Story/Task generation — with GPT-4 Turbo reserved for complex HLD/LLD generation where output quality justifies the cost.
How Do You View Monthly Costs Per Product?
The Cost Tracker Dashboard is accessible to Global Admins and Product Admins from Admin → Cost Tracker. The dashboard displays: total input tokens, total output tokens, and total estimated cost for the selected period (daily, weekly, monthly, custom range), broken down by product, feature, model, and user. Charts show cost trends over time, and a top-10 highest-cost features table helps identify optimization opportunities.
Product Admins see only their own product's data; Global Admins see all products with a cross-product comparison view. The dashboard data is exportable as CSV for integration with FinOps tools, internal billing systems, or Tableau/Power BI dashboards. Automated monthly cost reports can be emailed to specified addresses from Admin → Cost Tracker → Report Schedule — with a configurable cost threshold alert that emails the Global Admin if any product exceeds a defined monthly spend ceiling.
How Does FlowSquad Handle Zero-Cost Local Models (Ollama)?
Ollama models have a configured price of $0.00/1k tokens in the pricing table, which is accurate for API cost — but does not reflect infrastructure cost (GPU/CPU compute, electricity, server hardware). FlowSquad handles this via an optional infrastructure cost override for Ollama: Product Admins can set a per-1k-token infrastructure rate (e.g., $0.001) that represents the amortized compute cost of running the local model. This cost is tracked separately from API costs in the dashboard, labeled "Infrastructure (Ollama)" — allowing honest total-cost-of-AI accounting even for on-premise deployments.
This feature is particularly valuable for procurement teams who need to justify GPU server purchases by quantifying their AI cost displacement. A team spending $3,500/month on GPT-4 that switches to Ollama on a $15,000 GPU server recovers the hardware investment in 4.3 months — a calculation FlowSquad's cost tracker makes explicit and auditable.
Frequently Asked Questions
Does the cost tracker capture failed requests (e.g., rate limit errors)?
Yes. Requests that return a rate limit error (HTTP 429) after tokens have been consumed by the provider are still logged with their token count and marked success=false. This is important because some providers charge for tokens processed even on failed requests. Requests that fail before any tokens are consumed (network error, authentication failure) are logged with zero tokens and success=false for completeness.
Can I set spending limits per product to prevent runaway costs?
Yes. Admin → Cost Tracker → Spending Limits allows Global Admins to set a monthly token budget per product. When a product approaches its limit (configurable warning at 80%), a notification email is sent to the Product Admin. When the limit is reached (100%), AI generation features are soft-blocked (with an "over budget" message) until the next month or until the limit is raised — preserving read-only access to existing content.
How accurate are token count estimates for Ollama models?
FlowSquad uses the OpenAI tiktoken tokenizer (cl100k_base encoding) to estimate token counts for Ollama models, since Ollama's HTTP API does not return token counts in its response metadata. This estimate is accurate to within ±5% for most English-language prompts. For non-English codebases or high-symbol-density code, tokenization may diverge more significantly. A future Ollama adapter version will use Ollama's /api/show endpoint to detect the model's actual tokenizer.
Can cost data be exported to external FinOps tools like Apptio or Cloudability?
FlowSquad's Cost Tracker CSV export is compatible with standard FinOps import formats. For deeper integration, the /api/v1/admin/cost-tracker/export endpoint returns paginated JSON with all usage log rows, queryable by date range, product, model, and feature. This endpoint is used by several FlowSquad enterprise customers to feed cost data into internal FinOps dashboards built on Grafana, Tableau, or Power BI.
Does cost tracking introduce latency to LLM requests?
No measurable latency. The cost tracking write is asynchronous — it is dispatched to a background task queue immediately after the LLM response is returned to the user, without blocking the response delivery. Under normal conditions, the cost record is written within 50ms of request completion. The database write uses a bulk-insert buffer that flushes every 5 seconds to minimize DB write amplification during high-concurrency periods.