The average was green. One customer was on fire.

Code: github.com/RajdeepKushwaha5/TrueSLA
Demo video: youtu.be/z8B42TCVReQ
Live: true-sla.vercel.app

I did not set out to build an SLA tool. I set out to answer a smaller question: if my dashboard says the service is at 99.9%, which customer is that 99.9% lying to?

SigNoz already had the answer sitting in its traces. It just was not saying it out loud. TrueSLA is the thing that makes it testify.

Here is the shape of the whole project before I go day by day. My demo service ships OpenTelemetry traces into SigNoz, and every request carries the customer it belongs to. TrueSLA reads those traces back through the SigNoz MCP, computes availability for each customer, compares it with that customer’s contract, and opens a breach when the measured window crosses the target. It also estimates the current credit exposure, creates the customer’s alert in SigNoz, and writes a checksummed evidence bundle with sample failed traces and the dependency where most errors appeared.

Take SigNoz away and the product has no telemetry to measure and no trace to show as proof. SigNoz is not a logo I added to the architecture slide. It is the source of the measurements, the home of the alerts and cross-signal dashboard, and the place where TrueSLA’s own refresh and MCP calls can be inspected.

The demo uses a synthetic shop. A “credit” is a row in a local ledger, not a real invoice. In the demo it is a short-window estimate of contractual exposure, not a claim that accounting should automatically pay.

The killer moment, in one screen

If you only look at one thing, look at this. I have five customers running through the same service, and one dashboard number for the whole thing.

I break one customer. Not the service, just one tenant, acme. A fault switch makes only Acme's requests fail.

GLOBAL AVERAGE    99.9%   still green, still calm
  acme            97.9%   BREACH   $1,150 credit   burning toward the next tier
  globex         100.0%   ok
  initech        100.0%   ok
  umbrella       100.0%   ok
  hooli          100.0%   ok

Those are real numbers from my own run, not a mockup. The global average barely moved, because Acme is a small, high-value customer, a thin slice of total traffic. A team watching the average would see green and go back to lunch. Meanwhile Acme has blown through its tight 99.9% target, the demo schedule shows $1,150 of provisional exposure, and TrueSLA is estimating the time to the next tier.

Every measured number on that screen came out of SigNoz. The breach has a real alert behind it, plus evidence that points to the likely failing dependency. When I restore Acme, new failures stop; the rolling window keeps the incident visible until the failed requests age out.

This is not hypothetical

I did not invent this problem. Two things about it are already well documented.

The aggregate dashboard lies during real outages. In the big AWS us-east-1 outage on December 7, 2021, AWS’s own status page stayed green for over an hour while services were failing hard, partly because the outage had also knocked out the tooling behind the status page. The single green light said “fine” while customers were very much not fine. (ThousandEyes analysis)

Getting an SLA credit is often a claim process, not an automatic refund. The exact terms vary by provider, but the burden can sit with the customer: notice the incident, file before the deadline, identify the affected tenant or service, and provide evidence. Microsoft’s current Partner Center guidance, for example, asks for the customer tenant ID and outage ID and explicitly says that ordinary ticket details alone are not enough. (Microsoft: request a credit)

That is the part I wanted to attack. If the service already has tenant-tagged telemetry, the evidence should not have to be reconstructed from screenshots weeks later. It should be captured while the incident is happening.

So the problem is real on both ends. The average hides which customer is failing, and the customer who is failing usually cannot prove it in time to get paid back. TrueSLA computes the per-customer truth continuously and keeps the proof, from the traces you already have.

This is the build log, one day at a time, from the start of the Agents of SigNoz hackathon on July 20, 2026 to the end on July 26. Each day is a chapter, including the parts I got wrong and had to redo.

Day 1 (July 20, 2026): The gap I could not unsee

Day one was setup. I needed a service that looked like a real multi-tenant app, sending traces to SigNoz, with each request tagged by customer. I wrote a small “shopfront” with a real dependency chain: a checkout calls a payment step, which calls a ledger database query. Every span gets a customer.tenant attribute.

The first per-customer number

Once traffic was flowing, I asked SigNoz a simple question through the MCP: group the requests by customer, and count how many had errors. Five customers came back, each with its own request count and its own p95 latency, all computed from real spans and not from anything I made up.

globex   2043 reqs   p95 143ms   100.0%
initech  1694 reqs   p95 145ms   100.0%
umbrella 1115 reqs   p95 134ms   100.0%
hooli     558 reqs   p95 153ms   100.0%
acme      286 reqs   p95 149ms   100.0%

Everything was healthy, which was correct, because nothing was broken yet. Note that Acme is the smallest slice of traffic here but, as I set up later, the highest-value contract. That mix is deliberate, and it is exactly why the global average will lie about Acme. Seeing five separate numbers instead of one blended number was the whole idea in miniature: the moment I broke Acme, four of those rows would stay at 100 and one would fall, and the blended average would barely notice.

Why per-customer and not just per-service

A normal service dashboard tells you the service is up. It cannot tell you that the service is up for four customers and down for the fifth, because it adds them all together first. The information is in the traces the whole time. It just gets averaged away the moment you stop grouping by customer. Day one was me proving to myself that the grouping was the entire trick, and that SigNoz could do it on real data.

Day 2 (July 21, 2026): Reading the truth out of SigNoz

With traffic flowing, I built the part that reads per-customer health back out of SigNoz and turns it into a number I can trust.

Availability is a formula, not a count

My first instinct was to count errors. That is wrong, and it is wrong in a way that would embarrass me in front of a judge. Ten errors means nothing without the denominator. Ten errors out of ten thousand requests is a great day. Ten out of twenty is an outage.

So availability is computed as a real fraction, per customer, from the traces:

availability = 1 - (error spans / total spans)   for this customer, in the window

I pull the totals and error counts as two grouped aggregations through the SigNoz MCP, divide per customer, and that is the availability SLI. I also ask SigNoz for p95 latency instead of averaging it, because a mean can hide the slow tail. The current breach decision is based on availability; latency is shown beside it as operational context, not silently mixed into the credit calculation.

The queries use SigNoz Query Builder v5 rather than reaching into ClickHouse directly. That kept the project on the same public query path an operator uses inside SigNoz, and made the query stored in a breach understandable outside my Python code. SigNoz documents the same errors / total pattern for trace-based error-rate alerts. (Trace-based alerts)

The bug that made everything take 21 seconds

Every call to the MCP took about 21 seconds, no matter how small. I assumed the database was slow. It was not. On Windows, localhost resolves to the IPv6 address first, that attempt fails, and then it waits for the fallback to IPv4.

http://localhost:8000/mcp     ~21,000 ms
http://127.0.0.1:8000/mcp        ~575 ms

One word, localhost to 127.0.0.1, and every query got about forty times faster. I now use 127.0.0.1 for anything I open a socket to, and only keep localhost for links a human clicks.

SigNoz kept looking like it was crash looping

Every SigNoz container kept showing “Up 4 seconds” no matter how long I waited. I went digging for a broken cluster. There was none. WSL2 shuts its virtual machine down when nothing is running inside it, so each command I ran from Windows booted the whole stack, ran, and let it fall asleep again. The fix is to keep one process alive inside WSL so the machine stays up. My start script does that first now, because otherwise SigNoz would quietly die in the middle of a demo.

Day 3 (July 22, 2026): Contracts, credits, and one honest number

A per-customer availability number is only interesting next to what the customer was promised. So day three was contracts and money.

Contracts and credit tiers

Each customer gets a contract: an availability target, a latency target, and a monthly fee. Acme is on 99.9% for $4,600 a month. Hooli is on a cheaper 99.0% for $500. When a customer drops below their target, the demo credit schedule maps the measurement to a percentage of that customer’s monthly fee:

below 99.9%   ->  10% credit
below 99.0%   ->  25% credit
below 95.0%   ->  50% credit

The credit is a percentage of that customer’s fee, so the same measured failure can represent very different exposure for Acme and Hooli. That is the point. A blended availability number cannot tell me the risk attached to one customer’s contract. A per-customer measurement can.

There is an important qualifier: the demo applies those tiers to a rolling ten-minute measurement window so a judge can see the lifecycle in one sitting. A production contract would normally settle over a fixed billing period and apply exclusions from the actual agreement. I therefore label the number provisional exposure. It is useful for deciding how urgently to respond; it is not an invoice generator.

The consistency rule I made myself follow

Early on my headline availability and my credit amount disagreed by a rounding step, because they came from two slightly different reads. That is the kind of small bug that destroys trust in a money-shaped number. So I made a rule: one refresh produces the availability, tier, amount, summary inputs, and evidence snapshot together. The snapshot history is append-only; the breach record keeps the worst measurement seen so far. If two numbers on the page disagree, that is a bug, not a harmless rounding difference. There is a test for that rule.

Day 4 (July 23, 2026): The Credit Clock

Detecting a breach after the fact is what every SLA report already does, at the end of the month, when it is too late to do anything. I wanted the number that a support lead would actually act on: how long until this gets worse.

Counting down to the next credit tier

When a customer is failing, TrueSLA watches how quickly new errors are arriving and estimates when the rolling error fraction could cross the next credit boundary. That estimate is the Credit Clock.

acme is at 98.3%, target 99.9%
burning toward the 99.0% tier
next tier: 25% credit ($1,150 more)
time until it becomes unavoidable: about 20 seconds

It reframes the screen from “here is a report” to “this is getting more expensive while we are looking at it.” When new errors stop, the clock stops advancing. I keep calling it an estimate because the current version extrapolates recent error arrivals; production forecasting should also model incoming successful traffic and the exact billing-period boundary. That is one of the pieces I would harden before anyone used the clock for settlement.

Being honest when there is not enough data

A customer with three requests in the window does not have a trustworthy availability number, and I refuse to open a breach or quote a credit off three requests. Below a minimum request count a customer is marked low-data, not breached. It is a small thing, but quoting a credit off noise is exactly how you lose the room.

Day 5 (July 24, 2026): Proof, cause, and the alert in SigNoz

A number that moves money has to be defensible. Day five was about making every breach carry its own proof and its own alert.

Evidence as code

When a breach opens, TrueSLA builds an evidence snapshot. It holds the query expression, measured values, configured window, sampling disclosure, calculation version, sample failing trace IDs, and a checksum over the bundle. Later worsening measurements are appended to the snapshot history, and the breach record is updated to the worst point seen. The checksum makes accidental or casual edits visible. It is not a substitute for signed, off-host evidence, but it is much stronger than “trust me, Acme was down.”

evidence: {
  query_expr: "1 - errors/total WHERE service='shopfront' ...",
  measured:   { availability: 79.5%, errors: 50, total: 244 },
  likely_dependency: { span: "ledger.db.query", error_spans: 58,
                       co_failing: ["payment.authorize"] },
  sampling:   "100%",
  checksum:   "sha256:50870dae09d21216"
}

The evidence points to the likely dependency

The part I did not expect to be so satisfying: the evidence does not stop at “Acme is down.” TrueSLA ranks the names of failing spans for that customer. In my demo, ledger.db.query rises to the top, below the checkout the customer actually called. That gives the responder a useful first place to look.

I am careful with the word causal here. The current ranking is evidence of where errors concentrate; it does not yet walk one trace’s parent_span_id chain and prove the exact propagation path. The report links representative failed traces so a human can inspect the hierarchy in SigNoz. Walking that hierarchy automatically is the next version.

The alert lives in SigNoz, and it is created before the breach

This is a fix I made after an early version got it backwards. At first I created the alert when a breach opened, which is post-mortem alerting: by the time the alert exists, the contract has already failed. So now TrueSLA provisions a per-customer error-rate alert for every contract at startup, and when a new contract is added, through the SigNoz MCP. The alert is already watching before anything goes wrong. Each customer gets a formula of errors / total and a threshold derived from that customer's allowed error fraction. A page about Acme is therefore genuinely about Acme, not a global threshold its small traffic may never move. The rule ID is stored on the breach so the evidence page can open the live alert in SigNoz.

The code and UI originally called this a “burn-rate” alert. Strictly speaking, the metric I emit is recent errors per second, while a normalized SRE burn rate is observed error fraction divided by allowed error fraction. The trace alert’s formula and threshold are correct for detecting a budget violation; normalizing the exported metric and adding multi-window burn alerts is still on my production list.

The local model narrates, it does not decide

There is a plain-English summary on each breach, written by a small local model through Ollama. It only ever narrates numbers the deterministic engine already computed, and if the model is down the summary falls back to a template. It also only re-writes the summary when a breach hits a new worst point, so the fast refresh loop is not paying for a language model on every pass. No third-party model API key is required.

Day 6 (July 25, 2026): Making it live, and watching itself

By day six the engine worked. This day was about making it feel alive and making SigNoz genuinely load bearing, not decorative.

The loop is the agent

I did not want to put a chat box beside a dashboard and call it an agent. TrueSLA’s loop runs whether anybody is looking at the page:

1. Observe: query tenant-scoped traces and derived telemetry from SigNoz.
2. Decide: apply deterministic SLI, support, contract, and tier rules.
3. Act: open or update the local breach record, emit telemetry, and keep the tenant’s SigNoz alert and dashboard ready.
4. Explain: attach evidence and optionally ask the local model to turn the measured facts into plain English.

The MCP is what lets that loop both read from and act on SigNoz. I use it for Query Builder requests plus alert, notification-channel, and dashboard operations; I do not query SigNoz’s ClickHouse tables behind its back. (SigNoz MCP server)

The dashboard TrueSLA builds for itself

On startup TrueSLA provisions its own SigNoz dashboard through the MCP, so the per-customer view exists in SigNoz itself and not only in my UI. An SRE should be able to live in SigNoz and still see the per-customer story. It is a cross-signal board: availability, recent error arrival rate, credit exposure and active breaches as metric series, the raw request, error and p95 traces the SLIs are computed from, and the breach log stream, all on one screen.

I had duplicate generated dashboards during development, so the current demo reconciles them by replacing older TrueSLA boards at startup. That gives me one known board for judging. It is deliberately simple rather than the production strategy: the safer long-term version would keep a stable dashboard UUID and update it in place so bookmarks and manual edits are not broken.

The screen moves without a manual refresh

The main ledger arrives over Server-Sent Events. The engine refreshes its SigNoz measurement on a configurable cadence, then pushes changed snapshots to the browser. In my demo configuration that means the screen usually changes within one refresh cycle, without reloading the page. There is also a fallback poll if the event stream disconnects. Watching Acme go from green to red and then seeing the exposure change is what makes the demo land.

TrueSLA observes itself

Every refresh pass, and every call TrueSLA makes to the SigNoz MCP, is itself a span in SigNoz under its own service name. So the thing that watches your service health is watchable with the same tools. If a refresh is slow, that shows up as a span, in the same place as everything else.

It shows up on the services list next to the shop it is measuring, which I like more than I expected to. You can see how long the thing judging your reliability takes to make up its mind.

All three signals, not just traces

This started as a trace-only project, and I want to be honest about that because fixing it is one of the things I am most glad I did. Early on TrueSLA read everything from traces, created alerts and a dashboard through the MCP, and self-instrumented, but it never emitted its own metrics or logs. For a hackathon that rewards leaning on all of SigNoz, using one of three signal types was a real gap. So I closed it.

TrueSLA now emits the SLA domain as first-class metric series: truesla.sli.availability, truesla.error_budget.burn_rate, truesla.credit.exposure, truesla.breach.active, and truesla.engine.refresh.duration. Those are the "how much and how fast over time" numbers a metric dashboard and metric alerts can chart without re-scanning traces.

And every breach emits a structured log when it opens and closes, carrying the tenant, credit, evidence checksum, alert id, the failing dependency, and later on the S3 key of the immutable copy. That one line ends up tying together every signal in the system, which is the bit I would show someone first:

{
  "event": "breach_open", "tenant": "acme", "availability": 99.153,
  "credit_amount": 460.0,
  "alert_rule_id": "019f7b27-273c-7a2e-97cc-2983f5508495",
  "evidence_checksum": "sha256:c3dd4e8d75da50620b841eafb2447803...",
  "causal_span": "ledger.db.query",
  "evidence_s3_key": "evidence/acme/1785111605-c3dd4e8d75da5062.json"
}

At this point in the week traces were still the source of the SLI, and metrics only made the derived state chartable. That turned out to be the wrong way round, and fixing it on the last day is the thing I would most want a reviewer to look at. It gets its own section further down.

What did not change is the division of labour. Metrics answer “how much and how fast”, logs mark lifecycle changes, and traces remain the place to inspect what actually failed. SigNoz’s dashboard supports moving among those signal types, which is the workflow I was trying to build rather than three isolated telemetry checkboxes. (SigNoz dashboard interactivity)

Day 7 (July 26, 2026): Proving the number, not just showing it

By this point the loop worked. A customer broke, a breach opened, a credit appeared, and there was evidence behind it. That was the moment I started worrying about the opposite problem.

Showing one breach proves almost nothing

Every demo of this kind shows the same thing: something goes red, look, it caught it. That tells you nothing about the case I would actually be nervous about, which is billing a customer who was fine.

A guardrail that cries wolf gets switched off. And here the stakes are worse than noise, because the output is attached to somebody’s invoice. Getting it wrong in that direction is not an annoyance, it is a refund I owe for a failure that never happened.

So I wrote down a set of windows where I already knew the right answer, and made the engine replay them. Not a copy of the logic, the same function the live page calls. If I change the math tomorrow, the score changes with it.

The cases are deliberately awkward, because the easy ones prove nothing:

• A customer whose every request failed, but who only made five requests. Five failures is not enough evidence to charge anyone for. It must stay quiet.
• A customer sitting exactly on their target. Meeting a contract is not missing it, so an off-by-a-hair comparison here would bill a compliant customer.
• A customer in genuine breach who is owed nothing, because their contract is tighter than the coarsest credit tier. This one is my favourite. A 99.95% contract measured at 99.9% is a real breach, but 99.9% has not crossed the 99.9 credit boundary yet. Quietly inventing a credit there is the easiest way to overcharge somebody by accident, and it would look completely reasonable in the code.
• Exact tier boundaries, where slipping one way turns a 10% credit into 25%.
• A customer who succeeded on every single request and is still in breach, because they answered twice as slowly as their contract promised.

Twenty-one cases, ten genuine breaches and eleven healthy windows.

breach recall        100.0%   (10 genuine breaches)
healthy pass rate    100.0%   (11 healthy windows)
false breaches         0.0%
credit accuracy      100.0%
status accuracy      100.0%
decision p50        0.0031 ms

It grades one more thing than it looks. A case can declare why it should breach, and the engine has to agree. Being right by accident is not good enough: a latency breach reported as an availability breach would put the wrong explanation in front of a customer who is being paid a credit.

That third line is the one I care about. Recall on its own is easy to fake: an engine that screams breach at everything scores 100% and is useless.

The whole thing runs offline. No SigNoz, no model, no traffic. It is a plain function and a list of known answers, which is exactly why I trust it. The result also gets written into SigNoz as its own span, so the accuracy claim lives next to the live data instead of only in a README.

I also made the harness prove it can fail. There are tests that deliberately break the decision function, one that flags everything and one that inflates every credit, and check the benchmark catches them. A scoring harness that cannot fail is decoration.

A receipt the customer can keep

The evidence page was good, but it had a flaw I did not like: it needed my server running to render. If I am telling a customer they are owed money, and the proof of that only exists while my backend is up, that is not really proof.

So every breach now also produces one self-contained HTML file. Everything is inlined. It opens with nothing running.

It carries what was measured, the exact query the number came from, the contract, the credit ladder that was applied, the dependency that actually failed, links to real failing traces, and the sampling disclosure.

The part I am happiest with is that it re-checks its own checksum when you open it. It does not print a stored string and ask you to trust it. It recomputes the digest from the numbers on the page and tells you whether they still match. Edit a figure and it says MISMATCH.

It also refuses to read like an invoice. It says provisional exposure, and it says final eligibility is subject to contract exclusions and human review, because that is the truth and pretending otherwise would be the sort of thing that gets a tool like this thrown out.

Putting the proof somewhere it cannot be edited

A checksum only helps if you still trust where the file lives. Someone with database access here could change a bundle and recompute the hash, and nothing would look wrong.

So there is an optional archive: every evidence bundle is also written to Amazon S3 with Object Lock switched on. In compliance mode, that object cannot be overwritten or deleted for the retention period, by anyone, including the account that wrote it. That turns “here is a hash” into “here is proof stored somewhere neither of us can quietly rewrite”.

Two things mattered to me here. First, it has to be entirely optional. With no bucket configured it is a clean no-op, and the whole local demo runs with no AWS account and no cost. Second, an archive problem must never break breach detection. If S3 is unreachable, the local checksummed bundle is still the working evidence and the failure is recorded rather than thrown.

I built this against a real implementation of the S3 API running in-process first, because I wanted the failure modes covered before I pointed it at anything that costs money. Seventeen tests cover it now: that a bundle really lands, that it is written under Object Lock with a retention date, that reading it back re-verifies the checksum, that a doctored archive is caught, that two customers do not collide, and that a failure to reach S3 degrades quietly instead of taking the engine down.

Then I turned it on for real.

The setup script does one thing I am glad I wrote. After it creates the bucket it writes a test object under retention, then immediately tries to delete it, and only reports success if AWS refuses. A bucket that merely says Object Lock is enabled is not the claim. The claim is that deletion actually fails.

It does. Here is what happens when I try to remove a real evidence bundle from my own account, with full permissions on that bucket:

Attempting a HARD DELETE of this exact version, as the account that wrote it...
  AWS REFUSED: AccessDenied
  Access Denied because object protected by object lock.

Object still present after the delete attempt: 1713 bytes

That is the whole point of the feature in four lines. Not my code refusing. Amazon refusing, on my behalf, against me.

It goes the other way too. TrueSLA can pull a bundle back down out of S3 and re-verify it, which is the part that makes the archive an audit trail rather than a write-only pile:

{
  "checksum_claimed":    "sha256:95d1f7ae804e8c24945f092c8493c243bae0c4b7...",
  "checksum_recomputed": "sha256:95d1f7ae804e8c24945f092c8493c243bae0c4b7...",
  "checksum_valid": true,
  "object_lock_mode": "COMPLIANCE",
  "tenant": "acme"
}

There is also Terraform in cloud/ for a much bigger version: a multi-tenant service on ECS Fargate with a real Postgres dependency, traced into self-hosted SigNoz through the ADOT collector, with AWS Fault Injection Service breaking one customer's dedicated shard for real. The design decision I like there is the sharding. Chaos tools break infrastructure, which is normally broad, so Acme runs on its own shard and the load balancer routes it there by header. That is how a real chaos experiment produces a single-customer breach instead of an outage.

I have not applied that Terraform. It creates genuinely billable infrastructure and I did not need it to prove the idea. The README says so plainly rather than implying I ran it. The S3 archive is the part that is actually live.

The bug that would have refunded money nobody earned

This is the one I would put at the top if I could only keep one thing from the whole week.

Everything above measures availability by counting traces. Failed spans over total spans. It works perfectly on my laptop, and it is exactly the wrong thing to ship.

Nobody keeps every trace in production. It is far too expensive. So people tail sample: keep the traces that failed, throw away most of the ones that succeeded. That is a completely sensible thing to do, and it is great for debugging.

Now divide those two numbers. Your denominator has been gutted and your numerator has not. The error rate you compute is far worse than the error rate that actually happened.

For most observability tools that is a chart being a bit pessimistic. For this one, that number becomes a refund. I would be paying customers back for outages they never had, and I would have no idea, because the pipeline doing it to me is the one I trust.

The fix is not clever, it is just deliberate. The service now increments a counter, once per request, in memory, before any sampling decision exists:

REQUESTS = _meter.create_counter(
    "truesla.requests",
    description="Requests served, per customer, counted before any sampling "
                "decision. outcome=success|error.")

Every request adds one, tagged with the customer and whether it succeeded. TrueSLA reads availability from that counter through signoz_query_metrics, asking for the increase over the window grouped by customer and outcome. Traces stay exactly where they are, doing the thing traces are genuinely best at, which is explaining why something failed.

Metrics count the breach. Traces explain it.

Two details I would have got wrong if I had not checked the live response.

A cumulative counter has to be read as its increase over the window, not its running total, or every window inherits all of history and availability slowly drifts toward perfect.

And when a customer has no failures at all, the error series simply does not exist in the response. There is no zero row. If you treat a missing series as missing data rather than as zero errors, a perfectly healthy customer silently falls out of the ledger. That one took me a while to see, because everything looked fine.

The honest part is what happens when the counter is unavailable. TrueSLA falls back to counting traces, but the evidence bundle records which source produced the number and whether that source survives sampling:

"sli_source": {
  "kind": "metrics",
  "metric": "truesla.requests",
  "sampling_safe": true,
  "detail": "Counted from an unsampled per-request counter, so the figure is
             exact regardless of trace sampling."
}

If it ever falls back, the receipt says so in plain English and the report carries a visible warning that the error rate may be overstated. I would rather publish a weaker number with a label on it than a confident number that is quietly wrong in the direction that costs someone money.

Blaming the right span

The first version of “likely cause” grouped failing spans by name and picked the most common one. It gave the right answer on my demo, which is why it survived as long as it did.

It is still guessing. Look at what a single failed checkout actually looks like once you expand it:

GET /checkout          Server    FAILED   96.4ms
  payment.authorize    Internal  FAILED    2.0ms
    ledger.db.query    Client    FAILED    1.4ms   <- the actual problem
  GET /checkout send   Internal  ok

Three spans failed. Only one of them broke. The other two are just honest reporting: their child failed, so they failed. Counting names cannot tell those apart, and on a service with a different shape it would confidently blame the entry point.

So TrueSLA now pulls one real failing trace with signoz_get_trace_details, rebuilds the parent/child tree from the span ids, and takes the deepest failing span, the one with no failing children. Then it walks back up to the entry point to recover the path. Where several independent leaves failed, it prefers the slowest, because a fast propagated error is rarely the thing that actually hurt.

What comes out is specific enough to act on:

root cause : ledger.db.query (Client)
error      : RuntimeError: ledger.db.query timed out for tenant=acme
dependency : postgresql / INSERT charge
path       : GET /checkout -> payment.authorize -> ledger.db.query

The bundle records method: trace-hierarchy or name-ranking, so the strength of the claim is visible rather than assumed. Name ranking is still the fallback when there is no trace to walk.

One bug worth admitting. The first version looked back over the SLA window only, so a trace captured near the edge of that window sat just outside the fetch and came back with zero spans. It fell back to name ranking every single time and looked like it was working. Widening the trace lookback fixed it, and I only noticed because I went looking for the field in the API response.

Two alerts per customer, not one

The original alert was a single threshold per customer. That is the classic way to get SLO alerting wrong, and I knew it while I was writing it.

Make the window short and every brief blip pages somebody at three in the morning, so they mute the alert and then it protects nothing. Make it long and a slow steady leak drains the entire month’s budget without ever tripping.

The standard answer is to alert on how fast the budget is burning, over more than one window at once. So every customer now gets two:

RuleBurn rateWindowSeverityCatchesfast14.4x budget5mcriticala sharp outage, within minutesslow6x budget30mwarningthe quiet leak a short window smooths away

Both are scoped to one customer and to that customer’s own contract, so Acme on 99.9% is held to a tighter threshold than Hooli on 99.0%. And both read the same unsampled counter the ledger reads, which matters more than it sounds: if the alert counted traces while the page counted requests, the two could disagree about whether a customer was breaching, and I would have no idea which to believe.

This is also where I found something embarrassing. Rules were being reused whenever the name matched, so the five fast rules I had created days earlier were never rebuilt when I moved the SLI to metrics. They were still counting trace spans. The alerts page looked completely fine and was quietly measuring a different thing from the ledger.

Rules now carry a burn_speed label. Its absence marks the old shape, and those get deleted and recreated. There is also a prune step that clears rules for tenants whose contract no longer exists, which cleaned out ten rules my own test suite had left behind in live SigNoz for customers named testcorp and gone. That was a second bug: the tests redirected the database to a temp file, but the alert creation was still hitting the real stack. The suite now blocks the MCP client outright unless a test opts in.

Latency is half the contract

latency_target_ms had been sitting on every contract since day three, and all it did was get displayed.

But a contract normally promises two things: that requests succeed, and that they come back fast enough. A service answering every single request in four seconds is up by the first measure and useless by the second.

So latency can now open a breach on its own. The case that makes the point:

1000 requests, 0 errors, p95 of 900ms against a 400ms promise
  -> availability 100.0%
  -> BREACH

Availability alone calls that perfect.

The honest part is the credit. The credit ladder in the contract is written against availability, so a latency-only breach reports a $0 credit rather than having a figure invented for it. Each row also names which promise was missed, availability, latency, or both, so a customer never gets told the wrong reason for their own outage.

The check that caught me out

The verifier does something I nearly did not bother with. It hard-fails if the global average drops below 99.5% while one customer is breaching.

That sounds redundant. It is the entire thesis of the project. If breaking one customer visibly moves the overall number, then the overall number was not hiding anything and there is nothing here worth building.

It caught me. On one run it failed, with global at 99.369%. I had been experimenting earlier with a much heavier fault rate, and because the view is a rolling ten-minute window, that earlier damage was still inside it and dragging the average down.

The fix was not to lower the threshold, which would have been moving the goalposts. Acme was about 5% of traffic, and that is not thin enough to survive a demo being run twice in a row.

I actually sat down and did the arithmetic instead of guessing, because the relationship is simple: the global average only moves by the failing customer’s share of traffic multiplied by their failure rate. At 5% of traffic and a 25% failure rate, that is 1.25% off the global number, which is enough to sink it. At 1.25% of traffic, the same brutal failure rate costs the global average 0.31%, so it stays comfortably green even if I leave the fault running for a full window.

So Acme is now 1.25% of traffic, one request in eighty. Which is also more honest about the real situation. The customer your average hides is usually a small, high-value one, not a twentieth of your traffic.

I would rather a check embarrass me in private than have someone notice the claim does not hold while I am recording.

Where it stands, and what I think

The last part is the honest inventory and the command I run before showing it to anyone.

Where it stands

Working and verified against live SigNoz:

• A multi-tenant demo service shipping per-customer traces, with a tenant-scoped fault switch so I can break exactly one customer.
• Per-customer availability computed from an unsampled request counter, so the figure stays exact no matter how the tracing pipeline is sampled, with a disclosed fallback to traces and a sampling-safety flag inside every bundle.
• Per-customer p95 latency, and latency able to open a breach on its own, with the credit still priced off availability and reported as $0 when nothing is owed.
• Contracts, a credit schedule, and an append-only breach snapshot history, with a test that guards the one-measurement consistency rule.
• The Credit Clock: a live estimate of the next tier, with a low-data guard so it never quotes exposure from a tiny sample.
• Causal analysis that walks a real failing trace, rebuilds the span tree and blames the deepest failing span rather than the ancestors that propagated the error, recording which method it used.
• All three SigNoz signals: the SLA domain emitted as metric series (availability, burn rate, credit exposure, active breaches), a structured breach log on open and close, and the traces behind every number.
• Two burn-rate alerts per customer, fast and slow, provisioned in SigNoz through the MCP before anything breaks, both reading the same unsampled counter the ledger reads, both scoped to that customer’s own contract.
• Alert firing history read back from SigNoz into the breach record.
• A self-provisioned, cross-signal SigNoz dashboard of nine panels that reconciles to one board, and self-instrumentation of the backend and every MCP call.
• A live UI over Server-Sent Events, and a verify-demo.ps1 that runs the whole loop, breaks a customer, hard-checks that the global average stayed green while that customer failed, checks the breach, alert, evidence and portable receipt, and restores. It currently passes 29 checks.
• A benchmark that replays 21 labelled windows through the same decision function the live page uses, reporting breach recall, healthy pass rate, false breaches, credit accuracy and which promise was broken. Currently 100% recall, 0% false breaches, exact credits, and it emits its own result into SigNoz.
• A self-contained evidence receipt per breach that re-verifies its own checksum when opened and needs nothing running.
• A live S3 Object Lock archive. Every bundle is written to a real bucket in compliance mode, and AWS refuses to delete it. Reading one back re-verifies the checksum against what S3 is storing.
• Write protection on the endpoints that change contracts or break a service, with a constant-time key comparison, and /api/health reporting which mode it is in.
• 210 backend tests covering the contract verdict, the credit ladder and its boundaries, the Credit Clock, the SLI source and its fallback, the causal hierarchy walk, latency SLO, burn-rate alert shape, evidence integrity, the HTTP surface, the portable report, write protection, and which measurement reaches the archive.

I checked this end to end. In the last full run I broke Acme and watched it fall to 90.4% while the global average sat at 99.88%. TrueSLA showed $2,300 of provisional exposure, the breach log carried the evidence checksum and the S3 key, ledger.db.query was identified as the root cause by walking the trace, and the portable receipt re-verified its own checksum on open. Then I restored the fault and new failures stopped. Because the view is rolling, the breach remains visible until those failed requests age out of the window. I prefer that to a demo switch which magically erases historyAnd here is the thing I actually run before showing this to anybody. It breaks a customer for real, waits for the breach, checks the alert, the evidence, the portable receipt and the archive, then restores. It refuses to say “demo ready” if the global average moved while one customer was failing:

6. Breach proof (the money moment)
  PASS acme is in BREACH avail 90.37% < target 99.9%
  PASS global average stayed GREEN while one customer failed global 99.88%
  PASS credit computed from measured telemetry $2300
  PASS evidence bundle checksummed sha256:cb75011a4b7507f254b5af85...
  PASS causal dependency identified ledger.db.query
  PASS report re-verified its own checksum on open
  PASS credit labelled provisional, not an invoice

  Result
  29 passed, 0 failed, 0 warnings
  Demo ready.

Running it

Everything is self-hosted. SigNoz runs through Foundry, with casting.yaml and casting.yaml.lock in the repo so the deployment is reproducible. The model runs locally on Ollama, so the incident narrative does not leave the machine for a hosted model.

cp .env.example .env      # add your SigNoz service-account key
cd backend && python -m venv .venv
# requirements.txt is pinned; requirements.lock.txt has the exact tree
.venv/Scripts/pip install -r requirements.txt
cd ../frontend && npm install
./start-all.ps1

Everything lives here:

• Code, including casting.yaml and casting.yaml.lock: github.com/RajdeepKushwaha5/TrueSLA
• The three-minute walkthrough: youtu.be/z8B42TCVReQ
• Hosted preview: true-sla.vercel.app

Once it is up, run ./verify-demo.ps1. It drives the whole loop, breaks a customer, and refuses to say "demo ready" unless the global average stayed green while that customer breached. I still check the browser and SigNoz dashboard manually before recording; the script validates the backend path, not the visual state of every panel.

To run it against a real service instead of the demo, point TRUESLA_SERVICE at your service, set TRUESLA_TENANT_ATTR to whatever span attribute carries the customer, and attach real contracts through the API or the UI. The engine re-reads SigNoz on the configured cadence and streams each changed snapshot to the UI. The demo is only the traffic and the fault switch; the measurement path is the same either way.

Why I would keep building it

This is not another observability dashboard competing with SigNoz. The buyer I have in mind is a B2B SaaS company with enterprise contracts: the SRE team owns the telemetry, support owns the escalation, finance owns the credit, and nobody has one shared incident record. TrueSLA turns an existing tenant attribute into that shared record.

The production path is fairly concrete: use unsampled request metrics for authoritative SLI accounting; retain traces as exemplars and causal evidence; separate the fast operational window from the fixed billing period; reconcile alerts and dashboards by stable IDs; sign and export evidence revisions; add authentication and audit roles; and deliver alert transitions to the customer’s real incident channel. None of that requires replacing the core demo. It hardens the boundaries around it.

What I actually think

The interesting problem was never collecting the traces. SigNoz and OpenTelemetry already do that well. The interesting problem is that the number everyone looks at, the blended average, is the one number that structurally cannot tell you which customer you are failing. The information was always in the traces. It just needed to be grouped by the customer instead of averaged across them, turned into each customer’s own contract, and kept as proof.

The part I keep coming back to is the demo moment. The average said everything was fine. Acme’s traces said otherwise. And TrueSLA said how many seconds we had to save the contract.