Root-cause analysis of customer credits
Every credit has a reason. Most of them are written in a sentence nobody has time to read.
Credit Signal reads the free text behind customer credits and complaints, extracts the failure mode with a language model, joins it to what the system already knows (packaging, carrier, warehouse, date), and ranks causes by what they cost the customer in downtime — not just by the refund.
01 · Try it
Paste a complaint. Watch it get filed.
This is the same classifier the numbers below were measured on, running live on the model the memo recommends. It returns the failure mode, how sure it is, the words it based that on, and whether it would file the record automatically or send it to a person.
Result appears here.
02 · What drives credits
Ranked by refund, the list looks one way. Ranked by customer downtime, it reorders.
A $40 credit on a damaged shaft collar is trivial. The four hours a line sat idle waiting for the reship is not. Downtime hours per failure mode are an operator's estimate (see Data); the toggle shows how much the ranking depends on them.
03 · The pattern keyword search misses
A packaging change in May quadrupled thread damage on bulk-bagged fasteners. Searching "damaged" barely moves.
Two patterns were deliberately seeded into the synthetic credit records so the method could be tested against a known answer. The model that wrote the customer notes never saw packaging, carrier, or warehouse — so if the pipeline finds the pattern, it found it in the language.
Thread-damage rate in fasteners, by month
Lift found: LLM extraction vs keyword search
Nobody told the scan where to look
Every failure mode × every packaging spec, carrier, and warehouse, within each product class — cells, ranked by significance. The seeded cell is where it should be.
All three checks, including the ones that didn't clear
Lift = rate inside the suspect group ÷ rate outside it. p is a one-sided two-proportion z-test. Two keyword baselines: naive is what a reason code or a search for "damaged" gives you; tuned is a regex written after you already suspect thread damage — it works, which is the point: keyword search finds what you already know to look for. Reporting the non-significant checks on purpose: the method should say "no" as clearly as it says "yes".
04 · What I would do about it
Finding the cause is the easy half.
The hard half is getting packaging engineering, carrier management, and DC operations to change something because a model read customer notes. That takes an owner, a fix small enough to say yes to, and a metric agreed before anyone starts. From the memo, for the top causes above:
| Cause | Owner | Fix | Metric that says it worked | Target |
|---|---|---|---|---|
| Thread damage · bulk-bag fasteners · post-May | Packaging engineering | Revert PB-2 for fastener SKUs above a size threshold, or add a divider. Test on the top 20 SKUs by credit count. | Thread-damage credits on those SKUs, next 8 weeks vs. the 8 before | −50% in 8 weeks |
| Aged elastomers · DC-4 | DC operations + inventory | FIFO audit on elastomer bins; date-code check at pick. | Aged-consumable credits, DC-4 vs. other DCs | Parity within one rotation cycle |
| Wrong part in the right bag | Fulfillment QA | Trace to bagging line or supplier lot; scan-verify at bagging for the top 50 SKUs. | Wrong-part credits per 10k lines shipped | −30% on top 50 SKUs |
Targets are assumptions for a pilot to correct, not promises. Program metric: credit recurrence rate — the share of this quarter's credits whose cause was already in last quarter's top ten.
The four-week pilot
Week 1: load six months of credits, redact PII, run. Week 2: two CS reps hand-label 300 records; set the confidence threshold from the curve. Week 3: take the top three causes, with evidence quotes, to the teams that own them. Week 4: agree the metric and the review cadence.
What the memo adds
Why rank by downtime, what the gold set changed about the model choice, what it would cost at scale, and the limitations stated plainly.
05 · What real customers say
Two thousand real one- and two-star reviews of industrial products, sorted into an operator's failure taxonomy.
The taxonomy started as 15 failure modes written from shop experience. Hand-labeling real reviews showed what a parts-receiving view misses — the part that works but doesn't do the job, the listing that was wrong, the customer who chose wrong — so v2 has 20. Expand a mode to read what the model quoted as its evidence.
06 · Coverage and precision
The model is allowed to say "I'm not sure." That's what makes it deployable.
Every classification carries a confidence. Above the threshold it's auto-filed; below, it goes to a person. Move the threshold and you trade coverage for precision. On clean synthetic text the trade is generous; on real reviews it is not — and the model's confidence knows the difference.
Synthetic ground truth (n = 1,500)
Real reviews vs. human labels (100 unseen, v2)
Review queue
07 · How accurate is it, and where does it fail
Measured against a human, twice — and the taxonomy got better in between.
Against synthetic ground truth every model looks great. The number that matters is agreement with a careful human on real reviews. The author hand-labeled 200 records under the original taxonomy (72%), found that a third of real complaints had nowhere to go, revised the taxonomy — merged two modes customers can't tell apart, added five the reviews demanded — and then labeled 100 fresh reviews no one had seen. That clean test is the headline: 78% with the classifier frozen beforehand, 85% once the labeling guideline was written down. Always guessing the most common label would score 40%. Model tier matters on real language in a way it does not on clean text: the written rules helped Opus and did nothing for Sonnet.
The 100 unseen reviews: agreement by how sure the human was
The 50 synthetic records in the gold set also check the generator: the human agreed with the hidden ground truth on .
Where it gets confused
On the 100 unseen reviews. What's left are neighbors an experienced person would also debate — a part that broke versus one that fell short of its rating.
Where the human and the model disagreed, with the human's note
08 · What it costs to run
Cents per thousand records.
Measured from this project's own API usage, with prompt caching on. A human reviewer reading 200 notes an hour costs more per record than either model by two orders of magnitude — the point is not to replace the reviewer, it's to send them only the records worth reading.
09 · Method
Five steps, no magic.
1. Collect the note
The customer's own words from the credit request, email, or review. PII would be redacted here before anything reaches a model.
2. Extract
Failure mode from a 15-item taxonomy, a confidence, a verbatim evidence quote, and where in the chain it likely originated.
3. Join
Attach what the order system already knows: product class, packaging spec, carrier, origin DC, date, credit amount.
4. Rank and test
Count, dollars, expected downtime. Lift and significance for every metadata cut. Trend by month.
5. Route
Confident results file automatically. Uncertain ones go to a reviewer, whose labels feed the next accuracy check.
10 · Data, honestly
None of this is McMaster-Carr data. Two sources, both disclosed everywhere they appear:
Real: public product reviews
2,000 one- and two-star reviews from the Industrial & Scientific category of the Amazon Reviews 2023 dataset (UCSD / McAuley Lab), stratified 70/30 toward reviews using shop vocabulary. They supply genuine, messy customer language. They have no credit amounts, carriers, or warehouses, so they appear only in the taxonomy and accuracy sections.
Synthetic: credit records
1,500 credit memos with product class, packaging spec, carrier, origin DC, date, and credit amount, generated to mimic the record structure a distributor would actually hold. Notes were written by a language model that saw only the failure mode and product class. Two patterns were seeded on purpose so the method could be checked against a known answer. Downtime hours per failure mode are an operator's estimate, not measured.
Limitations
- Synthetic notes were written and classified by models from the same family. The human-labeled sets are the check on that — and they show synthetic accuracy (96–99%) overstates real-review accuracy (78–85%). Treat synthetic numbers as an upper bound.
- The labeling guideline was refined while the clean set was being labeled, so the 85% is mildly tuned to it; the 78% frozen-protocol number is not. One labeler; a second would turn "one person's judgment" into inter-rater agreement.
- Public reviews are consumer-adjacent; the industrial share is real but the mix is not a distributor's mix.
- Downtime estimates drive the impact ranking. The sensitivity toggle exists because that assumption should be visible, not buried.
- No order-level truth exists for the public reviews; accuracy there rests on the gold set alone.