Not McMaster-Carr data. Public reviews + synthetic credits · Decision memo

Credit Signal

Root-cause analysis of customer credits

Every credit has a reason. Most of them are written in a sentence nobody has time to read.

Who built this, and why

Muireadhach Currie. I owned PrintWithin, an additive-manufacturing service bureau, ran production for a 3D-printer manufacturer, and ran machines in a family machine shop. When a shipment arrived damaged, the refund might be forty dollars. The bigger cost was a machine sitting idle until the replacement came, which is why this page ranks causes by downtime.

m@muireadhach.com · linkedin.com/in/muireadhach

Credit Signal uses a language model to read the free text on customer credits and complaints and pull out the failure mode. It joins that to what the order system already records (packaging, carrier, warehouse, date) and ranks causes by the downtime they cost the customer, alongside the refund.

Paste a complaint and see how it gets filed.

This is the same classifier the numbers below were measured on, running live on the model the memo recommends. It returns the failure mode, how sure it is, the words it based that on, and whether it would file the record automatically or send it to a person.

Result appears here.

What the analysis found

Loading results…

Rank the causes by customer downtime instead of refund dollars, and the order changes.

A $40 credit on a damaged shaft collar is trivial next to the four hours a line sat idle waiting for the reship. Downtime hours per failure mode are my estimates from running shops (see Data); the toggle shows how much the ranking depends on them.

Credit dollarsEstimated downtime cost (typical)

A packaging change in May quadrupled thread damage on bulk-bagged fasteners. A search for "damaged" doesn't pick it up.

I seeded two patterns into the synthetic credit records so the method could be tested against a known answer. The model that wrote the customer notes never saw packaging, carrier, or warehouse, so any pattern the pipeline finds had to come from the wording of the notes.

Thread-damage rate in fasteners, by month

Lift found: LLM extraction vs keyword search

Nobody told the scan where to look

The scan tests every failure mode against every packaging spec, carrier, and warehouse within each product class: cells, ranked by significance. The seeded cell comes out on top.

Show the top-ranked cells
scroll sideways →

All three checks, including the ones that didn't clear

scroll sideways →

Lift is the rate inside the suspect group divided by the rate outside it. p comes from a one-sided two-proportion z-test, and a check counts as significant at p < 0.01. There are two keyword baselines. Naive is what a reason code or a search for "damaged" gives you. Tuned is a regex written after you already suspect thread damage; it works, but only because you already knew what to search for. The non-significant checks are included on purpose, to show the method returning "no" when the signal isn't there.

Finding the cause is easier than getting it fixed.

A fix means getting packaging engineering, carrier management, or DC operations to change something because a model read customer notes. That takes an owner, a fix small enough to say yes to, and a metric agreed before anyone starts. Here is the plan from the memo for three of the causes above:

scroll sideways →
CauseOwnerFixMetric that says it workedTarget
Thread damage · bulk-bag fasteners · post-MayPackaging engineeringRevert PB-2 for fastener SKUs above a size threshold, or add a divider. Test on the top 20 SKUs by credit count.Thread-damage credits on those SKUs, next 8 weeks vs. the 8 before−50% in 8 weeks
Aged elastomers · DC-4DC operations + inventoryFIFO audit on elastomer bins; date-code check at pick.Aged-consumable credits, DC-4 vs. other DCsParity within one rotation cycle
Wrong part in the right bagFulfillment QATrace to bagging line or supplier lot; scan-verify at bagging for the top 50 SKUs.Wrong-part credits per 10k lines shipped−30% on top 50 SKUs

The targets are assumptions for the pilot to correct. The program metric is credit recurrence rate: the share of this quarter's credits whose cause was already in last quarter's top ten.

Recommendation if this were real

The four-week pilot

  1. Week 1

    Load and run

    Load six months of credits, redact PII, and run the pipeline.

  2. Week 2

    Set the threshold

    Two CS reps hand-label 300 records, and the confidence threshold is set from the resulting curve.

  3. Week 3

    Take it to the owners

    Bring the top three causes, with evidence quotes, to the teams that own them.

  4. Week 4

    Agree the metric

    Settle what counts as success and how often the results get reviewed.

The memo explains why the ranking uses downtime, how the hand-labeled test changed the model choice, what this would cost at scale, and where the evidence is weak.

Read the decision memo →

Two thousand real one- and two-star reviews of industrial products, sorted into an operator's failure taxonomy.

The taxonomy started as 15 failure modes written from shop experience. Hand-labeling real reviews showed what that parts-receiving view missed: parts that work but don't do the job, listings that were wrong, customers who picked the wrong part. Version 2 has 20 modes. Open any mode to read the text the model quoted as evidence.

Failure mode · sharetap + to expand

The model can say "I'm not sure," which is what makes it safe to deploy.

Every classification comes with a confidence score. Records above the threshold are filed automatically, and the rest go to a person. Raising the threshold buys precision at the cost of coverage. On clean synthetic text you give up very little; on real reviews you give up more, and the confidence scores still track how often the model is right.

Synthetic ground truth (n = 1,500)

Real reviews vs. human labels (100 unseen, v2)
scroll sideways →

Review queue

Measured against a human twice, with a taxonomy revision in between.

Every model scores well against synthetic ground truth. The harder test is whether it agrees with a careful person reading real reviews. I hand-labeled 200 records under the original taxonomy, and the model matched me 72% of the time. A third of the real complaints had no category that fit, so I revised the taxonomy, merging two modes customers can't tell apart and adding five the reviews called for. Then I labeled 100 fresh reviews no one had seen. On that clean test the model agreed with me 78% of the time with the classifier frozen beforehand, and 85% once the labeling guideline was written down. Always guessing the most common label would score 40%. Model tier matters more on real language than on clean text: the written rules helped Opus and did nothing for Sonnet.

Where the human and the model disagreed, with the human's note

More disagreements
The 100 unseen reviews: agreement by how sure the human was
scroll sideways →

The 50 synthetic records in the labeled set also test the generator. My labels matched the hidden ground truth on .

Where it gets confused
scroll sideways →

On the 100 unseen reviews, the remaining confusions are between neighboring modes an experienced person would also argue about, such as a part that broke versus one that fell short of its rating.

Well under a cent per record.

scroll sideways →

Measured from this project's own API usage, with prompt caching on. A person reading 200 notes an hour costs about a hundred times more per record than either model. The model's job is to hand that person only the records worth reading.

The pipeline, in five steps

1. Collect the note

IN · free text

The customer's own words from the credit request, email, or review. PII would be redacted here before anything reaches a model.

2. Extract

LLM · structured output

Failure mode from a 20-item taxonomy, a confidence, a verbatim evidence quote, and where in the chain it likely originated.

3. Join

JOIN · metadata

Attach what the order system already knows: product class, packaging spec, carrier, origin DC, date, credit amount.

4. Rank and test

MATH · no model

Counts, dollars, and expected downtime per mode, plus lift, significance, and a monthly trend for every metadata cut.

5. Route

OUT · people

Confident results file automatically. Uncertain ones go to a reviewer, whose labels feed the next accuracy check.

None of this is McMaster-Carr data. It comes from two sources, both labeled wherever they appear:

Real: public product reviews

2,000 one- and two-star reviews from the Industrial & Scientific category of the Amazon Reviews 2023 dataset (UCSD / McAuley Lab), stratified 70/30 toward reviews using shop vocabulary. The language is genuine and messy. They have no credit amounts, carriers, or warehouses, so they appear only in the taxonomy and accuracy sections.

Synthetic: credit records

1,500 credit memos with product class, packaging spec, carrier, origin DC, date, and credit amount, generated to match the record structure a distributor would actually hold. A language model wrote the notes and saw only the failure mode and product class. I seeded two patterns on purpose so the method could be checked against a known answer. Downtime hours per failure mode are my estimates, not measurements.

Limitations

  • Models from the same family wrote the synthetic notes and classified them. The human-labeled sets check for that, and they show synthetic accuracy (96–99%) overstating real-review accuracy (78–85%). Treat the synthetic numbers as an upper bound.
  • I refined the labeling guideline while labeling the clean set, so the 85% is mildly tuned to it. The 78% from the frozen classifier is not. All the labels are mine; a second labeler would replace one person's judgment with a measured inter-rater agreement.
  • The public reviews lean consumer. The industrial share is real, but the mix is not a distributor's.
  • Downtime estimates drive the impact ranking, which is why the page has a toggle to show how much the ranking depends on them.
  • No order-level truth exists for the public reviews; accuracy there rests on the gold set alone.