Case study: product categorization
A drop-in replacement for a word-matching classifier: exact match, then a multilingual encoder, then kNN corroboration, then an LLM. Each layer catches what the previous one missed.
A retailer’s feed assigns every incoming product line to a fixed category tree. The legacy model was a random forest matching words, so “TK Pizza” and “Tiefkühlpizza” were unrelated strings and a typo broke everything. It had been trained once, years ago, with no retraining loop. It gave no usable confidence signal. And the business was about to enter Italy, with zero categorized Italian products for it to learn from.
The replacement is a cascade, ordered by cost. Normalization and exact match resolve everything seen before, for free. A multilingual transformer encoder, fine-tuned on the full history of correctly categorized products, handles the bulk of traffic in milliseconds, on CPU, with no per-request fees. An independent nearest-neighbor check corroborates every decision against a vector index of all labeled products. Only the few percent that stay uncertain go to an LLM, asynchronously, so no response ever waits on it.
The interesting part is why there are two models in the middle. The classifier generalizes: rules distilled from the whole history. The index remembers: direct comparison against known cases, and it doubles as evidence, the twenty labeled products nearest to this one, of which, say, eighteen vote the same way. They consume the same input but fail differently, so agreement is strong evidence the answer is right, and disagreement is a cheap, reliable uncertainty detector. Agreement plus high confidence assigns automatically. Anything else is provisional and gets re-examined in the background.
Confidence is calibrated, so a stated 95% empirically means 95% correct, and the threshold for automatic assignment becomes a configuration choice for the business. The Italian problem dissolves in the encoder’s pre-training: it read about a hundred languages into one shared space, so “Milch”, “latte” and “milk” already sit in the same region, and category boundaries learned from German data mostly hold for Italian inputs. Before launch that gets reinforced with machine-translated copies of the German training data and LLM pre-labeled Italian strings, verified by human spot checks.
Go-live is gated on a benchmark. The most recent slice of labeled data is held out to simulate tomorrow’s products, every component is scored on it per taxonomy level and per language, and a coverage-precision curve shows how much can be auto-assigned at any precision target. The operating threshold gets picked off that chart, by the business, before cutover. The whole service then runs in shadow mode against live traffic first.
Every prediction is logged with its confidence, which makes the log three things at once: the retraining set, the monitoring feed, and the future review queue. The legacy model aged. This one accumulates.