We shipped a ticket classifier with a 0.92 macro F1 and spent the next month wondering why the support team kept correcting it.

The model was unremarkable: nine categories, 41k historical tickets, a gradient-boosted tree over TF-IDF plus a handful of numeric features. The split was random, 80/20, stratified. The number looked plausible, which is exactly why nobody questioned it.

A month after launch I pulled 500 fresh tickets, labeled them by hand, and got 0.71. Same model, same categories, same code. The only thing that had changed was the data.

Two bugs wearing one coat

The first was duplication. Support tickets are heavily templated. “Refund for order #44192” and “Refund for order #44193” are different rows but the same document as far as a bag-of-words model is concerned. A random split puts one in train and one in test, and the model gets to memorize the template. I ran MinHash over 5-word shingles and found that 18% of the test rows had a near-duplicate (Jaccard above 0.8) in the training set.

The second was worse: one of the features was last_note, a free-text field the agent fills in after resolving the ticket. It contains things like “issued refund” and “escalated to billing”. I had effectively handed the model the label in a text column. The giveaway is embarrassingly simple — train a model on that column alone. It scored 0.89. Any single feature that gets you most of the way to your best number is not a feature, it is the answer.

What I changed

Three things, in order of how much they mattered:

  1. Dropped every field written after the ticket was resolved. If a column would not exist at prediction time, it does not exist in training either.
  2. Deduplicated the whole corpus before splitting, not after. Dedupe-then-split is the only order that works; splitting first guarantees the same document ends up on both sides.
  3. Split by time. Train on everything before January, test on February and March. The random split had been flattering the model for a year.

The clean number was 0.86. Production settled at 0.84 after a month, which is the first number I have believed in this project.

The rule I keep

The first accuracy number you should trust is the one from data you have never touched, collected after you wrote the code. Everything before that is a measurement of how well the model memorized the past — useful for catching bugs, useless for predicting next month.

We now keep a locked holdout that nobody looks at until a release is cut. It is small, it is boring, and it is the only set I care about.

The code, such as it is

The dedupe step is smaller than the argument about it:

from datasketch import MinHash, MinHashLSH

def shingles(text, k=5):
    words = text.lower().split()
    return {" ".join(words[i:i+k]) for i in range(len(words) - k + 1)}

lsh = MinHashLSH(threshold=0.8, num_perm=128)
for row in tickets:
    m = MinHash(num_perm=128)
    for s in shingles(row["subject"] + " " + row["body"]):
        m.update(s.encode())
    lsh.insert(row["id"], m)

dupes = {k: v for k, v in lsh.items() if len(v) > 1}

Two details that matter more than the library. First, dedupe has to run over the whole corpus, before the split — otherwise you only remove duplicates inside each half and the cross-half pairs survive. Second, keep one representative per cluster, preferring the earliest ticket, so the training set does not drift toward whatever was popular last month.

The time split was three lines of pandas and a rule written on the wall: no ticket newer than the cutoff is ever allowed in training, even for a quick experiment. That rule has saved me from at least two more 0.9s that meant nothing.