Tutorials for “summarise customer chats with an LLM” end at the JSON. That’s where the work starts. These are the rules our claims-logging bot needed, each one written the day a row came out wrong and someone said my name in a tone. They’re specific to a food-distribution company, but the shape of them isn’t.

Structure

1. One row per product. A chat about three bad products is three rows with three product ids, not one row with a shopping list in it.
2. At most eight items per chat, or discard. More than eight means the model stopped reading the complaint and started copying the order list. Models love a list.
3. Ten fixed keys, no extras. Cause, detail, resolution, item, quantity, unit, quantity basis, delivery date, amount, note. Extra keys are dropped; missing keys are empty, never guessed.
4. A reopened chat is a new entry. Keyed by chat id plus closing timestamp, so it can’t overwrite the first record.

Classification

5. “Not everything arrived” is a missing item, not a quality claim. Different cause, different responsibility, different cost. “Unhappy” is not a category.
6. Same product, missing vs quality: separate claims. We tried the broader rule “different cause means different claim” and it split 26 cases that belonged together. The narrow rule stays. The broad one is in a drawer.
7. A failed exchange is a new row. Replacement sent, replacement also defective: second claim.
8. An unordered product that arrived is a second row. Logged as a wrong-item delivery next to the original claim. An exchange is still one row.
9. Who’s responsible is a lookup. Wrong item or transit damage: logistics. Quality: supplier. The model doesn’t get a vote; the cause category decides.
10. Our own mis-advice is a cause only when the transcript shows it. If an agent gave wrong information on an earlier case and that’s visible in the chat, it’s “CS error”. Otherwise it’s whatever the customer is reporting. The bot is not allowed to assume we messed up. That’s my job.

What is not a claim

11. A zero-amount order is a resolution, not a claim. Replacement shipments appear as orders at unit price zero with a memo. Skip them, or your complaint count doubles overnight and someone schedules a meeting.
12. A customer re-sending within 24 hours is the same case. Not a duplicate, not a new claim, no alert.
13. An agent-initiated call with no answer is an empty chat. Skip it; it’s not a bot failure, it’s voicemail.

Matching

14. A transcribed call can’t add products. If a product isn’t on any order, don’t create a row for it. A misheard word is far likelier than a secret unrecorded order.
15. Match product names by sound. Speech-to-text spells products however they sound that day. Compare phonetically against the order items.
16. If the window has no orders, look back 30 days. Customers complain about last week’s delivery this week. Sometimes last month’s.
17. Boxed vs single: divide the unit price by pack size. A claim about one item from a box of twelve is priced per item, not per box, or the refund gets exciting.

Timing and state

18. Unknown company: wait, don’t guess. Pending row, retry every ten minutes until the profile has a company.
19. “On hold” only when the tool says snoozed. An open chat is “in progress”. Words mean things.
20. Evening unshipped-order calls: tomorrow’s cancellation, or today’s re-delivery. A call at 7pm about a missing box is either “cancel tomorrow’s line” or “send a courier now”. The bot picks by what the agent said, never by default.

How rules get added

A human corrects a cell the bot wrote. The bot logs the correction (bot value → human value, row, column). Every evening a task summarises the day’s corrections into a shared “rule change notes” document. When the same correction shows up three times, it becomes a rule in the prompt, with the date and the row that caused it. The prompt is, at this point, mostly a list of things that once went wrong. So is wisdom.

Twenty rules in, the team corrects rows instead of writing them. That’s the entire goal. Everything else is decoration.

One line from me

None of this is hard. It’s the tedious, soul-sucking grunt work nobody wants to do. Hand every bit of it to the bot.

Written by Seungjoo Choi. Who I am and why I run these bots.