My AI agent keeps sending sync failed alerts, but nothing failed

What happened

We ran an hourly sync job between two machines. When something went wrong, an alert went to a chat channel so a person could look at it.

In one day the channel got 4 "sync failed" alerts. We checked every sync that day, and every one finished with exit code 0, which means it worked. Nothing had failed. The alerts were made up.

The first fix didn't hold. We made the wrappers around the job stricter and stopped passing them any arguments. That cut the fake alerts down but didn't stop them.

Why the AI did it

A small local AI model sat in the middle of the alert path. Its job was to read the job output and pass along a short status message. Language models are built to produce likely-sounding text. If you ask one "how did the sync go?" often enough, it will sometimes write a failure message, because failure messages are a normal thing to see in that spot.

The tell was the format. Real failures from the job always carried the exit code and the end of the log, like rc=1 :: <last lines of log>. The fake ones had neither. They read like an alert, but nothing backed them up.

The same risk applies to any agent that summarizes system status: a Salesforce sync, an integration health check, a nightly data load. If a model writes the message, the model can write the wrong message.

The fix, step by step

You don't need a special plan for this. The fix is to take the AI out of the part that should never be a judgment call.

1. List every alert that passes through an AI. For each one, ask: is this a fact (it ran, it failed, here's the code) or an opinion (this looks risky)? Facts don't need a model.
2. Replace the model with a plain script for the facts. Ours became a basic scheduled script: run the sync, stay silent if it works, and on failure post the real exit code and the log tail to a webhook.
3. Make real alerts carry proof. Every alert includes the exit code and the actual log lines. An alert without them should be treated as suspect.
4. Keep the AI for the opinion part, if you want it. If you like a model explaining a failure in plain English, let it read the real log after the script has already posted the facts. It can add notes. It can't create an alert.
5. Write the rule down. Ours: status relays are plain scripts and webhooks, never model judgment.

If you use ChatGPT or Claude to help, ask it to write the script for you, then read it yourself. That part is fine. The point is that the model writes the code once, and the code runs without the model.

How to check it worked

Run a planted-mistake test both ways:

• Plant a real failure. Point the sync at a folder that doesn't exist, or break it on purpose some other way. You should get exactly one alert, with a non-zero exit code and the real log lines.
• Plant a quiet day. Let it run normally for a day. You should get zero alerts.

Our test run came back clean. If you get an alert on a quiet day, something in the path is still writing its own text.