Field note
38 days, six reviews, one script that says no
In July, a Salesforce team at an enterprise software company had a 31-slide concept deck, a mockup and a business case for an internal product. Thirty-eight days later, the first version was installed in the company's production org. AI builders wrote the code. People decided what shipped. This is what the team did, what it would do again, and what it would not.
$ python3 review-gate.py ./release verdicts.txt PROD-REVIEW-GATE: REFUSED — candidate tree ff4df36df11b9b94 is not the reviewed tree 1c0cb002f3448f96 — the code changed after review (exit 3)
That line is the whole method. A script that says no, for a reason it can print, and a person who has to answer to it. Everything else in this piece is the record of one team using it: how long it took, what the reviews caught, what they missed, and what the team would not do again.
The 38 days
The deck and mockup were saved on July 8. A product requirements document and an architecture write-up followed within the hour. The production install was the evening of August 15.
It was not one big push. The team staged a rollback package first, then deployed in dependency order: custom permissions, objects and fields, tabs, configuration records, a flow, permission sets. Then one production validation, which ran 95 components and 516 tests with zero errors, and a quick deploy of that exact validated job. The flow arrived in Draft, because production does not activate flows on deploy. The team stopped there, and a person activated it. Two independent reviewers had signed off on the final code beforehand, each with no critical or major findings.
It launched as a pilot, not a company-wide rollout. That was by design.
The rule
Nothing ships unless two reviewers who did not build it pass the exact code, and a person says go.
“Exact” is the part most teams skip. Before a review starts, the code is frozen: every file is hashed. The review records the hash before and after. A script reads the verdicts and refuses to release unless two different reviewers returned GO or GO-with-changes, the hash did not move during the review, and the hash still equals the code about to deploy. Edit a line after the review and the release is refused.
The script does not press go. That is a rule the team follows, not code it wrote. We say so because teams tend to overstate what their automation enforces.
What six reviews look like
One feature, a notification built to send messages alongside the existing tasks, went through six review rounds, each on a different frozen version. The first reviewer’s findings per round were 9, 6, 2, 2, 1 and 0. The first three rounds were NO-GO. Round 1 had seven major findings. The second reviewer’s counts were 5, 3, 3, 2, 0 and 0.
| Round | Reviewer A | Findings | Reviewer B | Findings |
|---|---|---|---|---|
| 1 | NO-GO | 9 | GO-WITH-CHANGES | 5 |
| 2 | NO-GO | 6 | GO-WITH-CHANGES | 3 |
| 3 | NO-GO | 2 | GO | 3 |
| 4 | GO-WITH-CHANGES | 2 | GO-WITH-CHANGES | 2 |
| 5 | GO-WITH-CHANGES | 1 | GO | 0 |
| 6 | GO | 0 | GO | 0 |
The feature was launched one afternoon and released to production the next, with its switch off. Findings are a mix of severities, so read those numbers as a trend, not as nine bugs. The trend is the point: each round ended with a frozen version and a hash someone could check.
Some findings were small on purpose. One reviewer returned NO-GO because a test comment said the package installs two permission tiers when it changes three. Comments ship into the customer’s org, so the team treats a wrong comment as a major defect.
Trying to beat the gate
A rule is only as good as the ways it can be dodged, so the team tested the release script by trying to get past it. One review instead of two. A NO-GO. No verdicts file. Code changed during the review. The same reviewer twice. Code edited after the review. Eight cases in all: seven refused, one passed.
One review instead of two$ python3 review-gate.py ./release cases/one-review.txt PROD-REVIEW-GATE: REFUSED — need exactly 2 landed reviews, found 1 A NO-GO from either reviewer$ python3 review-gate.py ./release cases/a-no-go.txt PROD-REVIEW-GATE: REFUSED — REVIEWER_A verdict is NO-GO No verdicts file$ python3 review-gate.py ./release cases/missing.txt PROD-REVIEW-GATE: REFUSED — no verdicts file cases/missing.txt — reviews have not landed Code changed during the review$ python3 review-gate.py ./release cases/changed-mid-review.txt PROD-REVIEW-GATE: REFUSED — tree changed during review (d5cabe39f029af2a -> 1c0cb002f3448f96) The same reviewer twice$ python3 review-gate.py ./release cases/same-reviewer-twice.txt PROD-REVIEW-GATE: REFUSED — both reviews are from REVIEWER_A — need two different reviewers One line edited after the review$ python3 review-gate.py ./release verdicts.txt PROD-REVIEW-GATE: REFUSED — candidate tree ff4df36df11b9b94 is not the reviewed tree 1c0cb002f3448f96 — the code changed after review Two different reviewers, both GO, hash unchanged$ python3 review-gate.py ./release verdicts.txt PROD-REVIEW-GATE: PASS — REVIEWER_A GO, REVIEWER_B GO, tree 1c0cb002f3448f96
The pass is the boring one, and it is meant to be. Two different reviewers, both GO, the hash the reviewers signed equal to the hash of the bytes about to leave. Anything else, and the script says no and stops. It does not warn, and it does not ask.
What the reviews missed
Reviews are not enough. Here is where the team was wrong.
- A filter that matched nothing. A repair read tasks of a certain type. That type does not exist in the org’s list of 44 values, and never had. Both reviews passed it, and the tests were green, because the tests created their own matching records. A review checks code against itself. Someone has to check it against the org. The team now verifies every reviewer finding, and every filter value, against the live org before a fix is written.
- A green deploy that was a dead switch. One deploy reported success and reset two prompt templates to Draft, which switched off the in-product AI assistant. The team learned it from a log row with a blank intent. Activation is not part of deploy success. A post-deploy check is now a required step.
- A sandbox that did not match production. The production validation of one release failed four times in a row, with 36, 6, 1 and 5 failures, before a later run passed 935 of 935 tests. Production’s record-sharing setting had been changed, and the sandbox still had the old one. A check-only validation costs time, not an outage, which is why it comes first.
What the team would do again
- Freeze and hash the code before review, and make a script, not a person, check the hash at release.
- Use two reviewers who did not build it, from different sources.
- Validate in production first, check-only, then deploy the validated job, not a fresh package.
- Deploy in dependency order. One vague production error cost five failed attempts before someone read the team’s own notes. It was ordering.
- Start data loads small. One load covered the first 100 records, stopped for a review, and only then ran the rest, with zero failures.
- Write a lesson the day it is learned, in one file, so the next person has it the next morning.
What the team would not do again
- Let exceptions in. In the first five weeks after installation the team recorded at least 24 production deploys. The team’s own tracker marks two of them as released without both reviews. The two-review check was added to the release script on September 25, and the script now refuses.
- Treat a review as the check against the org. See the filter above.
- Run production validation at the default test level. One validation did, and 12 tests failed in unrelated org classes. Nothing was deployed, but it broke a standing rule.
- Tell a builder to apply every audit finding. A builder rewrote help text the business owner had already approved. The whole text-fix effort used more than 76 million AI tokens; what reached production was a five-line change.
- Wait until week eleven to ask whether the deck’s rules could work on real data. On day 78 an audit found a checklist step that required a record status the org never uses, so no record could ever pass it.
Production is two steps
A check-only validation first. Nothing deploys; the org runs the tests and reports. Then a person reads the result and says go, in writing, for this exact release. Then the same validated job is deployed. Not a fresh package, not a rebuild: the bytes that were validated.
$ validate ./release --target production --check-only components 95/95 tests 516/516 errors 0 check-only: nothing deployed validated job id: PRACTICE-JOB-0001 $ quick-deploy PRACTICE-JOB-0001 --target production deploying the validated bytes only components 95/95 status Succeeded
What it costs
This is not cheap. A later sandbox-only feature took 18 review rounds, 12 of which had at least one NO-GO, before the last round returned GO-with-changes and GO on an unchanged hash. The first version of a rule always feels slower. The release on October 13, nearly two months after first install, still ran under it: three quick deploys, each on a validated job, with two reviewers on the final versions.
If your team has switched on AI for Salesforce development, the minimum is small: a review on the exact code, by something that did not write it, and a person who decides.
Sources
This piece cites no outside pages. Its numbers come from the team’s own records, as set out below.
Where the numbers come from
- Dates and counts are from the team’s own daily logs and its production change tracker, read at the time of writing.
- The six-round verdicts were re-read from the six real verdict files. Reruns on an unchanged hash are not counted.
- Terminal output is from the real gate script run in a clean practice repo with the real numbers. No org was touched and no work system was captured.
- The employer, product, customers and people are not named, and no business figures from the employer appear.
Independent. Not affiliated with Salesforce.