Connecting Codex to GitHub for Automated Review
In one line: on an agent-built codebase the bot reviewer is not a nice-to-have alongside human review — measured over 348 pull requests it is the review function, and the rule it needs is "merge when the reviewer has gone quiet on the current head", not "merge when CI is green".
Measured on the flagship project, over every inline review comment the repository holds:
Review comments by chatgpt-codex-connector[bot] | 1,153 |
| Review comments by a human | 13 |
| Ratio | 89 : 1 |
| Pull requests that received findings | 348 |
| Findings per PR | median 2, max 24 |
| Review rounds per PR (distinct head commits reviewed) | median 1, mean 1.4, max 8 |
| PRs needing more than one round | 51 / 348 (14%) |
| PRs needing more than three rounds | 17 / 348 (4%) |
Earlier revisions of this document wrote "Codex, CodeRabbit or equivalent", which reads as a menu of interchangeable options. At 89:1 that phrasing misdescribes how the methodology actually runs. Codex is named here because the measurement names it; an adopter running a different reviewer should substitute freely, but should not read the plural as evidence that the choice is unimportant or that a human reviewer is covering the same ground.
Setup — four steps, none of which setup/bootstrap.sh can do for you (it
makes zero GitHub API calls; see §7.6 for the same limitation on rulesets):
- Install the app. Settings → GitHub Apps → Codex → Configure, scoped to the repository, or from the ChatGPT side under Codex → Environments → Connect GitHub. It needs read access to the code and write access to pull requests. This is a console click-path, and §5.7 says to prefer a CLI — there is no CLI for app installation, so this is one of the few genuinely manual steps. Record that it was done; nothing downstream can detect it.
- Enable review on pull requests in the app's settings so it fires on
openedandsynchronizewithout being asked. If you skip this, reviews only happen when you request them by hand, which is a reminder, not a mechanism. - Verify it is actually attached — presence in a settings screen is not
evidence it ran:
A repository that believes it has bot review and shows zero bot comments has an unconfigured app, not a quiet one.gh api "repos/<owner>/<repo>/pulls/comments?per_page=100" --paginate \--jq '.[].user.login' | sort | uniq -c | sort -rn
- Know the re-trigger. After pushing fixes, a new round is not always
automatic:
gh pr comment <n> --body "@codex review"
The merge rule this creates. CI green and review clear are different questions, and the reviewer answers the one CI cannot. Do not merge on a green tick while the reviewer is mid-round: 14% of PRs needed a second round and 4% needed a fourth, so "it passed CI" is silent about a defect class that showed up on one PR in seven. The discipline is:
- Push the fix, then wait for the reviewer to speak on the new head before merging.
- A stalled or rate-limited bot looks exactly like a bot that found nothing. Treat silence as unknown, not as clear — the same fail-closed reading the rest of this document applies to evidence.
- Every finding still takes one of the three §3.3 dispositions — fixed, acknowledged with a tracking issue, or disputed with technical reasoning. "The bot is wrong" without reasoning is not a disposition.
Honest limits. This is a review, not a gate: it reports no status, so it cannot block a merge and must never be counted as an enforced control in a compliance or conformity artifact (§7.1, §14). It has no observation proof in the §7 sense — nothing detects the case where it was silently uninstalled except step 3 run by hand. And the reviewer sees the diff, not the system: it is reliably good at the defect classes this document keeps naming (a guard that cannot fire, a claim with no check, an absent input read as a benign value) and reliably blind to whether the feature was worth building.
Evidence: the ratio, round distribution and finding counts above are read from the live GitHub API over 1,166 review comments, not estimated. In the session that produced this section the reviewer found real defects on essentially every pull request, including defects inside the quality gates being written in those same PRs.