Should AI QA block a merge in CI/CD?

Only the fast, deterministic part of it. An AI review of the diff plus a three-to-five-test critical-path smoke suite can reasonably block a pull request. Model-judged results — visual comparisons, exploratory findings, accessibility judgement calls — should post a report against the preview deploy instead of setting the exit code, because a check that is occasionally wrong will get disabled if it can hold up a teammate's merge.

Can I run AI QA without a CI/CD pipeline?

Yes. The three stages map onto what small teams already do: review the diff before merging your own work, run the exploratory pass against the preview URL your host generates, and schedule a critical-path run against production after deploy. Adding a pipeline later formalises stages you were already using informally.

AI QA in Your CI/CD Pipeline: Where It Fits and What to Run

Q: How long should the blocking QA stage take?

Budget it in minutes, not tens of minutes. The pull-request gate should cover lint, typecheck, unit tests, a diff review and a smoke suite narrow enough that you can name every test in it out loud. Everything slower or broader moves to the preview deploy, where nobody is waiting on a spinner.

Q: What should a failing pipeline test produce?

The same evidence bundle you would demand in a bug report: a screenshot or video at the point of failure, console errors, failed network requests with status codes, the URL and viewport, and the branch and commit. A failure that emits only a stack trace and a job URL turns into an hour of local guessing.

AI QA in a CI/CD pipeline belongs in three distinct places, not one: a fast deterministic gate that blocks the merge, a broader AI-driven exploration pass that runs against a preview deploy without blocking it, and a continuous check against production that catches what no pre-deploy test could. The mistake most teams make is wiring an AI testing tool in as a single blocking step — it either slows every pull request to a crawl or gets disabled the first week it flags something cosmetic. Split by stage and by blocking authority, and AI QA becomes a safety net instead of a tax.

Where does AI QA actually fit in a CI/CD pipeline?

A pipeline already has natural gates. The job is to match each kind of check to the gate where it is cheap enough to run and consequential enough to matter.

Pipeline stage	What AI QA does here	Blocks?
Pre-commit / local	Review the diff for logic the author did not intend — unhandled states, missing error paths, auth checks dropped from a copied handler	No (advisory)
Pull request	Static review of the diff, plus a small smoke suite on the critical path (load, log in, one core action)	Yes — keep it minutes, not tens of minutes
Preview / staging deploy	Persona-driven exploration of real flows, cross-browser rendering, forms and checkout, accessibility regressions	No — report, don't gate
Post-deploy (production)	Scheduled runs of the same critical-path suite, plus in-app bug capture from real users and clients	Alerts, not gates

The column that matters most is the third one. Blocking authority is the scarce resource in a pipeline: every check that can hold a merge needs to be fast, deterministic, and almost never wrong. Checks that are valuable but occasionally noisy — anything model-judged, anything visual, anything exploratory — still earn their place, just not with a veto.

What should run on every pull request, and what should wait?

Put two things on the pull request and resist adding a third. First, a review pass over the diff itself: it needs no environment, finishes in seconds, and catches the category of defect that AI-assisted code produces most — plausible-looking code with a missing branch. Second, a smoke suite narrow enough that you can name every test in it out loud. If the app does not load, a user cannot log in, or the one action the product exists for fails, the merge should stop.

Everything else waits for the preview deploy, where you have a real URL and no one is watching a spinner. That is the right place for breadth: multiple browsers and viewports, long multi-step journeys, the half of your app that nobody wrote tests for. These runs take longer and sometimes surface a judgement call rather than a clear bug, which is exactly why they should produce a report a human skims rather than a red X that blocks a teammate.

A useful ordering rule: a check graduates to blocking only after it has run non-blocking for a while and been right every time it complained. Promote checks into the gate; don't start them there. Our AI QA workflow guide walks through the same split as a day-to-day loop rather than a pipeline config.

How do you stop AI QA from blocking the pipeline?

Three failure modes account for nearly all abandoned AI testing setups, and each has a concrete fix.

Noise from model-judged results

If a check can flag "this looks wrong" rather than "this assertion failed", it will occasionally be wrong. Route those results to a report or a review comment with evidence attached, and keep them out of the exit code. A finding a developer can dismiss in five seconds costs almost nothing; a finding that blocks their merge costs a context switch and a grudge.

Flaky tests treated as real failures

Nothing erodes trust in a pipeline faster than a suite that fails for reasons unrelated to the change. Fix the flake or quarantine the test explicitly — never leave a known-flaky test in the blocking gate, and never let "just re-run it" become the team's reflex. Self-healing selectors help with the specific flake caused by markup churn, but they do not fix a race condition or a test that depends on yesterday's seed data. We go deeper on telling those apart in fixing flaky end-to-end tests.

Failures you cannot reproduce

A pipeline failure that produces only a stack trace and a job URL turns into an hour of local guessing. Make every failing run emit the same evidence bundle you would demand in a bug report: screenshot or video at the point of failure, console errors, failed network requests with status codes, the URL and viewport, and the branch and commit. When the artifact is complete, the fix starts at the cause instead of at the reproduction.

What if you don't have a CI/CD pipeline at all?

Plenty of teams shipping real products — especially those building with Cursor, Lovable, v0, Bolt or Replit — deploy straight from a push or a button in a hosting dashboard. The stage model still applies; the stages are just not YAML files.

Before you merge your own work: read the diff with an AI reviewer. This is the pull-request gate, minus the pull request.
On the preview URL your host generates: run the exploration pass. Preview deploys are the single most underused QA surface available to small teams — a real environment, a real URL, and no consequence to breaking it.
After deploy: schedule the critical-path run against production and make it easy for a client or user to file a bug with evidence attached, rather than by sending a screenshot in a chat.

Adding a pipeline later does not change the thinking. You are formalising stages you were already using informally, which is a much smaller job than designing a QA strategy and a pipeline at the same time.

How does this map to Klavity's three pieces?

Klavity is built around this split rather than around a single blocking step.

AutoSim covers the deterministic layer — end-to-end tests that repair their own selectors when markup changes, so the suite you put in the blocking gate stays trustworthy instead of decaying into re-runs.
Sims covers the exploratory layer against a preview deploy — AI personas work through real flows the way different kinds of users would, including the paths your written tests never cover.
Snap covers what escapes anyway — a right-click bug report from the page itself, with console, network and environment captured automatically, filed to your tracker.

For the full picture of how these fit together, including where human review still belongs, start with our complete guide to AI QA, or read continuous QA for the cadence side of the same idea.

A minimal setup worth copying

On every pull request: lint, typecheck, unit tests, an AI review of the diff, and a smoke suite of three to five critical-path tests. Blocking. Budget it in minutes.
On every preview deploy: the full end-to-end suite plus an exploratory persona pass across your two or three real browser/viewport combinations. Non-blocking, posts a report.
On a schedule against production: the critical-path suite, hourly or daily depending on traffic. Alerts a human; does not gate anything.
Always on: in-app bug capture, so a bug a client finds arrives with evidence instead of as "the site is broken".

That is four lanes, one of which can stop a merge. Review the boundary every few months: a non-blocking check that has been reliably right is a candidate for promotion, and a blocking check that cried wolf twice should be demoted the same day.

Ready to wire this up? Try Klavity free — point it at a preview URL and see what the exploratory pass finds before your next deploy goes out.