Incident explainer / Vercel deployment queue
The healthy target is 50 or fewer. The queue is holding 450 too many deployment records, with an average wait of 1 hour 26 minutes and an oldest wait of 3 hours.
Each square is one queued deployment record. Green is the entire desired queue; red is overload.
+450 OVERIt is an author stamp on deployment records, not a count of unique changes made by one bot.
The same automation identity authored or triggered all 500 queued deployment records.
One bot performed 500 unique app changes. It also does not mean AdamoBot is a Vercel project with 500 builds.
It ties the records to one automation channel. That makes the origin traceable, but the project and commit fields explain how a small number of repository changes multiplied into hundreds of deployment rows.
In a monorepo, Vercel projects watch the same repository. A routine maintenance commit can be interpreted as work for many projects, even when it genuinely changed only a few apps.
One repository event lands.
~51 Vercel projects evaluate the same commit. Gold marks ally, the current leader with 14 queued rows.
Top commit 2047d2d462a1
The top commit generated 51 deployments. That is fanout: one commit, many project-level queue records.
Each repeated commit produced another fanout wave. Newer rows piled up behind older rows for the same projects, creating duplicate, superseded work.
This is accumulation, not 500 separate ideas. Repeated repository events multiplied across projects while earlier deployment rows were still waiting.
For a given project, an older queued build may already be obsolete because a newer commit is waiting behind it. Unless the older row is prevented or safely canceled, Vercel continues treating it as pending work.
The current Ignored Build Step can cancel after a deployment reaches build processing. The better control is Skip unaffected projects, which prevents unrelated deployment records before they enter the queue.
Late filter: the record already exists and has entered deployment processing.
Reduces build work, but still creates queue and concurrency pressure.
Early filter: Vercel checks whether the project changed before creating the deployment.
Prevents unrelated deployments before they can consume queue capacity.
57 audited / 24 need repair
The reducer only considers QUEUED rows. Per project, it preserves the newest queued deployment and removes older queued copies that the newest one supersedes.
Preserve the newest queued row per project. Cancel only older queued rows for that same project.
Cancel queued rows for projects the commit did not genuinely touch. Preserve affected apps.
Candidate groups can overlap. Do not add 405 and 490 together. The dry run identifies records; it has not mutated production.
Cleanup restores capacity once. Prevention keeps routine repository activity from rebuilding the same queue.
Enable Skip unaffected projects for all 24 projects currently disabled or unset.
Classify top-level channels, docs, and .forgebot content as workspace packages so affected-project logic can reason about them.
Do not land routine-generated app artifacts unless the app change is intentional.
Fail deployment configuration audits whenever affected-project skipping is absent.
Repeated maintenance commits fanned out across many Vercel projects faster than the platform could drain them, while the current build filter acted after queue creation; safe cleanup removes obsolete queued records, and early affected-project skipping prevents the next pileup.