Plan 07 — Cutover Runbook
This is NOT a code plan. It's an ops runbook. Run by a human operator (on-call engineer or release captain) after Plans 03-06 have all merged + integration tests pass in test env.
For agentic workers: REQUIRED SUB-SKILL — none. This plan is procedural. An agent must not execute it autonomously; each step requires human verification before the next step runs.
Goal: Cut over from Phase 1 transitional schema(3-value status enum + legacy columns)to final state(2-value status + legacy columns dropped). After this runbook completes,@retaintive/common@2.0.0 ships with final schema,and the Phase 1 invariants (§5.0 of normative-spec) hold in test env.
Architecture: All caller code in Plans 03-06 already writes 'open' for new tasks and reads with status IN ('pending', 'open'). This runbook only:
- Backfills data (
UPDATE pending → open) - Drops the now-unused legacy enum value, columns, and CHECK clauses
- Bumps
@retaintive/commonmajor (1.2.x → 2.0.0) because the enum shape is breaking
Tech Stack: PostgreSQL 16 (Neon) / drizzle-kit / psql / GitHub Actions release workflow
Spec source: docs/product-design/v2/unified-pipeline/implementation-plan/normative-spec.md §6.3 cutover preflight + §6.4 cleanup
Dependency: Plans 03, 04, 05, 06 all merged + their integration test suites green + test env smoke tests pass for ≥1 hour without errors.
Why this is a separate runbook
The destructive operations (UPDATE, DROP COLUMN, enum narrow) cannot be put in a normal PR because:
- Order-sensitive. Caller code must be deployed first (writing
'open', readingIN (...)); then DB; then bump common to drop enum value. - Not atomic across services.
UPDATE pending → openruns once at the DB level; before it runs, some caller might still be reading'pending'. Order matters. - Rollback playbook differs. A code PR rolls back via
git revert. This runbook rolls back via specific SQL. - Human gate required. Before each destructive step, the operator visually verifies the prior step succeeded and no caller is still writing legacy values.
That's why this is a runbook (per-step manual verification) rather than a PR (atomic commit).
Preflight (do all of these before running ANY destructive step)
This section is checkbox because the operator literally checks each item.
P1: All caller PRs merged
- Plan 03 contacts-analyzer migration PR is merged to its repo's main branch
- Plan 04 lead-processor migration PR is merged
- Plan 05 message-processor migration PR is merged
- Plan 06 studio-api migration PR is merged
P2: All caller integration tests green
-
cd /Users/maxwsy/workspace/callytics-infrastructure/lambda/contacts-analyzer && bun test --grep integration— all pass -
cd /Users/maxwsy/workspace/callytics-infrastructure/lambda/lead-processor && bun test --grep integration— all pass -
cd /Users/maxwsy/workspace/callytics-infrastructure/lambda/message-processor && bun test --grep integration— all pass -
cd /Users/maxwsy/workspace/studio-website-monorepo/apps/api && bun test --grep integration— all pass (E2Eclose/progress/reopen/postponeendpoints) -
cd /Users/maxwsy/workspace/lead-tracking && bun test --grep integration(Codex audit 2026-06-02 抓的 missing — lead-tracking 也写 tasks)
P3: Test env Lambdas deployed with new code
- CloudFormation / CDK deploy stack for
contacts-analyzershows latest commit (newer than Plan 03 merge) - Same for
lead-processor - Same for
message-processor - studio-api stack shows latest commit (newer than Plan 06 merge)
-
lead-trackingstack shows latest commit(Codex audit 抓的 missing)
Verify via:
P4: No caller writing legacy values (no 'pending', no action_needed=true, no task_type set)
First verify the repo, not only recent DB writes:
Expected output: no active writer writes status='pending', tasks.action_needed, or tasks.task_type. Read paths may still contain transitional status IN ('pending','open') until final cutover; those should be annotated as read-only transitional compatibility.
Run smoke read against test Neon over the past 1 hour of writes:
Expected output: Only 'open' and 'closed' should appear for tasks created in the last hour. If 'pending' shows up:
- Identify which Lambda is writing it (
SELECT contact_phone, source_type, created_at FROM tasks WHERE status = 'pending' AND created_at >= NOW() - INTERVAL '1 hour' LIMIT 10— then trace bysource_typeand look at the writer Lambda) - Verify that Lambda's deployment includes the Plan 03-06 changes
- Re-deploy if not, then re-run P4 after 30 minutes
Expected output: > 0 is OK (legacy callers transitioning). But verify by querying:
contact_analyzerwritesaction_needed = trueduring Plan 03 transitional window (intended; removed in this runbook)- No code path should write
action_needed = false
P5: CloudWatch logs show no recent old-path errors
Expected: No matches.
P6: Dashboard parallel-query verification
Before cutover, routes/v3/dashboard-*.ts should already be using the new task_progress_events count for "Attempt workload" metric. Quickly compare:
Old way (still works during transitional):
New way:
- Numbers within ±10% of each other (some drift is expected because
attemptedwas set at close time,progress_eventsare per-attempt and finer-grained) - Trend over the past week is consistent (no sudden divergence)
If drift > 10% or trend diverges, STOP. Investigate before proceeding.
P7: Backup test Neon
- Confirm branch exists in Neon dashboard
Runbook steps
Operate one step at a time. After each step, verify it succeeded before running the next.
Step 1: Apply 0011_phase1_cutover.sql (UPDATE data + change CHECK)
This is the data-and-CHECK migration. Schema columns stay (drop happens later).
1.a Generate the migration file
Create drizzle/0011_phase1_cutover.sql manually (not via drizzle-kit generate, because this migration needs ordered statements):
1.b Run dry-run via manual transaction wrapper
Expected: UPDATE N where N is the count of remaining pending rows. If N = 0, the backfill is already done (good).
1.c Apply for real
1.d Verify
Expected: only 'open' and 'closed'. Zero 'pending' rows.
Expected: the CHECK definition contains status = 'open' (not IN ('pending', 'open')).
1.e If 1.d fails
Then investigate the source of the writes that re-added 'pending'. Common cause: a Lambda still on old code.
Step 2: Narrow TASK_STATUS enum in @retaintive/common
Now that the DB has no 'pending' rows, the TypeScript enum can be narrowed.
2.a Bump version + edit schema
In the phase-1/cutover-data branch on callytics-common:
Edit src/db/schema/tasks.ts line 16:
This is breaking for any external consumer that imports TASK_STATUS. Bump version 1.2.x → 2.0.0.
Edit package.json:
2.b Update other transitional-only constants
In src/db/schema/task-ui.ts, remove the legacy 'pending' entry from STATUS_OPTIONS:
2.c Run typecheck and test
Expected: All passes. If a test fails because it still asserts TASK_STATUS.length === 3, update the test.
2.d Build + check exports
Expected: 0 (no 'pending' string literal in the compiled output).
2.e Commit and PR
2.f Pre-publish lockstep verify(防 Lambda auto-update 拉 2.0.0 break legacy caller)
Why this step exists(Codex audit 2026-06-02 抓的 ordering bug):
Publish 2.0.0 后,任何还没升级到 Phase 1 caller code 的 Lambda 在下次 deploy
(bun install 时拉最新 2.x)会立刻 break — 老 caller 还在写 status='pending',
但 2.0.0 已是 narrow status enum ['open','closed'],运行时 throw。
Lockstep gate(全部满足才能 publish):
-
All caller package.json 已 pin 当前 1.x range 上限 — 防 auto-update:
-
All Phase 1 caller PR merged to main — Plan 03 / 04 / 05 / 06 全 merged 状态确认。lead-tracking 也写 tasks(Codex 抓的另一个 missing),scope 列入 verification:
gh search code --owner retaintive 'applyTaskActions\|buildTaskActionSQL'预期返回:contacts-analyzer / lead-processor / message-processor / studio-api / lead-tracking。 -
CI 已通过 所有 caller repo main branch 最新 commit 的 build + test。
2.g Merge PR + publish 2.0.0
After lockstep gate 2.f 全通过:
2.h Verify 2.0.0 reachable
Expected: Lists 2.0.0 in versions array.
Step 3: DROP tasks.action_needed and tasks.task_type columns
Now the schema can shed the legacy columns. Caller code has been writing legacy shim values during Plans 03-06 and reading via EXISTS. Time to drop.
3.a Verify no caller still reads the columns
Expected: No matches in code paths actually used by Phase 1. Some legacy migration files / tests may still reference. Investigate any production-path match.
Expected: No matches in active code. Especially verify routes/tasks/list.ts:164 (Plan 06 should have removed t.task_type) and routes/v3/contacts.ts:112 (now uses EXISTS, not c.action_needed).
If any match exists in a production path, STOP and re-check Plans 03-06 PRs to ensure complete removal.
3.b Generate destructive migration
Create drizzle/0012_phase1_drop_legacy.sql:
3.c Dry-run
Expected: No errors. Tables look correct after rollback (since we rolled back).
3.d Apply
3.e Verify
Expected: Zero rows (columns are gone).
Expected: Zero rows (constraint is gone).
3.f Commit migration to common
After review: bump @retaintive/common@2.1.0 and publish.
Step 4: Narrow TASK_CLOSE_RESULT to 15 final values
Now that no caller writes progress values to closeResult (they go through record_progress), shrink the enum.
4.a + 4.b 同事务执行 backfill + CHECK 收紧
Why 合并事务(Codex audit 2026-06-02 抓的 race window):
如果 4.a backfill COMMIT 后、4.b CHECK install 之前,有 writer 写 'attempted',
4.b 新 CHECK 直接 reject 这些刚写入的行(已 invalid)。4.a + 4.b 必须同事务,
DDL + DML 一起 atomic — 或在 4.a 之前 freeze writers。
The 4 progress-style values (no_answer, left_voicemail, callback_later, attempted) might still exist in historical rows of the tasks table. These have already-closed tasks; they don't represent active work.
Expected: Some count for attempted (historical), possibly small counts for the others.
4.c Narrow the schema enum(TS side — callytics-common@2.x PR)
In callytics-common schema(这步在 PR 而非 psql,跟 4.a+4.b 不需要同事务):
Note: TS enum 收紧的 CHECK 跟 4.b SQL CHECK 已对应。下面 psql 重复 CHECK 写出
仅作 idempotent 验证,可跳过(运行也无害):
Also update CLOSE_RESULT_OPTIONS in task-ui.ts to remove the 4 progress entries.
4.c Bump and publish 2.2.0
Step 5: tasks.store_id SET NOT NULL
Now that all caller code passes storeId (verified by Plan 06's getAuthorizedStoreNeon() and Plans 03-05's lookup paths), enforce at DB level.
5.a Verify no NULL rows
Expected: Zero rows in test env (Plan 01 Step 5 already DELETEd them during the 0010 migration; this re-verifies no stray writes recreated them).
If non-zero, identify and DELETE before proceeding:
5.b Apply NOT NULL
5.c Verify
Expected: is_nullable = NO.
Expected: index definition no longer contains WHERE.
5.d Update common schema
Edit src/db/schema/tasks.ts:
Commit + publish 2.3.0.
Step 6: Final invariant check
After Steps 1-5, run the full §5.0 invariant test suite from normative-spec.
Plus the schema-level invariants directly via psql:
- All caller integration tests pass
- All direct DB invariants verified
- Integration scenarios explicitly cover DNC hard stop, cross-store taskId rejection, deterministic lead path, and progress endpoint idempotency
Step 7: Cleanup transitional code
These are not destructive but should happen now for hygiene.
7.a Remove .transform() shim in TaskDecision Zod schema
The shim in contacts-analyzer/src/core/models.ts that mapped legacy create → create_open is no longer needed because prompt-engineer has shipped the new prompt (Plans 03-06 assumed this).
Verify prompt is shipped:
If the new prompt is in place, remove the .transform() from models.ts. Otherwise leave it.
7.b Delete shared/utils/dnc-cascade.ts
This helper is no longer called by anyone (Plan 05 message-processor migrated to closeAllOpenForContact() which inlined the logic).
Expected: Only matches inside dnc-cascade.ts itself or its test file. If others, investigate.
Commit with a chore: remove unused dnc-cascade helper message.
Step 8: Ship release notes
Document what changed for downstream awareness.
Update docs/product-design/v2/unified-pipeline/unified-pipeline-final.md or create a release note in docs/release-notes/2026-06-XX-phase-1-cutover.md:
- Release note committed and pushed
Rollback procedure
If any step fails halfway and the system is unstable, follow these rollback paths.
Rollback after Step 1 (data backfill failed)
Verify SELECT status FROM tasks GROUP BY status returns 3-value enum again. Investigate root cause before re-attempting.
Rollback after Step 2 (common 2.0.0 published but DB unhealthy)
Common is already published; cannot unpublish. Mitigation:
- Pin downstream consumers to 1.2.x in their
package.json - Fix the DB issue
- Re-attempt Step 2 with a
2.0.1patch if needed
Rollback after Step 3 (DROP COLUMN — destructive)
Cannot restore columns cheaply. Restore the whole Neon branch:
This loses any data written between cutover start and rollback. Coordinate with team before doing this. Better path is to verify P4 / P5 thoroughly before Step 3 so this never triggers.
Rollback after Steps 4-5 (closeResult narrow / NOT NULL)
Restore Neon branch. Same caveat as Step 3.
Estimated time
- Preflight checklist: 30-60 minutes
- Steps 1-3 (data + enum + drop columns): 30-60 minutes (waiting for Neon, common publish, etc.)
- Steps 4-5 (closeResult + NOT NULL): 30 minutes
- Step 6 (invariants): 15 minutes
- Steps 7-8 (cleanup + release notes): 30 minutes
Total: 2-3 hours under happy path. Allow a half-day window. Schedule outside of business hours if test env is shared with other dev work.
Sign-off
After completion, the operator confirms:
- All preflight items P1-P7 verified
- Steps 1-6 completed without unplanned rollback
- Step 6 invariants all green
- Steps 7-8 cleanup and release notes shipped
- Operator:
__________________Date:__________
Phase 1 is done. Move to Phase 2 backlog.