Skip to content

Migrate Sponsor CRM GSIs in a protected stage-only workflow #136

Description

@alexeygrigorev

Migrate Sponsor CRM GSIs in a protected stage-only workflow

Status: blocked — #140 research lifecycle is complete; #143 ownership reconciliation and the separate HUMAN protected-environment gate remain open
Tags: bug, infra, testing, data, migration, P1
Depends on: #140 completed with Tester PASS and PM acceptance; #143 accepted ownership remediation; completed/effective #138 and no-change #139 via aws-infra d370385556344bb6841f88473c12397c326a6b5a; completed/effective #129/#130/#135; application stack terminal; #141 is not a blocker
Blocks: #114, #115, #121, #122, #124, #125, #126, #127, #128, #129, #131, #134, and #137
Next owner: authorized HUMAN AWS operator on #143 under #143’s existing Support-guidance resume signal; independently, the protected-environment administrator owns that separate explicit gate
Resume condition: #143 records the accepted ownership/integrity outcome and the protected-environment proof is recorded; then resume #136 with fresh Architecture/Security review of any identity or guard assumptions invalidated by #143 before HUMAN dispatch
Evidence: Latest Architecture BLOCK, Latest Security BLOCK, #138 effective HUMAN gate, #139 no-change closure, failed attempt 2, #140 Tester PASS, and #140 PM acceptance

Outcome

The live dataops-v1-sponsor-crm table is ACTIVE with zero GSIs. The application template contains four GSIs; one ordinary CloudFormation update attempted all four and DynamoDB rejected it with Cannot perform more than one GSI creation or deletion in a single update.

Use two disjoint workflows:

  1. A protected migration-only workflow_dispatch discovers exact prefix 0..4, creates and executes one exact table-only GSI change set at a time, waits for the index to become ACTIVE with no backfill, proves retained table/stack state, reaches exact prefix four, and stops.
  2. The existing deploy-dataops-v1 workflow remains the sole application deploy. Immediately after OIDC and before build, a bounded read-only guard independently proves current stack ownership and exact live prefix four. Incomplete or unowned state fails before build/mutation with migration instructions; valid state permits the unchanged SAM deployment path.

After accepted code reaches main: HUMAN dispatches migration, On-Call verifies the table-only result, then On-Call reruns the ordinary main deploy under its existing OIDC role. No manual app deploy.

Superseding scope

All earlier final-application handoff requirements remain removed from #136:

  • no custom final SAM/package/deploy path;
  • no direct final application change set;
  • no local SAM translator/toolchain;
  • no final package/upload/manifest/full-app binding;
  • no final app no-change/pre-execute/postdeploy implementation.

Migration contains no application deployment. Normal deployment contains no staged migration. The existing application SAM command remains the sole app deploy.

Rejected fingerprint

The latest repaired simplified candidate is rejected for commit, integration, dispatch, or execution:

  • base/HEAD: 79ed0adb3c9f020a435f9604879be2331b6968fd
  • exact 11-path tree: 1aaff729053e32ea809ee841ef2947a6ace3cd7f
  • full staged-equivalent binary diff SHA-256: ce26c77a19e94dd941595da32eacaaa9cd8c1c53022ba796503593bd5cbb69bb
  • unchanged original application sam deploy command SHA-256: 5a6f6391d8be4c5b6219e5e9c94d7249d312ccab1d743b4d7bf20cb6d082d270

Keep the accepted two-workflow split, immediate post-OIDC guard placement, read-only ownership/GSI guard, protected migration boundary, one-GSI generator/state machine, bounded processes/waiters, exact-ID template/change validation, retained-table snapshot, and final pagination.

Repair exactly four groups:

  1. exact fresh-dispatch recovery of one prior-run unexecuted candidate;
  2. all seven missing CloudFormation request/default fields;
  3. stable PITR normalization that excludes volatile restore timestamps;
  4. complete immediate pre-execute stack/Processed/live/retained-state reread.

Canonical stages

Starting from base PK/SK, add exactly:

  1. GSI-Communication: string GSI1PK HASH and GSI1SK RANGE; projection ALL.
  2. GSI-SponsorSendDue: string GSI2PK HASH and GSI2SK RANGE; projection ALL.
  3. GSI-SponsorSendLookup: string GSI3PK HASH and GSI3SK RANGE; projection ALL.
  4. GSI-SponsorBookingCommunication: string GSI4PK HASH and GSI4SK RANGE; projection ALL.

Each transition adds exactly one GSI and two string attribute definitions. It preserves the same table identity for the migration run, base keys, billing, stable SSE/PITR configuration, current TTL, current stream configuration/identity, retention policies, tags, and all non-stage template content. It never replaces the table, deletes/alters a GSI, or changes another resource.

Two identities: deploy guard versus migration run

Reviewers and tests must use this distinction.

Cross-workflow deploy guard identity

The ordinary deploy guard starts independently on a later workflow run. It has no trustworthy pre-migration TableId to carry across workflows and must not hardcode or compare against a historical TableId: a legitimate stack-owned table replacement in some separately accepted future deployment can change TableId.

Instead, the guard establishes current ownership atomically through:

  • exact caller/account/role/region;
  • exact stack name and current exact StackId;
  • exact Sponsor logical resource to current physical table mapping;
  • the current table ARN matching that exact physical table/account/region;
  • canonical CloudFormation system tags matching the same ownership:
    • aws:cloudformation:stack-name=dataops-v1;
    • aws:cloudformation:stack-id=<current exact StackId>;
    • aws:cloudformation:logical-id=<exact Sponsor logical ID>;
  • a present, syntactically valid TableId used consistently within that guard observation;
  • exact healthy prefix four.

This is the deploy guard's replacement/ownership proof. A test must not demand equality to a TableId from a different workflow. Reviewers must assess the guard against the full StackId/logical/physical/ARN/system-tag binding, not against a nonexistent cross-run TableId constant.

The guard must not require future target TTL or stream settings. The following existing SAM deploy owns adding/updating them. Guard acceptance is current ownership plus healthy prefix four.

Within-run migration identity

Migration has a preflight observation before any stage. It must pin the exact current TableId and ARN, then compare both exactly at every pre-stage, post-stage, waiter-resume, immediate pre-execute, and final boundary. Any change during that migration run is a replacement/drift and fails closed.

Migration also snapshots current TTL/stream and proves those values remain unchanged; it does not require the future application target.

Protected migration workflow

  • A separate workflow has only workflow_dispatch; no push, schedule, reusable-call, or app-deploy trigger.
  • It uses dataops-v1-production, shared non-cancelling deployment concurrency, and the existing OIDC deploy role.
  • Before mutation it validates exact event, actor alexeygrigorev, repository DataTalksClub/dataops, refs/heads/main, SHA, OIDC principal/account, partition, region, stack name/StackId, logical resource, physical table, ARN, and TableId.
  • It only reads stack/table state and creates/describes/gets/deletes/executes exact migration-owned stage change sets.
  • It contains no SAM build/package/deploy, app package/upload/change set, smoke/seed, workflow chaining, or app-deploy helper.
  • At exact healthy prefix four, it proves completion and stops.

Existing deploy workflow and immediate guard

  • deploy-dataops-v1 retains the original SAM deploy command and is the only app deployment workflow.
  • OIDC credential setup is followed immediately by the guard, with no intervening action/run step.
  • Guard completes before make sam-build and every package/upload/change-set/deploy/hook or credentialed source/build action.
  • It uses only bounded read APIs required for caller, stack/resource mapping, tags, and DynamoDB schema/status.
  • It proves the cross-workflow identity contract above and exact prefix four: canonical GSI names, string key attributes, key schemas, ALL projections, all ACTIVE, all Backfilling=false, exactly base plus eight GSI attributes, no extra/missing/transitional index.
  • Failure occurs before build with clear migration-and-rerun instructions.
  • Success falls through only to the existing app path. Guard never calls migration or mutates.

Effective prerequisites and HUMAN boundary

Full baseline and stable normalization

Before stage one, snapshot and bind:

  • complete exact-stack Processed template object and digest;
  • stack/StackId and logical-to-physical resource identity;
  • physical table, ARN, TableId, and base keys;
  • billing;
  • complete stable non-secret SSE configuration;
  • stable PITR configuration;
  • current TTL status/attribute;
  • current stream enabled/view and current ARN/label;
  • canonical complete tags;
  • deletion/update-replace retention policies.

PITR normalization must compare stable configuration/state only:

  • ContinuousBackupsStatus;
  • PointInTimeRecoveryStatus;
  • configured RecoveryPeriodInDays when returned/applicable.

It must deliberately omit EarliestRestorableDateTime and LatestRestorableDateTime, because they advance during a healthy multi-stage run. No other security/configuration field may be dropped merely to make equality pass.

Tests must prove advancing restore timestamps are accepted while a change to continuous-backups status, PITR status, or recovery period fails.

For each stage, deep-copy the latest accepted complete Processed template and add only the next two attributes plus canonical GSI. A canonical structural diff must prove no other table path or resource changes. The accepted full Processed baseline advances only by that exact stage delta.

Prefix and resume state

At every required boundary compare the full current stack/Processed template, live ownership/retained snapshot, and Processed/live GSI schemas.

Accept only:

  • terminal exact prefix N, 0 <= N <= 4, all present indexes ACTIVE, none backfilling; or
  • one interrupted transition where Processed/live both have exact prefix N, indexes 1..N-1 are ACTIVE/not backfilling, only newest N is CREATING/backfilling with exact schema, and table is ACTIVE or UPDATING.

Interrupted transition creates/executes nothing; resume bounded waiter and repeat full checks. Reject drift, disagreement, earlier non-active index, multiple transitions, gap, extra/wrong index or attribute, deletion state, replacement, or unsupported stack/table state.

At terminal prefix N below four create only N+1. At four, run final proof and stop.

Closed stage request, including all defaults

Freeze one canonical stage request. Bind stack/type/name/description/client token, exact processed stage template/reference/digest, complete parameters/UsePreviousValue choices, capabilities, and every explicit/default request decision.

Post-create and immediate pre-execute Describe validation must include the previously missing seven fields exactly:

  1. ChangeSetType — exactly UPDATE;
  2. RollbackConfiguration — exact normalized empty/default configuration;
  3. Tags — exact frozen expected normalized set/absence;
  4. OnStackFailure — exact expected absence/default for this UPDATE;
  5. ImportExistingResources — exactly false/absent per the frozen request normalization;
  6. ParentChangeSetId — absent for this non-nested change set;
  7. RootChangeSetId — absent for this non-nested change set.

Also retain exact validation of:

  • structured create response Id and StackId, including ARN partition/account/region/stack/name/UUID;
  • parameters;
  • capabilities;
  • absent RoleARN;
  • empty NotificationARNs;
  • IncludeNestedStacks=false;
  • name/description/client token identity;
  • status/execution status;
  • full exact-ID TemplateStage=Processed equality;
  • exactly one non-replacing Sponsor Modify for the next GSI/attribute transition and no other resource action.

Reject an unknown/undeclared field rather than silently accepting it. Validate after creation/discovery and repeat immediately before executing the immutable exact ARN.

Fresh prior-run recovery

Before every create/execute, fully paginate active change sets for exact StackId and reject token cycles/repeats/truncation/excessive pages.

A fresh authorized dispatch, including attempt 1 with a different GITHUB_RUN_ID, may delete one earlier-run candidate only when it is the sole active candidate and full proof shows:

  • its closed identity parses as this migrator and binds the same repository/ref/SHA, StackId, current next stage, old-prefix baseline digest, and generated-template digest;
  • exact-ID Describe is CREATE_COMPLETE and ExecutionStatus=AVAILABLE;
  • all request fields, including the seven above, parameters, capabilities, no-role/no-notifications/nested=false, agree exactly;
  • exact-ID Processed template equals the regenerated stage;
  • exact change list is the single next table transition;
  • live and current stack Processed state remain the exact old prefix/baseline;
  • it has never begun execution.

Recognition must not derive allowable prior candidates only from the current run ID or lower attempts. Prior run ID/attempt are parsed and validated as provenance, but content/metadata/state proof establishes safe ownership. Name or description alone is never sufficient.

Then delete only that exact ARN, wait boundedly until absent, fully paginate and require zero candidates, and create a fresh candidate bound to the current run/attempt. Never execute/reuse the prior-run candidate.

Any ambiguity, second candidate, mismatch, unsafe status, different source/stage/baseline, or incomplete proof blocks without deletion. Post-execute interruption resumes from exact Processed/live state; never reverse a completed GSI.

Immediate pre-execute full reread

Immediately before every ExecuteChangeSet, after all candidate waiting and without an intervening mutation-capable action:

  1. validate exact caller/account/role/region and exact stack/StackId terminal state;
  2. retrieve the current stack TemplateStage=Processed and require complete equality to the expected old-prefix full baseline;
  3. re-resolve the Sponsor logical resource to the same physical table;
  4. re-read DescribeTable, PITR, TTL, tags, stack resource/template retention, and every live/retained property;
  5. require exact within-run ARN/TableId, keys, billing, SSE, stable PITR normalization, current TTL, current stream view/ARN/label, tags, retention, and old live prefix;
  6. fully paginate and require the exact candidate is the only active item;
  7. re-Describe the exact candidate and validate all closed request/default/status/change fields;
  8. re-fetch the exact candidate Processed template and require full generated-stage equality;
  9. execute only the immutable exact ARN.

Candidate-only rereads are insufficient. Any current stack, Processed, ownership, live, retained, request, content, status, inventory, or change-list drift stops before execution.

Perform the same full stack/Processed/live/retained checks at preflight, before and after each stage creation/wait, after interrupted resume, after each completed stage, and final completion.

Completion, timeout, and cleanup

After prefix four:

  • prove exact same within-run ARN/TableId/logical/physical ownership;
  • prove keys, billing, SSE, stable PITR, current TTL, current stream identity/view, tags, and retention equal baseline;
  • prove exact four canonical GSIs and eight GSI attributes, all ACTIVE, no backfill;
  • prove the complete latest Processed template equals the preflight full template plus only the four ordered stage deltas;
  • after all final table/stack reads, make a final fully paginated active-change-set call as the last AWS read before success and require empty inventory.

Every AWS subprocess/waiter has argv-only execution, hard timeout, output cap, TERM→KILL, cancellation handling, bounded backoff, and redacted errors. Cleanup touches only exact owned local/S3 artifacts and a fully proven unexecuted candidate. Never glob/bulk-delete/alter unrelated state.

Required tests

  • Workflow split and original app SAM-command digest.
  • Guard immediately after OIDC and before make sam-build/all deploy-capable actions.
  • Guard exact StackId/logical/physical/ARN/system-tag/prefix-four proof, including wrong mappings/tags/ARN and malformed TableId.
  • Guard test explicitly does not hardcode a historical TableId; a currently stack-owned replacement identity may pass if all current ownership bindings and prefix four are exact.
  • Migration test pins one exact TableId/ARN and rejects any valid substitution from preflight through every stage/final boundary.
  • Stage generation changes only next two attributes/GSI; every other template mutation fails.
  • Prefixes 0..4, interrupted resume, mismatch/gap/extra/wrong schema/rollback/replacement tests.
  • Fresh-dispatch cancel-after-create recovery with different run ID and attempt 1 deletes only a fully proven AVAILABLE candidate, proves empty inventory, and creates fresh.
  • Forged/ambiguous/unsafe/different-stage or baseline prior candidate is never deleted.
  • Seven killed request-field mutations: ChangeSetType, RollbackConfiguration, Tags, OnStackFailure, ImportExistingResources, ParentChangeSetId, RootChangeSetId, at post-create and pre-execute.
  • Existing request/Id/StackId/parameter/capability/role/notification/nested/template/change-list mutations remain killed.
  • PITR timestamps advance with stable configuration and pass; status/recovery-period mutations fail.
  • Immediate pre-execute mutations to stack identity/status, full current Processed baseline, logical/physical mapping, valid TableId/ARN, keys, billing, SSE, PITR, TTL, stream identity/view, tags, retention, prefix, candidate inventory/request/template/change list all fail before execute.
  • Late residual candidate after final retained reads fails the last empty-inventory proof.
  • Process timeout/flood/TERM-ignore/cancel/waiter/cleanup/redaction coverage.
  • End-to-end: normal deploy blocks before build at prefix zero; migration 0→1→2→3→4 and stops; normal rerun passes guard and reaches only unchanged SAM deploy.
  • Full repository workflows required by _docs/PROCESS.md.

Acceptance gates

  • Software Engineer implements only these four repair groups without commit.
  • Freeze new exact base/tree/diff/path inventory.
  • Architecture reviews exact fingerprint against the cross-workflow-versus-within-run TableId distinction and records PASS.
  • Security/IAM reviews the same fingerprint and distinction and records PASS.
  • Tester verifies every criterion/full workflow on that fingerprint and records PASS.
  • PM performs final operator acceptance on the same fingerprint.
  • Only then Software Engineer commits with Closes #136.
  • [HUMAN] Protected environment is verified before integration.
  • Orchestrator integrates/pushes only accepted commit.
  • [HUMAN] alexeygrigorev dispatches migration.
  • On-Call verifies exact prefix four/no app deploy, then reruns/monitors ordinary deploy.

Grooming authorizes no commit, integration, push, dispatch, AWS mutation, or deployment.

Out of scope

  • Custom final application deployment machinery.
  • Historical/cross-workflow TableId pinning in the ordinary deploy guard.
  • Changing application SAM behavior beyond the immediate read-only ownership/GSI guard.
  • Manual DynamoDB mutation or manual app deploy.
  • Source-repo or canonical-GSI changes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1ImportantbugSomething is brokendataData model, migration, storageinfraDeployment and infrastructuremigrationImport or migration worktestingTests and QA

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions