Handle bulk data access 2.0

[apurvabhaleMS]

Description

This PR improves Bulk Export (Bulk Data Access 2.0) cancellation behavior by introducing a CancelRequested flag on the orchestrator/group export job. This enables consistent cancel semantics even when the orchestrator job has already Completed or Failed, and aligns job retrieval behavior when cancellation is requested.

Key changes

• Core

  • Added CancelRequested to ExportJobRecord and JobRecordProperties.
  • Updated CancelExportRequestHandler to:
    • Return 404 (JobNotFound) when canceling an already-canceled export job.
    • For Completed/Failed export jobs: check the group/orchestrator job’s CancelRequested and return 404 if already requested; otherwise return 202 Accepted without changing job status to Canceled.
    • For Queued/Running export jobs: continue to set status to Canceled and return 202 Accepted.
  • Updated FhirOperationDataStoreBase to treat JobStatus.Cancelled and group/orchestrator CancelRequested as JobNotFound when reading export jobs, and to propagate Id/GroupId consistently.
  • Updated UpdateExportJobAsync to cancel by group id and throw JobNotFound when the job does not exist.

• Cosmos DB

  • Updated CosmosQueueClient.CancelJobByGroupIdAsync to set CancelRequested = true for the export orchestrator job even if it is already Completed/Failed.

• SQL Server

  • Updated PutJobCancelation sproc to set CancelRequested = 1 for the export orchestrator job (JobId == GroupId) when status is Completed/Failed.
  • Added schema migration V104 and updated schema version constants / csproj latest schema version.

Tests

• Added/updated unit tests for CancelExportRequestHandler covering:
  • Unauthorized user
  • Already canceled jobs
  • Completed/failed jobs with/without group CancelRequested
  • JobId vs GroupId fetching behavior
• Added/updated integration tests for:
  • Export job read/update behavior around cancelled / cancel-requested jobs
  • Queue client behavior for export orchestrator cancellation

Impact / Notes

• Behavioral change: Canceling an already canceled export job now results in 404 (JobNotFound).
• Schema: SQL schema bumped to V104 (SQL scripts included).
• PR is currently Draft; CI status to be validated before merge.

Related issues

Addresses 179823.

Testing

Describe how this change was tested.

FHIR Team Checklist

• Update the title of the PR to be succinct and less than 65 characters
• Add a milestone to the PR for the sprint that it is merged (i.e. add S47)
• Tag the PR with the type of update: Bug, Build, Dependencies, Enhancement, New-Feature or Documentation
• Tag the PR with Open source, Azure API for FHIR (CosmosDB or common code) or Azure Healthcare APIs (SQL or common code) to specify where this change is intended to be released.
• Tag the PR with Schema Version backward compatible or Schema Version backward incompatible or Schema Version unchanged if this adds or updates Sql script which is/is not backward compatible with the code.
• When changing or adding behavior, if your code modifies the system design or changes design assumptions, please create and include an ADR.
• CI is green before merge
• Review squash-merge requirements

Semver Change (docs)

Patch|Skip|Feature|Breaking (reason)

[Copilot]

Pull request overview

This PR updates Bulk Export cancellation behavior across SQL and Cosmos queue implementations, including schema changes and expanded test coverage, to better handle “cancel requested” semantics for export orchestrator jobs that have already completed or failed.

Changes:

• Adds logic to mark Export orchestrator jobs as CancelRequested even when already Completed/Failed (SQL sproc + Cosmos queue client).
• Updates export job status retrieval/cancellation flows to treat canceled/cancel-requested jobs as “not found” in more cases.
• Introduces SQL schema version 104 and adds/updates integration + unit tests covering these scenarios.

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
test/Microsoft.Health.Fhir.Shared.Tests.Integration/Persistence/QueueClientTests.cs	Adds integration tests validating CancelRequested behavior for completed/failed export orchestrator jobs.
test/Microsoft.Health.Fhir.Shared.Tests.Integration/Features/Operations/FhirOperationDataStoreExportTests.cs	Adds extensive integration test coverage for export job retrieval and cancellation-related outcomes.
src/Microsoft.Health.Fhir.SqlServer/Microsoft.Health.Fhir.SqlServer.csproj	Bumps SQL LatestSchemaVersion to 104.
src/Microsoft.Health.Fhir.SqlServer/Features/Schema/Sql/Sprocs/PutJobCancelation.sql	Updates cancellation sproc logic to set CancelRequested for completed/failed export orchestrator jobs.
src/Microsoft.Health.Fhir.SqlServer/Features/Schema/SchemaVersionConstants.cs	Updates max schema version constant to V104.
src/Microsoft.Health.Fhir.SqlServer/Features/Schema/SchemaVersion.cs	Adds schema enum value V104.
src/Microsoft.Health.Fhir.SqlServer/Features/Schema/Migrations/104.diff.sql	Adds migration for updated PutJobCancelation sproc (but needs adjustment).
src/Microsoft.Health.Fhir.CosmosDb/Features/Storage/Queues/CosmosQueueClient.cs	Mirrors export orchestrator CancelRequested behavior for completed/failed jobs in Cosmos.
src/Microsoft.Health.Fhir.Core/Features/Operations/JobRecordProperties.cs	Adds CancelRequested job-record property name constant.
src/Microsoft.Health.Fhir.Core/Features/Operations/FhirOperationDataStoreBase.cs	Adjusts export job retrieval to incorporate group cancellation semantics and propagate group id into outcomes.
src/Microsoft.Health.Fhir.Core/Features/Operations/Export/Models/ExportJobRecord.cs	Adds CancelRequested field and loosens setter for GroupId for internal population.
src/Microsoft.Health.Fhir.Core/Features/Operations/Export/CancelExportRequestHandler.cs	Updates cancel handler behavior for canceled/completed/failed export jobs.
src/Microsoft.Health.Fhir.Core.UnitTests/Features/Operations/Export/CancelExportRequestHandlerTests.cs	Updates and expands unit tests for updated cancel/export behavior.

In general, to fix a “missed ternary opportunity” where both branches of an if assign the same variable, we replace the if/else with a single assignment that uses the conditional (?:) operator to select between the two values.

Here, both branches assign record.FailureDetails:

• If record.FailureDetails == null, assign new JobFailureDetails(Core.Resources.ProcessingJobHadNoResults, HttpStatusCode.InternalServerError).
• Else, assign new JobFailureDetails(record.FailureDetails.FailureReason + "\r\n" + Core.Resources.ProcessingJobHadNoResults, record.FailureDetails.FailureStatusCode).

We can replace the whole if/else with:

record.FailureDetails = record.FailureDetails == null
    ? new JobFailureDetails(Core.Resources.ProcessingJobHadNoResults, HttpStatusCode.InternalServerError)
    : new JobFailureDetails(record.FailureDetails.FailureReason + "\r\n" + Core.Resources.ProcessingJobHadNoResults, record.FailureDetails.FailureStatusCode);

This preserves all behavior: the same condition is checked and the same constructor arguments are used; we only change syntax. No new imports, methods, or definitions are required. The change is confined to the else branch inside the foreach loop in FhirOperationDataStoreBase.cs around original lines 132–139.