[CI] Sonoma corrupts mac-test-package.7z during test expansion

[dalexsoto]

Failure signature

The X64 - Mac Sonoma (14) Expand tests task can read mac-test-package.7z, but extraction reports widespread ERROR: Data Error entries and exits with code 2 on XAMBOT-1049.Sonoma. Other macOS lanes expand the same build artifact successfully.

Stable signature:

X64 Mac Sonoma Expand tests extracts mac-test-package.7z with widespread ERROR: Data Error entries and exits 2 on XAMBOT-1049 while sibling workers expand the same artifact

Confirmed occurrences

PR #26481

• Head: af51a02611066b9bd44db7a6ed8f1f1fc8069e70
• Build 15181206, Sonoma / Expand tests
• Worker: XAMBOT-1049.Sonoma
• Extraction produced 1,627 data errors and exited 2.
• Tahoe and Sequoia expanded the same artifact successfully. A targeted Sonoma retry on XAMBOT-1006.Sonoma succeeded, and the PR later merged.

PR #26535

• Head: 3d2e0c0665e69b90d74f2b5d9a186d9b475a283f
• Build 15183044, Sonoma / Expand tests
• Worker: XAMBOT-1049.Sonoma
• Extraction produced 1,352 data errors and exited 2.
• VSM-XAM-47.Tahoe.arm64 and VSM-XAM-75.Sequoia.arm64 expanded the same artifact successfully in the same build.

First observed: 2026-09-02T07:06:24Z.
Latest confirmed occurrence: 2026-09-02T12:30:22Z.
Recurrence: 2 jobs across 2 distinct monitored PRs.

Classification and impact

This is a worker-local CI artifact transfer/storage corruption, not a source or deterministic test failure. The affected PRs are unrelated, the archive remains readable enough for 7-Zip to enumerate entries, and sibling workers successfully expand the same pipeline artifact.

The corruption prevents Sonoma tests from starting and fails the entire stage.

Investigation performed

• Inspected Azure timelines, task logs, worker assignments, and 7-Zip summaries.
• Counted the retained ERROR: Data Error records for each occurrence.
• Compared extraction results across Sonoma, Tahoe, and Sequoia within the same builds.
• Verified the successful [msbuild] Embed on-demand resource asset packs in the app bundle for the simulator. Fixes #26218 #26481 Sonoma retry on a different worker.
• Searched all open and closed dotnet/macios issues using the archive name, exact data-error text, task name, worker, and corruption terms. No matching issue existed.

Workaround

After the full run is terminal and retry gates permit it, rerun the failed Sonoma stage so the artifact is transferred and expanded on a healthy worker.

Azure artifacts and task logs are retention-limited.

[dalexsoto]

Active retry status for PR #26535 @ 3d2e0c0665e69b90d74f2b5d9a186d9b475a283f:

• After refreshing the unchanged conflict-free head, terminal run, zero prior retries, and equivalent-attempt state, Azure accepted build 15183044 mac_14_x64 attempt 2.
• The Sonoma macOS tests job is active on XAMBOT-1006.Sonoma, rerunning the failed archive-expansion path.
• A separate request reruns the unrelated assembly-processing timeout tracked by [CI] assembly-processing times out at 600 seconds after all NUnit tests pass #26528.
• Retry usage is 2 PR-lifetime / 2 current-head requests across both accepted stage requests.

No equivalent retry will be submitted while attempt 2 is active. The issue body has no Additional info to collect checklist.

[dalexsoto]

Terminal retry outcome for PR #26535 @ 3d2e0c0665e69b90d74f2b5d9a186d9b475a283f:

• Build 15183044, mac_14_x64 attempt 2 succeeded on XAMBOT-1006.Sonoma at 2026-09-02T14:51:04Z.
• The mac-test-package.7z archive expanded successfully and the worker-local data corruption did not recur.
• The separate assembly-processing retry remains active. Retry usage remains 2 PR-lifetime / 2 current-head requests.

The canonical count remains 2 occurrences across 2 PRs. The issue body has no Additional info to collect checklist.