Skip to content

CI flakiness inventory: Nov 4-9, 2026 (follow-up to #13278) #13429

Open

Description

@simonrozsival

Android framework version

Other: net12.0-android on main; net11.0-android on release/11.0.1xx. The pipeline also exercises previous-SDK application builds and net10.0 host-side test assemblies.

Affected platform version

dotnet/android public Azure DevOps dotnet-android pipeline: dnceng-public, project public, definition 333, macOS/Linux/Windows.

Primary snapshot: PR builds queued 2026-11-04T00:00:00Z through 2026-11-09T14:37:24Z, enumerated at 2026-11-09T14:37:53Z. Results are frozen at collection; later retries/completions can change the live pages.

Description

Follow-up to #13278, the October 16-27 inventory opened October 27. The earlier chain is #13168, #12929 and #12704. This issue supersedes #13278 as the current tracking snapshot. Closing the predecessor does not resolve its remaining test, runtime or infrastructure problems.

The Unix .NET-home permission repairs are holding in observed fixed-source CI. The same emulator archive stall, Windows project-move sharing violation and unchanged-AAB timing assertion still recur. The upload/preferences memory-pressure tests continue to pass, but two other process crashes remain, so "all device crashes are fixed" would be incorrect.

This is a bounded post-merge verification, supplemented by the intervening October 27-November 3 analysis, not a claim that every independent root of every red build has been exhaustively diagnosed.

Population and reliability trend

The primary snapshot contains 72 retained PR builds: 67 completed, 44 succeeded and 23 failed; four in progress and one cancelling are excluded from completed-build rates. A separate deleted-record query found 31 additional records, all canceled.

The completed builds published 2,683 test runs, 575,152 test executions and 134 failed-result entries across 15 builds. Earlier stage attempts are included, including incidents in builds that eventually turned green. Collection completed with zero outstanding errors; retry-timeline references and per-run failed-result counts were reconciled.

Metric Oct 16-27 inventory Oct 27-29 bridge Oct 30-Nov 3 interim Nov 4-9
Completed non-canceled PR builds 104 54 70 67
Final red builds, all causes 47 (45.19%) 14 (25.93%) 16 (22.86%) 23 (34.33%)
Builds with stage retries 33 (31.73%) 11 (20.37%) 14 (20.00%) 17 (25.37%)
Builds with explicit test-process crash signature 36 (34.62%) 0 1 (1.43%) 2 (2.99%)

These are descriptive build-level rates, not a regression-excluded flakiness ratio. The crash metric uses the same explicit The test process crashed: signature across snapshots; a crash is not automatically a flaky test or one shared root cause. The old inventory's confirmed-incident catalog was not exhaustively reclassified for this bounded follow-up, so no comparable current catalog-union flakiness percentage is claimed.

Final-red and retry rates rose relative to the interim snapshot, but remain below the October inventory. Candidate PR failures, release/main and runtime mix, the .NET transition, reduced duplicate work and changed test coverage prevent treating that movement as a clean causal measure of infrastructure reliability. The large drop in explicit crashes is encouraging; source-aware test execution below is stronger evidence for the specific memory repairs.

Fixes with positive execution evidence

Change Actual tested-source exposure and assessment
#13376, move Unix bootstrap SDK installation into agent temporary storage; #13397, isolate DOTNET_CLI_HOME without changing real HOME No original .dotnet-home permission failure among 36 completed sources containing #13397. Their macOS tasks include 806 successful bootstrap-install attempts across 36 builds and 770 successful build-server shutdown attempts across 33 builds, with zero failures of those operations. Attempts include configurations/retries, not independent builds.
#13110, bounded upload-test copying; #13310, SharedPreferences test pressure repair Both merges occur in 63 completed build sources, including 57 device-exercising builds. PutAndGetManyValues and RepeatedEditorsDoNotLeakReferences each report 230 passes; the two upload-reset completion-option cases each report 228 passes, with no observed failures of those four targets in inspected fixed-source suite logs. These are reported test events, not internal repeat iterations or independent builds.
#13265, remove the unsuitable SDK-resolution timing guard The unreliable correctness gate was removed while correctness/output assertions remained. The intervening verification found positive correctness execution; this is not evidence that SDK resolution became faster.

The two observed first-run sentinel denials, 1663841 and 1663916, both contain #13376 but lack #13397. The SDK-location change alone did not isolate first-run state; these failures do not show the combined fix failing.

Positive example 1670836: bootstrap log 63 installs into the agent-temp dotnet-bootstrap root, and log 275 executes a successful build-server shutdown using that SDK. These logs do not independently print DOTNET_CLI_HOME; no unobserved value is inferred.

Disposition: do not duplicate the landed permission implementation. Reopen that exact problem only with a matching fixed-source recurrence.

Persistent problems: next investigation order

Priority Problem Fresh evidence and boundary
1 Emulator archive operation stalls until job cap 1665928, initial macOS MSBuild+Emulator8 attempt: last substantive progress is the 488,731,099-byte emulator-darwin_x64-16428233.zip download at Nov 5 16:21:54 UTC, then silence until cancellation at 19:15:34 UTC under the 180-minute job cap. Same last-logged archive as six October 27-29 capped builds. Source contains both permission fixes and the lane-reduction fix; later stage retry makes the final build green. The last log does not establish which network/stream/install operation blocked. Keep the existing cap and integrity checks while identifying the blocking operation.
2 Windows project-move sharing violation 1668977, queued Nov 7: BuildBasicApplicationThenMoveIt((True, CoreCLR)), run 46497074, result 100237, again fails with a sharing violation. Same family as earlier build 1657572; current source includes the setup/memory fixes. Identify the locked object and Windows handle owner before changing process lifetime.
3 Unchanged AAB incremental-install timing assertion IncrementalFastDeployment("aab") fails in 1665928 and 1670795, the latter queued Nov 9. Unchanged 15.96s versus incremental 15.78s, and unchanged 24.76s versus incremental 24.37s. These are measured Build,Install durations, not proof another package install occurred. Attribute phases before changing the assertion.
Separate bounded work Bootstrap-script DNS failure 1665740, log 796: curl: (6) Could not resolve host: builds.dotnet.microsoft.com while installing the SDK into bin/Release/dotnet. Retry recovers. This is the existing bootstrap-fetch failure surface, not a permission failure or proof of the emulator stall's cause.

The other capped build, 1666161, belongs to the callback-test hang/crash below. It must not be added to the emulator-download-stall count.

Remaining crashes: do not reopen the upload-memory diagnosis indiscriminately

Both current crash build sources contain #13110 and #13310:

Build / candidate context Observed outcome Interpretation
1666161, testing #13312 NativeAOT task log 856 reports a process crash while HandlerCancellationDoesNotScaleQuadratically(True) is active. Its Debug suite becomes silent after the False variant and reaches the job cap. A later build of the same PR, 1667641, publishes 26 failed ActionCallback result entries in a NullReferenceException family. Route the new callback-test failures and hang/crash evidence to #13312. Do not count this as an established recurrence of upload OOM. The last active test is a lead, not causal proof.
1665601, testing JNI-layout #12984 Debug task log 856 reports a process crash while PingLocalhost is active. Later stage retry recovers the build. Root remains unknown. Neither the last active method nor the candidate's JNI changes alone prove the cause.

Finalization and Gradle: exact signatures and source matter

  • The original Java.InteropTests.JavaObjectTest.Dispose_Finalized assertion really recurs in 1665919, run 46404854, result 100640. Debug log 1056 reports the expected Dispose(disposing:false) callback missing; later Debug log 2250 passes, and the final build is green. However, this is [release/11.0.1xx][TrimmableTypeMap] Validate generated top-level Java identifiers #13405 targeting release/11.0.1xx, whose source lacks the main setup/memory fixes. Keep the old CoreCLR test failure: Java.InteropTests.JavaObjectTest.Dispose_Finalized #10358 finalization work unresolved; do not describe this as a new main regression. This is the original exact test, unlike the interim snapshot's different registered-peer finalizer failure.
  • AGP 9.1.1 plugin resolution recurs in 1663707: the attached GradleFBProj((True, NativeAOT)) build log cannot resolve com.android.application:9.1.1. Its source lacks [tests] Enable GradleFBProj resolution diagnostics #13336, even though execution happens after that PR's merge time. No failed GradleFBProj result was observed among 48 completed sources containing [tests] Enable GradleFBProj resolution diagnostics #13336, but that PR adds diagnostics, not a resolver fix. The historical corrupt-ZIP incident was not reproduced in the bounded local probe; neither recurrence nor eradication is established here.
  • New source-build Maven inputs are distinct: 1667014, log 102, fails manifest-merger:32.5.1 dependency resolution with a mirror HTTP 401 explicitly requiring authenticated upstream package saving; 1667015, log 84, cannot resolve the AGP 9.5.1 marker. Investigate feed/package availability separately from the AGP 9.1.1 test failure and unidentified ZIP corruption.

Hosted-agent and checkout incidents

Five completed builds report lost agent contact: 1667560, 1665718, 1664036, 1663708, and 1663707.

Two report checkout failure: 1665714 and 1664036. These counts overlap; a checkout task interrupted by agent loss is not automatically a separate Git transport root.

Existing provider/pool-owner escalation remains appropriate for resolver/egress and agent-contact evidence; cross-.NET transport history includes dotnet/arcade#9009. Prior public GitHub-incident overlap was sparse and did not establish a stable outage effect or a shared root for these failures. No provider escalation or CI retry was executed as part of this verification.

Coverage and rollout checks

Carry-forward and follow-up

The predecessor's GREF/collection, finalization, PowerShell procargs, MAUI/device-lab, corrupt-ZIP and other unresolved entries are not closed by absence from this bounded report. In particular, preserve #12031/#13010, #13275, #10358, #12652, #13088 and #12658; this issue makes no current live-state or fixed-source resolution claim for trackers not separately reverified.

  • Diagnose the repeated emulator archive operation stall without increasing the job cap or adding a blanket retry layer.
  • Identify the Windows lock holder in the project-move failure.
  • Attribute unchanged-AAB Build,Install timing before changing thresholds or deployment behavior.
  • Monitor [build] net12.0 target framework #13376/[ci] Isolate Unix .NET CLI first-run state #13397; only resume duplicate permission work on a matching fixed-source recurrence.
  • Keep the targeted upload/preferences fixes; triage the callback and PingLocalhost crashes separately.
  • Assess bootstrap DNS mitigation and provider/pool-owner transport/contact-loss evidence independently.
  • Keep release finalization and other predecessor reference-lifetime assertions unresolved; preserve real correctness checks.
  • Separate Gradle plugin/feed failures from archive corruption and verify diagnostic coverage on actual source.
  • Verify [Microsoft.Android.Run] Report partial results after interruption #13313 after a source containing its merge actually exercises a crash; continue coverage-aware reporting.

Steps to Reproduce

  1. Enumerate public pipeline definition 333 PR-source builds from https://dev.azure.com/dnceng-public/public/_apis/build/builds with the UTC queue bounds above, all continuation pages, and a separate deletedFilter=includeDeleted query. Preserve initial build states rather than mixing later completions into the frozen denominator.
  2. Read each build's timeline and recursively retain all referenced previous-attempt timelines. Deduplicate attempts by record identity/start/attempt, preferring completed records. Keep recovered green-build incidents.
  3. Collect run summaries and per-run failed results with $skip, repeated-page detection and count reconciliation. Do not trust ResultsByBuild pagination; it can repeat the first page. Count explicit crash task logs separately because crashed suites may not publish failed test rows.
  4. Resolve task/job/configuration ancestry; inspect failed logs, attached Gradle logs and positive setup/suite output. A skipped task with a timestamp/log is not executed coverage. Strip terminal escapes before parsing reported test events.
  5. Compare each relevant merge SHA with the actual build sourceVersion using GitHub compare; only ahead/identical establishes merge inclusion. Queue/execution time after merge is insufficient. Check release-targeted sources separately.
  6. Count incidents once per completed build, keep category overlap explicit, and distinguish operation/configuration/retry counts from independent builds. Compare like-defined metrics across snapshots; do not turn all red builds or failed-result entries into a flaky-build rate.
  7. Check actual lane starts and restored previous-SDK result outcomes before interpreting coverage reductions or zero failures as fixes.

Did you find any workaround?

Use source containing both #13376 and #13397 for the old Unix home-permission failure. Keep the bounded upload-copy and preferences test repairs in #13110/#13310. Raising Java heap limits is not an established general correction.

A targeted retry may unblock a verified transient incident, as several examples did here, but it does not fix a stall, lock or runtime error. Do not blanket-retry PR regressions, unknown crashes or every test failure, raise caps, relax assertions, or disable meaningful coverage to improve headline rates.

No product changes, new retry framework, CI reruns, toolchain/device writes or child-session assignments were made during this read-only verification.

Relevant log output

Permission failures on source containing #13376 but not #13397, builds 1663841/1663916:

System.UnauthorizedAccessException: Access to the path '/Users/cloudtest/.dotnet/11.0.100-rc.3.26522.148.toolpath.sentinel' is denied.

Bootstrap fetch, build 1665740/log 796:

curl: (6) Could not resolve host: builds.dotnet.microsoft.com

For exact task/result evidence, use the linked build pages and the run/result/log identifiers above. The full frozen raw logs, source-ancestry comparisons, SQL analysis and SHA-256 manifest are preserved with this investigation; GitHub does not expose those session-local files, so they are not presented as public attachments.

Activity

  1. added
    needs-triageIssues that need to be assigned.
    flaky-ciIntermittent CI failures and work to improve CI reliability
    on Nov 9, 2026
  2. simonrozsival commented on Nov 13, 2026

    @simonrozsival
    MemberAuthor

    Flaky recurrence: Java.InteropTests.JavaObjectTest.Dispose_Finalized (exact #10358 signature)

  3. simonrozsival commented on Nov 13, 2026

    @simonrozsival
    MemberAuthor

    Flaky recurrence: Xamarin.Android.Build.Tests.InstallAndRunTests.GradleFBProj((True, CoreCLR)): Gradle corrupt ZIP

  4. simonrozsival commented on Nov 13, 2026

    @simonrozsival
    MemberAuthor

    Flaky recurrence: macOS PowerShell startup Call to 'procargs' failed with errno 5 (tracked in #12652)

  5. simonrozsival commented on Nov 13, 2026

    @simonrozsival
    MemberAuthor

    Flaky recurrence: hosted-agent disconnect (We stopped hearing from agent)

  6. simonrozsival commented on Nov 13, 2026

    @simonrozsival
    MemberAuthor

    Flaky recurrence: macOS PowerShell procargs failure (Determine if we need to rerun failed tests)

    Evidence: Tests ran and results were published. No failed test results were found. The PowerShell rerun-decision task failed with a procargs/errno 5 host error, the known macOS signature (#12652). This was the only failed job in the run.

  7. simonrozsival commented on Nov 13, 2026

    @simonrozsival
    MemberAuthor

    Flaky recurrence: checkout git fetch failure (exit 128)

    Evidence: Checkout failed before any PR code ran. The log shows RPC failed; curl 56, early EOF, Could not resolve host: github.com and git fetch exit code 128. The later upload test results failure was a consequence of the checkout failure. This was the only failed job in the run. The signature matches the known checkout/git transport family.

  8. simonrozsival commented on Nov 14, 2026

    @simonrozsival
    MemberAuthor

    Flaky CI report: Java.InteropTests.JavaObjectTest.Dispose_Finalized (NativeAOT): Expected JavaDisposedObject.Dispose(disposing: false) to run. Expected: True But was: False

    Test run 46837024 (Mono.Android.NET_Tests-NativeAOT), result 100855, stage retry attempt 2 of macOS > Tests > APKs 2. This is the same signature as the attempt-1 report above, and the known finalization flake (#10358). The PR changes Handler/View action posting, not JNI finalization.

    Automated triage; no retry was requested (job-scoped retry could not be verified).

  9. simonrozsival commented on Nov 14, 2026

    @simonrozsival
    MemberAuthor

    Flaky CI report: Java.InteropTests.JavaObjectTest.Dispose_Finalized (NativeAOT): Expected JavaDisposedObject.Dispose(disposing: false) to run. Expected: True But was: False

    Test run 46858018 (Mono.Android.NET_Tests-NativeAOT), result 100815, stage retry attempt 3 of macOS > Tests > APKs 2. This is the third consecutive attempt with the same signature. It may be strongly reproducible on NativeAOT for this head.

    Automated triage; no retry was requested (job-scoped retry could not be verified).

  10. simonrozsival commented on Nov 14, 2026

    @simonrozsival
    MemberAuthor

    Flaky CI report: Java.InteropTests.JavaObjectTest.UnreferencedInstanceIsCollected (NativeAOT): Expected the unreferenced instance to be collected. Expected: True But was: False

    Test run 46858018, result 100829, attempt 3 of macOS > Tests > APKs 2. This matches the tracked flake #13275. It did not fail in attempts 1 or 2.

    Automated triage; no retry was requested (job-scoped retry could not be verified).

  11. simonrozsival commented on Nov 14, 2026

    @simonrozsival
    MemberAuthor

    Flaky CI report: Java.InteropTests.JavaObjectTest.Dispose_Finalized (Mono.Android.NET_Tests-Debug): Expected JavaDisposedObject.Dispose(disposing: false) to run. Expected: True But was: False

    Result 100673, macOS > Tests > APKs 1. This PR targets release/11.0.1xx and has the same signature as the release-branch recurrence already listed in this inventory (#10358). The PR changes R8 rules and typemap targets, not finalization.

    Note: this build also has PR- or branch-related NativeAOT arm build failures. Those are not reported here as flaky.

    Automated triage; no retry was requested (job-scoped retry could not be verified).

  12. 1 remaining item

  13. simonrozsival commented on Nov 14, 2026

    @simonrozsival
    MemberAuthor

    Flaky: PerformanceTest.Build_XAML_Change(True) timing threshold (MSBuildDeviceIntegration, macOS)

  14. simonrozsival commented on Nov 14, 2026

    @simonrozsival
    MemberAuthor

    Flaky: PerformanceTest.Build_XAML_Change(False) timing threshold (MSBuildDeviceIntegration, macOS)

  15. simonrozsival commented on Nov 14, 2026

    @simonrozsival
    MemberAuthor

    Known infra: emulator archive download stall → 180-minute job cap (macOS MSBuild+Emulator)

  16. simonrozsival commented on Nov 14, 2026

    @simonrozsival
    MemberAuthor

    Flaky: Xamarin.Android.Build.Tests.PerformanceTest.Build_XAML_Change(False) exceeded expected time

    Evidence: this is the only failed job/test in the build. The same timing-threshold signature has failed on unrelated PRs (#13434 build 1679890 at 4221ms, #13550 build 1680482), and it is already tracked here. Caveat: #13472 changes Microsoft.Android.Sdk.Tooling.targets (Java version validation moves from a task to targets), which is on the build path. Given the small overrun, a PR contribution cannot be fully excluded. No retry was requested because job-scoped retry semantics could not be verified.

  17. simonrozsival commented on Nov 14, 2026

    @simonrozsival
    MemberAuthor

    Flaky (recurring): Java.InteropTests.JavaObjectTest.Dispose_Finalized (NativeAOT)

    Evidence: this is the only failed job/test in the build. It is a known GC/finalizer-timing flake (see #10358), and it also hit unrelated #13558 (build 1680359). Note: this is the 4th consecutive occurrence on #13312 (build 1679688 attempts 1-3, now 1681395). #13312 changes Handler/View/Thread callback code and not finalization, but the repeated NativeAOT recurrence on this PR is worth watching. No retry was requested.

  18. simonrozsival commented on Nov 15, 2026

    @simonrozsival
    MemberAuthor

    CI infrastructure: Azure Artifacts restore failures in #13409, build 1677951, attempt 1.

    Other failures remain uncertain, including a test timeout; no retry was requested.

  19. simonrozsival commented on Nov 15, 2026

    @simonrozsival
    MemberAuthor

    CI infrastructure: Linux apt DNS failure in #13507, build 1675508, attempt 1.

    Linux Build job and install-gcc task, log 40: apt reported Temporary failure resolving for multiple Ubuntu and Launchpad hosts, then could not locate gcc-10. Linux Build attempt 2 already succeeded.

    The same DNS errors occurred independently in build 1675509, PR #13508 (log 39), and build 1675442 (log 642). The current APK failures are separate, PR-related errors; no retry was requested.

  20. simonrozsival commented on Nov 15, 2026

    @simonrozsival
    MemberAuthor

    PublishTestResults DNS failure resolving the test-results service

    Evidence:

    • nodename nor servname provided, or not known (tcmprodcus3.vstmr.visualstudio.com:443) while publishing test results.
    • Infrastructure DNS failure, not a test failure. The job passed on attempt 2 of the same build and commit badfe20af7, and the build is green.
  21. simonrozsival commented on Nov 15, 2026

    @simonrozsival
    MemberAuthor

    Checkout git submodule update fails with exit code 128 (github.com connect/DNS)

    Evidence:

    • Checkout: Failed to connect to github.com port 443 after 75003 ms, then Could not resolve host: github.com. The submodule fetch for external/android-api-docs failed, and git submodule update exited with 128.
    • Network infrastructure failure, in the same family as checkout-git-fetch-128. The job passed on attempt 2 of the same build and commit 2a4a1303aa, and the build is green.
  22. simonrozsival commented on Nov 15, 2026

    @simonrozsival
    MemberAuthor

    macOS PowerShell procargs errno 5 (run JcwGen_Tests)

    Evidence:

    • Call to 'procargs' failed with errno 5; PowerShell exited with code 'null'.
    • Known macOS agent procargs family. APKs 2 later passed on attempt 3 of the same build and commit 2a4a1303aa, and the build is green.
  23. simonrozsival commented on Nov 15, 2026

    @simonrozsival
    MemberAuthor

    DownloadPipelineArtifact DNS failure resolving dev.azure.com

    Evidence:

    • Attempt 2 task 07079253-83cb-5f56-2e9d-c4bdc45f7f76 (no log ID was recorded): nodename nor servname provided, or not known (dev.azure.com:443).
    • Infrastructure DNS failure. APKs 2 passed on attempt 3 of the same build and commit 2a4a1303aa, and the build is green.
  24. simonrozsival commented on Nov 15, 2026

    @simonrozsival
    MemberAuthor

    macOS PowerShell procargs errno 5 (set JDK variables)

    Evidence:

    • Call to 'procargs' failed with errno 5; PowerShell exited with code 'null'. The later upload test results step failed only because TestRelease did not exist.
    • Known macOS agent procargs family. The job passed on attempt 2 of the same build and commit 588edaa723, and the build is green.
  25. simonrozsival commented on Nov 15, 2026

    @simonrozsival
    MemberAuthor

    Xamarin.Android.Build.Tests.InstallTests.IncrementalFastDeployment("aab") unchanged-install timing assertion (inventory row 3)

    Evidence:

    • Test run 46819682 (buildUri=vstfs:///Build/Build/1679285, build ID verified): Second unchanged install: '00:00:23.2600000' should be faster than clean install: '00:02:38.8900000' and incremental install: '00:00:22.3700000'.
    • Same signature as inventory row 3 (unchanged vs incremental within ~1s). The job passed on attempt 2 of the same build and commit 8d70cd48b7, and the build is green.
    • PR change: removes unused FileUtil/AndroidAppManifest.WriteToFile helpers and the uname Mac probe. Nothing in it touches FastDeploy or install timing.
  26. simonrozsival commented on Nov 16, 2026

    @simonrozsival
    MemberAuthor

    Xamarin.Android.RuntimeTests.JavaMarshalGCBridgeTests.ManagedPeerCycleSurvivesJavaRootAndIsCollectedAfterRelease: Java-rooted cycle missing from bridge processing (new signature)

    Evidence:

  27. simonrozsival commented on Nov 16, 2026

    @simonrozsival
    MemberAuthor

    System.NetTests.PingTest.PingLocalhost: ping to localhost TimedOut (NativeAOT)

    Evidence:

    • Test run 46892342, Mono.Android.NET_Tests-NativeAOT (buildUri=vstfs:///Build/Build/1681205, build ID verified): Ping to localhost failed with status: TimedOut. Expected: Success But was: TimedOut.
    • The same job passed on attempt 2 of the same build and commit, and the build is green.
    • Definition 333 test history shows the identical TimedOut failure in NativeAOT on unrelated PR [build] Honor inherited analyzers and remove blanket warning waivers #13428 (build 1672500).
    • PR change: TrimmableTypeMap and JavaPeerProxy. Nothing in it touches networking.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    flaky-ciIntermittent CI failures and work to improve CI reliabilityflaky-testsneeds-triageIssues that need to be assigned.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions