Repository navigation
Description
Android framework version
Other: net12.0-android on main; net11.0-android on release/11.0.1xx. The pipeline also exercises previous-SDK application builds and net10.0 host-side test assemblies.
Affected platform version
dotnet/android public Azure DevOps dotnet-android pipeline: dnceng-public, project public, definition 333, macOS/Linux/Windows.
Primary snapshot: PR builds queued 2026-11-04T00:00:00Z through 2026-11-09T14:37:24Z, enumerated at 2026-11-09T14:37:53Z. Results are frozen at collection; later retries/completions can change the live pages.
Description
Follow-up to #13278, the October 16-27 inventory opened October 27. The earlier chain is #13168, #12929 and #12704. This issue supersedes #13278 as the current tracking snapshot. Closing the predecessor does not resolve its remaining test, runtime or infrastructure problems.
The Unix .NET-home permission repairs are holding in observed fixed-source CI. The same emulator archive stall, Windows project-move sharing violation and unchanged-AAB timing assertion still recur. The upload/preferences memory-pressure tests continue to pass, but two other process crashes remain, so "all device crashes are fixed" would be incorrect.
This is a bounded post-merge verification, supplemented by the intervening October 27-November 3 analysis, not a claim that every independent root of every red build has been exhaustively diagnosed.
Population and reliability trend
The primary snapshot contains 72 retained PR builds: 67 completed, 44 succeeded and 23 failed; four in progress and one cancelling are excluded from completed-build rates. A separate deleted-record query found 31 additional records, all canceled.
The completed builds published 2,683 test runs, 575,152 test executions and 134 failed-result entries across 15 builds. Earlier stage attempts are included, including incidents in builds that eventually turned green. Collection completed with zero outstanding errors; retry-timeline references and per-run failed-result counts were reconciled.
| Metric | Oct 16-27 inventory | Oct 27-29 bridge | Oct 30-Nov 3 interim | Nov 4-9 |
|---|---|---|---|---|
| Completed non-canceled PR builds | 104 | 54 | 70 | 67 |
| Final red builds, all causes | 47 (45.19%) | 14 (25.93%) | 16 (22.86%) | 23 (34.33%) |
| Builds with stage retries | 33 (31.73%) | 11 (20.37%) | 14 (20.00%) | 17 (25.37%) |
| Builds with explicit test-process crash signature | 36 (34.62%) | 0 | 1 (1.43%) | 2 (2.99%) |
These are descriptive build-level rates, not a regression-excluded flakiness ratio. The crash metric uses the same explicit The test process crashed: signature across snapshots; a crash is not automatically a flaky test or one shared root cause. The old inventory's confirmed-incident catalog was not exhaustively reclassified for this bounded follow-up, so no comparable current catalog-union flakiness percentage is claimed.
Final-red and retry rates rose relative to the interim snapshot, but remain below the October inventory. Candidate PR failures, release/main and runtime mix, the .NET transition, reduced duplicate work and changed test coverage prevent treating that movement as a clean causal measure of infrastructure reliability. The large drop in explicit crashes is encouraging; source-aware test execution below is stronger evidence for the specific memory repairs.
Fixes with positive execution evidence
| Change | Actual tested-source exposure and assessment |
|---|---|
#13376, move Unix bootstrap SDK installation into agent temporary storage; #13397, isolate DOTNET_CLI_HOME without changing real HOME |
No original .dotnet-home permission failure among 36 completed sources containing #13397. Their macOS tasks include 806 successful bootstrap-install attempts across 36 builds and 770 successful build-server shutdown attempts across 33 builds, with zero failures of those operations. Attempts include configurations/retries, not independent builds. |
| #13110, bounded upload-test copying; #13310, SharedPreferences test pressure repair | Both merges occur in 63 completed build sources, including 57 device-exercising builds. PutAndGetManyValues and RepeatedEditorsDoNotLeakReferences each report 230 passes; the two upload-reset completion-option cases each report 228 passes, with no observed failures of those four targets in inspected fixed-source suite logs. These are reported test events, not internal repeat iterations or independent builds. |
| #13265, remove the unsuitable SDK-resolution timing guard | The unreliable correctness gate was removed while correctness/output assertions remained. The intervening verification found positive correctness execution; this is not evidence that SDK resolution became faster. |
The two observed first-run sentinel denials, 1663841 and 1663916, both contain #13376 but lack #13397. The SDK-location change alone did not isolate first-run state; these failures do not show the combined fix failing.
Positive example 1670836: bootstrap log 63 installs into the agent-temp dotnet-bootstrap root, and log 275 executes a successful build-server shutdown using that SDK. These logs do not independently print DOTNET_CLI_HOME; no unobserved value is inferred.
Disposition: do not duplicate the landed permission implementation. Reopen that exact problem only with a matching fixed-source recurrence.
Persistent problems: next investigation order
| Priority | Problem | Fresh evidence and boundary |
|---|---|---|
| 1 | Emulator archive operation stalls until job cap | 1665928, initial macOS MSBuild+Emulator8 attempt: last substantive progress is the 488,731,099-byte emulator-darwin_x64-16428233.zip download at Nov 5 16:21:54 UTC, then silence until cancellation at 19:15:34 UTC under the 180-minute job cap. Same last-logged archive as six October 27-29 capped builds. Source contains both permission fixes and the lane-reduction fix; later stage retry makes the final build green. The last log does not establish which network/stream/install operation blocked. Keep the existing cap and integrity checks while identifying the blocking operation. |
| 2 | Windows project-move sharing violation | 1668977, queued Nov 7: BuildBasicApplicationThenMoveIt((True, CoreCLR)), run 46497074, result 100237, again fails with a sharing violation. Same family as earlier build 1657572; current source includes the setup/memory fixes. Identify the locked object and Windows handle owner before changing process lifetime. |
| 3 | Unchanged AAB incremental-install timing assertion | IncrementalFastDeployment("aab") fails in 1665928 and 1670795, the latter queued Nov 9. Unchanged 15.96s versus incremental 15.78s, and unchanged 24.76s versus incremental 24.37s. These are measured Build,Install durations, not proof another package install occurred. Attribute phases before changing the assertion. |
| Separate bounded work | Bootstrap-script DNS failure | 1665740, log 796: curl: (6) Could not resolve host: builds.dotnet.microsoft.com while installing the SDK into bin/Release/dotnet. Retry recovers. This is the existing bootstrap-fetch failure surface, not a permission failure or proof of the emulator stall's cause. |
The other capped build, 1666161, belongs to the callback-test hang/crash below. It must not be added to the emulator-download-stall count.
Remaining crashes: do not reopen the upload-memory diagnosis indiscriminately
Both current crash build sources contain #13110 and #13310:
| Build / candidate context | Observed outcome | Interpretation |
|---|---|---|
| 1666161, testing #13312 | NativeAOT task log 856 reports a process crash while HandlerCancellationDoesNotScaleQuadratically(True) is active. Its Debug suite becomes silent after the False variant and reaches the job cap. A later build of the same PR, 1667641, publishes 26 failed ActionCallback result entries in a NullReferenceException family. |
Route the new callback-test failures and hang/crash evidence to #13312. Do not count this as an established recurrence of upload OOM. The last active test is a lead, not causal proof. |
| 1665601, testing JNI-layout #12984 | Debug task log 856 reports a process crash while PingLocalhost is active. Later stage retry recovers the build. |
Root remains unknown. Neither the last active method nor the candidate's JNI changes alone prove the cause. |
Finalization and Gradle: exact signatures and source matter
- The original
Java.InteropTests.JavaObjectTest.Dispose_Finalizedassertion really recurs in 1665919, run 46404854, result 100640. Debug log 1056 reports the expectedDispose(disposing:false)callback missing; later Debug log 2250 passes, and the final build is green. However, this is [release/11.0.1xx][TrimmableTypeMap] Validate generated top-level Java identifiers #13405 targetingrelease/11.0.1xx, whose source lacks the main setup/memory fixes. Keep the old CoreCLR test failure:Java.InteropTests.JavaObjectTest.Dispose_Finalized#10358 finalization work unresolved; do not describe this as a new main regression. This is the original exact test, unlike the interim snapshot's different registered-peer finalizer failure. - AGP 9.1.1 plugin resolution recurs in 1663707: the attached
GradleFBProj((True, NativeAOT))build log cannot resolvecom.android.application:9.1.1. Its source lacks [tests] Enable GradleFBProj resolution diagnostics #13336, even though execution happens after that PR's merge time. No failedGradleFBProjresult was observed among 48 completed sources containing [tests] Enable GradleFBProj resolution diagnostics #13336, but that PR adds diagnostics, not a resolver fix. The historical corrupt-ZIP incident was not reproduced in the bounded local probe; neither recurrence nor eradication is established here. - New source-build Maven inputs are distinct: 1667014, log 102, fails
manifest-merger:32.5.1dependency resolution with a mirror HTTP 401 explicitly requiring authenticated upstream package saving; 1667015, log 84, cannot resolve the AGP 9.5.1 marker. Investigate feed/package availability separately from the AGP 9.1.1 test failure and unidentified ZIP corruption.
Hosted-agent and checkout incidents
Five completed builds report lost agent contact: 1667560, 1665718, 1664036, 1663708, and 1663707.
Two report checkout failure: 1665714 and 1664036. These counts overlap; a checkout task interrupted by agent loss is not automatically a separate Git transport root.
Existing provider/pool-owner escalation remains appropriate for resolver/egress and agent-contact evidence; cross-.NET transport history includes dotnet/arcade#9009. Prior public GitHub-incident overlap was sparse and did not establish a stable outage effect or a shared root for these failures. No provider escalation or CI retry was executed as part of this verification.
Coverage and rollout checks
- [tests] Remove duplicate integration builds and one CI lane #13390: 34 completed build sources contain the reduction from 12 to 11 emulator lanes and removal of duplicate builds. 31 actually start 11 emulator jobs each; three never reach them. Reduced work is not by itself proof of improved reliability or faster builds.
- [tests] Restore previous Android SDK device coverage on .NET 12 #13406: seven completed sources contain restored previous-SDK coverage. In successful 1670836, all three restored
DotNetInstallAndRunPreviousSdkcases actually pass: CoreCLR Debug (run 46555534/result 100015), CoreCLR Release (46555728/100005), NativeAOT Release (46555794/100019). Thirteen integration test runs, including two auto-retry runs, were reconciled to their total-result counts. - [Microsoft.Android.Run] Report partial results after interruption #13313: merged Nov 9 at 14:34:06 UTC, but zero sampled build sources contain its merge. The partial-crash-reporting change cannot yet receive a post-merge CI effectiveness verdict.
- Only four latest-attempt MAUI Helix tasks actually execute. That does not establish restored main coverage or a healthier device lab. Preserve Restore MAUI Integration lane when dotnet/maui net12.0 branch exists #13088/CI: MAUI R2R Helix device-lab failures across Android queues #12658 until coverage and relevant device outcomes justify a change.
Carry-forward and follow-up
The predecessor's GREF/collection, finalization, PowerShell procargs, MAUI/device-lab, corrupt-ZIP and other unresolved entries are not closed by absence from this bounded report. In particular, preserve #12031/#13010, #13275, #10358, #12652, #13088 and #12658; this issue makes no current live-state or fixed-source resolution claim for trackers not separately reverified.
- Diagnose the repeated emulator archive operation stall without increasing the job cap or adding a blanket retry layer.
- Identify the Windows lock holder in the project-move failure.
- Attribute unchanged-AAB
Build,Installtiming before changing thresholds or deployment behavior. - Monitor [build]
net12.0target framework #13376/[ci] Isolate Unix .NET CLI first-run state #13397; only resume duplicate permission work on a matching fixed-source recurrence. - Keep the targeted upload/preferences fixes; triage the callback and
PingLocalhostcrashes separately. - Assess bootstrap DNS mitigation and provider/pool-owner transport/contact-loss evidence independently.
- Keep release finalization and other predecessor reference-lifetime assertions unresolved; preserve real correctness checks.
- Separate Gradle plugin/feed failures from archive corruption and verify diagnostic coverage on actual source.
- Verify [Microsoft.Android.Run] Report partial results after interruption #13313 after a source containing its merge actually exercises a crash; continue coverage-aware reporting.
Steps to Reproduce
- Enumerate public pipeline definition 333 PR-source builds from
https://dev.azure.com/dnceng-public/public/_apis/build/buildswith the UTC queue bounds above, all continuation pages, and a separatedeletedFilter=includeDeletedquery. Preserve initial build states rather than mixing later completions into the frozen denominator. - Read each build's timeline and recursively retain all referenced previous-attempt timelines. Deduplicate attempts by record identity/start/attempt, preferring completed records. Keep recovered green-build incidents.
- Collect run summaries and per-run failed results with
$skip, repeated-page detection and count reconciliation. Do not trustResultsByBuildpagination; it can repeat the first page. Count explicit crash task logs separately because crashed suites may not publish failed test rows. - Resolve task/job/configuration ancestry; inspect failed logs, attached Gradle logs and positive setup/suite output. A skipped task with a timestamp/log is not executed coverage. Strip terminal escapes before parsing reported test events.
- Compare each relevant merge SHA with the actual build
sourceVersionusing GitHub compare; onlyahead/identicalestablishes merge inclusion. Queue/execution time after merge is insufficient. Check release-targeted sources separately. - Count incidents once per completed build, keep category overlap explicit, and distinguish operation/configuration/retry counts from independent builds. Compare like-defined metrics across snapshots; do not turn all red builds or failed-result entries into a flaky-build rate.
- Check actual lane starts and restored previous-SDK result outcomes before interpreting coverage reductions or zero failures as fixes.
Did you find any workaround?
Use source containing both #13376 and #13397 for the old Unix home-permission failure. Keep the bounded upload-copy and preferences test repairs in #13110/#13310. Raising Java heap limits is not an established general correction.
A targeted retry may unblock a verified transient incident, as several examples did here, but it does not fix a stall, lock or runtime error. Do not blanket-retry PR regressions, unknown crashes or every test failure, raise caps, relax assertions, or disable meaningful coverage to improve headline rates.
No product changes, new retry framework, CI reruns, toolchain/device writes or child-session assignments were made during this read-only verification.
Relevant log output
Permission failures on source containing #13376 but not #13397, builds 1663841/1663916:
System.UnauthorizedAccessException: Access to the path '/Users/cloudtest/.dotnet/11.0.100-rc.3.26522.148.toolpath.sentinel' is denied.
Bootstrap fetch, build 1665740/log 796:
curl: (6) Could not resolve host: builds.dotnet.microsoft.com
For exact task/result evidence, use the linked build pages and the run/result/log identifiers above. The full frozen raw logs, source-ancestry comparisons, SQL analysis and SHA-256 manifest are preserved with this investigation; GitHub does not expose those session-local files, so they are not presented as public attachments.
Activity
[Xamarin.Android.Build.Tasks] Generate trimmable typemap after Compile
[tests] Assert incremental deployment work instead of elapsed ordering
simonrozsival commentedon Nov 13, 2026
Flaky recurrence:
Java.InteropTests.JavaObjectTest.Dispose_Finalized(exact #10358 signature)3b72002d)smoke_tests(Package Tests), attempt 1Mono.Android.NET_Tests-NativeAOT): 1 of 1007 failed:Expected JavaDisposedObject.Dispose(disposing: false) to run. Expected: True But was: FalseJava.InteropTests.JavaObjectTest.Dispose_Finalized#10358 and prior inventories (CI flakiness inventory: Oct 2-16, 2026 #13168, CI flakiness inventory: Oct 16-27, 2026 (follow-up to #13168) #13278, CI flakiness inventory: Nov 4-9, 2026 (follow-up to #13278) #13429), which record same-configuration recoveries. The PR changes Handler/View/Thread Action→Runnable mapping, not JavaObject finalization. Not retried, because no job-scoped retry mechanism was verified.simonrozsival commentedon Nov 13, 2026
Flaky recurrence:
Xamarin.Android.Build.Tests.InstallAndRunTests.GradleFBProj((True, CoreCLR)): Gradle corrupt ZIP3b72002d)msbuilddevice_tests(MSBuild Emulator Tests), attempt 1build.log:gradlew Library:assembleRelease→A problem occurred configuring project ':Library'. > java.util.zip.ZipException: zip END header not found(from KotlinClasspathEntrySnapshotter/ Gradle build-script cache), thenMSB6006: "gradlew" exited with code 1simonrozsival commentedon Nov 13, 2026
Flaky recurrence: macOS PowerShell startup
Call to 'procargs' failed with errno 5(tracked in #12652)5eb2fb72)macOS > Tests > MSBuild+Emulator 11, task "Slice unit tests with filter": https://dev.azure.com/dnceng-public/public/_build/results?buildId=1679800&view=logs&j=3170e47c-863f-5c15-e18f-1da6ba1af89d&t=86046c01-d695-5308-2da5-0b128c136e72PowerShell exited with code 'null'+ManagedPSEntry+StartupExceptionbefore any test ran; no failed test results. This is the only failure in the run. The PR touches only MamJsonToXml build-task code/tests, not pipeline or PowerShell.simonrozsival commentedon Nov 13, 2026
Flaky recurrence: hosted-agent disconnect (
We stopped hearing from agent)2b5c5d0f)macOS > Tests > APKs 1: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1679890&view=logs&j=c603587c-45e1-580d-a0f5-ab650547d049MSBuild+Emulator 8failingBuild_XAML_Change(False)on a 3000ms timing threshold; that one is not classified as flaky here (uncertain; the PR changes pipeline/test infrastructure).simonrozsival commentedon Nov 13, 2026
Flaky recurrence: macOS PowerShell procargs failure (
Determine if we need to rerun failed tests)<JavaDependencyVerification/>to Microsoft.Android.Build.Tasks #13470macOS > Tests > MSBuild+Emulator 4: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1680055&view=logs&j=c6143a1d-3295-5c1c-c6c3-bc1c4b69d703&t=1fc7fff0-bc8b-5f3e-bf2b-66bfa3850b08Evidence: Tests ran and results were published. No failed test results were found. The PowerShell rerun-decision task failed with a procargs/errno 5 host error, the known macOS signature (#12652). This was the only failed job in the run.
simonrozsival commentedon Nov 13, 2026
Flaky recurrence: checkout git fetch failure (exit 128)
macOS > Tests > MSBuild+Emulator 11checkout: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1680029&view=logs&j=3170e47c-863f-5c15-e18f-1da6ba1af89d&t=2e461c23-fe86-56c1-e407-236085996171Evidence: Checkout failed before any PR code ran. The log shows
RPC failed; curl 56, early EOF,Could not resolve host: github.comand git fetch exit code 128. The laterupload test resultsfailure was a consequence of the checkout failure. This was the only failed job in the run. The signature matches the known checkout/git transport family.simonrozsival commentedon Nov 14, 2026
Flaky CI report:
Java.InteropTests.JavaObjectTest.Dispose_Finalized(NativeAOT):Expected JavaDisposedObject.Dispose(disposing: false) to run. Expected: True But was: FalseTest run 46837024 (
Mono.Android.NET_Tests-NativeAOT), result 100855, stage retry attempt 2 ofmacOS > Tests > APKs 2. This is the same signature as the attempt-1 report above, and the known finalization flake (#10358). The PR changes Handler/View action posting, not JNI finalization.Automated triage; no retry was requested (job-scoped retry could not be verified).
simonrozsival commentedon Nov 14, 2026
Flaky CI report:
Java.InteropTests.JavaObjectTest.Dispose_Finalized(NativeAOT):Expected JavaDisposedObject.Dispose(disposing: false) to run. Expected: True But was: FalseTest run 46858018 (
Mono.Android.NET_Tests-NativeAOT), result 100815, stage retry attempt 3 ofmacOS > Tests > APKs 2. This is the third consecutive attempt with the same signature. It may be strongly reproducible on NativeAOT for this head.Automated triage; no retry was requested (job-scoped retry could not be verified).
simonrozsival commentedon Nov 14, 2026
Flaky CI report:
Java.InteropTests.JavaObjectTest.UnreferencedInstanceIsCollected(NativeAOT):Expected the unreferenced instance to be collected. Expected: True But was: FalseTest run 46858018, result 100829, attempt 3 of
macOS > Tests > APKs 2. This matches the tracked flake #13275. It did not fail in attempts 1 or 2.Automated triage; no retry was requested (job-scoped retry could not be verified).
simonrozsival commentedon Nov 14, 2026
Flaky CI report:
Java.InteropTests.JavaObjectTest.Dispose_Finalized(Mono.Android.NET_Tests-Debug):Expected JavaDisposedObject.Dispose(disposing: false) to run. Expected: True But was: FalseResult 100673,
macOS > Tests > APKs 1. This PR targetsrelease/11.0.1xxand has the same signature as the release-branch recurrence already listed in this inventory (#10358). The PR changes R8 rules and typemap targets, not finalization.Note: this build also has PR- or branch-related NativeAOT arm build failures. Those are not reported here as flaky.
Automated triage; no retry was requested (job-scoped retry could not be verified).
1 remaining item
simonrozsival commentedon Nov 14, 2026
Flaky:
PerformanceTest.Build_XAML_Change(True)timing threshold (MSBuildDeviceIntegration, macOS)macOS > Tests > MSBuild+Emulator 9: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1680482&view=logs&j=3fa46a7c-e001-5541-0f75-574e14078f00Exceeded expected time of 3750ms, actual 3880.066ms. This is the only failing job in the build.(True), and 1679809, 1677394, 1672808 and 1671799 for(False). It is also listed in Flaky test analysis: Xamarin.Android-PR pipeline, April 10–24 2026 #11203.simonrozsival commentedon Nov 14, 2026
Flaky:
PerformanceTest.Build_XAML_Change(False)timing threshold (MSBuildDeviceIntegration, macOS)macOS > Tests > MSBuild+Emulator 8: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1679890&view=logs&j=4b924f79-a52f-52d1-6add-282111deb881Exceeded expected time of 3000ms, actual 4221.105ms(False)overrun appears in unrelated PR builds 1679809 ([Xamarin.Android.Build.Tasks] Replace GetExtraPackages with XmlPeek #13486), 1677394 ([Xamarin.Android.Build.Tasks] Link asset packs with Exec #13503), 1672808 ([generator] Avoid repeated dictionary lookups #13449) and 1671799 ([build] Remove unused checked-build and native build settings #13444). It is also listed in Flaky test analysis: Xamarin.Android-PR pipeline, April 10–24 2026 #11203. The only other failure in this build is the APKs 1 agent loss, which was already reported above.simonrozsival commentedon Nov 14, 2026
Known infra: emulator archive download stall → 180-minute job cap (macOS MSBuild+Emulator)
macOS > Tests > MSBuild+Emulator 6: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1680376&view=logs&j=3ae2673a-0ec7-57f7-5141-29fd33bfe4b1install OpenJDK and accept Android SDK licensesloggedDownloading from ".../emulator-darwin_x64-16428233.zip"at 22:48:32Z, then made no progress until it was canceled at 01:37:59Z. This matches the inventory's emulator-archive stall signature. An earlier .NET bootstrap step also loggedcurl: (6) Could not resolve host, which was retried.ILongValueAnnotationInvoker) look PR-related and are not reported here.simonrozsival commentedon Nov 14, 2026
Flaky:
Xamarin.Android.Build.Tests.PerformanceTest.Build_XAML_Change(False)exceeded expected time4edad88b)Exceeded expected time of 3000ms, actual 3354.643msEvidence: this is the only failed job/test in the build. The same timing-threshold signature has failed on unrelated PRs (#13434 build 1679890 at 4221ms, #13550 build 1680482), and it is already tracked here. Caveat: #13472 changes
Microsoft.Android.Sdk.Tooling.targets(Java version validation moves from a task to targets), which is on the build path. Given the small overrun, a PR contribution cannot be fully excluded. No retry was requested because job-scoped retry semantics could not be verified.simonrozsival commentedon Nov 14, 2026
Flaky (recurring):
Java.InteropTests.JavaObjectTest.Dispose_Finalized(NativeAOT)21363144)Expected JavaDisposedObject.Dispose(disposing: false) to run. Expected: True But was: FalseEvidence: this is the only failed job/test in the build. It is a known GC/finalizer-timing flake (see #10358), and it also hit unrelated #13558 (build 1680359). Note: this is the 4th consecutive occurrence on #13312 (build 1679688 attempts 1-3, now 1681395). #13312 changes
Handler/View/Threadcallback code and not finalization, but the repeated NativeAOT recurrence on this PR is worth watching. No retry was requested.simonrozsival commentedon Nov 15, 2026
CI infrastructure: Azure Artifacts restore failures in #13409, build 1677951, attempt 1.
503 (Service Unavailable)from multiple existing Azure Artifacts feeds. The same 503 from the dnceng feed occurred independently in Cannot build the sdk due to HTTP 503 #7304.Unable to read data from the transport connection: Connection reset by peerin MSBuild+Emulator 3 (log 867) and MSBuild+Emulator 5 (log 820); [CI] 'extract workload packs' failures should fail the job fast (currently succeedsWithIssues, then runs the whole suite against a broken SDK) #11815 records the same transport error independently during CI workload setup. The PR adds a feed, but these failures occurred on the existing feeds.Other failures remain uncertain, including a test timeout; no retry was requested.
simonrozsival commentedon Nov 15, 2026
CI infrastructure: Linux apt DNS failure in #13507, build 1675508, attempt 1.
Linux Build job and install-gcc task, log 40: apt reported
Temporary failure resolvingfor multiple Ubuntu and Launchpad hosts, then could not locate gcc-10. Linux Build attempt 2 already succeeded.The same DNS errors occurred independently in build 1675509, PR #13508 (log 39), and build 1675442 (log 642). The current APK failures are separate, PR-related errors; no retry was requested.
simonrozsival commentedon Nov 15, 2026
PublishTestResultsDNS failure resolving the test-results serviceEvidence:
nodename nor servname provided, or not known (tcmprodcus3.vstmr.visualstudio.com:443)while publishing test results.badfe20af7, and the build is green.simonrozsival commentedon Nov 15, 2026
Checkout
git submodule updatefails with exit code 128 (github.com connect/DNS)Evidence:
Failed to connect to github.com port 443 after 75003 ms, thenCould not resolve host: github.com. The submodule fetch forexternal/android-api-docsfailed, andgit submodule updateexited with 128.checkout-git-fetch-128. The job passed on attempt 2 of the same build and commit2a4a1303aa, and the build is green.simonrozsival commentedon Nov 15, 2026
macOS PowerShell
procargserrno 5 (run JcwGen_Tests)Evidence:
Call to 'procargs' failed with errno 5; PowerShell exited with code 'null'.2a4a1303aa, and the build is green.simonrozsival commentedon Nov 15, 2026
DownloadPipelineArtifactDNS failure resolving dev.azure.comEvidence:
07079253-83cb-5f56-2e9d-c4bdc45f7f76(no log ID was recorded):nodename nor servname provided, or not known (dev.azure.com:443).2a4a1303aa, and the build is green.simonrozsival commentedon Nov 15, 2026
macOS PowerShell
procargserrno 5 (set JDK variables)Evidence:
Call to 'procargs' failed with errno 5; PowerShell exited with code 'null'. The laterupload test resultsstep failed only becauseTestReleasedid not exist.588edaa723, and the build is green.simonrozsival commentedon Nov 15, 2026
Xamarin.Android.Build.Tests.InstallTests.IncrementalFastDeployment("aab")unchanged-install timing assertion (inventory row 3)Evidence:
buildUri=vstfs:///Build/Build/1679285, build ID verified):Second unchanged install: '00:00:23.2600000' should be faster than clean install: '00:02:38.8900000' and incremental install: '00:00:22.3700000'.8d70cd48b7, and the build is green.FileUtil/AndroidAppManifest.WriteToFilehelpers and theunameMac probe. Nothing in it touches FastDeploy or install timing.simonrozsival commentedon Nov 16, 2026
Xamarin.Android.RuntimeTests.JavaMarshalGCBridgeTests.ManagedPeerCycleSurvivesJavaRootAndIsCollectedAfterRelease: Java-rooted cycle missing from bridge processing (new signature)fail if any issues occurredrun Mono.Android.NET_Tests-NoAab)Evidence:
Mono.Android.NET_Tests-NoAab(buildUri=vstfs:///Build/Build/1679429, build ID verified):The Java-rooted cycle must participate in global-to-weak bridge processing. Expected: not <empty> But was: <empty>.simonrozsival commentedon Nov 16, 2026
System.NetTests.PingTest.PingLocalhost: ping to localhostTimedOut(NativeAOT)fail if any issues occurredrun Mono.Android.NET_Tests-NativeAOT)Evidence:
Mono.Android.NET_Tests-NativeAOT(buildUri=vstfs:///Build/Build/1681205, build ID verified):Ping to localhost failed with status: TimedOut. Expected: Success But was: TimedOut.TimedOutfailure in NativeAOT on unrelated PR [build] Honor inherited analyzers and remove blanket warning waivers #13428 (build 1672500).