Summary
On .NET 10.0.9 (Linux x86_64), dotnet build (and dotnet test) of a multi-project solution intermittently crashes the host process with a CoreCLR fatal execution-engine error (0x80131506 / "Internal CLR error"). A full minidump shows the crash is preceded by many System.Net.Sockets.SocketException: "Cannot assign requested address /tmp/MSBuild<pid>" thrown from MSBuild's out-of-proc node IPC (Unix domain sockets at /tmp/MSBuild<pid>), with faulting threads inside Microsoft.Build.BackEnd.NodeProviderOutOfProcBase.TryConnectToProcess. Forcing a single in-proc MSBuild node (-m:1) eliminates the crash entirely (0/25 vs ~30% baseline).
Environment
- OS: Linux x86_64 (Ubuntu, kernel 6.x).
/tmp is tmpfs, drwxrwxrwt, 32 GB, plenty of free space/inodes.
- .NET SDK:
10.0.109
- .NET Runtime:
Microsoft.NETCore.App 10.0.9 (/usr/lib/dotnet/shared/Microsoft.NETCore.App)
- CoreCLR version (from
dotnet-dump → eeversion): 10.0.926.27113 @ commit 901ca941248413c79832d2fdbd709da0c4386353
- SOS version:
9.0.13.11903
- GC mode: Workstation (default)
- Only one SDK installed (no
global.json pinning to an older version)
Reproduction
A solution of ~11 C# projects (mix of class libraries and test projects, .NET 10.0). In a loop:
for i in $(seq 1 30); do dotnet build MySolution.slnx || echo "crashed on iter $i"; done
Within ~20 iterations the process aborts:
Fatal error.
Internal CLR error. (0x80131506)
Observed crash rates (this solution, Linux)
| Command |
Crashes |
Total |
dotnet build Virelia.slnx (11 projects) |
6 |
20 |
dotnet test Virelia.slnx (same solution) |
2 |
15 |
dotnet build/test of a 3-project solution |
0 |
32 |
dotnet build Virelia.slnx -m:1 (single in-proc node) |
0 |
25 |
dotnet build Virelia.slnx -nodereuse:false (still multiproc) |
5 |
15 |
So the crash correlates with multiproc out-of-proc node spawning (larger solutions spawn more nodes): a 3-project solution did not reproduce in 32 iterations, while the 11-project solution crashes ~30% of the time. -m:1, which disables out-of-proc nodes entirely, eliminates it (0/25).
Downstream symptom (possibly a separate test-runner bug)
When the crash hits during dotnet test, the faulted process sometimes wedges (the runtime's createdump reports Target process is alive after writing the dump, and the process does not exit). A foreground dotnet test invocation then appears to hang — a spawned child (testhost) inherits the stdout pipe and keeps it open, so the caller never sees EOF. Under a clean harness (process in its own session, stdout to a file), dotnet test exits on every crash (2 crash-exit, 0 hang in 15). I have not fully isolated whether the hang is an independent dotnet test/vstest robustness bug or purely a side effect of the wedged process + inherited pipe; flagging it here in case the test team wants to harden the runner against a dead/wedged build child.
Analysis (from the full minidump)
Captured via the runtime's built-in dumper (DOTNET_DbgEnableMiniDump=1, DOTNET_DbgMiniDumpType=4), analyzed with dotnet-dump analyze:
- No current managed exception on the main thread — the fault is a native CoreCLR execution-engine error.
0x80131506 corresponds to System.ExecutionEngineException.
dumpexceptions shows the pre-allocated EE/stackoverflow/OOM exceptions on the heap plus many System.Net.Sockets.SocketException: Cannot assign requested address /tmp/MSBuild<pid>, several carrying the stack frame System.Net.Sockets.Socket.DoConnect(System.Net.EndPoint, System.Net.SocketAddress).- Thread 0 (the
dotnet → MSBuild forwarder) is in Microsoft.Build.Execution.BuildSubmission.Execute() → WaitOneNoCheck.
- Several worker threads are inside
Microsoft.Build.BackEnd.NodeProviderOutOfProcBase:
TryConnectToProcess → TryConnectToPipeStream → NamedPipeClientStream.ConnectInternal + SpinWait. (Despite the NamedPipeClientStream type name, on Unix this is Unix-domain-socket IPC at /tmp/MSBuild<pid>.)- reached via
System.Threading.Tasks.Parallel.For over GetNodes/StartNewNode — parallel out-of-proc node spawning.
- One thread is in
Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.InitializeManifests().
- Each crash leaves ~10 stale
/tmp/MSBuild<pid> socket files behind (the node IPC endpoints), confirming the sockets were created but never cleaned up because the process faulted. (Cleaning these between runs does not prevent the next crash — the litter is a symptom, not the cause.)
Expected behavior
The .NET runtime must not crash with a fatal execution-engine error (0x80131506) regardless of what managed code does with a socket. A SocketException (EADDRNOTAVAIL on a Unix domain socket) is a normal, recoverable managed failure and must be surfaced as a managed exception, never race the runtime into an unrecoverable CoreCLR fault.
Workaround
-m:1 (force a single in-proc MSBuild node) eliminates the crash (0/25 in my testing) by never spawning out-of-proc nodes and thus never creating/connecting the /tmp/MSBuild<pid> sockets. No perceptible perf cost for this solution (build/test times stay ~1–3s).
Related
Minidump
A full CoreCLR minidump (467 MB raw → 66 MB zstd-compressed), produced by the runtime's createdump on the 0x80131506 fault, is available for download:
dotnet-dump analyze loads it cleanly; eeversion confirms CoreCLR 10.0.926.27113 @ commit 901ca941248413c79832d2fdbd709da0c4386353.
Summary
On .NET 10.0.9 (Linux x86_64),
dotnet build(anddotnet test) of a multi-project solution intermittently crashes the host process with a CoreCLR fatal execution-engine error (0x80131506/ "Internal CLR error"). A full minidump shows the crash is preceded by manySystem.Net.Sockets.SocketException: "Cannot assign requested address /tmp/MSBuild<pid>"thrown from MSBuild's out-of-proc node IPC (Unix domain sockets at/tmp/MSBuild<pid>), with faulting threads insideMicrosoft.Build.BackEnd.NodeProviderOutOfProcBase.TryConnectToProcess. Forcing a single in-proc MSBuild node (-m:1) eliminates the crash entirely (0/25 vs ~30% baseline).Environment
/tmpistmpfs,drwxrwxrwt, 32 GB, plenty of free space/inodes.10.0.109Microsoft.NETCore.App 10.0.9(/usr/lib/dotnet/shared/Microsoft.NETCore.App)dotnet-dump→eeversion):10.0.926.27113@ commit901ca941248413c79832d2fdbd709da0c43863539.0.13.11903global.jsonpinning to an older version)Reproduction
A solution of ~11 C# projects (mix of class libraries and test projects, .NET 10.0). In a loop:
Within ~20 iterations the process aborts:
Observed crash rates (this solution, Linux)
dotnet build Virelia.slnx(11 projects)dotnet test Virelia.slnx(same solution)dotnet build/testof a 3-project solutiondotnet build Virelia.slnx -m:1(single in-proc node)dotnet build Virelia.slnx -nodereuse:false(still multiproc)So the crash correlates with multiproc out-of-proc node spawning (larger solutions spawn more nodes): a 3-project solution did not reproduce in 32 iterations, while the 11-project solution crashes ~30% of the time.
-m:1, which disables out-of-proc nodes entirely, eliminates it (0/25).Downstream symptom (possibly a separate test-runner bug)
When the crash hits during
dotnet test, the faulted process sometimes wedges (the runtime'screatedumpreportsTarget process is aliveafter writing the dump, and the process does not exit). A foregrounddotnet testinvocation then appears to hang — a spawned child (testhost) inherits the stdout pipe and keeps it open, so the caller never sees EOF. Under a clean harness (process in its own session, stdout to a file),dotnet testexits on every crash (2 crash-exit, 0 hang in 15). I have not fully isolated whether the hang is an independentdotnet test/vstest robustness bug or purely a side effect of the wedged process + inherited pipe; flagging it here in case the test team wants to harden the runner against a dead/wedged build child.Analysis (from the full minidump)
Captured via the runtime's built-in dumper (
DOTNET_DbgEnableMiniDump=1,DOTNET_DbgMiniDumpType=4), analyzed withdotnet-dump analyze:0x80131506corresponds toSystem.ExecutionEngineException.dumpexceptionsshows the pre-allocated EE/stackoverflow/OOM exceptions on the heap plus manySystem.Net.Sockets.SocketException: Cannot assign requested address /tmp/MSBuild<pid>, several carrying the stack frameSystem.Net.Sockets.Socket.DoConnect(System.Net.EndPoint, System.Net.SocketAddress).dotnet→ MSBuild forwarder) is inMicrosoft.Build.Execution.BuildSubmission.Execute()→WaitOneNoCheck.Microsoft.Build.BackEnd.NodeProviderOutOfProcBase:TryConnectToProcess→TryConnectToPipeStream→NamedPipeClientStream.ConnectInternal+SpinWait. (Despite theNamedPipeClientStreamtype name, on Unix this is Unix-domain-socket IPC at/tmp/MSBuild<pid>.)System.Threading.Tasks.Parallel.ForoverGetNodes/StartNewNode— parallel out-of-proc node spawning.Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.InitializeManifests()./tmp/MSBuild<pid>socket files behind (the node IPC endpoints), confirming the sockets were created but never cleaned up because the process faulted. (Cleaning these between runs does not prevent the next crash — the litter is a symptom, not the cause.)Expected behavior
The .NET runtime must not crash with a fatal execution-engine error (
0x80131506) regardless of what managed code does with a socket. ASocketException(EADDRNOTAVAILon a Unix domain socket) is a normal, recoverable managed failure and must be surfaced as a managed exception, never race the runtime into an unrecoverable CoreCLR fault.Workaround
-m:1(force a single in-proc MSBuild node) eliminates the crash (0/25 in my testing) by never spawning out-of-proc nodes and thus never creating/connecting the/tmp/MSBuild<pid>sockets. No perceptible perf cost for this solution (build/test times stay ~1–3s).Related
NodeProviderOutOfProcBase.TryConnectToPipeStreamcode path this crash hits. On .NET 10.0.9 the failure manifests not just as the 30s hang that issue targets, but as a CoreCLR fatal crash.Minidump
A full CoreCLR minidump (467 MB raw → 66 MB zstd-compressed), produced by the runtime's
createdumpon the0x80131506fault, is available for download:zstd -d prog_full.zst(yieldsprog_dotnet_pid_558566.dmp, ~446 MB)dotnet-dump analyze prog_dotnet_pid_558566.dmpdotnet-dump analyzeloads it cleanly;eeversionconfirms CoreCLR10.0.926.27113 @ commit 901ca941248413c79832d2fdbd709da0c4386353.