Repository navigation
App hangs after taking crashdump in official .NET 9 x64 glibc docker images #109779
Description
Description
If a .NET 9 app running using the official .NET 9 glibc x64 images crashes and you take a crash dump, the process hangs indefinitely after writing the dump.
Note that AFAICT the issue only reproduces for the following tags, on x64
9.0.100-bookworm-slim9.0.100-noble9.0.100-azurelinux3.0
It does not reproduce on arm64, or on the following tags:
9.0.100-alpine3.20
The same issue is seen with both mcr.microsoft.com/dotnet/sdk and mcr.microsoft.com/dotnet/runtime
Reproduction Steps
Create a simple dockerfile as follows, that just throws an exception in Program.cs:
FROM mcr.microsoft.com/dotnet/sdk:9.0.100
WORKDIR /src
RUN mkdir /dumps \
&& dotnet new console -n SmokeTest -o . \
&& echo 'throw new System.Exception("Expected");' > Program.cs \
&& dotnet publish -c Release --framework net9.0 -o /src/publish
# Capture dumps
ENV COMPlus_DbgEnableMiniDump=1
ENV COMPlus_DbgMiniDumpType=4
ENV DOTNET_DbgMiniDumpName=/dumps/coredump.%t.%p
ENTRYPOINT ["dotnet", "/src/publish/SmokeTest.dll"]You can build and run this using:
docker build -t smoketest .
docker run smoketestWhich produces the following output:
> docker run smoketest
Unhandled exception. System.Exception: Expected
at Program.<Main>$(String[] args) in /src/Program.cs:line 1
[createdump] Gathering state for process 1 dotnet
[createdump] Crashing thread 0001 signal 6 (0006)
[createdump] Writing full dump to file /dumps/coredump.1731508490.1
[createdump] Written 146817024 bytes (35844 pages) to core file
[createdump] Target process is alive
[createdump] Dump successfully written in 132msbut the process never ends, it is stuck at this point.
Note that I failed to reproduce this by running directly in Debian (in WSL). I could only reproduce it when running with the docker images
Expected behavior
The process should exit after createdump is called.
Actual behavior
The process does not exit.
docker exec into the process and running cat /proc/1/status shows it's frozen:
cat /proc/1/status
Name: dotnet
Umask: 0022
State: S (sleeping)Regression?
This does not happen in .NET 8.
Known Workarounds
None, other than "don't collect a crash dump".
Configuration
This only reproduces with
- .NET 9 (both RC2 and GA)
- x64
- glibc (
bookworm-slim,noble,azurelinux3.0) - docker
Other information
@kevingosse noted this stack is strange, in that abort triggers the sigsegv handler instead of sigabrt:
#5 0x00007f83778e7125 in sigsegv_handler (code=11, siginfo=0x7f8376a9abf0, context=0x7f8376a9aac0)
at /__w/1/s/src/coreclr/pal/src/exception/signal.cpp:677
#6 <signal handler called>
#7 __GI_abort () at ./stdlib/abort.c:107
#8 0x00007f8377912399 in PROCAbort (signal=<optimized out>, siginfo=siginfo@entry=0x0)
at /__w/1/s/src/coreclr/pal/src/thread/process.cpp:2562
chatgpt suggests abort could raise a segfault if the sigabort information is invalid (and so a segfault is triggered when calling the sigabort handler), which sounds... plausible?
We hypothesized that this commit is the suspect, but mostly just because that's changed since .NET 8.
Activity
dotnet-policy-service commentedon Nov 13, 2024
Tagging subscribers to this area: @tommcdon
See info in area-owners.md if you want to be subscribed.
hoyosjs commentedon Nov 13, 2024
cc: @mikem8361 @dotnet/dotnet-diag
mikem8361 commentedon Nov 14, 2024
@janvorli do you think this has anything to do with PR #105578?
janvorli commentedon Nov 14, 2024
@mikem8361 it is theoretically possible, although the SIGABORT should result in termination of the process, the fallback in the abort that uses SIGSEGV should never kick in. Let me debug the repro case.
janvorli commentedon Nov 14, 2024
Without any testing done, I actually believe this happens due to the docker assigning PID=1 to the process. Linux handles signals differently for PID=1 as without docker, it is the init process. Such a process has special responsibilities. There is an article describing why running your own process as the only one in Docker is problematic: https://daveiscoding.hashnode.dev/why-do-you-need-an-init-process-inside-your-docker-container-pid-1
In the case of PID=1, the default action for Abort is "SIG_IGN". The
abort()sendsSIGABRTto the current process, but due to this, it is just ignored. So theabort()code falls back to killing the process using an instruction that triggersSIGSEGV. But since we don't uninstall theSIGSEGVhandler after the PR #105578, this fallback doesn't terminate the process either.I'll take a look if we can do something about it in our sigsegv handler. The workaround is to pass
--initto docker when starting the container. That adds a "dummy" init process to the container and the actual .NET process would get PID=2 and everything would work as expected.I've tried this workaround and it worked.
janvorli commentedon Nov 14, 2024
I've attached debugger to the process to see if we can fix that problem. What happens is that the SIGSEGV triggered by the fallback path in abort() ends up calling
PROCCreateCrashDumpagain. And it ends up spinning here:runtime/src/coreclr/pal/src/thread/process.cpp
Lines 2235 to 2238 in 1e98c05
The reason is that the
g_crashingThreadIdstill contains the current thread id that was set when thePROCCreateCrashDumpwas called.@mikem8361 I think the fix would be to return with false if the
previousThreadId == currentThreadIdtommcdon commentedon Nov 15, 2024
fyi @dotnet/dotnet-monitor for awareness
Add more smoke testing for .NET 9 (#6296)
mikem8361 commentedon Dec 4, 2024
PR that fixes the recursion problem: #110379