Towards the end of last month, I started a build job on my self-hosted GitHub Actions runner. It was a job that normally took 10-15 minutes, but this time it just wouldn't finish. The job seemed stuck, and I wasn't getting any response from the runner. When I tried to connect to the server via SSH, the connection was refused. It felt similar to the OOM scenarios I'd experienced on my VPS where sshd couldn't accept connections, but this time my RAM usage was normal.
After some digging, I realized my runner's heart had stopped beating. Looking at the GitHub Actions panel, I saw the runner was "Offline". The interesting thing was that the server itself was up, and my other Docker containers were running without issues. The only problem was my CI runner.
Getting to the Root of the Problem: A Cleanup Script Murder
To understand why the runner had died, I connected to the server via console. My first task was to check the dmesg output. There was nothing surprising there; no kern
Discussion
Jump in and comment!
Get the ball rolling with your comment!