Software, technology, sysadmin war stories, and more. Feed
Thursday, March 15, 2018

Bus errors, core dumps, and binaries on NFS

Here's another bit of Linux arcana that comes in handy every five years or so, when a fresh group of people run into a problem that just won't go away. Maybe it's happened to you and you never knew why. Hopefully someone will find this helpful.

Imagine a scene. You have a continuous build system set up. It generates fresh builds of your internal software regularly. Maybe it's once a day or maybe it's once an hour. Then, as part of that process, something copies the output to a NFS filer so that other hosts can use it.

This isn't a great way to do software distribution, but it's super easy and mostly works, so people tend to do it over and over. It does, however, have an interesting failure mode that is frequently chalked up to anything but what it really is.

If you run your programs directly from NFS, once in a while, you will probably see where they just keel over and die. Without any warning, you'll see either "Bus error" or "Bus error (core dumped)" depending on your core settings, and then it'll be toast. What happened?

Well, in this situation, it has little to do with the actual "bus". It's not the thing on wheels or thing with all of the slots in your server. It's just a recycled error that is coming from the depths of the kernel: SIGBUS. The kernel is trying to tell you that it's most unhappy about something that was done to it, and that it can't continue, so your process is now dead.

What happened? I bet you just pulled the rug out from under your program. That is, someone replaced the binary of the running process on that NFS-exported filesystem, and it no longer matched up with what was running. Then your machine needed to access a page, and it wasn't available, and the house of cards came tumbling down.

Normally you can't do this. If you are running a program from local storage and try to overwrite it, the write will fail with "text file busy". This is the kernel trying to keep you from kneecapping yourself. Note that you can unlink this file and replace it with a new one, but the original inode is still there as long as some process has its hooks in it. You may think you're cheating the system but you're not. You just removed the pointer from the name to the inode, then put up another inode and pointed the same name at it. In this context, the name doesn't matter, so nothing happens.

But, over NFS, things are different. It's entirely possible the NFS server has no idea that anyone has pages from this binary mapped into memory and might be coming back for more. As far as it's concerned, it's just another ordinary file, and it's more than happy to let you do whatever you want to it: replace it, truncate it, edit it, append to it, and all of that good stuff. There's no EBUSY because it has no reason to protect it.

Granted, it takes the right confluence of events to make this happen. If your binaries are relatively small and always fit into memory, you may never notice this. If the programs running from NFS don't stick around for very long, it's not likely they will be running when the file gets changed.

If you see this, you probably have very large binaries that restart infrequently and then usually stick around for a long time.

This probably also varies based on what sort of system your client is, and what sort of system (or appliance/filer) the server is. You might get lucky and never see it at all.

But hey, if you're getting "bus error (core dumped)" ever since moving to NFS for your server processes, this is probably why. Now you know what's going on, and can take steps to avoid it.

Final thought: this is not the only way to trigger a SIGBUS on Linux. There's one involving alignment but if you're on an Intel processor then you're not likely to see it. Raspberry Pi and other ARM systems, however, are more interesting. This is a story for another time.