Writing

Software, technology, sysadmin war stories, and more. Feed
Monday, October 17, 2011

Don't overreact to a seemingly undeletable file

I saw some interesting overreactions to problems in my days of working tech support. Most of them came from other employees.

One time, a bunch of people were trying to figure out what had gone wrong with a system. They were trying to delete a file but simply could not succeed. Even as root, the file would not go away. I think they even went to the trouble of taking the box down, removing it from the rack and booting it into a rescue environment.

Even then, it would not let them remove the file. They were freaked out by this and started talking about how it was "probably hacked" or "corrupted" or something else equally ridiculous. They started talking about having the customer make plans to get a freshly installed drive and then either restore from backups or migrate data from the old disk.

Somehow, I heard about this and popped over to the box with ssh to take a look. This is about how it proceeded from there:

# rm foo
rm: cannot remove `foo': Permission denied

Okay, hoofbeats. I think "horses, not zebras".

# lsattr foo
--------------- foo

Okay, that's neat. But what about...

# touch bar
touch: cannot touch `bar': Permission denied

Aha, so it's not just the file. I think I tried doing a rmdir on it next (yes, even though it had a file in it). This was useful:

# rmdir test
rmdir: failed to remove `test': Operation not permitted

Aha! If it had something in it, it would have looked like this:

# mkdir x; touch x/y; rmdir x
rmdir: failed to remove `x': Directory not empty

Nope, something else was going on there.

# lsattr -d test
----i---------- test

Very nice. Someone set the immutable bit on a directory. Note that just plain old "lsattr test" is insufficient since it will display what's in the directory, just like ls.

Obviously, the customer did not wind up reinstalling after I found this.