Don't overreact to a seemingly undeletable file
I saw some interesting overreactions to problems in my days of working tech support. Most of them came from other employees.
One time, a bunch of people were trying to figure out what had gone wrong with a system. They were trying to delete a file but simply could not succeed. Even as root, the file would not go away. I think they even went to the trouble of taking the box down, removing it from the rack and booting it into a rescue environment.
Even then, it would not let them remove the file. They were freaked out by this and started talking about how it was "probably hacked" or "corrupted" or something else equally ridiculous. They started talking about having the customer make plans to get a freshly installed drive and then either restore from backups or migrate data from the old disk.
Somehow, I heard about this and popped over to the box with ssh to take a look. This is about how it proceeded from there:
# rm foo rm: cannot remove `foo': Permission denied
Okay, hoofbeats. I think "horses, not zebras".
# lsattr foo --------------- foo
Okay, that's neat. But what about...
# touch bar touch: cannot touch `bar': Permission denied
Aha, so it's not just the file. I think I tried doing a rmdir on it next (yes, even though it had a file in it). This was useful:
# rmdir test rmdir: failed to remove `test': Operation not permitted
Aha! If it had something in it, it would have looked like this:
# mkdir x; touch x/y; rmdir x rmdir: failed to remove `x': Directory not empty
Nope, something else was going on there.
# lsattr -d test ----i---------- test
Very nice. Someone set the immutable bit on a directory. Note that just plain old "lsattr test" is insufficient since it will display what's in the directory, just like ls.
Obviously, the customer did not wind up reinstalling after I found this.