There won't be too much detail here, just an ugly page with enough to show what things are happening. This page will disappear if things go well.
Capsule summary: one of our fileservers lost a boot drive. The boot drive has little to do with the files being served but is needed to make the server run. Initial attempts at reviving the boot drive failed so we brought the system to where we could work on it, and replaced that drive. The next steps involved getting the system to recognize the filesystem on the array of drives dedicated to that. The system, for whatever reason, erred in that process. It made an incorrect determination of the arrangement of drives in the filesystem, and instead of assembling the filesystem, created errors instead. This was pretty disheartening. It's been a tedious process to very carefully try to work this out. That, and attending to daytime jobs that have nothing to do with this puts limits on available time.
Update Nov 1: Most, if not all, web sites including shared pages are back in clone form.
Update Oct 29, 8PM-ish: Progress was made in getting the filesystem's configuration correct, and the files visible. It needed some repair and there was worry that the errors generated as mentioned above would make that very difficult. However the filesystem has been recovered and declared healthy. It will still take some time to finish the system configuration and put it back in place. We are working with, as they say, an abundance of caution, and because of that as well as with the other time constraints stated earlier, it may still be some days before it's back in place.
Update Oct 30, 10PM: While the web files themselves are accessible, the fileserver still has some issues such that it requires a lot of hands-on stuff, for lack of a better word, to get it to boot. It can't be reinstalled in this state, and will require more work and headscratching to figure it out, so we are doing the next best thing and transferring website files via sneakernet (actually car and USB net) to the datacenter. This is again a fairly lengthy process. At this point most standalone websites on the server have been replaced in this way. If you're still seeing the failure message, the site you're seeing it on will be updated soon. The shared pages (e.g. user pages) are among these still needing to be done. When the fileserver issues have been figured out and dealt with, the fileserver will be reinstalled and its files updated to reflect any changes that have been made in the copies.