Backup server brought online after crash

By Rachel Carlson     4/17/08 7:00pm

One of Rice's three e-mail servers suffered a hardware failure and a subsequent operating system collapse at 10 a.m. Monday. The failure severely slowed e-mail access for about 50 percent of users until after 2 p.m. Director of System Architecture and Infrastructure Barry Ribbeck said a replicate mail service was implemented in lieu of the failed server and functions exactly like the original system.Ribbeck said the failure at 10 a.m. was repaired with a replacement part within 30 minutes, but a flood of recovered information led to the corruption of the operating system software at 2 p.m. E-mail access was restored at 2:30 pm. Ribbeck said the replicate e-mail service used to restore services was installed during Winter Break in response to previous e-mail outages in the fall semester.

"When messages are mailed to your account, they go to two independent systems," Ribbeck said. "When one system fails, we can turn on the second one."

The Information Technology Department planned to test the replica this summer but was forced to implement it during Monday's system failure, Ribbeck said.

---

---

The backup server differs from the original in that it does not contain a quarantine system for spam, Ribbeck said. The replica will still include the registered blacklist system, RBL, which continually screens incoming messages, but currently lacks the DSPAM defense system, which tags and removes spam.

Ribbeck said that the replica e-mail server is a new and innovative method of system recovery. Only two other institutions in the world, Carnegie Mellon University and University of Michigan, use this backup feature.

"When you look at peer institutions, we are way far removed from what they're doing, Ribbeck said. "Some don't even back up their mail."

IT is still deciding whether to revert back to the original system or use the replica as the primary e-mail server. One of the disadvantages of using the replica is that the system is based in the Mudd Laboratory instead of the Primary Data Center located off campus, which is more secure in the event of a storm.

"The Primary Data Center has more resources and is built to withstand a category three hurricane," Ribbeck said. "Mudd Lab has a wooden roof."

Ribbeck said the replica system worked effectively, and IT is continuing efforts to make user access more convenient.

"We used this as an opportunity to learn what we did well and what we're supposed to do better," Ribbeck said. "Overall, in terms of the continuity process, everybody's very happy with the processing of it.

---