Hello all,

I have a 2-machine cluster of RHEL 5.1 servers (Dell PE 2850s) that serve up a whole bunch of web sites. One of them is the primary web server, svn repository, and mysql slave, the other is the mysql master and backup web server (heartbeat service is used to fail over web traffic if the primary web server goes down, and a homebrew solution is used to fail over mysql).

3 times in the last two weeks, the main web server has crashed. I suspected hardware until this afternoon, when after disabling what I thought was the offending piece of hardware in the BIOS (i'd seen a few hda-related errors in /var/log/messages a few weeks back and disabled the IDE controller last night after its 3rd crash) the machine stayed up all night while the other server -- the primary database server, which was now serving up our websites as well -- crashed in the same manner today just after noon.

What happens is the "crashed" machine remains pingable but does not respond to any other requests, and at the console it only prints memory status messages every few seconds but doesn't allow logins (IE it looks like all processes are stopped). On reboot everything looks hunky dory and there's nothing out of the ordinary in /var/log/messages or any other log that I could think to examine.

The interesting thing is that the primary database / backup web server had never crashed in this manner before today around noon, and it took over the web serving last night around 10pm. So it appears to me that apache is a likely culprit, since before it just happily ran mysqld without any problems.

I apologize if this isn't the right forum to ask about this, but I don't believe it to be hardware-related, and since the problem suddenly began occuring 2 weeks ago and has occurred with increasing frequency since (once two weeks ago, 2 times yesterday, and now once already today) on two RHEL 5.1 machines running apache, I thought maybe a RHEL update might be at fault. I'm a pretty inexperienced linux admin and so far just looking through logs hasn't turned up anything.

Has anybody else seen anything like this? Any pointers on where to begin looking?

--
Matthew Rich
Senior Web Application Developer
Northwestern University
School of Education and Social Policy
Annenberg Hall, Room 249
+1 847 467 2819
m-rich@northwestern.edu

_______________________________________________
rhelv5-list mailing list
rhelv5-list@redhat.com
https://www.redhat.com/mailman/listinfo/rhelv5-list