Archive for December, 2006
2/4 failed drives in a raid5 array??
Posted by aaron in Hardware, Linux, Troubleshooting on December 30th, 2006
Apparently raid arrays don’t like it when you kill power to the machine. We’ve been doing remodeling in the house, and I’ve been turning on and off breakers. Apparently I forgot which breaker the servers were on, and turned it off accidentally a few times.
One of the drives apparently did fail for some reason. I accept this. This happens all the time, and that is exactly why I have a raid 5 array. I was like ok no big deal, I’ll just send it in for an RMA. But shortly after, another drive looked like it failed. I saw this in /proc/mdstat:
…[U__U]
meaning only two of the four drives were left. Future attempts at rebooting the machine resulted in the raid volume not being accessible at all. Other clues that indicated a drive failure:
From /var/log/messages:
Dec 29 19:29:06 onyx kernel: Buffer I/O error on device md0, logical block 0
Dec 29 19:29:06 onyx kernel: lost page write due to I/O error on md0
Dec 29 19:29:06 onyx kernel: EXT2-fs error (device md0): ext2_readdir: bad page in #2
There was also some output in dmesg I found by typing
dmesg | less
(But I didn’t write it down, and now dmesg outputs information from the last boot which successfully brought up the array with 3/4 drives.)
I was convinced I hadn’t actually lost 2/4 drives at the same time, and set out to figure out a way to bring it back.
After several hours of looking through forums and reading the mdadm documentation, I was able to get the array back running on 3/4 drives.
I created the configuration file /etc/mdadm.conf:
DEVICE /dev/sd[abcd]1
ARRAY /dev/md0 devices=/dev/sda1,/dev/sdb1,/dev/sdc1,/dev/sdd1
Then ran:
/sbin/mdadm –assemble -f /dev/md0
mdadm: forcing event count in /dev/sdc1(2) from 1077319 upto 1077330
mdadm: clearing FAULTY flag for device 1 in /dev/md0 for /dev/sdc1
mdadm: /dev/md0 has been started with 3 drives (out of 4).
Now I just have to RMA this drive very quickly before another drive actually does fail.
Virtual Users with Postfix and MySQL
I recently discovered an excellent tutorial on setting up Postfix and Courier to use virtual users read from a MySQL database. This caught my attention because it is a much better setup than having real system users for all email addresses, as it allows much more flexibility. I finally set aside some time to set up a real mail server.
I first went through the tutorial installing everything on a virtual server running on my pc. I used a VMware image of Fedora Core 5. You can find that and many other VMware images on VMware’s appliance directory page.
I ran through the tutorial, and got everything working relatively quickly. There was one thing I had to add that was not in the tutorial. That was this command:
postconf -e ‘mailbox_transport = virtual’
After that, I was able to send and receive emails from my virtual server.
I then went out and bought another VPS account with Tektonic, planning to have another quick and easy install of the mail software. After running through the tutorial twice, I was still unsuccessful. I kept getting an error in postfix’s log:
Dec 25 06:41:28 mail postfix/smtpd[27714]: fatal: no SASL authentication mechanisms
Dec 25 06:41:29 mail postfix/master[15875]: warning: process /usr/libexec/postfix/smtpd pid 27714 exit status 1
Dec 25 06:41:29 mail postfix/master[15875]: warning: /usr/libexec/postfix/smtpd: bad command startup — throttling
These three lines show up in my mail log repeatedly. When I try to initiate a connection to my mail server on port 25, there is no response from Postfix:
telnet mail.parecki.com 25
Trying mail.parecki.com…
Connected to mail.parecki.com.
Escape character is ‘^]’.
It just hung after that.
I eventually figured out that I was missing a critical package.
yum install cyrus-sasl-plain
After installing that package, everything works perfectly! I guess it already came installed on my VMware image for some reason.
Thanks to Falko Timme for writing a great tutorial, and for responding to my forum posts asking for help!