Unable to open '/dev/zap/pseudo' after reboot (resolved)

[ronator]

Hello there,

I just wanted to share my experience with a nasty message i got after I rebooted vicidialnow. First I have to mention, that one hard disk had an hardware failure, but hence, it was a raid ... after syncing the new hdd everything seemed fine. Since the machine was running for 50 days I thought a reboot might be good (its a hardware-raid controlled by bios). I did that remotely in the night and I just wondered why asterisk wasn't started automatically after reboot (but httpd/mysqld/crond were). I did it manually. In the end I could not really test functionality since at night no calls are routed to that machine.
The next morning agents could log-in but could not make any calls; Asterisk CLI said something like

Code: Select all

Unable to open '/dev/zap/pseudo: No such device

And indeed, there was no /dev/zap at all ! So I had to activate my failover system to search for the error in the original system.
First thing I found out was that "lsmod" did not show either zaptel nor ztdummy. A modprobe with both worked fine. After that all needed "devices" under /dev/zap (inclusively) were created "on the fly". So i thought something got weird with udev, but here everything seemed fine,too.
Then I was checking for all start scripts and : surprise, surprise :
/etc/rc.d/rc.local was not executable (644)
Since this is the file where I have "modprobe zaptel" (etc.) I chmod'ed it to 755, made a reboot and zaptel/ztdummy was loaded on reboot.
I do not have the lightest clue why, when, what or even who changed the file setting of such an important start-script, and it took some hours to hunt it down. May this thread save someone's time to get his / her machine running again.

If anyone has a possible answer to the question how that could have happened, I'll be quite thankful ...

best wishes, r0n

[williamconley]

if your drive had any sort of failure, it may have been that this file was copied from one system to the other and had not had permissions brought over with the sync. could have been a RAID problem entirely.

BTW: i do not recommend RAID on VICIdial systems at all. a GOOD backup copy (perhaps even a mirror that is shut off), but not a RAID unless you have at least RAID 5 and lots of drives.

keeping a spare mirror copy on drive you can fire up or "switch to" at boot by altering your grub has its merit. but RAID has a tendency to copy flaws in the live system more often than it resolves dead drive issues.

so keeping a spare complete copy of the system on a drive that's not in use and perhaps even copying your database to it nightly (mount, copy, unmount), that could be useful

[ronator]

Dear Bill,

your statement concerning to RAID is somehow interesting and disturbing to me at the same time. I use RAID-1 and just mirror the primary hdd. One big point for updating to vicidial 1.3 was the feature-list for me, where it says
VicidialNOW CE 1.3 Released!!!
Here are some of the major changes:
New Vicidial agent interface
Astguiclient/Vicidial 2.0.5.1rc1
CentOS 5.4 as base
64 gig memory support
Phpmyadmin
RAID support

So if you limit this to RAID-5 ... that makes me a little nervous right now :) Could you please give me a little more insight, what might be the problem with mirroring the disks ? Because it`s somehow like a company-standard for us to keep business-relevant systems on RAID, so I do need really a good argument to switch to a spare backup system.

Furthermore I have to add that the system was running continously for 2 months on RAID-1 without any flaws and then a temperature problem in the server room hit one of the hdd. After replacing the damaged hdd and waiting for syncing everything was fine again. Without the reboot I wouldn't even have noticed the file permission problem on rc.local ...

thanks for your statement and
best wishes

[mflorell]

I completely disagree with the anti-RAID statement, especially the partial endorsement of RAID-5(which is the worst form of RAID and should only be used for archive servers, and only if you can't do RAID-6).

Every server we sell and deploy has at least a RAID-1 with two drives. Database servers should use a RAID-10 with a caching hardware raid controller if possible. We have found this to be an extremely robust and safe way of running a system that offers instant and non-crashing redundancy should a drive fail.

Yes, you should backup your system at regular intervals(which is why I wrote the ADMIN_backup.pl script), but RAID is a very good idea on a ViciDial system.

To reiterate how bad RAID-5 is here is a website dedicated to it:
http://www.miracleas.com/BAARF/RAID5_versus_RAID10.txt
http://www.miracleas.com/BAARF/

[williamconley]

my experience with mirrored disks is that the time-loss (the system MUST wait to write all the data twice, there is no exception to this rule in mirroring unless you have a cache that can hold ALL your data at all times) will slow the system and any systemic errors (mysql is prone to them) will be duplicated across the mirrored disk rendering both useless from a "restore" standpoint. While you get single disk failure capacity, you have slowed down your system and for most errors will be stuck with a "recovery" (translates to "hire a geek to sort all this out") anyway.

my experience with RAID5 (or 10) is similar ... BUT with 3 disks, your likelihood of slowing the system down is dramatically smaller. The RAID controller can allow one disk to "catch up" at any given moment in time and in tests most RAID5 systems are FASTER than single disk systems, not slow (which is what tests show in a mirror).

there are all sorts of studies that show cost comparisons per gigabyte and argue over which one is better, but in the end it's not about how much the drive itself cost (consider the cost of the drive to the revenue generated by the system, the drive cost is virtually inconsequential).

I prefer to focus on "did this cool setup save me TIME" (specifically DOWNTIME). while either one will do so, in the case of a dead drive the mirror will slow the entire system down while protecting against this while the RAID5 will not. Neither will protect against MyISAM MySQL scrambles.

in EITHER case, if your system has a fault, the fault (Bad Data caused by MySQL is the most common) will be passed to the "live copies" instantly and you will enter restore/repair mode.

this leads me to my preferred method of a "dormant" mirror. fully installed duplicate of the original with daily data backups written to it and (best case scenario) mysql logs stored on it in case of a requirement for an actual restore "to the minute".

mirror is not "bad", but anything that slows down the system in any way is not good, either. in addition to that the concept that you CAN add LOTS of disks to RAID 5 and have serious redundancy is intriguing, especially with the cost of a 1T drive under $100. you can now have the capability to have multiple drive failures and/or increase your write speed by having those extra disks available (if 2 of the 6 are busy, there are still 4 drives available to write the data this means the RAID controller should never have to wait for a place to put the data, which isn't even true on a single drive system, much less a mirror).

the only downsides for RAID5 can be resolved by stepping up to RAID10 (yet another redundancy) or an offline mirror (in case of data scrambling).

there are also hybrid methods for the truly geeky, where the OS can reside on a RAID5 (since the OS is NOT prone to scrambling like the MySQL MyISAM tables are) and put the MyISAM data on a single freakishly fast drive with offline backup and log shipping. Then (in most expected cases) you'll never experience multiple drive outages and your OS will never need reinstallation and your data is securely backed up via offline mysqldumps and log shipping. And none of it will slow the system in any way.

But that's me. I rarely get a client who will actually pay for RAID unless it comes with the motherboard, and most motherboards only come with MIRROR RAID. Software RAID5 is worse than No RAID, so MIRROR (unless you get an ACTUAL Server Level System) is what they are stuck with. But you gotta wonder why most enterprise level systems are at LEAST RAID5 with a specific card installed just to handle the RAID itself and lots of drives.

A system with no interaction and no downtime (except for a technician popping out the dead drives and popping in the fresh ones whenever one dies, after all they are wear parts) required is the ultimate goal. No "hire a guy to repair my data" should be required. But IF this is a requirement because of a freaky MyISAM moment, an up-to-date perfect copy (in whatever format) is essential for a (for instance) bank. NO data loss.

RAID5 (or above), offline log shipping. No Data Loss. No slow down.

But that's me. LOL (I don't have Matt's Customers or experience, most of my RAID customers are on Microsoft OSs and have RAID5 because MSSQL never seems to scramble its data, and if it does it generates a clean copy from the logs quite nicely, not quickly but nicely.)

[mflorell]

RAID on Windows is a whole different animal compared to RAID on Linux.

On Linux, the RAID-1 performance hit is usually from 5% to incalculable it's so small. We have run tests with a single disk system and a RAID-1 system and noticed no change in the average load. Now when you run something with crazy writes going on (like dedicated MySQL server for 200 ViciDial seats) and you are using a SOFTWARE RAID-1 you can see inflated load and a performance hit, but for a DB you should be using a hardware caching RAID controller anyway which eliminates that performance hit.

Also, RAID-10 is much faster than a single disk, because the data writes to two partitions at the same time, depending on how you set one up(hardware or software) you can get from 5-40% speed increase using RAID-10.

RAID-5 is faster the first time you write through, but over time and for many small writes it can get VERY slow, even using a hardware caching controller. Never use RAID-5 for a DB server.

[ronator]

hello everyboy,

thank you for this interesting conversation. Normally, I'd mention some thoughts here after re-reading the postings, but today is my last work day before holiday and in the evening I'll be sittin' in a plane on the way to spain, so please forgive me not joining the conversation. I truly need some time to recover my energy ...

in this sense : have a good time !

all the best wishes,
r0nator