Server Crashed and Now unusable

[brainwhistler]

Hi,

I have single server setup of ViciDial/ Asterisk:

Asterisk 1.2.24
ViciDial 2.0.4

Last night my server crashed due to sudden power failure and when the server started its hard drive keeps busy making the server deadly slow.

After running the following commang

/usr/share/astguiclient/AST_VDauto_dial.pl --debugX

I received the following DB error:


[root@localhost ~]# /usr/share/astguiclient/AST_VDauto_dial.pl --debugX
TIME DEBUG: |1|1| GMT: 05:54
2008-05-13 06:54:03|LOGGED INTO MYSQL SERVER ON 1 CONNECTION||
TIME DEBUG: -5|-4|1| GMT: 10:54

DBD::mysql::st execute failed: Can't open file: 'vicidial_live_agents.MYI'. (errno: 144) at /usr/share/astguiclient/AST_VDauto_dial.pl line 308.

executing: SELECT user,server_ip,campaign_id,conf_exten,status FROM vicidial_live_agents where status IN('READY','QUEUE','INCALL','DONE')

and server_ip='x.x.x.x' and last_update_time > '20080513065354' order by last_call_time Can't open file: 'vicidial_live_agents.MYI'. (errno: 144) at /usr/share/astguiclient/AST_VDauto_dial.pl line 308.
[root@localhost ~]#


It seems that the table 'vicidial_live_agents' got corrupted.

Does anyone know how fix this table.

Thanks

[mflorell]

bring MySQL server down "mysqladmin shutdown" and go to the mysql data directory where all of those MYI files live and run myisamchk on all of them then start up mysql and "optimize table" all of the tables, then you should be fine. Something like this:

cd /usr/local/mysql/var/asterisk/
myisamchk -r ./call_log.MYI
myisamchk -r ./conferences.MYI
myisamchk -r ./inbound_numbers.MYI
myisamchk -r ./live_channels.MYI
myisamchk -r ./live_inbound.MYI
myisamchk -r ./live_inbound_log.MYI
myisamchk -r ./live_sip_channels.MYI
myisamchk -r ./park_log.MYI
myisamchk -r ./parked_channels.MYI
myisamchk -r ./phone_favorites.MYI
myisamchk -r ./phones.MYI
myisamchk -r ./recording_log.MYI
myisamchk -r ./server_performance.MYI
myisamchk -r ./server_updater.MYI
myisamchk -r ./servers.MYI
myisamchk -r ./system_settings.MYI
myisamchk -r ./vicidial_agent_log.MYI
myisamchk -r ./vicidial_auto_calls.MYI
myisamchk -r ./vicidial_call_times.MYI
myisamchk -r ./vicidial_callbacks.MYI
myisamchk -r ./vicidial_campaign_agents.MYI
myisamchk -r ./vicidial_campaign_hotkeys.MYI
myisamchk -r ./vicidial_campaign_server_stats.MYI
myisamchk -r ./vicidial_campaign_stats.MYI
myisamchk -r ./vicidial_campaign_statuses.MYI
myisamchk -r ./vicidial_campaigns.MYI
myisamchk -r ./vicidial_campaigns_list_mix.MYI
myisamchk -r ./vicidial_closer_log.MYI
myisamchk -r ./vicidial_conferences.MYI
myisamchk -r ./vicidial_dnc.MYI
myisamchk -r ./vicidial_hopper.MYI
myisamchk -r ./vicidial_inbound_group_agents.MYI
myisamchk -r ./vicidial_inbound_groups.MYI
myisamchk -r ./vicidial_ivr.MYI
myisamchk -r ./vicidial_lead_filters.MYI
myisamchk -r ./vicidial_lead_recycle.MYI
myisamchk -r ./vicidial_list.MYI
myisamchk -r ./vicidial_list_pins.MYI
myisamchk -r ./vicidial_lists.MYI
myisamchk -r ./vicidial_live_agents.MYI
myisamchk -r ./vicidial_live_inbound_agents.MYI
myisamchk -r ./vicidial_log.MYI
myisamchk -r ./vicidial_manager.MYI
myisamchk -r ./vicidial_pause_codes.MYI
myisamchk -r ./vicidial_phone_codes.MYI
myisamchk -r ./vicidial_postal_codes.MYI
myisamchk -r ./vicidial_remote_agents.MYI
myisamchk -r ./vicidial_scripts.MYI
myisamchk -r ./vicidial_server_trunks.MYI
myisamchk -r ./vicidial_state_call_times.MYI
myisamchk -r ./vicidial_stations.MYI
myisamchk -r ./vicidial_status_categories.MYI
myisamchk -r ./vicidial_statuses.MYI
myisamchk -r ./vicidial_user_groups.MYI
myisamchk -r ./vicidial_user_log.MYI
myisamchk -r ./vicidial_users.MYI
myisamchk -r ./vicidial_xfer_log.MYI
myisamchk -r ./web_client_sessions.MYI

mysql asterisk

optimize table call_log;
optimize table conferences;
optimize table inbound_numbers;
optimize table live_channels;
optimize table live_inbound;
optimize table live_inbound_log;
optimize table live_sip_channels;
optimize table park_log;
optimize table parked_channels;
optimize table phone_favorites;
optimize table phones;
optimize table recording_log;
optimize table server_performance;
optimize table server_updater;
optimize table servers;
optimize table system_settings;
optimize table vicidial_agent_log;
optimize table vicidial_auto_calls;
optimize table vicidial_call_times;
optimize table vicidial_callbacks;
optimize table vicidial_campaign_agents;
optimize table vicidial_campaign_hotkeys;
optimize table vicidial_campaign_server_stats;
optimize table vicidial_campaign_stats;
optimize table vicidial_campaign_statuses;
optimize table vicidial_campaigns;
optimize table vicidial_campaigns_list_mix;
optimize table vicidial_closer_log;
optimize table vicidial_conferences;
optimize table vicidial_dnc;
optimize table vicidial_hopper;
optimize table vicidial_inbound_group_agents;
optimize table vicidial_inbound_groups;
optimize table vicidial_ivr;
optimize table vicidial_lead_filters;
optimize table vicidial_lead_recycle;
optimize table vicidial_list;
optimize table vicidial_list_pins;
optimize table vicidial_lists;
optimize table vicidial_live_agents;
optimize table vicidial_live_inbound_agents;
optimize table vicidial_log;
optimize table vicidial_manager;
optimize table vicidial_pause_codes;
optimize table vicidial_phone_codes;
optimize table vicidial_postal_codes;
optimize table vicidial_remote_agents;
optimize table vicidial_scripts;
optimize table vicidial_server_trunks;
optimize table vicidial_state_call_times;
optimize table vicidial_stations;
optimize table vicidial_status_categories;
optimize table vicidial_statuses;
optimize table vicidial_user_groups;
optimize table vicidial_user_log;
optimize table vicidial_users;
optimize table vicidial_xfer_log;
optimize table web_client_sessions;

[brainwhistler]

The table has been fixed. Thanks a lot.

But the Hard drive busy issue is still there. It reboots normally and gradually takes up the system and response deadly slow.

The memory usage is below the Physical memory. Processor usage stays between 5-20 %.

Dont know which is taking up the Hard Drive. Is there any issue with the recording files or some log file got corrupted.

thanks

[mflorell]

There could be any number of problems,

what kind of hard drive(s) are you using?

What is the filesystem? (ext2/ext3/reiser/xfs/etc...)

[brainwhistler]

Its IDE 7200 RPM, and filesystem is ext3.

There is not load on the system, max 5 users at this moment.

Everything was working fine before the crash due to power failure.

This is the avg load of the system

1.70 1.15 0.52 1/129 4203 (at boot time)

then

27.97 11.87 4.80 1/149 4815 (after some time)

[mflorell]

Ouch, that load hurts to look at.

I would suggest booting to a boot CD/DVD and running:

e2fsck -vfy /dev/hda0

might need to change that for your device setup and options on the version of e2fsck that you have on the boot disk, but it might be a problem with corrupt data on the hard drive.

What is the Linux distro?

[brainwhistler]

Red Hat AS4

[mflorell]

I don't know the e2fsck version that would be using, but you can jut google e2fsck and find out. I would recommend running it tonight if you can.

[brainwhistler]

Below is the error while running the command

[root@localhost /]# e2fsck -vf -y /dev/hdc2

e2fsck 1.35 (28-Feb-2004)
Couldn't find ext2 superblock, trying backup blocks...
e2fsck: Bad magic number in super-block while trying to open /dev/hdc2

The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>




It seems both superblock and alternate superblock got corrupted.

Any idea ???

Also, I googled for this error, a guy advised to run mke2fs with -S option. Any idea ??? I dont want to destroy all the data at this stage.

thanks

[mflorell]

are you sure hdc2 is the right partition for your root partition?

I would try every combination of partition possible:

e2fsck -vf -y /dev/hda0
e2fsck -vf -y /dev/hda1
e2fsck -vf -y /dev/hda2
e2fsck -vf -y /dev/hdb0
e2fsck -vf -y /dev/hdb1
e2fsck -vf -y /dev/hdb2
e2fsck -vf -y /dev/hdc0
e2fsck -vf -y /dev/hdc1
e2fsck -vf -y /dev/hdc2
e2fsck -vf -y /dev/hdd0
e2fsck -vf -y /dev/hdd1
e2fsck -vf -y /dev/hdd2