ASTGUICLIENT and VICIDIAL lost access to asterisk.

[AIRAM]

Had a running server with version 1.1.11 and sudenly VICIDIAL stopped connecting calls to agents. After further check seems as if it lost connection to manager interface. There doesn't appear to be any errors on the CLI even with full debug and verbose. And the AGC is not able to place calls or connect conferences.

Tried upgrading to the snapshot released today (several nice new features) but still have the same problem.

Any ideas what could have gone wrong. I was not doing any changes to the server itself. The only change done was to add a couple of users and disabled user selectable on two statuses so agents could not access them.

[mflorell]

Is your AST_VDauto-dial.pl script running?

What is the output when you run "screen -r"?

[mr_mehul_shah]

Check`s your disk space of the system. Also check that your database didn`t get corrupted due to harddisk full.

Update us about your problem.

Regards

Mehul

[AIRAM]

It was the vicidial_manager table that somehow got corrupted, though it was not due to lack of space.
Didn't respond earlier because had another server that crashed on Saturday as well this one was running version 1.1.9 and actually ran out of space (recently enabled recordings on this one) but this was a production server so was more important to get up.

Thanks again for the help will test this one since I already upgraded it to the snapshot just released.

[AIRAM]

Well seems like the problem reocured. Yesterday on the afternoon about the same time the tables got corrupted again but this time more tables were affected. I restored them but today all morning I'm seeing a problem with the AST_VDauto_dial.pl script constantly crashing.

When it crashes it displays the following error:
Cannot determine peer address at /usr/lib/perl5/site_perl/5.8.5/Net/MySQL.pm line 277

I have seen this error previously and it seems to be related to wrong version of MySQL.pm since I just installed the latest snapshot I thought that maybe in the process of doing that it replaced that file; so I copied from the lib folder that comes with astguiclient.

I'll be monitoring to check if the datgabase corruptiong could be related to the MySQL.pm problem.

Do you have any ideas on what might be wrong and if something was changed on the MySQL authentication on this new snapshot?

Thanks for any assitance on this matter.

[mflorell]

You should make sure that you are using 0.08 version of Net::MySQL, not the most current version(0.09) There are instructions in the SCRATCH_INSTALL for installing 0.08.

This could very well be your problem. You need to make sure you install 0.08 through cpan according to the instructions, then restart the scripts and all should work.

[AIRAM]

Actually I tried both methods suggested on the scratch install. Copying the file from /home/cron/astguiclient/Net/ and installing from CPAN with:

install O/OY/OYAMA/Net-MySQL-0.08.tar.gz

Definetelly I have version 0.08; here's a list of all perl modules and their version:

[root@asteriskdev ~]# perl version.pl
Archive::Tar -- 1.29
Archive::Zip -- 1.16
CPAN -- 1.87
Compress::Zlib -- 1.41
Cwd -- 3.18
Digest::MD5 -- 2.36
Digest::SHA -- 5.39
Digest::SHA1 -- 2.11
File::HomeDir -- 0.58
File::Temp -- 0.16
IO::Zlib -- 1.04
MD5 -- 2.03
Module::Signature -- 0.54
Net::MySQL -- 0.08
Net::Telnet -- 3.03
OLE::Storage_Lite -- 0.14
PAR::Dist -- 0.09
Perl -- 5.8.5
Spiffy -- 0.30
Spreadsheet::ParseExcel -- 0.2603
Term::ReadKey -- 2.30
Term::ReadLine -- 1.01
Test::Base -- 0.50
Test::Simple -- 0.62
Text::Glob -- 0.06
Time::HiRes -- 1.87
YAML -- 0.58
asterisk-perl -- ???

I still have problems where every now and then I will get an error from /home/cron/AST_manager_kill_hung_congested.pl it is the same error:

send: Cannot determine peer address at
+/usr/lib/perl5/site_perl/5.8.5/Net/MySQL.pm line 277

Sometimes the autodialer will hang for about a minute (meaning that it stops processing and sending calls to agents) and at that point if you try to login it will not dial out to connect the user to the conference then it starts right away by itself.

On some of the ocassions it will corrupt the vicidial_manager table so I need to do a restore to the table before normal operation restarts.

[mflorell]

I have never had that occur on any VICIDIAL server I've ever set up.

Have you run a memory utility on your server like memtest to rule out a RAM corruption?

We had a problem with bad RAM that took us weeks to figure out that it was bad RAM. Once we swapped out the RAM we had no more problems.

Could you tell us a little more about the hardware and software/versions of MySQL?

[AIRAM]

This is a Dell 2650 server with 1x2.4 IntelXeon processor and 2GB of memory.

MySQL ver:
Ver 14.7 Distrib 4.1.20, for pc-linux-gnu (i686) using EditLine wrapper

When the sript crashes the loadavg is usually around 1.12 and using only around 475MB of mem.

I will run H/W tests and/or see if can get similar server and copy image to another server.

Thanks again.

[mflorell]

Have you tried installing base MySQL from source?

I haven't used Editline Wrapper with MySQL, it might not like the large number of transactions that VICIDIAL is trowing at it.

[AIRAM]

It seemed like we did have some type of H/W problem. We moved to a different server and the errors seemed to lower, we also stopped seeing a lot of weird MySQL transactions with unrecognized characters.

We have replaced the server and seems to be more stable. We did have to create all tables from scratch since the tables were extremly corrupted by the time we changes servers.

This version was built according to the scratch install the only difference might have been the place I got it from since the link on the scratch install was not available when I got it so simple got one from MySQL mirrors. We will look into the Base MySQL as well. Since we still got a table corruptions today again in the vicidial_manager table.

[AIRAM]

Well we ended up rebuilding the server from scratch. Seems like too much data was corrupted already. Now it's been running perfectly for the last few days, lets hope it stays that way.

Also I will suggest to ad --with-readline to the ./configure line of the scratch install since the MySQL default configure will install Editline Wrapper.

Thanks again for all the help and again great work on the new release, realy like the new auto recycle feature and thanks for placing the link to reports on each report page.

[mflorell]

Thanks for the note, I will add --with-readline to the SCRATCH_INSTALL doc.

Thank you again for the compliments. Keep letting us know how you are doing.