Split Dialer: DBI Error

[bmorrison]

We moved our well-behaving vicibox split to a co-location. Everything seems to be fine for the Web/DB server. Then I check boot.msg on the asterisk-only box:

Code: Select all

DBI connect('asterisk:XX.XXX.XXX.130:3306','cron',...) failed: Can't connect to MySQL server on 'XX.XXX.XXX.130' (113) at /usr/share/astguiclient/AST_reset_mysql_vars.pl line 65
Couldn't connect to database: Can't connect to MySQL server on 'XX.XXX.XXX.130' (113) at /usr/share/astguiclient/AST_reset_mysql_vars.pl line 65.
Starting Asterisk DBI connect('asterisk:XX.XXX.XXX.130:3306','cron',...) failed: Can't connect to MySQL server on 'XX.XXX.XXX.130' (113) at /usr/share/astguiclient/start_asterisk_boot.pl line 59
Couldn't connect to database: Can't connect to MySQL server on 'XX.XXX.XXX.130' (113) at /usr/share/astguiclient/start_asterisk_boot.pl line 59.
failed

What did I mess up? Okay, let's see if I can connect to the DB server from the CLI:

Code: Select all

mysql -u cron -p -h XX.XXX.XXX.130
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1190
Server version: 5.0.67-log SUSE MySQL RPM

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> quit
Bye

Uh. Okay. Let's see if this works:

Code: Select all

 /usr/share/astguiclient/AST_reset_mysql_vars.pl
 - conferences reset
 - vicidial_conferences reset
 - vicidial_manager delete
 - vicidial_auto_calls delete
 - vicidial_live_agents delete
 - vicidial__users delete
 - vicidial_campaign_server_stats delete
 - vicidial_hopper delete

So far so good, how about:

Code: Select all

/usr/share/astguiclient/start_asterisk_boot.pl
started screen
changed directory
started new screen session
detached screens
raised ulimit open files
Asterisk started... screen logging on

So it works? But, this unit fails to connect to the remote mysql server on every reboot. I'm obviously missing something. I can hack the mysql_vars.pl and boot.pl to run every morning via cron, but I'd like the unit to boot correctly each and every time.

What am I missing? As always, please excuse whatever obviousness has exposed my newbie failure to grasp what must be a simple solution.

[williamconley]

when you post, please post your entire configuration including (but not limited to) your installation method, vicidial version and build, asterisk version, telephony hardware (model number is helpful here), cluster information if you have one, and whether any other software is installed in the box.

Similar to This:
Vicibox X.X from .iso | Vicidial X.X.X Build XXXX | Asterisk X.X.X | Single Server | No Digium/Sangoma Hardware | No Extra Software After Installation

this IS a requirement for posting along with reading the stickies (at the top of each forum) and the manager's manual (available on EFLO.net, both free and paid versions)
________

it may be firing that script before other processes are ready. in which case it may be a failure only temporarily. but if it only attempts to fire once ... then it's dead. how often does this script run?

if that script only needs to run during boot, it may be best to run it on the db server only, or change it to a timed daily script instead of boot, but you'd have to see if it was daily maintenance or repair to decide when to run it.

[bmorrison]

These were installed from vicibox 2. So what's that like Asterisk 1.4.27.1-vici and I upgraded these boxes to the latest version of 2.4 from SVN.

These are the scripts that are failing at boot.

/usr/share/astguiclient/AST_reset_mysql_vars.pl

and

/usr/share/astguiclient/start_asterisk_boot.pl

As I showed in my email, they failed at boot. Asterisk doesn't start and it's an ugly mess. But I can then log into the box, run exactly the same scripts, and everything is fine. Before we moved these boxes to their new co-location, these errors did *not* appear in boot.msg and asterisk would happily start up.

UPDATE: Now only AST_reset_mysql_vars.pl errors on boot:

Code: Select all

DBI connect('asterisk:XX.XXX.XXX.130:3306','cron',...) failed: Can't connect to MySQL server on 'XX.XXX.XXX.130' (113) at /usr/share/astguiclient/AST_reset_mysql_vars.pl line 65
Couldn't connect to database: Can't connect to MySQL server on 'XX.XXX.XXX.130' (113) at /usr/share/astguiclient/AST_reset_mysql_vars.pl line 65.
Starting Asterisk started screen
changed directory
started new screen session
detached screens
raised ulimit open files
Asterisk started... screen logging on
done

Started Vicidial
done
done
Master Resource Control: runlevel 5 has been reached
Skipped services in runlevel 5: nfs
<notice>killproc: kill(1886,3)

So at least asterisk starts up now. I don't know why, however.

[williamconley]

when you post, please post your entire configuration including (but not limited to) your installation method, vicidial version and build, asterisk version, telephony hardware (model number is helpful here), cluster information if you have one, and whether any other software is installed in the box.

Similar to This:
Vicibox X.X from .iso | Vicidial X.X.X Build XXXX | Asterisk X.X.X | Single Server | No Digium/Sangoma Hardware | No Extra Software After Installation

this IS a requirement for posting along with reading the stickies (at the top of each forum) and the manager's manual (available on EFLO.net, both free and paid versions)

i'm a little unclear on this. how many servers do you have, and is mysql running on the same box?

if it is not, have you verified that you have connectivity during boot? if the mysql server is rebooting at the same time, you must wait for it to reboot successfully before rebooting any dialers. also, this script may be a "database cleanser", which would then be necessary only on the mysql machine (if it is a one-shot run during boot and only performs a task on the sql server ... it should probably only be running on the mysql server?)

and where did you get your clustering instructions (a link is nice in case someone else has similar issues from the same instruction set).

also "vicibox 2" is not detailed enough, the version of the .iso is required. We also need the version with build of your final Vicidial (the only version information that is detailed is Asterisk, which is a start, let's get all of it).

[bmorrison]

Vicibox 2.0.0 that Kumba thoughtfully released. These units are running opensuse 11.2. The version of vicidial is the latest off trunk as of last Friday.

This is a split. One DB/Web and one asterisk.

There isn't a connectivity issue, as you can see I can remotely connect to the DB server's mysql using using the cli. All the scripts that fail at boot can run without issue moments after boot.

This is really strange, and if I have to edit which scripts run at boot, it's no big deal but there is *something* going on.

[williamconley]

Vicibox 2.0.0 that Kumba thoughtfully released. These units are running opensuse 11.2. The version of vicidial is the latest off trunk as of last Friday.

please do not use "the latest" in lieu of listing the version (and build for vicidial). this is not just for the "help you right now" portion, this is a post that will still be on this forum next year. if another person experiences the same issues as you ... "latest version as of today" will not help much.

This is a split. One DB/Web and one asterisk.

thanks for listing

There isn't a connectivity issue, as you can see I can remotely connect to the DB server's mysql using using the cli. All the scripts that fail at boot can run without issue moments after boot.

The fact that you can reach mysql later does not mean there isn't a connectivity issue when the script runs. that's an assumption (which is rarely a good idea when troubleshooting )

do you reboot the sql server at the same time or is it already "up"? (you didn't address that question from my post)

where did you get your clustering insructions? (you didn't answer that from my post)

if you want to be sure you have connectivity at that moment in the vici scripts, add another line that runs a command line mysql stmt to change a test value in a log table in vici (with those same credentials). if your networking is not "online" yet when this script runs, networking won't run.

[Kumba]

Are you rebooting the database server at the same time you reboot the dialer?

The only other thing I can think of is that your vicidial system is up and running before network has completely initialized (slow managed switch port negotiation, slow dhcp, etc).

[bmorrison]

This isn't a cluster. Short trunks are not being filled by a script. This is just a straight split.

In any case, the DB server is *not* being rebooted at the same time. The DB server is rebooted a half hour before the asterisk box.

These two boxes happily lived for months without any boot.msg errors. This started only on the asterisk box after the move to the colo. On a lark I upgraded just to make sure that wasn't the issue. It had no effect on this really strange problem.

I might try to through a pause into the runlevel and see if that cleans up the log.

[bmorrison]

Are you rebooting the database server at the same time you reboot the dialer?

The only other thing I can think of is that your vicidial system is up and running before network has completely initialized (slow managed switch port negotiation, slow dhcp, etc).

Yeah, I suspect a slow managed switch too. There's a cisco we had the colo install and set up that I think is the source of the issue.

[bmorrison]

Changes starting on lines 322 of /etc/init.d/vicidial:

Code: Select all

        sleep 10

        $VICI_DIR/AST_reset_mysql_vars.pl

        sleep 10

I changed the sleeps from 2 to 10. This fixed the problem for the box.

[williamconley]

Posted: Mon Aug 16, 2010 10:39 am

if your networking is not "online" yet when this script runs, networking won't run

at least we're all on the same channel now.

THANK YOU for posting your solution, somebody's gonna owe you dinner!

[Kumba]

Yeah, my core switch as my colo is like that. It can take upwards of 30 seconds for it to turn the port loose on the general network. It's busy doing port discovery, and seeing if anything on what was just plugged in conflicts with what is already on the network.

I've ran into the same issue before where I thought I had a bad server/NIC when it was just the switch doing it's thing.