How to increase performance on asterisk servers

[bobbymc]

Run the AST_update.pl scritps on a seperate server.. pissibly the server that runs AST_VDauto_dial.pl runs on or maybe the database server.

You can have the keep_alive scrript look through the servers table and use them to create and maintain the AST_update.pl processes. this way you can keep the load down on the asterisk servers to handle more calls =)

example snippets:

Code: Select all

use DBI;

$dbhA = DBI->connect("DBI:mysql:$VARDB_database:$VARDB_server:$VARDB_port", "$VARDB_user", "$VARDB_pass")
 or die "Couldn't connect to database: " . DBI->errstr;


my @server_list;
my $server_list_count = 0;
### Grab all server to run AST_update.pl on
$stmtA = "SELECT server_ip FROM servers where server_ip != '$VARserver_ip' and server_description not like \"%master%\";";
$sthA = $dbhA->prepare($stmtA) or die "preparing: ",$dbhA->errstr;
$sthA->execute or die "executing: $stmtA ", $dbhA->errstr;
$sthArows=$sthA->rows;
$rec_count=0;
while ($sthArows > $rec_count)
        {
        @aryA = $sthA->fetchrow_array;
        $server_list[$rec_count]=               $aryA[0];
        $rec_count++;
        }
$sthA->finish();


Code: Select all

$i=0;
foreach (@psoutput)
{
chomp($psoutput[$i]);
if ($DBX) {print "$i|$psoutput[$i]|     \n";}
@psline = split(/\/usr\/bin\/perl /,$psoutput[$i]);


        $server_list_count = 0;
        foreach (@server_list)
        {
                if ($psline[1] =~ /$REGhome\/AST_update\.pl $server_list[$server_list_count]/)
                        {
                                undef $server_list[$server_list_count];
                                if ($DB) {print "AST_update $server_list[$server_list_count] RUNNING:              |$psline[1]|\n";}
                        }
                $server_list_count++;
        }

        if ($psline[1] =~ /$REGhome\/AST_manager_send_hangup\.pl/)
                {
                $runningAST_send_hangup++;
                if ($DB) {print "AST_send_hangup RUNNING:                |$psline[1]|\n";}
                }
  .....


Code: Select all

@psoutput2 = `/bin/ps -o "%p %a" --no-headers -A`;
$i=0;
foreach (@psoutput2)
        {
                chomp($psoutput2[$i]);
        if ($DBX) {print "$i|$psoutput2[$i]|     \n";}
        @psline = split(/\/usr\/bin\/perl /,$psoutput2[$i]);



        $server_list_count = 0;
        foreach (@server_list)
        {
                if ($psline[1] =~ /$REGhome\/AST_update\.pl $server_list[$server_list_count]/)
                        {
                                undef $server_list[$server_list_count];
                                if ($DB) {print "AST_update $server_list[$server_list_count] RUNNING:              |$psline[1]|\n";}
                        }
        }




Code: Select all

$server_list_count = 0;
foreach (@server_list)
{
        $server_list[$server_list_count];
        if ($DB) {print "starting AST_update on server $server_list[$server_list_count]...\n";}
        if($AST_update > 0)
        {
                `/usr/bin/screen -d -m -S $server_list[$server_list_count] $PATHhome/AST_update.pl $server_list[$server_list_count]`;
        }
        $server_list_count++;
}

if ( ($AST_send_listen > 0) && ($runningAST_send_hangup < 1) )
        {
        #if ($DB) {print "starting AST_manager_send_hangup...\n";}
        # add a '-L' to the command below to activate logging
        `/usr/bin/screen -d -m -S ASTsend_hangup $PATHhome/AST_manager_send_hangup.pl`;
        }



[williamconley]

removing that one script that may take very little cpu seems like an unlikely way to increase output on the dialers.

have you studied the server to see which processes use the most "continual" cpu and which ones have the highest Peak usage?

[bobbymc]

of course thats how i came to this conclusion. the ultimate goal would be to setup a farm of FASTagi servers to handle all the perl and agi scripts to be executed so the asterisk server will do nothing but handle voip requests. In my opinion this would save the asterisk box quite a load. what do you guys think?

[williamconley]

Sounds interesting. Perhaps you should give it a shot (Not to the level of Farm, but on a single setup ...). See if you can run the perl/agi scripts on a box otherwise unrelated to the Vicidial system (ie: not running mysql, apache, or asterisk) and see if that both Works and Decreases the load on your dialer to allow more simultaneous calls. Perhaps that could end up being a 4th server type (DB/Web/Asterisk/Scripts).

But you should show the actual CPU usage under both circumstances (without a baseline, there's no way to tell if you have improved anything).

Of course, then there comes the requirement to have that server online ... and when it reaches "full", another of that type. So the question then is whether it's simpler to just have another dialer when the dialer fills without this 4th serve type ... and/or more cost-effective. If a single "Script" server could "handle" 10 "Dialers" before overloading, and allow each dialer 20% more capacity (agents/calls), with the same hardware cost as the Dialers ... that's cool. But if it only handles 5 ... the result is break even (but lots of extra work).

[bobbymc]

good point. i will investigate more and come back when i have some hard evidence.

[mcargile]

Don't forget the load / network traffic caused by connecting across the network rather than to local host. For instance I wrote a version of the FastAGI logging script that could accept calls from any server. It worked, but on an overloaded network it starts having issues. Calls end up being delayed because some manager decided to download torrents at work, or because the CEO is watching HD porn from his office (both I have seen before). Network problems are something that we run into frequently. Having a customer pay for a large complex cluster setup only to have them bitch and moan about the how crappy it is performing because their network is not up to the task is rather annoying and leaves a bad taste in peoples mouth. Admittedly the network needs to be fixed, but if we can mitigate the problems such things cause we do so. There are a number of things in Vicidial that could improve performance, but the reliability, especially in harsh environments, goes down.

I think you are focusing on improving the performance of small meaningless elements, when there are much bigger hogs on the system. First off the outbound and inbound agi scripts are just regular agis. Every time a regular AGI script gets calls, the kernel has to launch a new instance of it. These scrips being perl scripts means that they have to be compiled, then launched. If you do a 'ps -ef' on a heavily loaded outbound system you will see quite a few of these AGI scripts. Converting these over to FastAGI scripts like what Matt did with the call_log should cause a noticeable performance improvement. A FastAGI script is started on boot and can handle multiple concurrent AGI requests. This means that the load from start up is not there.

[mflorell]

When I first wrote vicidial I set it up so all scripts could be run from other servers, and it worked just fine at the time, but it became too difficult to manage and replicate setups, as well as the load issues, and what would happen if the single process server went down. In the interests of simplicity, redundancy and overall reliability I moved all scripts to the local asterisk machine early on. It isn't always the perfectly efficient solution, but on the whole it is a good solution that has served the project well and made multi-server systems easier to set up, grow and maintain as well as better redundancy for when a server goes down.

[bobbymc]

i understand and both of you make sence.. but since i also maintain the network the idea of a farm of servers handleing the FastAGI calls and possibly converting the outbound/inbound agi scritp into FastAGI would not be a bad idea.. it would require a farm of FastAGI servers and a proxy infront of it that has heart beat incase the proxy goes down there is a backup. this would fix the redundant and scale-ability issue but might cause a management nightmare for some.

[Op3r]

Let me chime in on this about the "network issues between the servers"

They have their own network segment. 1) Public IP 2) Local IP 3) VICI IP.

More cumbersome to setup and prone to major fups if not implemented properly but it gave me so much time for facebook and then some.

Of course nothing beats a very good leads. Dialing with just 1:1 ratio is like sipping that very cool coconut water while on a tropical beach instagramming.