Vicidial 8.1 not working properly

[manos]

Hello folks.. we are a small call center with a max of 30 operator. Until a month before we were using Vicibox 4.0.0 and it worked well. We decided to buy a new server with these specifics:
CPU - Intel(R) Xeon(R) CPU E5-2650L 0 @ 1.80GHz 32 cores
RAM: 64Gb
Asterisk: Asterisk 13.21.1-vici
VERSION: 2.14-692a
BUILD: 180927-0018
Vicibox 8.1 iso
install mode express

The prb with this version is that each day after reboot it begin working good but after a while it starts with the problems, it sends two calls simultaneously, the call don't hang up sometimes but it happens rarely, the calls sometimes don't get recorded, but the biggest problem for us is that we have a back office that receive inbound transfer call from the other operators to fix an appointment but like i said when the problems starts the calls don't get transferred. The operator clicks the "Quick Transfer" button, but the calls stays at his side, vicidial even shows the status table for that specific call but the call is still on operator side. When you click a status of the call then the call will hangup.

First thing that i thought about is the my.cnf file and from the default configuration i changed these options to the one written for optimized hardware specs since the server is powerful enough:

### Change the following if you have the Optimized HW Spec for a dedicated database/slave
# key_buffer_size = 16G
# max_connections = 4096
# table_definition_cache = 16192
# table_open_cache = 4096
# sort_buffer_size = 8M
# read_buffer_size = 8M
# read_rnd_buffer_size = 32M
# myisam_sort_buffer_size = 256M
# tmp_table_size = 256M
# myisam_repair_threads = 8

then i has noticed that in root email i receive very often a mail that says:
pattern match read eof at /usr/share/astguiclient/AST_conf_update.pl line 272
or
pattern match read eof at /usr/share/astguiclient/AST_conf_update.pl line 438

and when i reboot sometimes this one:
problem connecting to "localhost", port 5038: Connection refused at /usr/share/astguiclient/AST_conf_update.pl line 271

but this doesn't happens all the time, sometimes it can pass an hour and don't receive a mail

plus another thing that i noticed in /va/log/mysql/mysqld.log logs are thess warnings

2018-10-29 14:24:33 139613961099008 [Warning] Aborted connection 307407 to db: 'asterisk' user: 'cron' host: 'localhost' (Got an error reading communication packets)
2018-10-29 14:24:40 139613972313856 [Warning] Aborted connection 307847 to db: 'asterisk' user: 'cron' host: 'localhost' (Got an error reading communication packets)
2018-10-29 14:24:41 139613975041792 [Warning] Aborted connection 307914 to db: 'asterisk' user: 'cron' host: 'localhost' (Got an error reading communication packets)
2018-10-29 14:24:42 139613973223168 [Warning] Aborted connection 307952 to db: 'asterisk' user: 'cron' host: 'localhost' (Got an error reading communication packets)
2018-10-29 14:24:42 139613971101440 [Warning] Aborted connection 307974 to db: 'asterisk' user: 'cron' host: 'localhost' (Got an error reading communication packets)
2018-10-29 14:24:43 139613957158656 [Warning] Aborted connection 307995 to db: 'asterisk' user: 'cron' host: 'localhost' (Got an error reading communication packets)
2018-10-29 14:24:48 139613955946240 [Warning] Aborted connection 308267 to db: 'asterisk' user: 'cron' host: 'localhost' (Got an error reading communication packets)
2018-10-29 14:24:51 139613958067968 [Warning] Aborted connection 308377 to db: 'asterisk' user: 'cron' host: 'localhost' (Got an error reading communication packets)
2018-10-29 14:24:52 139613964736256 [Warning] Aborted connection 308426 to db: 'asterisk' user: 'cron' host: 'localhost' (Got an error reading communication packets)
2018-10-29 14:24:52 139613964433152 [Warning] Aborted connection 308453 to db: 'asterisk' user: 'cron' host: 'localhost' (Got an error reading communication packets)


For what i saw in forum the problem related to AST_conf_update.pl is the configuration of manager.conf
Here it is its content:

[general]
enabled = yes
port = 5038
bindaddr = 0.0.0.0

[cron]
secret = 1234
read = system,call,log,verbose,command,agent,user,originate
write = system,call,log,verbose,command,agent,user,originate

[updatecron]
secret = 1234
read = command,reporting
write = command,reporting

eventfilter=Event: CoreShowChannel


[listencron]
secret = 1234
read = system,call,log,verbose,command,agent,user,dtmf
write = command

eventfilter=Event: Shutdown
eventfilter=Event: DTMFBegin
eventfilter=Event: DTMFEnd
eventfilter=Event: NewCallerid
eventfilter=Event: Newstate
eventfilter=Event: Hangup
eventfilter=!Event: HangupRequest


[sendcron]
secret = 1234
read = command
write = system,call,log,verbose,command,agent,user,originate

Hope that you'll help me with the solution.
Thanks in advanced!!

[williamconley]

There is no "8.1". There is "8.1.0/8.1.1/8.1.2" and even "8.0.1" if that's what you mean. Each of those has it's own "bug set" which may or may not be related to your issue.

Each of the errors you mentioned can be tracked down and killed, but the only one I see possibly related is the mysql error. If this is not a multi-server system you shouldn't be having communication problems with the DB. So you may need to research that error (which I've not personally seen). But more importantly on all those errors: Do they occur when you're having your problem?

What about Average Server Load? Does high server load occur just before the problems start? Any othe patterns? Any errors in the /var/log/astguiclient logs? Do you have vicidial logging set to "FILE" so those logs will be populated?

If you imported your old DB into the new system, did you remember to verify ALL the settings in admin->servers and admin->system settings (to account for the new asterisk version and anything else that may have changed)?

[manos]

Thanks for the replay William!

Well the errors are permanent in fact. For example the mysql error starts showing up as soon as the operators begin working and receiving calls.
And as for the Average Server Load i can say that its quit low for the server potential. I thought that too in fact that maybe we missed something during the process of passing the data from the old server to the new one but i have to mention that we only import the users, the list-leads, campaigns, user-group, and sip phones nothing else. As for the system settings it was totally configured manually. I tried to take a look at /var/log/astguiclient too but there are many logs there and i'm a noob to understand all of it. But can you please tell me if i can show you some logs that maybe can help you to give me a direction what else can i do?

[williamconley]

... the mysql error starts showing up as soon as the operators begin working and receiving calls. ...
its quit low for the server potential...

Track down the mysql error. Check network and sockets to see what is causing the issue. It's likely the problem.

And "it's quite low" is not an answer. rough equivalent of "we're having a lot of temperature today". The answer to average server load should be the average server load experienced during one of these problems, and to avoid misunderstandings it's a good idea to state where you got the numbers ("uptime"? vs "real time screen"?)

[manos]

Well what i usually see is uptime and htop but htop in fact mostly, since there i see if the processor is full or not but i think it works very fluently. Right now for example it is: Load average: 6.78 5.03 4.21

[manos]

It happend just right now P They are saying me,, "Look the calls get stack in Zoiper, they don't pass to back office" and as you can see this is the load average right now. The one that i just showed you

There are 12 operators logged in and 111 calls placed for example.
I can tell you something else that we use another server that doesn't has even half the power of this server with vicidial 4.0.0 and works ok.

and the vicibox is v 8.1.2 just saw it

[williamconley]

Load average: 6.78 5.03 4.21

How many cores does this server have? (this is visible in htop, of course)

"Look the calls get stack in Zoiper, they don't pass to back office"

I'm lost. We're talking about Vicidial. Calls don't get to zoiper directly, there's no such thing as back office. So ... try a wee bit more technical description if you could.

[manos]

The server has 32 cores, and the operators don't know much of course and say something like that. Zoiper is the softphone that we use for receiving the live calls from vicidial server to the simple operators that are waiting for outbound calls. Back office are called the two operators that receive these transferred calls from other outbound operators. These simple operators asks the client to stay online and click the Quick Transfer button to pass this call to back office for confirming the appointment with the client. Like i said the calls don't pass to back office (like we call it) and get stack to this and even they continue to hear the conversation and can talk too still with the clients even that the table where you can give the status for the call is shown. After you click a status, lets it be Confer or Callback or Sale or whatever the call gets hanged up.

The call is something like this:

SERVER pass the call to Outbound operator.
Outbound operator ----> talks with the client.

IF they get agree together for product that we sale the call is passed to backoffice to get confered

Outbound operator click Quick Transfer call to pass it to backoffice but it get stack to the operator side.
Hope that i explained it more clear.

[williamconley]

Much better. So what you're saying is the fronters can't transfer calls to the closers during certain situations.

Do they sometimes go through "late" or is it always just "nope, no transfer"?

check for lengthy mysql commands that may be stopping the system from running until they finish.

Code: Select all

mysql -u cron -p1234 -e "show processlist"


[manos]

No in fact is nope no transfer..
I keep seeing the show processlist in fact with a difference, logged as root not as cron user and now that i'm looking at processlist i notice difference at this list comparing it to what is used to see with root but never thought to look at that specific moment. Ill do that to see if i can find something like you are saying.
Thanks!

[manos]

I just made a test and in fact the queries during that moment get finished immediately and don't take any long time to finish the querying process

[manos]

show processlist is showing to many waiting table to lock. You were right in fact.

516549 | cron | localhost | asterisk | Query | 1 | Waiting for table level lock | UPDATE vicidial_log FORCE INDEX(lead_id) set status='AB' where lead_id = '10898379' and uniqueid LIK | 0.000 |
| 516579 | cron | localhost | asterisk | Query | 1 | Creating sort index | SELECT location,recording_log.user,recording_log.length_in_sec,end_time FROM recording_log inner joi | 0.000 |
| 516588 | cron | localhost | asterisk | Query | 1 | Waiting for table level lock | UPDATE vicidial_log set status='NI' where lead_id='4515521' and user='sebastian' and call_date > "2 | 0.000 |
| 516613 | cron | localhost | asterisk | Query | 1 | Waiting for table level lock | UPDATE vicidial_log set user='erina', comments='AUTO', list_id='558', status='INCALL', user_group='A | 0.000 |
| 516646 | cron | localhost | asterisk | Query | 0 | Waiting for table level lock | UPDATE vicidial_log FORCE INDEX(lead_id) set status='AB' where lead_id = '8556262' and uniqueid LIKE | 0.000 |
| 516647 | cron | localhost | asterisk | Query | 1 | Waiting for table level lock | UPDATE vicidial_log set status='FUORI' where lead_id='6751941' and user='vehbina' and call_date > " | 0.000 |
| 516679 | cron | localhost | asterisk | Sleep | 0 | | NULL | 0.000 |
| 516708 | cron | localhost | asterisk | Query | 0 | Waiting for table level lock | INSERT INTO vicidial_log (uniqueid,lead_id,campaign_id,call_date,start_epoch,status,phone_code,phone | 0.000 |
| 516740 | cron | localhost | asterisk | Query | 0 | Waiting for table level lock | SELECT start_epoch,term_reason,uniqueid,campaign_id,status FROM vicidial_log where uniqueid='1540916 | 0.000 |
| 516743 | cron | localhost | asterisk | Query | 0 | Waiting for table level lock | SELECT start_epoch,status,user,term_reason,comments FROM vicidial_log FORCE INDEX(lead_id) where lea | 0.000 |
| 516752 | cron | localhost | NULL | Query | 0 | init | show processlist | 0.000 |
+--------+------+-----------+----------+---------+------+------------------------------+------------------------------------------------------



What is recommended to do in these cases?

[williamconley]

No in fact is nope no transfer..

Wait now: No transfers AT ALL is different than what you described before where agents would suddenly say "look, calls are stuck now".

If transfers don't happen at all, you have either an installation or configuration issue. If transfers happen for a while and then stop working for a while, you could have a problem with a delay which may be caused by a mysql delay. But we have to know which way to go here before continuing.

[manos]

Maybe was an installation problem or something like that, The transfers happen for a while and then stop working. We installed the version 8.0.0 and now is working perfectly.
Thanks for you time William, really appreciate it.