Hopper QUEUEd calls

[proper]

I am working on solving strange problem with Queued calls in hopper growing in number for servers without agents, wanted to see if anyone has encountered this.

Vicidial tags calls in the hopper with QUEUE to schedule them to be called by specific servers, once call is complete they should be removed, however, for "dialer-only" attached servers they seem to stick around and not get properly removed after call has been processed. Causing hopper size to baloon overtime.
There must be a process that does this.

Vici version is 2.12-552a and asterisk is 11.22, William do NOT ask me to post my full configuration.

If you have understanding of how QUEUE status calls are removed from the hopper for dialer only servers, I would greatly appreciate the help.

Update
It seems that the issue has to do with rate at which calls are being added to QUEUE and not the rate at which they are being removed

[mflorell]

I haven't run into this before, and I have worked on many clusters with dialer-only servers.

How did you install and configure those servers?

[proper]

Servers were installed using vicibox iso, connected using standard cluster formation process using expert installation and cluster performs as expected with this one exception.

Looking further in to this issue it seems that too many calls are being scheduled to be called, while pace of dialing is slower. So if system schedules 15 calls for server every 10 seconds and server calls 11 in that 10 seconds 4 calls would just hang around in QUEUE and never get called, because system will add 15more.
As time goes by this continues to accumulate.
This emerged only when I began offloading significant number of calls to dialer only servers, but does happen on any server that is dialer only without agents at any call volume.
Server load is not an issue ether, it happens at 5 or 80 trunks, just at different pace.

I will be checking if this issue is present on other deployments

I feel like there is a logic to keep tap on this, to check how many calls are in QUEUE for given server before adding more in to QUEUE but it is not working. I guess I will have to deconstruct the whole process to understand where it goes wrong.

[mflorell]

Just so we have a better idea of how everything is configured:
- Are you using Remote Agents at all?
- Are there any common elements to the leads that are stuck in QUEUE status, like number of digits in the phone number or invalid phone numbers?

[proper]

No remote agents.

I am working finding common element for these leads, they look completely normal, just not dialed, I will let you know if I find something.
Strangest thing is that if there are agents on the server, these calls get cleaned up properly, something runs differently when no agents are present on the server.

[proper]

I was able to pull apart the way system works and found the issue to be caused by configuration conflict. Keep alive 7 - process responsible for fill of calls was also running on archive server(sight).
Basically calls fill was processed twice, which caused more calls to be added to QUEUE than necessary, Simple thing but never expected for this process to hide on the archive server, which I never bothered to check, until i saw it was the culprit.

If you are someone who ran in to similar issue - ssh in to your machine and open /etc/astguiclient.conf look for line VARactive_keepalives => 1234567etc make sure only one server on the cluster runs process 7

And dont forget to check your archive server as well

Update after watching it for some time, I still see small number of calls cumulating overtime so I will look further in to how balancing is done and post if I find anything. If you have a cluster with dialer only server attached run this SQL query to see fi you are accumulating these calls
SELECT `user`, count(*) FROM `vicidial_hopper` WHERE `status`='QUEUE' GROUP BY `user`


Update- Fully resolves after cluster reboot.

[mflorell]

Thanks for posting your resolution!