vicidial.org

Hi,

I am having a vicidial cluster of vicibox 10.0.1 and asterisk 13. The problem is the asterisk is not reloading automatically after making any changes into phones or careers or inhroups. AGI script is not reloading automatically.

vicibox: VERSION: 2.14-859a
Vicidial : 10.0.1
Asterisk : Asterisk 13.38.2-vici

This could be caused by a number of things, couple things I would check right away would be:

1. Crontab has ADMIN_keepalive_ALL.pl set to run every minute on every server.
2. The VARactive_keepalives are set correctly on each server within /etc/astguiclient.conf (bear in mind if making changes that keepalives 5, 7, 9 and E must only be active on one of the servers within the cluster)
3. Server entries in the GUI have the correct IP, asterisk version listed and have the Active Asterisk server set to Y and Generate Conf Files set to Y.
4. The AMI user credentials are correct for the server entry and match what's in /etc/asterisk/manager.conf

If this all looks good, I'd comment out temporarily the ADMIN_keepalive_ALL.pl crontab entry so I can run it manually after making some changes that would trigger an asterisk reload (such as changing phone details).
I'd make some changes to a phone or something else that would trigger a reload.
I'd then check the server had "Rebuild Conf Files" set to "Y" (This should happen automatically when you make changes affecting the asterisk config for that server).
I'd open two sessions to the server, one to issue the below command, and the other to watch the output of the asterisk cli when I do so. (much easier when no dialing in progress).

Code: Select all

/usr/share/astguiclient/ADMIN_keepalive_ALL.pl --debugX

Then in the output look for "generating new auto-gen conf files", "writing auto-gen conf files" and "reloading asterisk modules" along with any errors that may appear during those processes,and look at asterisk cli to see if there was any authentication failures/errors or if the reload happened.

(If you resolve the issue, remember to uncomment the ADMIN_keepalive_ALL.pl entry in the crontab so it runs every minute as it should).

1) prabinnair: Please remember to include your Vicidial Version With Build. Note that your versions as posted are kinda backwards. Vicibox 10.0.1, Vicidial 2.14-859a, not the other way around, and the Build is missing from the Vicidial version. Onward:

2) Be sure that your Local IP from admin -> servers is the same as the IP in /etc/astguiclient.conf on the server in question. And that the DB server's local IP is also correct and that server has access to make DB requests. Run one from the CLI to verify, just in case something changed after installation. If you have altered your IP address for any servers, be sure to run the IP update script (which is on the console splash screeen when you log in).

3) Then jump into troubleshooting mode as mentioned by jamiemurray.

4) And remember to check the logs! They're often very helpful along with the root user's email.

Hi,

This is the output of the above command. All the above points are correctly set.

vici187:~ # /usr/share/astguiclient/ADMIN_keepalive_ALL.pl --debugX

----- SUPER-DUPER DEBUGGING -----

Asterisk version: 1 13
AST_update set to keepalive
AST_send_listen set to keepalive
AST_VDauto_dial set to keepalive
AST_VDremote_agents set to keepalive
AST_VDadapt set to keepalive
FastAGI_log set to keepalive
AST_VDauto_dial_FILL set to keepalive
ip_relay set to keepalive
Check to see if Timeclock auto logout should run
0| 1 /usr/lib/systemd/systemd --switched-root --system --deserialize 29|
1| 2 [kthreadd]|
2| 3 [rcu_gp]|
3| 4 [rcu_par_gp]|
4| 6 [kworker/0:0H-events_highpri]|
5| 8 [mm_percpu_wq]|
6| 9 [ksoftirqd/0]|
7| 10 [rcu_sched]|
8| 11 [migration/0]|
9| 12 [idle_inject/0]|
10| 14 [cpuhp/0]|
11| 15 [cpuhp/1]|
12| 16 [idle_inject/1]|
13| 17 [migration/1]|
14| 18 [ksoftirqd/1]|
15| 20 [kworker/1:0H-kblockd]|
16| 21 [cpuhp/2]|
17| 22 [idle_inject/2]|
18| 23 [migration/2]|
19| 24 [ksoftirqd/2]|
20| 26 [kworker/2:0H-events_highpri]|
21| 27 [cpuhp/3]|
22| 28 [idle_inject/3]|
23| 29 [migration/3]|
24| 30 [ksoftirqd/3]|
25| 32 [kworker/3:0H-kblockd]|
26| 33 [cpuhp/4]|
27| 34 [idle_inject/4]|
28| 35 [migration/4]|
29| 36 [ksoftirqd/4]|
30| 38 [kworker/4:0H-events_highpri]|
31| 39 [cpuhp/5]|
32| 40 [idle_inject/5]|
33| 41 [migration/5]|
34| 42 [ksoftirqd/5]|
35| 44 [kworker/5:0H-events_highpri]|
36| 46 [kdevtmpfs]|
37| 47 [netns]|
38| 48 [rcu_tasks_kthre]|
39| 49 [kauditd]|
40| 51 [khungtaskd]|
41| 52 [oom_reaper]|
42| 53 [writeback]|
43| 54 [kcompactd0]|
44| 55 [ksmd]|
45| 56 [khugepaged]|
46| 111 [kintegrityd]|
47| 112 [kblockd]|
48| 113 [blkcg_punt_bio]|
49| 114 [tpm_dev_wq]|
50| 115 [edac-poller]|
51| 116 [devfreq_wq]|
52| 117 [watchdogd]|
53| 121 [kswapd0]|
54| 122 [kworker/u13:0]|
55| 124 [kthrotld]|
56| 125 [ipv6_addrconf]|
57| 130 [kstrp]|
58| 135 [charger_manager]|
59| 406 [ata_sff]|
60| 408 [scsi_eh_0]|
61| 409 [scsi_tmf_0]|
62| 410 [scsi_eh_1]|
63| 411 [scsi_tmf_1]|
64| 414 [scsi_eh_2]|
65| 415 [scsi_tmf_2]|
66| 420 [kworker/1:1H-kblockd]|
67| 731 [kworker/3:2-mm_percpu_wq]|
68| 774 [kworker/3:1H-kblockd]|
69| 777 [jbd2/sda4-8]|
70| 778 [ext4-rsv-conver]|
71| 848 [kworker/0:1H-kblockd]|
72| 849 [kworker/5:1H-kblockd]|
73| 850 [kworker/4:1H-kblockd]|
74| 851 [kworker/2:1H-kblockd]|
75| 853 /usr/lib/systemd/systemd-journald|
76| 868 /usr/sbin/haveged -w 1024 -v 0 -F|
77| 869 /usr/lib/systemd/systemd-udevd|
78| 938 /sbin/auditd|
79| 958 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only|
80| 963 /usr/sbin/irqbalance --foreground|
81| 966 /usr/sbin/numad -i 15 -F|
82| 969 /usr/lib/polkit-1/polkitd --no-debug|
83| 986 /usr/lib/systemd/systemd-logind|
84| 987 /usr/lib/wicked/bin/wickedd-auto4 --systemd --foreground|
85| 990 /usr/lib/wicked/bin/wickedd-dhcp4 --systemd --foreground|
86| 991 /usr/lib/wicked/bin/wickedd-dhcp6 --systemd --foreground|
87| 1034 /usr/sbin/wickedd --systemd --foreground|
88| 1039 /usr/sbin/wickedd-nanny --systemd --foreground|
89| 1528 [kworker/3:0]|
90| 1580 /usr/sbin/httpd-prefork -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/apache2/sysconfig.d//global.conf -f /etc/apache2/httpd.conf -c Include /etc/apache2/sysconfig.d//include.conf -DSYSTEMD -DFOREGROUND -k start|

91| 1589 /usr/bin/python3 -Es /usr/sbin/tuned -l -P|
92| 1630 /usr/sbin/mysqld --defaults-file=/etc/my.cnf --user=mysql|
93| 1673 /usr/sbin/httpd-prefork -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/apache2/sysconfig.d//global.conf -f /etc/apache2/httpd.conf -c Include /etc/apache2/sysconfig.d//include.conf -DSYSTEMD -DFOREGROUND -k start|

94| 1674 /usr/sbin/httpd-prefork -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/apache2/sysconfig.d//global.conf -f /etc/apache2/httpd.conf -c Include /etc/apache2/sysconfig.d//include.conf -DSYSTEMD -DFOREGROUND -k start|

95| 1675 /usr/sbin/httpd-prefork -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/apache2/sysconfig.d//global.conf -f /etc/apache2/httpd.conf -c Include /etc/apache2/sysconfig.d//include.conf -DSYSTEMD -DFOREGROUND -k start|

96| 1676 /usr/sbin/httpd-prefork -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/apache2/sysconfig.d//global.conf -f /etc/apache2/httpd.conf -c Include /etc/apache2/sysconfig.d//include.conf -DSYSTEMD -DFOREGROUND -k start|

97| 1677 /usr/sbin/httpd-prefork -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/apache2/sysconfig.d//global.conf -f /etc/apache2/httpd.conf -c Include /etc/apache2/sysconfig.d//include.conf -DSYSTEMD -DFOREGROUND -k start|

98| 1678 /usr/sbin/httpd-prefork -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/apache2/sysconfig.d//global.conf -f /etc/apache2/httpd.conf -c Include /etc/apache2/sysconfig.d//include.conf -DSYSTEMD -DFOREGROUND -k start|

99| 1679 /usr/sbin/httpd-prefork -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/apache2/sysconfig.d//global.conf -f /etc/apache2/httpd.conf -c Include /etc/apache2/sysconfig.d//include.conf -DSYSTEMD -DFOREGROUND -k start|

100| 1680 /usr/sbin/httpd-prefork -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include
114| 1694 /usr/sbin/httpd-prefork -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/apache2/sysconfig.d//global.conf -f /etc/apache2/httpd.conf -c Include /etc/apache2/sysconfig.d//include.conf -DSYSTEMD -DFOREGROUND -k start|

115| 1695 /usr/sbin/httpd-prefork -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/apache2/sysconfig.d//global.conf -f /etc/apache2/httpd.conf -c Include /etc/apache2/sysconfig.d//include.conf -DSYSTEMD -DFOREGROUND -k start|

116| 1696 /usr/sbin/httpd-prefork -DSYSCONFIG -DSSL -C PidFile /var/run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/apache2/sysconfig.d//global.conf -f /etc/apache2/httpd.conf -c Include /etc/apache2/sysconfig.d//include.conf -DSYSTEMD -DFOREGROUND -k start|



543| 2372 /usr/lib/postfix/bin//master -w|
544| 2374 qmgr -l -t fifo -u|
545| 2430 /usr/sbin/cron -n|
546| 2651 /usr/bin/SCREEN -d -m -S ASTVDremote /usr/share/astguiclient/AST_VDremote_agents.pl --debug|
547| 2653 /usr/bin/perl /usr/share/astguiclient/AST_VDremote_agents.pl --debug|
AST_VDremote_agents RUNNING: |/usr/share/astguiclient/AST_VDremote_agents.pl --debug|
548| 2987 /sbin/agetty -o -p -- \u --noclear tty1 linux|
549| 3000 bpfilter_umh|
550| 5397 [kworker/2:1]|
551|10170 sshd: Ogie [priv]|
552|10203 /usr/lib/systemd/systemd --user|
553|10204 (sd-pam)|
554|10212 sshd: Ogie@pts/8|
555|10219 -bash|
556|10278 su|
557|10296 bash|
558|10654 /usr/bin/SCREEN -d -m -S ASTVDadapt /usr/share/astguiclient/AST_VDadapt.pl --debug|
559|10656 /usr/bin/perl /usr/share/astguiclient/AST_VDadapt.pl --debug|
AST_VDadapt RUNNING: |/usr/share/astguiclient/AST_VDadapt.pl --debug|
560|11667 [kworker/4:2-mm_percpu_wq]|
561|11699 [kworker/5:1-cgroup_destroy]|
562|14535 /usr/lib/systemd/systemd --user|
563|14536 (sd-pam)|
564|14585 /usr/bin/SCREEN -d -m -S ASTfastlog /usr/share/astguiclient/FastAGI_log.pl --debug|
565|14586 /usr/bin/perl /usr/share/astguiclient/FastAGI_log.pl --debug|
FastAGI_log RUNNING: |/usr/share/astguiclient/FastAGI_log.pl --debug|
566|14588 /usr/bin/SCREEN -d -m -S ASTVDadFILL /usr/share/astguiclient/AST_VDauto_dial_FILL.pl --debug|
567|14589 /usr/bin/perl /usr/share/astguiclient/AST_VDauto_dial_FILL.pl --debug|
AST_VDauto_dial_FILL RUNNING: |/usr/share/astguiclient/AST_VDauto_dial_FILL.pl --debug|
568|14593 ip_relay 40569 127.0.0.1 4569 9999999|
ip_relay RUNNING: |14593 ip_relay 40569 127.0.0.1 4569 9999999|
569|14594 ip_relay 41569 127.0.0.1 4569 9999999|
ip_relay RUNNING: |14594 ip_relay 41569 127.0.0.1 4569 9999999|
570|14595 ip_relay 42569 127.0.0.1 4569 9999999|
ip_relay RUNNING: |14595 ip_relay 42569 127.0.0.1 4569 9999999|
571|15608 SCREEN -L -S asterisk|
572|15609 /bin/sh|
573|15614 /usr/sbin/asterisk -vvvvvvvvvvvvvvvvvvvvvgcT|
asterisk RUNNING: |15614 /usr/sbin/asterisk -vvvvvvvvvvvvvvvvvvvvvgcT|
574|15771 SCREEN -L -S asterisk|
575|15772 /bin/bash|
576|15828 /usr/bin/SCREEN -d -m -S ASTupdate /usr/share/astguiclient/AST_update_AMI2.pl|
577|15830 /usr/bin/perl /usr/share/astguiclient/AST_update_AMI2.pl|
AST_update RUNNING: |/usr/share/astguiclient/AST_update_AMI2.pl|
578|15831 /usr/bin/SCREEN -d -m -S ASTsend /usr/share/astguiclient/AST_manager_send.pl|
579|15833 /usr/bin/perl /usr/share/astguiclient/AST_manager_send.pl|
AST_send RUNNING: |/usr/share/astguiclient/AST_manager_send.pl|
580|15834 /usr/bin/SCREEN -d -m -S ASTlisten /usr/share/astguiclient/AST_manager_listen_AMI2.pl|
581|15836 /usr/bin/perl /usr/share/astguiclient/AST_manager_listen_AMI2.pl|
AST_listen RUNNING: |/usr/share/astguiclient/AST_manager_listen_AMI2.pl|
582|15837 /usr/bin/SCREEN -d -m -S ASTVDauto /usr/share/astguiclient/AST_VDauto_dial.pl|
583|15839 /usr/bin/perl /usr/share/astguiclient/AST_VDauto_dial.pl|
AST_VDauto_dial RUNNING: |/usr/share/astguiclient/AST_VDauto_dial.pl|
584|20904 [kworker/0:1-events]|
585|21259 pickup -l -t fifo -u|
586|21393 /usr/bin/perl /usr/share/astguiclient/FastAGI_log.pl --debug|
FastAGI_log RUNNING: |/usr/share/astguiclient/FastAGI_log.pl --debug|
587|22522 /usr/bin/perl /usr/share/astguiclient/FastAGI_log.pl --debug|
FastAGI_log RUNNING: |/usr/share/astguiclient/FastAGI_log.pl --debug|
588|23101 [kworker/5:2-mm_percpu_wq]|
589|23399 [kworker/0:0]|
590|23636 [kworker/u12:1-events_unbound]|
591|23775 [kworker/2:2-events_power_efficient]|
592|23835 /usr/bin/perl /usr/share/astguiclient/FastAGI_log.pl --debug|
FastAGI_log RUNNING: |/usr/share/astguiclient/FastAGI_log.pl --debug|
593|24006 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups|
594|24374 [kworker/4:0]|
595|24758 /usr/bin/perl /usr/share/astguiclient/FastAGI_log.pl --debug|
FastAGI_log RUNNING: |/usr/share/astguiclient/FastAGI_log.pl --debug|
596|25402 /usr/bin/perl /usr/share/astguiclient/FastAGI_log.pl --debug|
FastAGI_log RUNNING: |/usr/share/astguiclient/FastAGI_log.pl --debug|
597|26615 [kworker/1:3-ata_sff]|
598|27103 sshd: Prabin [priv]|
599|27147 /usr/lib/systemd/systemd --user|
600|27148 (sd-pam)|
601|27155 sshd: Prabin@pts/11|
602|27156 -bash|
603|27203 su root|
604|27213 bash|
605|27249 [kworker/u12:0-events_unbound]|
606|27735 [kworker/1:2-ata_sff]|
607|28243 [kworker/1:0-ata_sff]|
608|28513 [kworker/u12:2-events_unbound]|
609|28731 [kworker/1:1-mm_percpu_wq]|
610|28768 /usr/sbin/CRON -n|
611|28774 /usr/bin/perl /usr/share/astguiclient/AST_manager_kill_hung_congested.pl|
612|28805 /usr/bin/perl /usr/share/astguiclient/ADMIN_keepalive_ALL.pl --debugX|
613|28806 /bin/ps -o %p %a --no-headers -A|
running Timeclock auto-logout process...
|SELECT count(*) from system_settings where timeclock_end_of_day LIKE "%0307%";|
voicemail server configuration: 0|N
Uptime debug 1: | 03:07:13 up 10 days 1:01, 2 users, load average: 0.13, 0.10, 0.09
|
Uptime debug 2: | 03:07:13 up 10 days 1:01|10 days 1:01|0| seconds: |867713.28|
Uptime debug 3: |1|UPDATE servers SET system_uptime='10 days 1:01' where server_ip='117.120.1.187';|
CONFIRM ASTERISK IS RUNNING: |Y|1|Y|N|0|
SELECT campaign_id,callback_useronly_move_minutes FROM vicidial_campaigns where active='Y' and callback_useronly_move_minutes > 0 order by campaign_id;
old live scheduled callbacks move active campaigns: 0
SELECT cid_group_id,cid_auto_rotate_minutes,cid_auto_rotate_minimum,cid_auto_rotate_calls,cid_last_auto_rotate,cid_auto_rotate_cid,UNIX_TIMESTAMP(cid_last_auto_rotate) FROM vicidial_cid_groups where cid_group_type='NONE' and cid_auto_rotate_minutes > 0 order by cid_group_id;
CID Groups with Auto-Rotate enabled: 0
Checking for Drop Lists: |0|SELECT dl_id FROM vicidial_drop_lists where ( (dl_server IN('117.120.1.187','active_voicemail_server') ) and ( ( (active='Y') and (dl_times LIKE "%0307%") and ( (dl_weekdays LIKE "%5%") or (dl_monthdays LIKE "%14%") ) ) or (run_now_trigger='Y') ) ) order by dl_id limit 1000;|
RESET LIST: 0|0307DONE

From that output, there were no pending changes at the time you fired the keepalive script.
If you had clicked submit on a phone entry for that server, then immediately run that script, immediately after the line "voicemail server configuration: 0|N" you should havesomething like this:

Code: Select all

generating new auto-gen conf files
Asterisk version: 1 16
<<<<<<[IVR Option Outputs]>>>>>>>>
writing auto-gen conf files
reloading asterisk modules: