Reload Dialplan on Servers not working

[tails84]

Hi guys,

Having an issue where the reload dialplan on servers (Admin > System Settings > Reload Dialplan on Servers = 1) is not actually reloading the dialplan although the last reload date is updated in the gui and the value is set back to 0. Reloading works fine if I issue dialplan reload from the CLI. Using Admin>Servers>Rebuild Conf Files=Y doesn't seem to work either, although in this case, the Y does not revert back to N.

If you know what php files/perl scripts I should be looking towards, happy to do a bit of debugging myself. Cheers.

We are running a 5 cluster system.
Vici: 2.14b0.5
Asterisk: 1.8.23.0-vici
OS: Centos 6
1 primary with web & db, 4 other dialers.

[williamconley]

1) POST the asterisk version in /etc/astguiclient.conf (exactly as shown in the file, including the "." if there is one)
2) POST the asterisk version in admin->servers for this server
3) POST the asterisk version from "asterisk -V"
4) POST the IP address from admin->servers for this server
5) POST the IP address from /etc/astguiclient.conf on this server
6) Manual run of keepalive with debug mode on the asterisk server:

Code: Select all

perl /srv/www/htdocs/vicidial/ADMIN_keepalive_ALL.pl --debugX

Happy Hunting!

[tails84]

1) 1.8
2) 1.8.23.0-vici
3) Asterisk 1.8.23.0-vici
4) I don't really feel like posting our IP on a public forum hope you can understand. But safe to say it is the correct IP and a public IP and it is appropriately firewalled.
5) Same IP as above
6) See below (redacted IP is the same IP as above). Also note: the keep alive script was located in /usr/share/astguiclient

Thanks!

Code: Select all

----- SUPER-DUPER DEBUGGING -----

Asterisk version: 1 8
AST_update set to keepalive
AST_send_listen set to keepalive
AST_VDauto_dial set to keepalive
AST_VDremote_agents set to keepalive
FastAGI_log set to keepalive
ip_relay set to keepalive
0|    1 /sbin/init|
1|    2 [kthreadd]|
2|    3 [migration/0]|
3|    4 [ksoftirqd/0]|
4|    5 [stopper/0]|
5|    6 [watchdog/0]|
6|    7 [migration/1]|
7|    8 [stopper/1]|
8|    9 [ksoftirqd/1]|
9|   10 [watchdog/1]|
10|   11 [events/0]|
11|   12 [events/1]|
12|   13 [events/0]|
13|   14 [events/1]|
14|   15 [events_long/0]|
15|   16 [events_long/1]|
16|   17 [events_power_ef]|
17|   18 [events_power_ef]|
18|   19 [cgroup]|
19|   20 [khelper]|
20|   21 [netns]|
21|   22 [async/mgr]|
22|   23 [pm]|
23|   24 [xenwatch]|
24|   25 [xenbus]|
25|   26 [sync_supers]|
26|   27 [bdi-default]|
27|   28 [kintegrityd/0]|
28|   29 [kintegrityd/1]|
29|   30 [kblockd/0]|
30|   31 [kblockd/1]|
31|   32 [kacpid]|
32|   33 [kacpi_notify]|
33|   34 [kacpi_hotplug]|
34|   35 [ata_aux]|
35|   36 [ata_sff/0]|
36|   37 [ata_sff/1]|
37|   38 [ksuspend_usbd]|
38|   39 [khubd]|
39|   40 [kseriod]|
40|   41 [md/0]|
41|   42 [md/1]|
42|   43 [md_misc/0]|
43|   44 [md_misc/1]|
44|   45 [linkwatch]|
45|   48 [khungtaskd]|
46|   49 [lru-add-drain/0]|
47|   50 [lru-add-drain/1]|
48|   51 [kswapd0]|
49|   52 [ksmd]|
50|   53 [khugepaged]|
51|   54 [aio/0]|
52|   55 [aio/1]|
53|   56 [crypto/0]|
54|   57 [crypto/1]|
55|   64 [kthrotld/0]|
56|   65 [kthrotld/1]|
57|   67 [kpsmoused]|
58|   68 [usbhid_resumer]|
59|   69 [deferwq]|
60|   99 [kdmremove]|
61|  100 [kstriped]|
62|  228 [scsi_eh_0]|
63|  229 [scsi_eh_1]|
64|  332 [jbd2/xvda1-8]|
65|  333 [ext4-dio-unwrit]|
66|  410 /sbin/udevd -d|
67|  686 [kauditd]|
68|  725 [ib_addr]|
69|  730 [infiniband/0]|
70|  731 [infiniband/1]|
71|  734 /usr/bin/perl /usr/share/astguiclient/FastAGI_log.pl --debug|
FastAGI_log RUNNING:             |/usr/share/astguiclient/FastAGI_log.pl --debug|
72|  740 [ib_mcast]|
73|  745 [iw_cm_wq]|
74|  751 [ib_cm/0]|
75|  752 [ib_cm/1]|
76|  757 [rdma_cm]|
77|  778 [ipoib_flush]|
78|  963 [flush-202:0]|
79| 1004 /sbin/dhclient -H ip-172-31-18-242 -1 -q -cf /etc/dhcp/dhclient-eth0.conf -lf /var/lib/dhclient/dhclient-eth0.leases -pf /var/run/dhclient-eth0.pid eth0|
80| 1060 auditd|
81| 1082 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5|
82| 1101 /sbin/udevd -d|
83| 1102 /sbin/udevd -d|
84| 1155 dbus-daemon --system|
85| 1193 /usr/sbin/acpid|
86| 1331 /usr/sbin/sshd|
87| 1342 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g|
88| 1379 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --pid-file=/var/run/mysqld/mysqld.pid --basedir=/usr --user=mysql|
89| 1532 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock|
90| 1627 /usr/libexec/postfix/master|
91| 1639 qmgr -l -t fifo -u|
92| 1643 /usr/sbin/httpd|
93| 1682 /bin/sh /usr/sbin/safe_asterisk|
94| 1686 /usr/sbin/asterisk -f -vvvg -c|
asterisk RUNNING:              | 1686 /usr/sbin/asterisk -f -vvvg -c|
95| 1687 crond|
96| 1710 /usr/bin/python -Es /usr/bin/fail2ban-server -s /var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail2ban.pid -x -b|
97| 1711 /usr/sbin/httpd|
98| 1712 /usr/sbin/httpd|
99| 1713 /usr/sbin/httpd|
100| 1714 /usr/sbin/httpd|
101| 1715 /usr/sbin/httpd|
102| 1716 /usr/sbin/httpd|
103| 1717 /usr/sbin/httpd|
104| 1718 /usr/sbin/httpd|
105| 1833 ip_relay 40569 127.0.0.1 4569 9999999|
ip_relay RUNNING:                | 1833 ip_relay 40569 127.0.0.1 4569 9999999|
106| 1834 ip_relay 41569 127.0.0.1 4569 9999999|
ip_relay RUNNING:                | 1834 ip_relay 41569 127.0.0.1 4569 9999999|
107| 1835 ip_relay 42569 127.0.0.1 4569 9999999|
ip_relay RUNNING:                | 1835 ip_relay 42569 127.0.0.1 4569 9999999|
108| 1874 sshd: centos [priv]|
109| 1877 sshd: centos@notty|
110| 1912 [bluetooth]|
111| 1919 sudo tcpdump -s 0 -U -K -n -w - -i eth0 not port 22 and not port 80 and not port 443 and not icmp and (not (port 445 or port 139 or port 69 or net 61.68.36.38))|
112| 1929 tcpdump -s 0 -U -K -n -w - -i eth0 not port 22 and not port 80 and not port 443 and not icmp and (not (port 445 or port 139 or port 69 or net 61.68.36.38))|
113| 1974 /usr/bin/SCREEN -d -m -S ASTfastlog /usr/share/astguiclient/FastAGI_log.pl --debug|
114| 1976 /usr/bin/perl /usr/share/astguiclient/FastAGI_log.pl --debug|
FastAGI_log RUNNING:             |/usr/share/astguiclient/FastAGI_log.pl --debug|
115| 2012 /usr/bin/perl /usr/share/astguiclient/FastAGI_log.pl --debug|
FastAGI_log RUNNING:             |/usr/share/astguiclient/FastAGI_log.pl --debug|
116| 2037 /usr/bin/SCREEN -S astshell20200301211117|
117| 2038 /bin/sh|
118| 2044 SCREEN -L -S asterisk|
119| 2045 /bin/sh|
120| 2062 /sbin/agetty /dev/ttyS0 115200 vt100-nav|
121| 2063 /sbin/mingetty /dev/tty1|
122| 2169 /usr/bin/perl /usr/share/astguiclient/FastAGI_log.pl --debug|
FastAGI_log RUNNING:             |/usr/share/astguiclient/FastAGI_log.pl --debug|
123| 6376 /usr/bin/SCREEN -d -m -S ASTupdate /usr/share/astguiclient/AST_update.pl|
124| 6378 /usr/bin/perl /usr/share/astguiclient/AST_update.pl|
AST_update RUNNING:              |/usr/share/astguiclient/AST_update.pl|
125| 6379 /usr/bin/SCREEN -d -m -S ASTsend /usr/share/astguiclient/AST_manager_send.pl|
126| 6381 /usr/bin/perl /usr/share/astguiclient/AST_manager_send.pl|
AST_send RUNNING:                |/usr/share/astguiclient/AST_manager_send.pl|
127| 6382 /usr/bin/SCREEN -d -m -S ASTlisten /usr/share/astguiclient/AST_manager_listen.pl|
128| 6384 /usr/bin/perl /usr/share/astguiclient/AST_manager_listen.pl|
AST_listen RUNNING:              |/usr/share/astguiclient/AST_manager_listen.pl|
129| 6385 /usr/bin/SCREEN -d -m -S ASTVDauto /usr/share/astguiclient/AST_VDauto_dial.pl|
130| 6386 /usr/bin/perl /usr/share/astguiclient/AST_VDauto_dial.pl|
AST_VDauto_dial RUNNING:         |/usr/share/astguiclient/AST_VDauto_dial.pl|
131| 6388 /usr/bin/SCREEN -d -m -S ASTVDremote /usr/share/astguiclient/AST_VDremote_agents.pl --debug|
132| 6390 /usr/bin/perl /usr/share/astguiclient/AST_VDremote_agents.pl --debug|
AST_VDremote_agents RUNNING:     |/usr/share/astguiclient/AST_VDremote_agents.pl --debug|
133| 6739 sshd: centos [priv]|
134| 6743 sshd: centos@pts/8|
135| 6744 -bash|
136| 6870 pickup -l -t fifo -u|
137| 8686 sshd: centos [priv]|
138| 8689 sshd: centos@pts/11|
139| 8690 -bash|
140|10567 sshd: centos [priv]|
141|10571 sshd: centos@pts/12|
142|10572 -bash|
143|10939 CROND|
144|10946 /usr/bin/perl /usr/share/astguiclient/AST_manager_kill_hung_congested.pl|
145|10993 /usr/bin/perl /usr/share/astguiclient/FastAGI_log.pl --debug|
FastAGI_log RUNNING:             |/usr/share/astguiclient/FastAGI_log.pl --debug|
146|11011 perl ADMIN_keepalive_ALL.pl --debugX|
147|11012 /bin/ps -o %p %a --no-headers -A|
148|16975 /usr/bin/SCREEN -d -m -S ASTconf3way /usr/share/astguiclient/AST_conf_update_3way.pl --debug|
149|16976 /usr/bin/perl /usr/share/astguiclient/AST_conf_update_3way.pl --debug|
AST_conf_3way RUNNING:           |/usr/share/astguiclient/AST_conf_update_3way.pl --debug|
150|30600 sshd: centos [priv]|
151|30603 sshd: centos@pts/7|
152|30604 -bash|
153|30652 sudo asterisk -r|
154|30653 java -jar Crawler.jar|
155|30654 rasterisk r|
156|32206 /usr/bin/perl /usr/share/astguiclient/FastAGI_log.pl --debug|
FastAGI_log RUNNING:             |/usr/share/astguiclient/FastAGI_log.pl --debug|
157|32454 /usr/bin/perl /usr/share/astguiclient/FastAGI_log.pl --debug|
FastAGI_log RUNNING:             |/usr/share/astguiclient/FastAGI_log.pl --debug|
|SELECT count(*) from system_settings where timeclock_end_of_day LIKE "%1100%";|
Uptime debug 1: | 11:00:44 up 13:50,  5 users,  load average: 0.07, 0.02, 0.00
|
Uptime debug 2: | 11:00:44 up 13:50|13:50|0|  seconds: |49804.61|
Uptime debug 3: |1|UPDATE servers SET system_uptime='13:50'  where server_ip='{REDACTEDIP}';|
CONFIRM ASTERISK IS RUNNING: |Y|1|N|N|0|
DONE


[williamconley]

You should be using a private IP for communication between the servers. Not a public one. And those servers should be sharing a physical separate network that is gigabit and not firewalled for fast inter-server communication. Having agents on the same local fast open network is optional.

But yes, the goal was to check that the IP was identical in all those places so the server would be requesting data for itself from the DB and would (of course) get it because the IP matched.

Do the database entries (set system_uptime for instance) succeed in updating the database?

Hard drive isn't full?

Database has no crashed tables?

If you make a change to a phone or carrier, does the sip-vicidial.conf file get rewritten with the new data?

Sorry, I don't have time to read the entire keepalive script.

[tails84]

Yeah for sure, I didn't set these up initially and would use definitely use private IP communication but it's a production server and can't really afford the time/cost for a switch at the moment. They are all on the same subnet, hosted on an AWS so routing between would hopefully be a non-issue. Firewall allows all ports between all of the servers.

HDD fine. No crashed tables. By all accounts everything is working as intended, database is getting updated fine from all servers. Changes to a phone does push to the sip-vicidial.conf, carrier pushes changes but need to issue a dialplan reload manually. Likewise a sip reload if changing a phone to a different server.

No problems. If you can even just point me in the right direction of how it updates, I'm willing and able to trace down myself. Eg, how is it communicating with the other servers, is it running a script, ssh, web api to vici and then through there, sending flag to database then polling from there.

[williamconley]

was the asterisk version changed after/during the install in any of those places? during the run of install.pl if the asterisk version is incorrect, it will load the wrong sample .conf files, and those incorrect files can cause failure in the issuance of the reload command.

that being said: the issuance of the command is logged during the keepalive script run. there should be an error in the debugX output if there is a reason for it to fail.

does the reload work from the linux cli? (asterisk -rx "sip reload")

[tails84]

At the time when I wrote this, the version wasn't changed, but we have upgraded now.

Reload works fine from CLI, have just made a batch script to ssh in and run the reload command manually.

Thanks for your help William.

[npaskov]

@talis84
Check if for some reason your asterisk service has been enabled - in my case that was the problem - disable it and restart server - vicidial has its own scripts to start properly asterisk using screen and later send commands to that specific screen session.