All servers inoperable after ADMIN_archive_log_tables

[Two8nine]

Vicidial version: 2.14-694a
Vicidial build: 181005-1738
Asterisk version: 13.21.1-vici

Hi hoping for some pointers.

System is down as of this morning, agents can log in, and we can manually dial calls, but agents cannot join sessions.

The only change made to the system was enabling this in crontab (same but set for yesterday's date rather than 1st):

Code: Select all

30 1 * * * /usr/share/astguiclient/ADMIN_archive_log_tables.pl

Now, this is the error output by the asterisk log when agents attempt to dial or log in: Connect to 'agi://127.0.0.1:4577/call_log' failed

The command 'sip reload' hangs indefinitely when attempted, and rebooting all servers has had no effect.

Code: Select all

mysqlcheck -r asterisk

command on DB server shows call_log as OK.

Code: Select all

tail /var/log/messages

Shows that the servers "/" directory is now full as well, which is interesting given it was at 10-12% yesterday, so I'm assuming some error or something is being duplicated?

Any suggestions?

EDIT: Investigating database tables shows where the bulk of the data lies.

vicidial_api_log.TMD
vicidial_api_log.MYD
vicidial_api_log_archive.MYD
vicidial_api_urls.MYD
vicidial_api_urls_archive.MYD

Are all 20+ GB, where the rest of the tables are a couple 100 mb at most.

Can and should data be removed from these tables? No idea how they got so huge so quick.