CPU over 300 percent

[dspaan]

Does anyone have an idea why this is happening? I have never seen this on any of my vicidial servers. This is the only vicibox 7 server i have. I had to restart the mysql service to get back to a normal CPU usage.



This is ViciBox v.7.0.3-160505

Path: /usr/src/astguiclient/trunk
Working Copy Root Path: /usr/src/astguiclient/trunk
URL: svn://svn.eflo.net/agc_2-X/trunk
Relative URL: ^/agc_2-X/trunk
Repository Root: svn://svn.eflo.net
Repository UUID: 3d104415-ff17-0410-8863-d5cf3c621b8a
Revision: 2553
Node Kind: directory
Schedule: normal
Last Changed Author: mattf
Last Changed Rev: 2553
Last Changed Date: 2016-06-22 01:03:57 +0200 (Wed, 22 Jun 2016)

Edit: I'm so evil 666 posts

[mflorell]

What is the mysql version?

What hardware do you have on this server(RAM, drives, RAID-card,etc...)?

[dspaan]

Server version: 10.1.6-MariaDB-log

However the database was imported from a much older vicibox server.

This is a Dell Poweredge 1950 with Intel Xeon E5420 CPU
http://ark.intel.com/products/33927/Int ... 33-MHz-FSB

The server has 8GB RAM and a single 240GB Solid State Drive. No raid config.

The CPU is between 70-100% right now and this is the realtime screen:

[mflorell]

Not really sure what is causing that, we have several clients on similar hardware using vicibox 7.0.3 and we don't see those issues.

It looks like it is mostly database load, so you should look at how many records you have in the vicidial_list and the log tables.

[boiken]

This is a all-in-one install ?
As shown on htop,you have a octa-core Procesor which means that 3.5 should not be a problem(even if its high for your actual configuration).With your CPU,it will be a problem if its over 8.00.

[dspaan]

The agents were not getting any more calls from the dialer and they even got an error message in their screen saying the system was slow or something. So this definitely was a show stopper.

@Matt -

vicidial_list has 1.287.195 records
vicidial_log has 4.154.002 records

[mflorell]

over 1 million leads is a lot for an all-in-one dialer with only one hard drive.

I would suggest removing some of those and also archiving your log tables.

[dspaan]

Thanks Matt, we will do that. Is this a manual procedure or is there a script we can use?

[mflorell]

For vicidial_list it is a manual procedure(unless you just go delete some lists).

For log archiving, there is the ADMIN_archive_log_tables.pl script:


# /usr/share/astguiclient/ADMIN_archive_log_tables.pl --help
allowed run time options:
[--daily] = only archives call_log, vicidial_log_extended and vicidial_dial_log tables, only last 24 hours kept
[--carrier-daily] = will also archive the vicidial_carrier_log table when --daily is run
[--vlog-daily] = will also archive the vicidial_log table when --daily is run
[--days=XX] = number of days to archive past, default is 732(2 years)
[--months=XX] = number of months to archive past, default is 24(2 years) If 'days' used then 'months' ignored
[--closer-log] = archive vicidial_closer_log records
[--queue-log] = archive QM queue_log records
[--only-trim-archive-level-one] = will not perform normal archive process, instead this will only delete records
that are older than XX months from least important log archive tables:
call_log_archive, vicidial_log_extended_archive, vicidial_dial_log_archive
[--only-trim-archive-level-two] = same as --only-trim-archive-level-one, except includes tables:
vicidial_carrier_log_archive, vicidial_api_log_archive
[--only-trim-archive-level-three] = same as --only-trim-archive-level-two, except includes tables:
vicidial_log_archive, vicidial_agent_log_archive, vicidial_closer_log_archive [--quiet] = quiet
[--calc-test] = date calculation test only
[--test] = test

[dspaan]

Is there a way to remove leads from vicidial_list and still dedupe against those leads when loading new leads?

[mflorell]

not currently. most of our clients that need to do things like that will use a more efficient external system.

[dspaan]

ok i deleted about 500.000 leads from vicidial_list and also used the archive script:

/usr/share/astguiclient/ADMIN_archive_log_tables.pl --carrier-vlog-daily --days=180

----- DAYS OVERRIDE: 180 -----



-- ADMIN_archive_log_tables.pl --

This program is designed to put all records from call_log, vicidial_log,
server_performance, vicidial_agent_log, vicidial_carrier_log,
vicidial_call_notes, vicidial_lead_search_log and others into relevant
_archive tables and delete records in original tables older than
180 days ( 2016-01-16 11:29:22|1452940162 ) from current date


Processing call_log table... (6348170|0)
6348170 rows inserted into call_log_archive table
4831943 rows deleted from call_log table
781 rows deleted from call_log_archive table

Processing vicidial_log table... (4199554|0)
4199554 rows inserted into vicidial_log_archive table
3161939 rows deleted from vicidial_log table

Processing vicidial_closer_log table... (113168|0)
113168 rows inserted into vicidial_closer_log_archive table
93115 rows deleted from vicidial_closer_log table

Processing vicidial_log_extended table... (4312810|0)
4312810 rows inserted into vicidial_log_extended_archive table
3255115 rows deleted from vicidial_log_extended table

Processing vicidial_dial_log table... (4221697|0)
4218116 rows inserted into vicidial_dial_log_archive table
3178651 rows deleted from vicidial_dial_log table

Processing vicidial_api_log table... (12677|0)
12677 rows inserted into vicidial_api_log_archive table
9568 rows deleted from vicidial_api_log table

Processing vicidial_log_noanswer table... (0|0)
0 rows inserted into vicidial_log_noanswer_archive table
0 rows deleted from vicidial_log_noanswer table

Processing vicidial_did_agent_log table... (0|0)
0 rows inserted into vicidial_did_agent_log_archive table
0 rows deleted from vicidial_did_agent_log table

Processing server_performance table... (0)
0 rows deleted from server_performance table

Processing vicidial_url_log table... (17)
17 rows deleted from vicidial_url_log table

Processing vicidial_dtmf_log table... (124)
0 rows deleted from vicidial_dtmf_log table

Processing vicidial_agent table... (2853371|0)
2853371 rows inserted into vicidial_agent_log_archive table
2088339 rows deleted from vicidial_agent_log table

Processing vicidial_carrier_log table... (4254373|0)
4254373 rows inserted into vicidial_carrier_log_archive table
3210601 rows deleted from vicidial_carrier_log table

Processing vicidial_call_notes table... (7812|0)
7812 rows inserted into vicidial_call_notes_archive table
7811 rows deleted from vicidial_call_notes table

Processing vicidial_lead_search_log table... (5971|0)
5971 rows inserted into vicidial_lead_search_log_archive table
4713 rows deleted from vicidial_lead_search_log table

Processing vicidial_outbound_ivr_log table... (0|0)
0 rows inserted into vicidial_outbound_ivr_log_archive table
0 rows deleted from vicidial_outbound_ivr_log table

script execution time in seconds: 3161 minutes: 52.6833333333333

A major difference the CPU is now at 1% with 20 agents and before it was 80-100% constantly.

[mflorell]

Sounds like a solution

[dspaan]

There are complaints from the agents again that the server is slow this morning and indeed it was back at 70% CPU with spikes of 100%:



Right now in vicidial_list there are 500.000 rows and vicidial_log 1.000.000 rows.