If you have 20 agents logged in, I will assume your statement of 4 calls is a miscommunication of some sort. If you are in autodial mode, the system is generating calls for all available agents. Those count. Especially if your room manager changed the dial ratio from 2.5 to 10 "a week ago". The total of all calls (agents, live prospects, calls ringing ... ) is what I was looking for.
It is also possible other functions are causing problems. You can use htop to see who the hog is. But if you have a single processor P4 @ 3.0 ghz, you are probably just experiencing an overload. Usually a Core2Quad at 2.3+ Ghz (ie: 4 processors) will handle 25 agents. If you have 20 agents on 1 processor ... it seems likely you are overloading.
But your system may be bogging down for other reasons as well. For instance: Have you backed up your DB to clear the DB binary logs? Have you archived your _log tables with the script in crontab for that purpose? Large log files (binary logs in mysql or log tables in the asterisk database) can slow down the system. Once "slow" has occurred, overload generally follows as the system fights to catch up (unsuccessfully). Especially in single processor systems.
NOTE: Always post your Vicidial Version with Build!