And let's not overlook system errors due to virtualization, configured apache limitations, or even bad DNS. There's a reason the "best practices" include physical server hardware and installing from the Vicibox.com .iso image installer. Best Practices can guide you down the narrow path that is "everything works". Any other path has so many variables that it becomes a hit-and-miss game of "wow, everything could be wrong".
Perhaps you should be checking logs (system and software packages alike) for errors to try to track down your problem. It could be very simple ... or any one of a myriad of minor to major failures. For most techs, this would be the "let's start killing error messages and hope to get lucky" and then trace: "when did each step of the process occur, and what could block the step that was delayed?"
Agent screen sends timed AJAX packet. Server gets AJAX packet. Server checks DB for a response to AJAX packet. Server sends response to AJAX packet. Agent screen received response to AJAX packet. Agent screen acts upon response to AJAX packet and shows data to agent. Each step can be tracked and/or debugged and time-checked to within one second. Something in that line is blocked to cause the delay. Happy Hunting.
But: Great job posting your specs.