HIGH PHYSICAL MEMORY USAGE

[georgem]

hello

i am using a vicidialnow 1.1 installation. vici 2.0.4 asterisk 1.2.27.

here is the h/w config. ( output of phpsysinfo)

RAM 4GB

Processors 2
Model Intel(R) Xeon(R) CPU E3110 @ 3.00GHz
CPU Speed 3 GHz
Cache Size 6.00 MB
System Bogomips 11971.45
PCI Devices
- Ethernet controller: Broadcom Corporation NetXtreme BCM5722 Gigabit Ethernet PCI Express
- Ethernet controller: D-Link System Inc DGE-530T Gigabit Ethernet Adapter
- Host bridge: Intel Corporation Server DRAM Controller
- IDE interface: Intel Corporation 82801I
- IDE interface: Intel Corporation 82801IR/IO/IH
- ISA bridge: Intel Corporation 82801IR
- PCI bridge: Intel Corporation 82801 PCI Bridge
- (3x) PCI bridge: Intel Corporation 82801I
- PCI bridge: Intel Corporation Server Host-Primary PCI Express Bridge
- SMBus: Intel Corporation 82801I
- (8x) USB Controller: Intel Corporation 82801I
- VGA compatible controller: Matrox Graphics, Inc. MGA G200e [Pilot] ServerEngines
IDE Devices
- hdd: HL-DT-STDVD-ROM GDRH20N
- hdc: GB0160EAFJE (Capacity: 149.05 GB)



10 agents are logged into the campaign. the ACD all the calls is 40 seconds ( aapprox).

the physical memory reaches 70 % within first 2 hours of calling.


TOP output.

[root@vicidialnow ~]# top
top - 19:50:00 up 1:22, 1 user, load average: 0.16, 0.34, 0.32
Tasks: 98 total, 1 running, 97 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.8%us, 0.5%sy, 0.0%ni, 96.2%id, 0.3%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 3761748k total, 901136k used, 2860612k free, 34084k buffers
Swap: 779144k total, 0k used, 779144k free, 740780k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2859 root 15 0 36900 13m 4784 S 5 0.4 9:03.94 asterisk
2372 mysql 15 0 141m 23m 4480 S 0 0.6 2:02.15 mysqld
2669 apache 15 0 58392 9144 5368 S 0 0.2 0:06.13 httpd
3141 root 15 0 12380 7180 2736 S 0 0.2 1:00.26 AST_update.pl
11396 apache 15 0 58396 7316 3592 S 0 0.2 0:01.45 httpd
11553 apache 15 0 61224 9612 5336 S 0 0.3 0:01.74 httpd
15450 apache 15 0 58276 8464 4848 S 0 0.2 0:01.09 httpd
1 root 15 0 2040 628 544 S 0 0.0 0:00.54 init
2 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/0
5 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/1
6 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
7 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/1
8 root 10 -5 0 0 0 S 0 0.0 0:00.15 events/0
9 root 10 -5 0 0 0 S 0 0.0 0:00.00 events/1
10 root 10 -5 0 0 0 S 0 0.0 0:00.00 khelper
11 root 10 -5 0 0 0 S 0 0.0 0:00.00 kthread




i have changed the web browser to mozilla firefox 1.0.5 as recommended.

is ther any way to clear the physical memory without server restart?
or do i have a problem in my installation.!!


kindly assist.

regards
GOERGE M

[williamconley]

does the memory hitting 70% cause any issues with the calls?

sql servers "eat" memory. they use it for paging. they release it for use in case it's needed, but if it isn't needed they will use it to more quickly respond to sql requests.

if you do not know if it is sql doing this, you can test by restarting your mysql service.

it is NOT a bad thing to use up that much memory as long as it does not affect performance (in fact, from the database's standpoint, the more memory it uses the more "optimized" your database is, the more pages it can access of data without having to read/write to the hard drive, which speeds up databasing).

[georgem]

hello william


thanks for responding.

yes it does effect the calls. agents are getting multiple calls before they hangup the call. and after a restart everything will be fine for the next 2 hours till the ram usage reaches 70-80 % again.


regards

george m

[brett05]

yes all williamconley has say is true
mysql eat very much the memory also if we speak about centos is not good with vicidial he have many bugs
i can say you that you can try to install a debian or ubuntu with the force of 64bit then try to séparate apache/mysql to the asterisk/vicidial this will decrease the load of your cpu for this you should install multiserver i think he is the best way with vicidial also the best way to increase your number of seats and avoid centos

[williamconley]

yes it does effect the calls. agents are getting multiple calls before they hangup the call. and after a restart everything will be fine for the next 2 hours till the ram usage reaches 70-80 % again.

My best advice would be to upgrade to a newer version and likely save yourself a lot of troubleshooting, which ultimately may require the upgrade anyway. Vicibox will install on a fresh ubuntu server for you. Alternately you can upgrade in place, but if it is an OS related issue, that may not resolve your problem.

If you can afford to "fresh install" you would probably greatly appreciate the new version of vicidial (2.0.5) and benefit from some superior changes that The Vicidial Group has managed to get into this release. If you need to bring your data with you, you can follow the upgrade instructions (in every release) to bring your data "up to date" and use it in a fresh install.

If upgrade isn't possible, you should find out what application is using this memory (htop or top will help with this). Then you'll be troubleshooting that application to resolve your memory issue.

[gardo]

First of all, CentOS is as good as any distribution when using Vicidial. As with other Linux distros, it needs to be tweaked to get maximum performance.

What's the output of "free -m"? Is the system swapping memory when it reaches 70%?

[williamconley]

Sorry gardo, i wasn't disparaging CentOS. It is a marvelous OS (My preference actually, I learned on it). I was just mentioning that Vicibox is a newer version than 2.0.4.

[gardo]

@williamconley: Hehehe! No harm done. I was actually referring to Brett05's post.

Not to be off-topic:

@georgem and @brett05: This might help you better understand memory management on Linux:

http://www.linuxhowtos.org/System/Linux ... gement.htm

[georgem]

hello


this is the output of free -m

[root@vicidialnow ~]# free -m
total used free shared buffers cached
Mem: 3673 2716 957 0 63 2475
-/+ buffers/cache: 178 3495
Swap: 760 0 760
[root@vicidialnow ~]#

[georgem]

hello

i have added 4GB more RAM into my hardware. but i see only 4 GB in it..

but during BIOS start up it shows 8 GB.

i did a centOS 5.3 installation on the same machine (i386) and it detected total of 8 gb. but cent os 5.1 ( VICIDIALNOW 1.1) is detecting only 4 GB of ram..

hw can i fix this bug?

regards

GEORGE M

[georgem]

hello

i got an interesting result.


my server was using almost 77 % of the total 4 GB RAM available.

[root@vicidialnow ~]# top
top - 21:22:31 up 5:25, 1 user, load average: 0.11, 0.11, 0.17
Tasks: 108 total, 1 running, 107 sleeping, 0 stopped, 0 zombie
Cpu(s): 7.8%us, 1.7%sy, 0.0%ni, 87.8%id, 2.3%wa, 0.1%hi, 0.4%si, 0.0%st
Mem: 3761748k total, 2901308k used, 860440k free, 74780k buffers
Swap: 779144k total, 0k used, 779144k free, 2640828k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18737 apache 15 0 58312 8324 4704 S 2 0.2 0:01.08 httpd
1 root 15 0 2040 636 544 S 0 0.0 0:01.02 init
2 root RT 0 0 0 0 S 0 0.0 0:00.04 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/0
5 root RT 0 0 0 0 S 0 0.0 0:00.01 migration/1
6 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
7 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/1
8 root 10 -5 0 0 0 S 0 0.0 0:00.55 events/0
9 root 10 -5 0 0 0 S 0 0.0 0:00.00 events/1
10 root 12 -5 0 0 0 S 0 0.0 0:00.00 khelper
11 root 10 -5 0 0 0 S 0 0.0 0:00.00 kthread
15 root 10 -5 0 0 0 S 0 0.0 0:00.31 kblockd/0
16 root 14 -5 0 0 0 S 0 0.0 0:00.06 kblockd/1
17 root 14 -5 0 0 0 S 0 0.0 0:00.00 kacpid
131 root 14 -5 0 0 0 S 0 0.0 0:00.00 cqueue/0
132 root 14 -5 0 0 0 S 0 0.0 0:00.00 cqueue/1
135 root 10 -5 0 0 0 S 0 0.0 0:00.00 khubd


then i gave this command

[root@vicidialnow ~]# service httpd restart
Stopping httpd: [ OK ]
Starting httpd: [ OK ]


and the usage came down to 21 %..

[root@vicidialnow ~]# free -m
total used free shared buffers cached
Mem: 3673 791 2881 0 73 543
-/+ buffers/cache: 175 3498
Swap: 760 0 760

[root@vicidialnow ~]#


wht can be the reason behind this heavy usage.. ? i think that MYSQL is using only a little share on RAM.


kindly assist.

regards
george M

[gardo]

As you can see from the output of "free -m", most of your RAM is either cached or buffered. To better understand what they mean, here's a snippet from the link I posted:

Traditional Unix tools like 'top' often report a surprisingly small amount of free memory after a system has been running for a while. For instance, after about 3 hours of uptime, the machine I'm writing this on reports under 60 MB of free memory, even though I have 512 MB of RAM on the system. Where does it all go?

The biggest place it's being used is in the disk cache, which is currently over 290 MB. This is reported by top as "cached". Cached memory is essentially free, in that it can be replaced quickly if a running (or newly starting) program needs the memory.

The reason Linux uses so much memory for disk cache is because the RAM is wasted if it isn't used. Keeping the cache means that if something needs the same data again, there's a good chance it will still be in the cache in memory. Fetching the information from there is around 1,000 times quicker than getting it from the hard disk. If it's not found in the cache, the hard disk needs to be read anyway, but in that case nothing has been lost in time.

I really recommend you read that link so you can better understand your system.

[williamconley]

the issue at present is that when memory usage gets up to 70-80%, the system becomes unstable. the installation is not standard vicidialnow or vicibox.

troubleshooting it to find out why apache is becoming a memory hog and somehow causing instability can be a nuisance, compared to a fresh install.

on the other hand, is there anything other than vicidial installed in this machine?

if not, my original advice is still good. reinstall (or you'll have to chase down your ghost).

[georgem]

hello

thanks gardo.

that was a clear explanation regarding memory management.

what should be done now to make all 8 GB of RAM visible! on the system.
from some googling i understand that 32 bit linux os only support a max of 4 gigs of RAM. but i made a centOS 5.3 i386 installation where total of 8 GB was detected.

kindly assist!!

regards
george m

[williamconley]

look up PAE kernel

[georgem]

hello William

i will do a new installation again.. but just understanding the possibilities of the bug in my system . i don't know why apache is becoming a memory hog . can u suggest something where could the problem be?

regards

george m

[georgem]

i think i dont have kernel PAE


[root@vicidialnow ~]# rpm -qa | grep kernel
kernel-2.6.18-53.el5
kernel-devel-2.6.18-53.el5
kernel-headers-2.6.18-53.el5
[root@vicidialnow ~]# uname -rm
2.6.18-53.el5 i686
[root@vicidialnow ~]#



but recompiling may lead to failure of zaptel according to this earlier post.
being a production server, i never compiled it.

http://www.vicidial.org/VICIDIALforum/v ... a4dcb210e7

[georgem]

hello

what is the use of the log files in var/log/httpd

cuz this folder is consuming a lot of disk space too. after i delete all the content in the folder .. the HDD usage goes down by 5 GB .

approx a total of 6 GB is created everyday in 8 hrs of dialing.

700 MB is for call recordings with 10 agents ( GSM format)

rest all is the content of /var/log/httpd

[gardo]

The logfiles in /var/log/httpd contains the access and error logs of the HTTPD server (apache). You can create a crontab that cleans the log files everynight. You can also edit /etc/php.ini so that only HTTPD errors show in the logfiles. This will keep the logfiles to a minimum.

[gardo]

You can install the kernel PAE and just download and recompile zaptel. I'd also recommend you upgrade to the latest VicidialNOW version (1.2). This uses Astguiclient/Vicidial 2.0.5 and brings lots of new features and bugfixes.

i think i dont have kernel PAE


[root@vicidialnow ~]# rpm -qa | grep kernel
kernel-2.6.18-53.el5
kernel-devel-2.6.18-53.el5
kernel-headers-2.6.18-53.el5
[root@vicidialnow ~]# uname -rm
2.6.18-53.el5 i686
[root@vicidialnow ~]#



but recompiling may lead to failure of zaptel according to this earlier post.
being a production server, i never compiled it.

http://www.vicidial.org/VICIDIALforum/v ... a4dcb210e7

[georgem]

thanks gardo.

i have a shell scripts now which cleans the logs in the folders.

clean.sh

#clear httpd

cd /var/log/httpd
rm * -fr
service httpd restart


i made it executable. and now the system works fine.

i will try kernel PAE also on a testing server.

thanks for all ur posts.

regards
george m

[gmcust3]

George, even I face the same issue like yours.

I have

VERSION: 2.0.5-173
BUILD: 90320-0424

clean.sh solved yr issue ?

Do you run that sh, every 2 hrs ?


[root@vici httpd]# ls -la
total 16713028
drwx------ 2 root root 4096 Aug 1 22:23 .
drwxr-xr-x 14 root root 4096 Sep 24 02:19 ..
-rw-r--r-- 1 root root 1245560378 Nov 9 11:23 access_log
-rw-r--r-- 1 root root 15851802049 Nov 9 11:23 error_log
-rw-r--r-- 1 root root 0 Aug 1 22:23 ssl_access_log
-rw-r--r-- 1 root root 35298 Nov 9 09:29 ssl_error_log
-rw-r--r-- 1 root root 0 Aug 1 22:23 ssl_request_log
[root@vici httpd]#


I see error_log as 15851802049.

Why so big and can we delete this ?

Is HIGH memory consumption is due to this ?

[gmcust3]

15851802049 ??

Whats the unit of this ??

So big ??

[williamconley]

both log files can be deleted without harm (as long as you stay away from mysql log files, most log files can be deleted at will).

as to why they are so large ... you should try reading what is in them to find out. as a rule a file called "error_log" may have useful information in it.

in fact: all the log files in your system have potential useful information. you should be wandering around your system and reading them until you understand how your system works. believe me, you'll profit from that. especially after you manage to "google" all your errors and get rid of the ones that need to be addressed.

[gmcust3]

But did u notice the size ?

15851802049

This server is hardly 2 months old and hardly 10 agent login daily !!!

[williamconley]

error_log files get large when there are a lot of errors to report or when there is excessive logging happening.

read the file. see which is the case. fix the errors or turn off the excessive logging.

then delete the file.

only really an issue if your drive space is being used up OR if the errors being reported are affecting system performance and being ignored (because noone is reading the error_log!)

[gmcust3]

So, need to make Server Log to NO ?

[williamconley]

that depends, did you read the file and see what the errors are?

of course, you could delete it and wait for it to repopulate (then you'd be reading a much smaller file) before deciding whether to turn it off (or repair the log rotator, which is another story entirely, if this is the log file with the broken rotator)

[Kamui]

For me the solution for high memory. Add the following line to your crontab

Code: Select all

###Clear cache every 1 hour
0 * * * * sync; echo 3 > /proc/sys/vm/drop_caches

Memory usage from 84% went down to 11% instantly. I suspect there is a memory leaking somewhere. I'm using Goautodial 3.0