Page 1 of 1

SERVER HANG

PostPosted: Fri Mar 27, 2009 1:20 pm
by globalme
I have Cent OS 5.1

I have asterisk running .

Now when I look at System Information ,

I see that \"Physical Memory\" keep increasing and at one point it reaches 96%. Then my sever get hang and then I have to restart it.

I have 4 GB RAM.

Processors 2
Model Intel(R) Core(TM)2 Duo CPU E7200 @ 2.53GHz
CPU Speed 2.53 GHz
Cache Size 3.00 MB
System Bogomips 10135.71

Kernel Version 2.6.18-53.el5 (SMP)
Distro Name CentOS release 5 (Final)
Load Averages 0.27 0.35 0.31


What could be the reason ?

PostPosted: Fri Mar 27, 2009 1:25 pm
by globalme
CentOS Forum suggest :

The ancient kernel you are running? No updates since 5.1? The newer kernels in 5.3 will have substantial changes in memory handling, but don\'t wait to upgrade!


I used VicidialNOW CD rc1.1 which comes with integrated CentOS 5.1 and Asterisk.

1) If I upgrade kernel , wont it break the functionality of any files ?
2) If it wont break, then how to upgrade ?


I am using VERSION: 2.0.4-121 BUILD: 80424-0442

PostPosted: Fri Mar 27, 2009 1:28 pm
by globalme
I am NOT recording any calls. and Hardly maximum 3-4 agents are logging.

PostPosted: Fri Mar 27, 2009 1:37 pm
by ykhan
Centos has issues with memory leaks as posted by Matt in a number of posts on this forum. Please look through the Managers Manual and then through the Forum and other support resources, you are very likely to find a resolution to your issue since others have likely had the same issue before.

This is essential especially true if you are a newbie to Vicidial and Linux.

PostPosted: Fri Mar 27, 2009 1:44 pm
by globalme
1) We should avoid CentOS ?

2) If so, then why not creator of ViciNow used some other Distro and why used Cent OS ??

3) If I have to use any other distro , then I have to follow scratch install. No time to go through that as of now as entire operation will down.

4) Any sol as of now ?

PostPosted: Fri Mar 27, 2009 1:52 pm
by ykhan
Reboot your server every so often. Normally doing this during a break time for the agents does the trick.

PostPosted: Fri Mar 27, 2009 1:53 pm
by globalme
Yeah, every 45 mins./

Though it was working perfectly for the last 6 months But today.. :-(

PostPosted: Fri Mar 27, 2009 2:20 pm
by globalme
Any version of ViciNow which uses Ubuntu or any other Distro as I see that CentOS is NOT stable , considering memory management.

How about if I upgrade kernel ?

PostPosted: Sat Mar 28, 2009 10:20 am
by gardo
CentOS is as stable as any distribution can get. You might take a look at what's consuming your memory. It might be a runaway software. By the way what do you mean hang? The server stops responding?

PostPosted: Sat Mar 28, 2009 12:46 pm
by globalme
Call get stop.

I see that \"Physical Memory\" keep increasing and at one point it reaches 96%

PostPosted: Sat Mar 28, 2009 1:15 pm
by TritenTech
Hi, I was having the same problem. After searching I found a thread here that fixed my problem. I added a cron job that cleared out the cache every 20 minutes and my system has been running much , much better. My memeory never gets above the 50% now.

Added the following as a cron job. I again got this from a nother thread but it did work for me.

#### Flush cache after every 20 minutes
29,59 * * * 1,2,3,4,5,6,7 sync; echo 3 > /proc/sys/vm/drop_caches

I went to /var/spool/cron/crontabs and added it.

Hope this helps.

PostPosted: Sat Mar 28, 2009 1:22 pm
by globalme
[root@vicidialnow ~]# cd /var
[root@vicidialnow var]# cd spool
[root@vicidialnow spool]# cd cron
[root@vicidialnow cron]# ls -la
total 20
drwx------ 2 root root 4096 Oct 30 09:07 .
drwxr-xr-x 12 root root 4096 Oct 29 15:54 ..
-rw------- 1 root root 2540 Oct 30 09:07 root
[root@vicidialnow cron]#

My folder is missing ??

PostPosted: Sat Mar 28, 2009 2:03 pm
by globalme
Changed root file.

Let me observe

PostPosted: Sat Mar 28, 2009 5:10 pm
by globalme
ahhh No It didn't work...

How to check for cronjob list running...

PostPosted: Sun Mar 29, 2009 11:34 pm
by williamconley
before going as far as a cron you could always execute it manually or as a script (when you see your load beginning to creep up on you), then if it works you execute the script with a cron

and you can put something else in the script to verify that the script ran, like putting "hey! I ran!" into a file in your home folder after running

PostPosted: Mon Mar 30, 2009 2:03 am
by Op3r
Funny I dont see the need to flush the cache.

I am also using vicidialnow on my installs. Even on clients that I have I used the said ISO.

You should really check who's hogging the resources.

PostPosted: Mon Mar 30, 2009 2:10 am
by globalme
Even I am using it for the last 6 months. and thnx to VicidialNOW team who made our work very easy.

But one fine morning I see this Problem poping up.

Let me know if you need any command output to analyse the problem.

PostPosted: Mon Mar 30, 2009 1:15 pm
by gardo
Here's a snippet of an article regarding linux memory management:

Traditional Unix tools like 'top' often report a surprisingly small amount of free memory after a system has been running for a while. For instance, after about 3 hours of uptime, the machine I'm writing this on reports under 60 MB of free memory, even though I have 512 MB of RAM on the system. Where does it all go?

The biggest place it's being used is in the disk cache, which is currently over 290 MB. This is reported by top as "cached". Cached memory is essentially free, in that it can be replaced quickly if a running (or newly starting) program needs the memory.

The reason Linux uses so much memory for disk cache is because the RAM is wasted if it isn't used. Keeping the cache means that if something needs the same data again, there's a good chance it will still be in the cache in memory. Fetching the information from there is around 1,000 times quicker than getting it from the hard disk. If it's not found in the cache, the hard disk needs to be read anyway, but in that case nothing has been lost in time.

To see a better estimation of how much memory is really free for applications to use, run the command:

$ free -m




The -m option stands for megabytes, and the output will look something like this:

total used free shared buffers cached
Mem: 503 451 52 0 14 293
-/+ buffers/cache: 143 360
Swap: 1027 0 1027



The -/+ buffers/cache line shows how much memory is used and free from the perspective of the applications. Generally speaking, if little swap is being used, memory usage isn't impacting performance at all.


What you're currently seeing is linux's effective memory management. Flushing the cache every so often negates the benefits mentioned above.

Here's the link:

http://www.linuxhowtos.org/System/Linux ... gement.htm

TritenTech wrote:Hi, I was having the same problem. After searching I found a thread here that fixed my problem. I added a cron job that cleared out the cache every 20 minutes and my system has been running much , much better. My memeory never gets above the 50% now.

Added the following as a cron job. I again got this from a nother thread but it did work for me.

#### Flush cache after every 20 minutes
29,59 * * * 1,2,3,4,5,6,7 sync; echo 3 > /proc/sys/vm/drop_caches

I went to /var/spool/cron/crontabs and added it.

Hope this helps.

PostPosted: Tue Mar 31, 2009 4:29 pm
by globalme
This did a wonder.

Now My memory usage doesn't go above 10%..

Thanks.

PostPosted: Wed Apr 01, 2009 11:14 am
by cristian
Awesome response gardo.

PostPosted: Wed Apr 08, 2009 2:59 pm
by globalme
#### Flush cache after every 20 minutes
29,59 * * * 1,2,3,4,5,6,7 sync; echo 3 > /proc/sys/vm/drop_caches

OR

#### Flush cache after every 15 minutes
0,15,30,45 * * * 1,2,3,4,5,6,7 sync; echo 3 > /proc/sys/vm/drop_caches



Which one is correct ?


Ref : http://www.scrounge.org/linux/cron.html



And also, I want to repair MySQL when my system is booting. How to do it ?

PostPosted: Wed Apr 08, 2009 3:19 pm
by williamconley
mysqlcheck --auto-repair asterisk -pXXXX

place your password (no space!) in place of the XXXX.

Since we're on this track ... you may want to "auto-backup" and wipe your logs from the sql tables while you're at it. They can grow IMMENSELY if you don't have an unlimited drive size. Here's a sample of that:

mysqldump --single-transaction --flush-logs --master-data=2 --all-databases --delete-master-logs > /YYYY.sql -pXXXX
(not to be used if replicating, keep a few days worth of logs for that, google it)

Once again, XXXX is your password and this time YYYY is the file name/path.

After that, not a bad idea to copy that file to another server before you begin your run for the day (if your system dies today ... you'll have your data!)

gzip /YYYY.sql

scp -P PORT /YYYY.sql.gz USER@SERVER:YYYY.sql.gz

Once again, YYYY is the filename/path, and we also have the ssh PORT and a USER & SERVER

oops. I forget how you automatically get the password in there. And i'm out of time. You may have to look that up or use another protocol to move the file. It WILL ask you for your password (not good in a startup script)

PostPosted: Wed Apr 08, 2009 3:40 pm
by globalme
Awesome article..!!

:-)

How about cron job ?