vicidial.org

Hello everyone

I have the following problem, I just happened to be working normally suddenly disconnected all the agents and could not connect, when we review we see that the score in the report menu is red indicating that it is with another time we enter see it but the time was correct, but asterisk was not started, something strange because before it was working well and at the time of starting it only indicated (does /var/run/asterisk.ctl exist?) even stranger, what we had to do was to restart the server and when everything started was normal but I do not know what the error was, at the time of the failure, the time was checked, everything was reset, the service was restarting everything well.

It only happened to one of the servers which is also web server


In the logs I find this: / var / log / messages


2018-09-27T09:45:04.973822-05:00 pbx1 vsftpd[11614]: pam_unix(vsftpd:auth): check pass; user unknown
2018-09-27T09:45:04.974102-05:00 pbx1 vsftpd[11614]: pam_unix(vsftpd:auth): authentication failure; logname= uid=0 euid=0 tty=ftp ruser=anonymous rhost=::ffff:192.168.1.62
2018-09-27T09:45:06.253656-05:00 pbx1 CRON[11403]: pam_unix(crond:session): session closed for user root
2018-09-27T09:45:06.663999-05:00 pbx1 vsftpd[11614]: [anonymous] FAIL LOGIN: Client "::ffff:192.168.1.62"
2018-09-27T09:45:06.845752-05:00 pbx1 CRON[11406]: pam_unix(crond:session): session closed for user root
2018-09-27T09:45:06.892990-05:00 pbx1 CRON[11402]: pam_unix(crond:session): session closed for user root
2018-09-27T09:45:15.269672-05:00 pbx1 CRON[11405]: pam_unix(crond:session): session closed for user root
2018-09-27T09:45:31.082237-05:00 pbx1 kernel: [15230.241983] ../mm/pgtable-generic.c:33: bad pmd ffff88040dbd4458(80000000c98008e7)
2018-09-27T09:45:31.082294-05:00 pbx1 kernel: [15230.242075] asterisk[11731]: segfault at 0 ip 00000000004f05f5 sp 00007f2c8874c650 error 4 in asterisk[400000+233000]
2018-09-27T09:45:33.920471-05:00 pbx1 kernel: [15233.080026] BUG: Bad rss-counter state mm:ffff88040d834800 idx:1 val:512
2018-09-27T09:45:33.920486-05:00 pbx1 kernel: [15233.080029] BUG: non-zero nr_ptes on freeing mm: 1
2018-09-27T09:45:45.846984-05:00 pbx1 CRON[11404]: pam_unix(crond:session): session closed for user root
2018-09-27T09:46:01.178281-05:00 pbx1 systemd-coredump[12410]: Process 1698 (asterisk) of user 0 dumped core.
2018-09-27T09:46:01.852897-05:00 pbx1 cron[12639]: pam_unix(crond:session): session opened for user root by (uid=0)
2018-09-27T09:46:01.853237-05:00 pbx1 cron[12641]: pam_unix(crond:session): session opened for user root by (uid=0)
2018-09-27T09:46:01.853475-05:00 pbx1 cron[12643]: pam_unix(crond:session): session opened for user root by (uid=0)
2018-09-27T09:46:01.853704-05:00 pbx1 cron[12642]: pam_unix(crond:session): session opened for user root by (uid=0)
2018-09-27T09:46:01.853932-05:00 pbx1 cron[12640]: pam_unix(crond:session): session opened for user root by (uid=0)
2018-09-27T09:46:01.873047-05:00 pbx1 systemd[1]: Started Session 1288 of user root.

/var/log/asterisk/messages

[Sep 27 09:45:31] VERBOSE[12370] manager.c: [Sep 27 09:45:31] == Manager 'sendcron' logged off from 127.0.0.1
[Sep 27 09:45:31] VERBOSE[12343][C-00006920] app_amd.c: [Sep 27 09:45:31] -- AMD: Channel [SIP/carrier2-00006110]. Word detected. iWordsCount:4
[Sep 27 09:45:31] VERBOSE[12343][C-00006920] app_amd.c: [Sep 27 09:45:31] -- AMD: Channel [SIP/carrier2-00006110]. ANSWERING MACHINE: iWordsCount:4
[Sep 27 09:45:31] VERBOSE[12343][C-00006920] pbx.c: [Sep 27 09:45:31] -- Executing [8369@default:4] AGI("SIP/carrier2-00006110", "VD_amd.agi,8369") in new stack

And here is where asterisk stops because there are no more logs until it starts again but from there nothing weird

VERSION: 2.14-678a
BUILD: 180613-0943

2 pbx servers one of them has the web server
1 server dedicated to the database
Installation with vicibox_v8.x86_64-8.0.1

Thanks for the support

Asterisk crashes which is not a Vicidial issue, asterisk is an open source linux package and actually has been known to die for many reasons ... even Great running servers may crash once or twice a year.

When asterisk crashes, the ASTupdate script (running in the ASTupdate screen) shuts down as it relies on Asterisk to run. When the update script shuts down, it stops updating the time in the report you were looking at. This has nothing to do with "time" per se, except that time is the value that is no longer being updated by the dead script.

In admin->servers (for every server) there is an option to auto-restart asterisk. This option will not stop asterisk from crashing, but it will restart asterisk without intervention by your IT staff and get you back online in about a minute. Calls will still have been lost, obviously, but the impact is dramatically reduced by using this function.

It's a good idea to keep track of the log file from that crash, however, as you may eventually find a pattern (if this happens more often than you like). If it always dies at or shortly after a certain process, you may have something to investigate. Also look at the Average Server Load and other processes that may have been running (was someone running a major report? did a large number of agents just log in? were there several call in an inbound queue?)

all that being said:

pam_unix(vsftpd:auth): authentication failure; logname= uid=0 euid=0 tty=ftp ruser=anonymous rhost=::ffff:192.168.1.62

Lock down your server (whitelist only). Turn off ipv6 if you don't need it.



PS: Good job posting your Vicidial Version with Build and your Installation Method with Version. Well done.

thank you very much William

I was very helpful, the option to restart the asterisk will work a lot for future events.

I would like very clearly your answer, a question, to see if this is a server as we said above with a lot of workload, I can use the option "Load system" from the Admin-Servers menu, if so, those 2 quantities are not I am very clear with the help menu, it helps me to understand it better please.

Best regards

Simpler method:

Log in to the server's command line.

Use:

Code: Select all

uptime

... for a simple, one-time output showing the 1, 5, 10 minute Average Server Load.

These represent the number of Cores busy (on average) for those time period. Thus if the numbers are:

Code: Select all

 load average: 2.12, 3.08, 1.09

Then you had:
2.12 cores busy average in the last 1 minute,
3.08 cores busy average in the last 5 minutes, and
1.09 cores busy average in the last 10 minutes.

Then the question is: Was that good ... or bad? The answer depends on how many cores your system has. If it has 8 cores, then you were well under 50% at all times and life is good. If your system has 4 cores, then it was fairly well loaded (over 50% at the 5 minute average, so it was sweating a little bit). But if it has 2 cores, you were overloaded for a while there and in danger of failure.

What normally happens if you go over 100% (ie: 4.0 if you have 4 cores), is that it begins to snowball. It will climb quickly after it hits overload, and can go to 20, 100, and even into the multiple hundreds as it flips out trying to catch up. Any time your Average Server Load exceeds your Core Count ... you need to begin worrying (and probably should do something to reduce that load). This applies equally to all servers in a cluster, and of course to a Standalone server.

If you get lucky, whatever caused the overload will complete and everything will go back to normal. But if not ... Cascade Failure and likely a need for a reboot.

Now go back and read the help and see if it makes more sense after that description.

HiWilliam

Thanks for explaining, reviewing I have these values ​​and monitoring all day never go over the account

The Database Server has 24Core and 64Gb ram
The Web and Telephone Server has 24Core and 32Gb ram
Server (Web, Telephony) 20:47 pm up 3:16, 2 users, load average: 0.50, 0.57, 1.03
Server (Database) 20:47 pm up 5 days 22:31, 1 user, load average: 0.69, 0.73, 0.85
So by hardware capacity I do not know if there is a problem

It is very rare because it is happening every 2 days or so, it does not have a schedule, it can be in the morning or in the afternoon. The only possible solution is to restart the server physically, by setting the option of Administrator / Servers / Auto-Restart Asterisk, it does not work automatically either.

I want to review the part of the web sessions but I do not know how to do it, I do not know if there is something that I am not seeing, what would be the recommendation to review.

Best regards

Asterisk crashing would only be related to web sessions if you were being attacked and one of the exploits they use dirties your memory. Thus: Lock that server down. Whitelist only. That will remove web/hacking from your list of possible reasons.

Do you reboot this server nightly? If not, this would be a very good time to being that process. Reboot the DB first, then (once the DB is up, say 5-10 minutes later) reboot everything else. This cleans out any problems and has been known to stabilize many problematic servers.

Next up: See if you can identify what was happening at the moment of crash (if the reboot doesn't fix it, of course). Turn on loggging in admin->servers and consider logging debug output in asterisk to a file. Issue "asterisk -rx 'logger rotate'" upon reboot (or two minutes later with a sleep 120; in front of it, to be sure asterisk is running) so your logs will always have the crash in an "easy to find" place.

When you say: "The only possible solution is to restart the server physically", but don't provide details of why you belive this to be true, I have to wonder if it's actually true. Is the "asterisk" screen still running? What's on it?

Are these virtual servers?

Asterisk seg faulting once a day isnt normal.

We had servers in the past that were sensitive to specific modules. I expect if we'd dug deeper we'd have found a specific "deep package" problem related to some specific "special" feature or configuration option, conflict/compatibility with something, but turning off the module in question (or daily reboot) resolved the issue so we never bothered to dig. In the last 10 years anyway. lol.

Hi William

Look, it makes a lot of sense what you comment, within the configuration of this environment, we have a daily scheduled restart for the Web Server / Bookmark but we did not put it for the Database, with which you comment we will add the task for the database and Then I told them how it was.

For "the doubt that the only possible solution is to restart the server" I have seen 2 scenarios, 1) where Asterisk is running at the time of the fall but does not make calls or connect anything (CLI remains open without traffic but is running ). 2) when entering asterisk -rvvvv appears the legend asterisk.ctl does not exist. In both cases the agents can not do anything and we have to restart the server physically, once started they can work without problem.

Thanks for the support and the restart of the database server will be added and I will see how it goes.

thephaseusa: the servers are not virtual

Best regards

You can't just start asterisk. You have to use the vicidial method to start it. There is a Screen running asterisk. Attach to that screen next time and see what the situation is. If asterisk has stopped, all you need to do is hit the up arrow and then enter. That will run the previous command in that "screen" cli history, which will be the proper command to start asterisk IN THAT SCREEN (which is essential). Of course, this is also the method that's used by the auto-restart. Report your results (for instance: the last asterisk commands or the segfault report, or whatever).