Can't start mysqld [Resolved]

[edooze]

Hi all,

See below.
Any help appreciated.

Code: Select all

mysql start
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysql/mysql.sock' (2)

Code: Select all

locate mysql.sock


This shows nothing.

Code: Select all

ps ax |grep mysqld
22581 pts/0    S+     0:00 grep --color=auto mysqld


Code: Select all

/usr/sbin/mysqld start
160104  0:57:56 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
160104  0:57:56 [Warning] The syntax '--log-slow-queries' is deprecated and will be removed in a future release. Please use '--slow-query-log'/'--slow-query-log-file' instead.
160104  0:57:56 [Note] /usr/sbin/mysqld (mysqld 5.5.46-MariaDB-log) starting as process 22033 ...
160104  0:57:56 [Warning] Could not increase number of max_open_files to more than 1024 (request: 17163)
160104  0:57:56 [Warning] Can't create test file /var/lib/mysql/dilr02.lower-test
/usr/sbin/mysqld: Can't change dir to '/var/lib/mysql/' (Errcode: 13)
160104  0:57:56 [ERROR] Aborting

160104  0:57:56 [Note] /usr/sbin/mysqld: Shutdown complete


Not sure what's happening here. Don't think it's a config issue, I'm sure there's something I'm missing, but I'm out of ideas.

In case these help:

Code: Select all

systemctl start mysql.service
Job for mysql.service failed. See 'systemctl status mysql.service' and 'journalctl -xn' for details.


Code: Select all

systemctl status mysql.service
mysql.service - MySQL server
   Loaded: loaded (/usr/lib/systemd/system/mysql.service; disabled)
   Active: activating (start-post) (Result: exit-code) since Mon 2016-01-04 01:05:06 AEDT; 10s ago
  Process: 24613 ExecStart=/usr/lib/mysql/mysql-systemd-helper start default (code=exited, status=1/FAILURE)
  Process: 24598 ExecStartPre=/usr/lib/mysql/mysql-systemd-helper upgrade default (code=exited, status=0/SUCCESS)
  Process: 24583 ExecStartPre=/usr/lib/mysql/mysql-systemd-helper install default (code=exited, status=0/SUCCESS)
 Main PID: 24613 (code=exited, status=1/FAILURE);         : 24614 (mysql-systemd-h)
   CGroup: /system.slice/mysql.service
           └─control
             ├─24614 /bin/bash /usr/lib/mysql/mysql-systemd-helper wait default
             └─24689 sleep 1



and

Code: Select all

service vicidial start
Job for vicidial.service failed. See 'systemctl status vicidial.service' and 'journalctl -xn' for details.


Code: Select all

systemctl status vicidial.service
vicidial.service - LSB: Vicidial Call Center Suit
   Loaded: loaded (/etc/init.d/vicidial)
   Active: failed (Result: exit-code) since Mon 2016-01-04 00:23:06 AEDT; 14s ago
  Process: 10140 ExecStart=/etc/init.d/vicidial start (code=exited, status=1/FAILURE)

Jan 04 00:23:02 dilr02 vicidial[10140]: Couldn't connect to database: Can't connect to local MySQL server through socket '/var/run/mysql/mysql.sock' (2) at /...pl line 65.
Jan 04 00:23:04 dilr02 vicidial[10140]: Starting Asterisk Unable to connect to remote asterisk (does /run/asterisk/asterisk.ctl exist?)
Jan 04 00:23:04 dilr02 vicidial[10140]: DBI connect('asterisk:localhost:3306','cron',...) failed: Can't connect to local MySQL server through socket '/var/ru...l line 105.
Jan 04 00:23:04 dilr02 vicidial[10140]: Couldn't connect to database: Can't connect to local MySQL server through socket '/var/run/mysql/mysql.sock' (2) at /...l line 105.
Jan 04 00:23:04 dilr02 vicidial[10140]: ..failed
Jan 04 00:23:06 dilr02 vicidial[10140]: Started Vicidial
Jan 04 00:23:06 dilr02 vicidial[10140]: ..done
Jan 04 00:23:06 dilr02 systemd[1]: vicidial.service: control process exited, code=exited status=1
Jan 04 00:23:06 dilr02 systemd[1]: Failed to start LSB: Vicidial Call Center Suit.
Jan 04 00:23:06 dilr02 systemd[1]: Unit vicidial.service entered failed state.
Hint: Some lines were ellipsized, use -l to show in full.


Thanks in advance. Let me know if you need any more info.

[williamconley]

1) Welcome to the Party!

2) As you are obviously new here, I have some suggestions to help us all help you:

When you post, please post your entire configuration including (but not limited to) your installation method and vicidial version with build.

This IS a requirement for posting along with reading the stickies (at the top of each forum) and the manager's manual (available on EFLO.net, both free and paid versions)

You should also post: Asterisk version, telephony hardware (model number is helpful here), cluster information if you have one, and whether any other software is installed in the box. If your installation method is "from scratch" you must post your operating system and should also post the .iso version from which you installed your original operating system. If your installation is "Hosted" list the site name of the host.

If this is a "Cloud" or "Virtual" server, please note the technology involved along with the version of that techology (ie: VMware Server Version 2.0.2). If it is not, merely stating the Motherboard model # and CPU would be helpful.

Similar to This:

Vicibox X.X from .iso | Vicidial X.X.X-XXX Build XXXXXX-XXXX | Asterisk X.X.X | Single Server | No Digium/Sangoma Hardware | No Extra Software After Installation | Intel DG35EC | Core2Quad Q6600

3) full hard drive?

[edooze]

1) Welcome to the Party!

2) As you are obviously new here, I have some suggestions to help us all help you:

When you post, please post your entire configuration including (but not limited to) your installation method and vicidial version with build.

This IS a requirement for posting along with reading the stickies (at the top of each forum) and the manager's manual (available on EFLO.net, both free and paid versions)

You should also post: Asterisk version, telephony hardware (model number is helpful here), cluster information if you have one, and whether any other software is installed in the box. If your installation method is "from scratch" you must post your operating system and should also post the .iso version from which you installed your original operating system. If your installation is "Hosted" list the site name of the host.

If this is a "Cloud" or "Virtual" server, please note the technology involved along with the version of that techology (ie: VMware Server Version 2.0.2). If it is not, merely stating the Motherboard model # and CPU would be helpful.

Similar to This:

Vicibox X.X from .iso | Vicidial X.X.X-XXX Build XXXXXX-XXXX | Asterisk X.X.X | Single Server | No Digium/Sangoma Hardware | No Extra Software After Installation | Intel DG35EC | Core2Quad Q6600

3) full hard drive?

Sorry, William, I normally do, it was just very late at night and I was highly frustrated.
I can't get in to the Web Interface to get the information, but this was the last information that I had to hand:

My build details:
Vicibox 6 from .iso
VERSION: 2.12-485a BUILD: 150512-2225
Asterisk 1.8.32.2-vici
Single Server
No Digium/Sangoma Hardware
No Extra Software After Installation
All phones & carriers are SIP-based.
Currently 2 agents live intermittently.
Please note that my servers are virtualised using VirtIO (KVM). I understand that this is not recommended, but a necessary evil for a small business in today's market.
Database/tel/web/archive server - 2 CPUs, 2GB RAM
I connect to my servers via a VPN

Code: Select all

df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda5        16G  4.1G   11G  28% /
devtmpfs        998M   48K  998M   1% /dev
tmpfs          1004M     0 1004M   0% /dev/shm
tmpfs          1004M   60M  945M   6% /run
tmpfs          1004M     0 1004M   0% /sys/fs/cgroup
tmpfs          1004M   60M  945M   6% /var/run
tmpfs          1004M   60M  945M   6% /var/lock
/dev/vda1        94M   48M   39M  56% /boot


disk space should be fine.

I managed to get it to throw up an issue with a file last night (something seemingly random like mysql.gd000), but I can't remember how I did it, I'll keep working on it.

Also, this occurred after an update. There were too many lines that have happened since, so I can't see exactly what was updated for you. I'm going to hazard a guess that mysql was among them - I wonder if it could be a permissions issue with an overwritten file?

[williamconley]

LOL: When you say "update", do you mean to say you updated the OS with Zypper?

If so: Fire whoever did that. They have too much time on their hands. There's no reason to update a Vicidial server that's in operation already. It's both a waste of time and dangerous.

If you were the one who did the update ... I believe you should be inflicting some sort of self-inflicted wound.

[edooze]

LOL: When you say "update", do you mean to say you updated the OS with Zypper?

If so: Fire whoever did that. They have too much time on their hands. There's no reason to update a Vicidial server that's in operation already. It's both a waste of time and dangerous.

If you were the one who did the update ... I believe you should be inflicting some sort of self-inflicted wound.

Guilty as charged! Although, the system has been in production since early 2014, with fortnightly zypper updates, and this is the first time something has been (seriously) broken. Unfortunately, my IT policies won't allow a machine to go un-updated for very long - (consider the heartbleed vulnerability etc)

Hoping to avoid a rebuild, but if needs must I've got the latest iso on hand. - I think the self-inflicted wound has already been inflicted...

Any other suggestions?

Thanks.

[williamconley]

If anyone has access to your server outside those in your organization (based on IP address), you'll eventually be attacked. If noone has access to it, then heartbleed (or any other theoretical exposures) won't matter at all. Look up Dynamic Good Guys or just settle for IP Whitelisting.

The fact that zypper didn't break anything once (or several times) in no way means it never will. In fact, it's pretty much inevitable, LOL.

A fresh installation of the same Vicidial SVN revision and a restoration of the database and configuration backup ... and you'll be back online. If'n you're lucky.

[edooze]

If anyone has access to your server outside those in your organization (based on IP address), you'll eventually be attacked. If noone has access to it, then heartbleed (or any other theoretical exposures) won't matter at all. Look up Dynamic Good Guys or just settle for IP Whitelisting.

The fact that zypper didn't break anything once (or several times) in no way means it never will. In fact, it's pretty much inevitable, LOL.

A fresh installation of the same Vicidial SVN revision and a restoration of the database and configuration backup ... and you'll be back online. If'n you're lucky.

Appreciated.

Do you have any suggestions of how to track down the source of the problem? The update will assumedly again break the system, so I'll need to know how to fix it anyway.

Thanks.

[edooze]

I have restored the backup from a snapshot prior to the update, but am having the same problem. Good to know it wasn't the updates.

See below:

Code: Select all

/usr/sbin/mysqld start
160104 21:16:01 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
160104 21:16:01 [Warning] The syntax '--log-slow-queries' is deprecated and will be removed in a future release. Please use '--slow-query-log'/'--slow-query-log-file' instead.
160104 21:16:01 [Note] /usr/sbin/mysqld (mysqld 5.5.44-MariaDB-log) starting as process 2274 ...
160104 21:16:01 [Warning] Could not increase number of max_open_files to more than 1024 (request: 17163)
160104 21:16:01 [Warning] Can't create test file /var/lib/mysql/dilr02.lower-test
/usr/sbin/mysqld: Can't change dir to '/var/lib/mysql/' (Errcode: 13)
160104 21:16:01 [ERROR] Aborting

160104 21:16:01 [Note] /usr/sbin/mysqld: Shutdown complete


Any help appreciated, I need this for production tomorrow.

[edooze]

After many restarts and throwing every command I could think of at the machine, mysql has come back up.

This finally worked:

Code: Select all

sudo /sbin/service mysql start

Code: Select all

sudo /sbin/service mysql status
mysql.service - MySQL server
   Loaded: loaded (/usr/lib/systemd/system/mysql.service; disabled)
   Active: active (running) since Mon 2016-01-04 22:04:08 AEDT; 7s ago


Then I had to run

Code: Select all

systemctl status vicidial.service

And finally

Code: Select all

sudo /etc/init.d/vicidial start
redirecting to systemctl start vicidial.service


And it seems to work for the time being.

Code: Select all

systemctl status vicidial.service
vicidial.service - LSB: Vicidial Call Center Suit
   Loaded: loaded (/etc/init.d/vicidial)
   Active: active (running) since Mon 2016-01-04 22:18:27 AEDT; 4min 42s ago
  Process: 2723 ExecStart=/etc/init.d/vicidial start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/vicidial.service
           ├─2759 /usr/bin/SCREEN -S astshell20160104221820
           ├─2760 /bin/sh
           ├─2764 SCREEN -L -S asterisk
           ├─2765 /bin/sh
           └─2770 /usr/sbin/asterisk -vvvvvvvvvvvvvvvvvvvvvgcT


Sorry I can't be of any help to others trying to solve a similar problem. I'll mark the thread as resolved anyway.

[williamconley]

If you did not change anything between "dead" and "working", I'd be a bit nervous. But I suspect you made a lot of changes and one of them worked. Save your bash history for posterity.