You are here
Hey everybody,
I use the Turnkey Wordpress appliance on a Server running Proxmox and everything has been running perfectly, but since yesterday, it stopped working and I am not able to fix it.
I did not install any updates or new plugins or didnt post content. Wordpress gives the "Error establishing a database connection" message, so I thought it might be a problem with mysql. When I tried to log in into the VW, I noticed it is running really slow, it is using over 900mb of the assigned 1000mb RAM. When I quit the mysql service with "service mysql stop", RAM usage drops to 200mb again. If I start it again, the system gets really slow again and high RAM usage is back.
All packages are up to date with apt update and apt upgrade.
I checked the error.log in /var/log/mysql, but I am not very familiar with mysql. I attached the error log as a text file.
I already thought about it being an error in the database, so I restored to a snapshot in Proxmox, where I know everything was working, but its still the same problem.
Any help would be higly appreciated, as I really don't know how to fix it anymore.
Thank you so much!
Hi Max
Hi Max, I can't see your log so it seems attaching it didn't work?!
Anyway, there are a few likely culprits which are worth excluding before we dig in too deep. All my instructions will assume that you have logged in as root at the commandline. Personally I prefer (and recommend) just using an SSH session, but the specifics of how you get a shell are more-or-less irrelevant. Seeing as your running on Proxmox, I'll assume that you'll be using root, but just in case someone else reads this (who has root disabled), you can either enable the root account and log in there, or get a root shell like this:
The first thing I'd try is checking it's statust:
Look for where it says "Active:" and if it's running it should be "active (running)". If it's "inactive (dead)" then it's not running and IIRC for MySQL "active (exited)" is also considered a failed state. If you're not sure, please feel free to post the output of that command and I can assist interpret.
If it's not running, then try starting it:
Or if it is running, try restarting it:
If WordPress is still failing to connect, then recheck the status and the most recent log entries which are also displayed. Hopefully this should give some hint on what has gone wrong. Again please share it if you want a hand interpreting.
If it refuses to start and you're not quite sure what the logs messages mean, the first thing to check is see if you've run out of disk space. This is a very common cause of MySQL (technically MariaDB in v15.x - but for ease, I'll just call it MySQL from here on) crashing and refusing to start. You can check that like this:
And to assist you to decode that, this is the output from a TKLDev (KVM) server I have running locally:
And in an effort to give you something closer to what you're likely to see, here's the output from a Proxmox LXC container that I have running locally:
The line that notes the root (i.e. '/') mount is the one you'll want to check (unless you've done something fancy with your mount points). So in my first example (KVM VM) I have /dev/mapper/turnkey-root mounted as '/' and it's 16GB total, 3.9GB used (26%) with 12GB free.
In my second (LXC) I have /dev/mapper/pve-vm--101--disk--0 mounted as '/' and it's 7.9GB total, 2.0GB (27%) used and 5.5GB free. Both my examples have plenty enough room. if you have less than 1GB, it's probably time to evaluate your options and either have a big cleanup, or extend your root partition. Although unless you have 1% or less free, that's unlikely to be the cause of MySQL/MariaDB issues.
It's also worth noting that on Proxmox, because by default it uses LVM-thin as a storage volume, it's possible to over allocate disk space. TBH, I'm not 100% sure, but I suspect that if the Proxmox storage your container volume resides on is full, within your container it would still report that there is free space (unless you've used it's allocation). So it'd also be worth double checking the host storage mount where your container's volume is stored.
The next thing to check is RAM. If MySQL starts again ok, but then continues to crash (seemingly) randomly, then running out of RAM might be the cause? Unfortunately this is a bit harder to test as with MySQL crashed, you likely have tons of free RAM. And if the issue is caused by a memory leak somewhere (e.g. a poorly written WordPress module), even with it running it may not be apparent. Regardless, here's how to check for free RAM:
On my VM:
On my container:
Because Linux uses free RAM to buffer disk writes, you need to consider the "available" value (not the "free" value). If the system needs the cached space back, it will flush it straight to disk. So I have 1754MB free in my VM and 684MB free in my container. Considering that's well over 50% of the RAM that each has (and they're both running fine) it's clearly not an issue on my machines. If MySQL is running and isn't crashing regularly, and you have a similar amount of headroom, then RAM is likely not the issue. Although as I say, if it is crashing every few days, then a memory leak may be the issue.
Again it's worth noting that you can also over allocate RAM on Proxmox. So if you have a ton of other servers running and the host system is reporting high RAM usage, then perhaps RAM is an issue?
Having said all that, re-reading your post, it seems that MySQL is not starting at all and as such RAM seems an unlikely culprit. Whilst lack of free space may still be a possibility, as you noted that you tried restoring an old backup and that didn't help. That makes me think that free disk space is less likely (or at least on the guest, host running out of space may still be a possibility) as IIRC Proxmox usually warns about lack of free space.
The only other likely possibility that occurs to me is data corruption. But again, that seems unlikely as you've restored to a "known good point" and it's still broken. So TBH, I reckone we'll need some more info.
FWIW you can get the last few lines of the log via "tail /path/to/logfile". If you'd like to post the output of that here (perhaps also with the output of the status command I gave earlier) and we'll see if there is anything that sticks out...
Fixed it
Hey Jeremy,
thank you so much for your detailed answer and trying to help me out!
I always use the root account via SSH, and already checked the storage before I asked in the forum, and the output was the following:
root@xxx ~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 487M 0 487M 0% /dev
tmpfs 100M 12M 89M 12% /run
/dev/mapper/turnkey-root 17G 2.2G 14G 14% /
tmpfs 499M 0 499M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 499M 0 499M 0% /sys/fs/cgroup
Anyway, I just wanted to let you, and everyone else who might run into this error tell you how I fixed the problem in the meantime:
I found an article explaining how to repair the mysql database automatically on https://www.thegeekstuff.com/2011/12/mysqlcheck/
I ran the commands shown there and now everything works perfectly again, however I am still surprised, why the error existed in the snapshot as well, which ran perfectly at the time I took the snapshot. Maybe the database was already corrupted at the time I took the snapshot.
Many thanks again!
Glad to hear you got it sorted...
Ah ok. So it was a corrupted DB after all. I'm glad to hear that you managed to resolve it.
Add new comment