Very Siberian's picture

Hi,

My TKL WordPress 15 server just became inaccessible following installation of security updates. I get a "database error" message when I try to connect to WordPress. I tried the usual rebooting but to no avail.

When I download a fresh copy of TKL 15 from this site and install it, everything works fine. If I follow the prompts at install time to install security updates, though, it kills WordPress and knocks it offline and I must start over again.

I have just downloaded another fresh copy of TKL WordPress 15 and restored my site from a backup. It is working now, but how do I disable automatic security updates until this problem is fixed?

Thank you!

Rob

Forum: 
Jeremy Davis's picture

Firstly thanks for the heads up. Someone will have a look into this ASAP so as to see if we can recreate the issue that you are experiencing. If we can, then we'll work out what is causing the issue. Assuming we can reproduce it, either it's a regression which we need to report to Debian ASAP, or it's something within our config which isn't as it should be...

Out of interest, did you do any troubleshooting on what the actual issue is? E.g. did you check that the database is running? Perhaps a database update just stops the service and it doesn't auto restart? Also, sometimes major updates require a server reboot, it's rare and usually only applies to kernel updates (and even then, the server should run fine until the reboot).

Regarding disabling auto security updates, they are triggered by the /etc/cron.d/cron-apt file. So moving that file out of /etc/cron.d will stop it from running, e.g.:

mv  /etc/cron.d/cron-apt  /root/cron-apt

Once the issue has been resolved then you can return the file to it's default place (to re-enable sec updates).

update: FWIW I just confirmed that Debian released a security update for MariaDB (it provides the MySQL service in v15.x) a few hours ago. Usually security updates are really well tested, but perhaps there is a regression which needs to be reported. Alternatively, perhaps there is some config which is sub-optimal by default which is incompatible with the security update?

I've just opened an issue for this on our tracker. A colleague is looking into it now.

Very Siberian's picture

Hi,

Thanks for the quick reply and note on disabling the automatic updates. I have done so and will wait until there is a fix available.

I did not do extensive searching for the cause (at least yet) since my priority was restoring the site to a functional state. However, a quick glance at the update output shows a bunch of stuff being removed but not clearly replaced. See below in case it helps. Happy to send logs or whatever else you might need. To replicate the problem, clean install TKL WordPress 15. Skip installing updates at initial install configuration. See that there is a working WordPress site. Then go and do the security updates. Boom, your site is dead. Yikes!

CRON-APT RUN [/etc/cron-apt/config]: Tue Nov 20 00:01:01 UTC 2018
CRON-APT SLEEP: 389, Tue Nov 20 00:07:30 UTC 2018
CRON-APT ACTION: 5-install
CRON-APT LINE: /usr/bin/apt-get -o quiet=1 dist-upgrade -q -y -o APT::Get::Show-Upgraded=true -o Dir::Etc::sourcelist=/etc/apt/sources.list.d/security.sources.list -o Dir::Etc::sourceparts=nonexistent -o DPkg::Options::=--force-confdef -o DPkg::Options::=--force-confold
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 70747 files and directories currently installed.)
Removing mysql-server (5.5.9999+default) ...
Removing default-mysql-server (1.0.2) ...
Removing mariadb-server-10.1 (10.1.26-0+deb9u1) ...
Removing mariadb-client-10.1 (10.1.26-0+deb9u1) ...
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 70580 files and directories currently installed.)
Preparing to unpack .../mariadb-common_10.1.37-0+deb9u1_all.deb ...
Unpacking mariadb-common (10.1.37-0+deb9u1) over (10.1.26-0+deb9u1) ...
Preparing to unpack .../mariadb-client-core-10.1_10.1.37-0+deb9u1_amd64.deb ...
Unpacking mariadb-client-core-10.1 (10.1.37-0+deb9u1) over (10.1.26-0+deb9u1) ...
Preparing to unpack .../mariadb-server-core-10.1_10.1.37-0+deb9u1_amd64.deb ...
Unpacking mariadb-server-core-10.1 (10.1.37-0+deb9u1) over (10.1.26-0+deb9u1) ...
Setting up mariadb-common (10.1.37-0+deb9u1) ...
Setting up mariadb-server-core-10.1 (10.1.37-0+deb9u1) ...
Setting up mariadb-client-core-10.1 (10.1.37-0+deb9u1) ...
Processing triggers for man-db (2.7.6.1-2) ...

 

 

 

Jeremy Davis's picture

We don't yet understand why, but it is clear that there are some important mariadb/mysql packages being removed! My guess is that the security update depends on newer versions of these packages, but they aren't being provided by the security repo, so the old versions are simply being removed.

Once we confirm a workaround, we'll need to consider what to do next to properly fix it. We'll probably discuss with Debian security team on how they do things to ensure that our auto security update mechanism is compatible. It may be that we need to reconsider how we install security updates. Or perhaps they will be willing to change the way that they do things. Another possibility is that we could include the required packages in our security repo, but that is probably my least favourite response. Either way, we need to fix the immediate issue and look to how we can ensure it doesn't happen again in the future!

Jeremy Davis's picture

If the auto updates have already run, then there are some packages that need to be reinstalled to get things working again:

apt update
apt install default-mysql-server

That should reinstall mariadb-client-10.1 & mariadb-server-10.1 - to be sure check:

apt policy mariadb-client-10.1 mariadb-server-10.1

Note that if you run that prior to running the auto updates, they should no longer break your server. Both of those scenarios have been tested and appear to work well.

If you have any further info, questions or concerns, please post back.

kaktux's picture

hi,

i just checked an installation i made with the current tk-core version.

For that i worked with a guide that had me install:

apt install mariadb-server php7.0-mysql mariadb-client

So after receiving the "Debian security update breaks v15.x LAMP based servers" email i looked up how to check if this package is installed on my instance

dpkg -s default-mysql-server
And it says: not installed.
So as i am no expert and mostly go with guides - reading your statemend that default-mysql-server reinstalls mariadb-client/server i am now confused that this package is not installed for me. But as i use maria-db on that instance i am unsure if i am now at risk or not.
Jeremy Davis's picture

The 'default-mysql-server' package is merely a metapackage (basically an empty package which installs other packages by declaring a set of dependencies). It depends on the default Debian "MySQL" server package, which is currently mariadb-server.

You can install it if you wish, but it shouldn't be a problem if you don't...

Assuming that you manually installed these packages recently and security updates didn't uninstall them for you (as per the issue), then you should be good. If MariaDB did get uninstalled, then you could simply reinstall the MariaDB packages and you should be all systems go again. Although if you wish to install default-mysql-server you can, it shouldn't really matter...

FWIW the only reason why I mention reinstalling default-mysql-server is that is the package we install by default for all servers that include MariaDB. It depends on both the mariadb-server & mariadb-client packages so saves a little typing... :)

kaktux's picture

well ;)

As i said i use a TL 15 Core - with manually added software using mariadb. I haven't used that in the past days. But trying it out the software was no longer accessible. According to the software logs the database problems appeared the day before yesterday.

But: Solved by installing the mentioned package. So not a big deal.

 

But an off topic question: if i remember right when installing turnkey you could add your email to be notified about security updates, right? Is there a way to test this? Because so far i did not get any (although using turnkey locally on a vm in a qnap-nas might be an unusual usercase).

 

Jeremy Davis's picture

Re not getting the email, I'm not sure?! I just checked and you're definitely subscribed.

I do have a ton of bounces (from people who have obviously changed their email addresses), but your email doesn't appear to be amongst those. So it seems that it was delivered to your email (or at least didn't bounce back to us).

I'd suggest double checking your spam folder (if you haven't already -or even double check if you have). FWIW the email would have come from "TurnKey GNU/Linux ". If it's definitely not there, then the other possibility is that your email provider "silently" blocked our email. Unfortunately, if that's the case, you'll need to take it up with them.

Unfortunately, without sending another newsletter, I don't have any way to test (but actually, it's not a bad idea really). But the newsletter sending uses the same email sending facility (and AFAIK the same "from" address) as the rest of the website. So at least in theory, using the password reset page (you'll need to log out first) should allow you to test if emails from the site are getting through (actually simply subscribing to a thread should send you notifications - another way to test).

Very Siberian's picture

Thank you very much for the fast response and workaround! It works perfectly and I'm back and running now with the security updates installed. 

Would you advise re-enabling the automatic security updates? I assume so but just wanted to check.

Thanks again!

Jeremy Davis's picture

I would definitely recommend re-enabling security updates.

I'm sure you know how to do that (or could guess - simply move the cron-apt file back), but just in case:

mv /root/cron-apt /etc/cron.d/cron-apt

Whilst this issue has demonstrated that they are not 100% foolproof, they have an extremely good track record. On balance IMO they provide significantly more value than potential harm. FWIW I watch the Debian security tracker fairly closely and they release at least 3 or 4 security updates a week (often many more). Obviously every TurnKey server doesn't rely on all of them (many are for GUI programs, so do not affect any TurnKey servers) but regressions are rare and severe breakages are extremely rare.

IIRC this is the 4th problematic security update to affect TurnKey in nearly 10 years, albeit the 2nd most significant one. FTR, the worst was a cron breakage back around 2010, which broke cron (and therefore automatic updates) on all appliances! That one was due to a poorly tested cron update which was our final straw with Ubuntu and finalised our decision to move to Debian. The others were fairly specific, one was a Java update that uninstalled Jenkins (only affected the Jenkins appliance) and a Samba update (that only affected the Samba based appliances, e.g. Domain controller, Fileserver ,etc.). These other 2 were somewhat similar to this, but a little different and obviously affected a lot less users.

As hinted in the blog post, I plan to use this situation as a spring board to work with Debian to reduce the likelihood of future issues. Exactly what that will look like I'm not yet sure, but it may possibly include us tweaking our default configuration. If that happens, ideally I'll look to do that via a pushed security update (so everyone gets it automatically). Although we'll need to do some careful and considered testing before we push that to ensure that it doesn't break anything! If/when we get to that, volunteers for testing would be warmly welcomed! Please feel free to put up your hand! :)

Jeremy Davis's picture

Firstly, if you haven't already, I would strongly advise you to register for a TurnKey website user account. FWIW, I sent out an email notification to all subscribers immediately after posting the blog post.

Next up, yes, if you experience an issue "out of the blue" such as this, we'd really appreciate it if you let us know. At the very least, assuming that we can recreate the issue, then we can assist you to resolve it. And probably more importantly, if it's a bug, then we can develop a workaround/fix, improve future appliances and in cases such as this, alert other users to the issue. Everybody wins! :)

FWIW if you were using TKLBAM to generate automatic daily offsite backups, you could have restored yesterday's backup to a new server (restores to the same version should work flawlessly, restores to same major version should also generally work flawlessly) and you'd be off and racing! ;) (Sorry for the shameless self promotion there...)

Anyway... Regarding your specific issue, so the DB appears to be working ok, but WordPress still isn't right? Is it still giving an error noting that it can't connect to the database? Or is it now giving a new/different error?

If it's still a DB connection issue, perhaps re-configuring the WordPress DB connection may fix the issue?

You could manually reset the password of the 'wordpress' MySQL user account and ensure that WordPress was using that same password. Although if you didn't want to try that manually, here's a script to do it for you (FWIW, I've taken the relevant parts from the TurnKey WordPress firstboot "secrets" regeneration script. You could simply re-run that script in it's entirety, although TBH I'm not sure of what other implications that might have?! (It might make things worse as the firstboot scripts are only intended to run on firstboot). If you try that, I'd urge you to ensure that you have a current backup of all your files and your DB, just in case. If you just update the WordPress DB user password, it's highly unlikely to make things worse.

So to regenerate the WP DB password and update the WP conf to use the new password, paste this directly into your commandline (i.e. in an SSH session):

updateconf() {
    CONF=/var/www/wordpress/wp-config.php
    sed -i "s/\(.*\)$1\(.*\)');/define('$1', '$2');/;" $CONF
}
PASSWORD=$(mcookie)
updateconf 'DB_PASSWORD' $PASSWORD
/usr/lib/inithooks/bin/mysqlconf.py --user=wordpress --pass="$PASSWORD"

If that doesn't work, I suggest that you start a new thread for your issue and give some more specific details about the error you are getting. The tail of your Apache log might also be useful (i.e. tail -30 /var/log/apache2/error/log). I'll respond ASAP. (PS new threads require you to be logged in).

Jeremy Davis's picture

I imagine that with the drama you've experienced so far, you may be questioning the value of TurnKey... However, I'd like to assure you that this particular issue is a rarity and unfortunately we can't account for bad functions! :)

Please feel free to post here in the forums if you have any further issues when running TurnKey. Actually any feedback at all (including/especially constructive criticism) is always warmly welcomed. Generally though it's best to start a new thread (requires login), unless of course your issue/feedback is completely identical and/or relevant to an existing thread.

I aim to respond to forum posts ASAP, usually within a few days, often quicker. In times like this, (when a major issue occurs) I tend to watch the forums a bit closer than usual. If you do choose to go forward with TurnKey and choose to subscribe to a paid plan on our Hub to manage TurnKey cloud servers and/or (TKLBAM) backups then we also provide support via that (expected response time one work day, but usually much quicker).

Good luck with it all and hopefully TurnKey gives you some value.

Very Siberian's picture

Hi, Pierre. Sorry to hear that you're still stuck. By chance do you have an external backup of your database or WordPress site? 

Jeremy Davis's picture

Thanks again for reporting this issue Rob. Whilst it didn't stop the damage, it did allow me to limit it significantly by issuing a blog post and email notification. I had many really thankful users who either fixed it before it broke, or were able to fix it quickly with minimal downtime. So big thanks to you on being so pro-active! :)

Whilst it likely won't make much difference to you guys directly, I thought it worth also mentioning that updated images have now been produced (as of last Friday) so fresh installs should no longer have this problem.

Cheers! :)

Very Siberian's picture

You're most welcome! Happy to help. TurnKey Linux is an outstanding creation. I am grateful to you all for quickly fixing everything so that downtime was minimized. 

Cheers,

Rob (a.k.a. Very Siberian)

Add new comment