Apologies on the slow response. I've been laser focussed on getting the RC (release candidate) of v15.0 ready for public consumption. It's been a long time coming, but it's really close now...!

Anyway, first thing, that link you posted gives me a 404?! I wonder if it's something to do with the website update?

Regarding your issue, that does sound a bit weird. I suggest that you check what the RAM and CPU are doing when it "hangs". I'm totally guessing, but I'm wondering if perhaps you are running out of RAM?

top is probably a good option to check the CPU/RAM usage (and what's using it). If you log in in 2 SSH sessions, and run TKLBAM in one and top in the other, hopefully that may at least give us some additional info.

Another thing worth doing is to check the MySQL logs to see if they are giving any additional info. They should be in /var/log although I forget the exact location OTTOMH.

Out of interest, how long have you let it run for to see if it eventually gets over it? I'm guessing quite a while...

Thanks for the additional info Bill.

It sounds like all my guesses were wrong then and the issue runs a little deeper than I had hoped. So sorry for my rant above. It sounds like you are already all over the points I noted...

Now I can see the top info you've posted (thanks for fixing that) I'm even more perplexed! Even when it freezes, there's still about 180MB RAM free and almost no CPU load! You are right that it is weird that it stops after dumping the DB and that restarting MySQL allows it to continue?!

At first glance, I noticed that when it freezes ksoftirqd is the second highest usage of CPU. TBH, I don't know a lot about it, but AFAIU excessive ksoftirqd CPU usage is often related to hardware issues. But on closer look, it's only using 0.7% CPU - which is hardly excessive!

So it doesn't appear to be RAM or CPU related (at least not on face value). It seems to me, like TKLBAM is waiting for MySQL to finish, but as far as MySQL is concerned, it's done...!

Did you check the MySQL logs to see if that is reporting anything amiss at that point? TBH, my guess is that there won't be much (if anything) of interest there, but probably worth a look. Beyond a MySQL and/or TKLBAM edge case bug, the only other thing that I can think to check would be I/O stuff (again I doubt it's that, but who knows...). I just discovered SysStat and it looks like it's in the Debian repos. TBH, I've never used it, but it looks like it may assist with double checking IO related stuff?!

If you run out of ideas and you're willing to share your backup with us, we can do some testing? We possibly just need a copy of your DB? If you do, please feel free to santise the data first (i.e. remove anything sensitive, e.g. user info). But either way, please make sure that the issue still occurs with the data you share. Your backup looks pretty big, so assuming that it's reproducible with just the DB, that's probably best. I'm not 100% sure on the best way to get it to me, but if you can upload it somewhere, then I can download it. Obviously don't post it somewhere public! If I need credentials to access it, you can email me directly via jeremy AT turnkeylinux.org. If you use google drive, then that could be an option (my google account is jeremy AT jeremydavis.org).

Can you please test just doing a tklbam dump to the local FS and see if the same thing happens?

mkdir /tklbam-dump
tklbam-backup --dump=/tklbam-dump

TBH, I don't expect it to give a different result, but I'd like to be sure...

I've just loaded your DB into MySQL on a locally installed v14.2 LAMP appliance (I figured that would that be close enough). FWIW the VM is running on KVM with a 15GB vHDD, 2GB RAM and 2 vCPU cores (the hardware CPU is an Intel Atom octo-core running @2.4GHz) - so should be much lower powered than your AWS instance, but it completes quite quickly and with no obvious issues.

[...]
table: wp_votetonya/wp_votetonya_wfScanners
table: wp_votetonya/wp_votetonya_wfStatus
table: wp_votetonya/wp_votetonya_wfThrottleLog
table: wp_votetonya/wp_votetonya_wfVulnScanners

UNCOMPRESSED BACKUP SIZE: 1.88 GB in 6978 files

TIP: test your backups with a trial restore BEFORE something bad happens.
root@lamp ~# 

Unfortunately I missed it finishing, so I'm not sure whether it paused or anything, but I will try again over the weekend. I may even do a screen recording so I can replay it.

It's probably also worth double checking that we're using the same versions. Please give me the output of:

apt-cache policy tklbam mysql-server | grep Installed

FWIW this is what I get:

  Installed: 1.4.1+17+g71478bd
  Installed: 5.5.59-0+deb8u1

(top one is tklbam; bottom one is mysql)

So the next step will be to try to reproduce the issue on AWS. Assuming we're on the same version etc, my hunch at this point is that there is some corner case AWS hypervisor issue, or perhaps . Can you please clarify the version of TurnKey you are using and the size and region the server is running. Hopefully that way I have a good shot of reproducing it.

This is just a hunch, but my guess is that it's something to do with your my.cnf

If this backup originally came from a previous major TurnKey version (i.e. my guess is v13.x if the original backup was 2013) then perhaps you are missing some config option(s) (or perhaps have some that are causing unexpected results)? When I first saw your changes due to deprecated features, I thought that may have been the issue, but on reflection, that may have been due to the version changes, but perhaps you missed something important?

The thing that I previously overlooked when I suggested checking the /etc git changes, was that the .git directory (where the changes are saved) would also have come from your backup! So whilst it's interesting, in this case it won't show what I was hoping it would show!

So IMO, what you'd want to do next is check what the default my.cnf for v14.x should be and compare that to your current one.

Unfortunately, I don't have that LAMP server I was working with previously, but I just quickly created a new one to double check. This should work, but no guarantees...

Make sure everything is committed in /etc (it should automatically commit when you run the package management commands I'm about to get you to use, but let's be sure...).

cd /etc
git add .
git commit -m "manual commit before reinstalling default config"

I've just doublechecked where all the MySQL config comes from, but for the record, here's how you do it:

dpkg -S /etc/mysql

On my test system that returned this:

mysql-server-5.5, mysql-common: /etc/mysql

So the contents of /etc/mysql comes from those 2 packages (mysql-server-5.5 & mysql-common). So now we'll reinstall those and specify that we want the current config files overwritten:

apt-get update
apt-get -o Dpkg::Options::="--force-confnew" install --reinstall mysql-server-5.5 mysql-common

Now the current HEAD of the git repo in /etc should be the new (original) config files from the mysql-server-5.5 & mysql-common packages. So this should do the trick:

git diff HEAD^ HEAD

FWIW "HEAD" is the latest commit and "^" is shorthand for the parent commit, so "HEAD^" is the parent of (i.e. one before) "HEAD". So essentially we are doing a diff between the latest commit and the one before (actually the other way round, but hopefully you know what I mean!).

My guess is that there is something in there that isn't as it should be. Either something missing that it needs, or something that doesn't act as it probably should. Whilst it won't be tuned to your ideal production scenario, I suggest doing a quick test as per above, i.e.:

mkdir /tklbam-dump
tklbam-backup --skip-files --skip-packages --dump=/tklbam-dump

That will at least confirm (or not) my suspicion.

Hopefully it does and the backup now works! If so, the next step will be troubleshooting which bit was actually problematic. I suggest that you adjust the defaults in your new my.cnf to what you note above. Although I'd recommend that you don't add any new ones (at least not at this stage). If you want to keep track of your changes as you go, you can manually commit to the /etc git repo and use git tools to rollback changes that break things.

Fingers crossed that works as I hope it does. If you can narrow it down to exactly what the issue was/is, that'd be super awesome, but even if you can confirm (or not) that I'm on the right track may assist others in the future!

If I'm completely off on the wrong track, then I'm not really sure what to fiddle with next. But let's cross that bridge if we get to it! :)

(PS hope you don't mind but I edited the layout of your git diff output above to make it a bit easier to read)