You are here
Serializing MySQL database to /TKLBAM/myfs
------------------------------------------
Traceback (most recent call last):
File "/usr/bin/tklbam-backup", line 510, in <module>
main()
File "/usr/bin/tklbam-backup", line 443, in main
opt_resume, True, dump_path if dump_path else "/")
File "/usr/lib/tklbam/backup.py", line 237, in __init__
self._create_extras(extras_paths, profile_paths, backup_conf)
File "/usr/lib/tklbam/backup.py", line 183, in _create_extras
limits=conf.overrides.mydb, callback=mysql.cb_print()) if self.verbose else None
File "/usr/lib/tklbam/mysql.py", line 553, in backup
mna = MysqlNoAuth()
File "/usr/lib/tklbam/mysql.py", line 698, in __init__
self.orig_varrun_mode = stat.S_IMODE(os.stat(self.PATH_VARRUN).st_mode)
OSError: [Errno 2] No such file or directory: '/var/run/mysqld'
Exception AttributeError: "MysqlNoAuth instance has no attribute 'orig_varrun_mode'" in <bound method MysqlNoAuth.__del__ of <mysql.MysqlNoAuth instance at 0x7fa2f396e560>> ignored
I have mysqld in the following location - /usr/sbin/mysqld.
Additionally, after trying to
Additionally, after trying to run this backup, it has done something to my set up so that my systems cannot connect:
A server restart, everything
A server restart, everything is fine. Does TKLBAM try to stop mysqld or something?
(This server runs applications off Tomcat btw. )
Any ideas on this? Stopping
Any ideas on this? Stopping Tomcat before the backup does not help either. Only other way this server differs from my others is that it is from OVH. Thanks.
Bump: OSError: [Errno 2] No
Bump: OSError: [Errno 2] No such file or directory: '/var/run/mysqld'
rest of error as per OP.
Can't work out why I can't get this server to backup on TKLBAM.
Sounds like you are running out of RAM and MySQL is crashing
When you reboot, everything seems fine again (MySQL is running again) until next time you run out of RAM...
Obviously I'm only guessing and I can't be 100% sure, but going from experience, I think it's likely I'm right...
This is a 6GB memory machine.
This is a 6GB memory machine. The highest memory has ever peaked is 55%. There's a fairly small database on it.
Looking at the monitor from yesterday when I was trying this backup again, memory peaked at 28.75%.
So, you appear to understandfrom the error message that mySQL is crashing. - But there has to be something else causing it aside from a memory issue.
Hmm ok, sounds like not memory then...
As a general rule most logs (especially system ones and most Debian packages) can be found in /var/log . Probably the one that will be most helpful here is the MySQL log. IIRC it's something like /var/log/mysql.log (or similar).
Hopefully that may give some clues.
Thanks for that.
Thanks for that.
It seems to be denying access. Do I need to somehow put the root password into TKLBAM?
2017-05-30T08:24:15.435239Z 23723 [Note] Access denied for user 'root'@'localhost' (using password: NO)
2017-05-30T08:24:15.441292Z 23724 [Note] Access denied for user 'root'@'localhost' (using password: NO)
2017-05-30T08:24:15.452058Z 23725 [Note] Access denied for user 'root'@'localhost' (using password: NO)
2017-05-30T08:24:15.457254Z 23726 [Note] Access denied for user 'root'@'localhost' (using password: NO)
2017-05-30T08:24:15.458883Z 0 [Note] Giving 0 client threads a chance to die gracefully
2017-05-30T08:24:15.458917Z 0 [Note] Shutting down slave threads
2017-05-30T08:24:15.458931Z 0 [Note] Forcefully disconnecting 0 remaining clients
2017-05-30T08:24:15.458955Z 0 [Note] Event Scheduler: Purging the queue. 0 events
2017-05-30T08:24:15.459033Z 0 [Note] Binlog end
2017-05-30T08:24:15.501800Z 0 [Note] Shutting down plugin...............................................
Shouldn't be TKLBAM
But it certainly looks like something is trying to log in to MySQL using the root account (but no password)!
Also perhaps there is a separate MySQL error log? Actually, after the service has crashed, but before you restart it, if you run 'service mysql status' that should give you the last few lines of log for the MySQL daemon. Even if that doesn't give you a ton of clues, it should also give you a hint on what command to run to get further daemon log entries. You could also manually check the daemon log, but using SystemD is probably easier as it will only give you the MySQL related stuff.
Thanks. I will look further
Thanks. I will look further into the logs.
The above was copied from the mySQL error log.
Could TKLBAM trigger some other process to try and log in to mySQL?
The only other user account in this installation of mySQL is mysql.sys@localhost - is that the system maintenance account you mean that TKLBAM uses?
I guess mysql.sys must be the system maintenance account
Sorry I thought that looked more like a general log rather than an error log. It certainly seems like it was shutting down though. So perhaps look higher up/further back in that log (if there is higher up/further back). It'd be interesting to see what else was happening at "08:24:15", or just before...
If that is all there is, then perhaps have a look for a log file with the same name but with a number appended on the end. There may also be files with a .gz on the end. The files ending in /gz have been compressed so you'll need to use zcat to view them (rather than cat or tail).
Also IIRC MySQL has 2 logs, an error.log and a more general log. So I just checked on a server that I have running here and I actually see 3 log files (all in /var/log): mysql.log, mysql.err and mysql/error.log. On my system both mysql.log, mysql.err are empty but mysql/error.log has a few entires, so I'm guessing that's the one you're looking at. It may be worth double checking those other 2 log files just in case.
Also there are other logs which may give additional relevant info. daemon.log (log of all services), syslog (general system log), dmesg (kernel messages - maybe there is something funky going on with hardware?). I'd be looking for time stamps that roughly match what is in your MySQL error log (i.e. around 08:24 2017-05-30). Note that some of the logs may have a slightly different timestamp format. dmesg actually uses a completely different time scale (time in milliseconds since boot).
Once you think you may have found something of interest, to check, reboot your server (so everything is running nicely). Then log into your server using multiple separate SSH sessions (one for each log you wish to watch, plus one extra to run the backup). In all but one of the sessions, tee up the log files to display in real time. Do that like this:
That should display the last few lines of the log file, then sit there waiting. When new log entries are written, they'll display here instantly. To exit, press <Ctrl>-;<C>.
Once you have all that set up, manually trigger a TKLBAM backup in the remaining session ('tklbam-backup'). Hopefully that will give you some clearer idea of what is actually happening and when...
If you find anything that seems relevant, please post (probably better to err on the side of sharing too much, than not enough).
Actually, one other thing that occurs to me is that perhaps you are running out of HDD space? That would cause similar behaviour (although TBH seems unlikely as there'd probably also be other warnings). To double check, try this:
Look for the volume mounted on '/' (should be the first entry under the headings). Ideally that shouldn't be too much more that about 70% (to leave headroom for TKLBAM backups and restores) although it probably shouldn't cause immediate issues so long there is enough free room to create your backup.
I have the exact same issue now trying to restore a server.
I still have a problem with this.
I never resolved the above, which meant I could not back up the server. I just left it, and backup manually.
I have the exact same issue now trying to restore a server.
They are different servers. The commonality is they are both running Apache Tomcat. So, this may underlie the problem.
Again, the main error returned in the console is:
And the mySQL log returns:
So, you said above that "... By default TKLBAM should be using the hidden "system maintenance" account (special root account only available locally which can access MySQL using a stored password)..."
It's obviously not doing this. Perhaps I would have switched this account off to harden the server in the past? - I don't recall doing so though.
How can I check that account referred to is available to TKLBAM? Exactly?
Or perhaps Liraz would have some ideas on this?
This is on Debian version 8.9
MySQL version 5.7.20
Tomcat 7
Thanks.
Ok, I seem to have disabled
Ok, I seem to have disabled the debian-sys-maint account when hardening. That might explain it.
I need to look into how to reenable it, and you mentioned a 'stored password'.
Hopefully I get it resolved. I will update this if so.
Any tips welcome. Thanks.
I don't recall removing that
I don't recall removing that account manually though. I wonder how it came about. Any advice on reinstating it welcomed. There does not seem to be a /etc/mysql/debian.cnf file there either.
Perhaps you removed it?!
TBH, I'm a bit surprised that anything works after removing that. You most likely wouldn't be able to start/stop MySQL service either (as the system wouldn't have permission).
This shouldn't be destructive to any data stored in your DB (or anything else), but as per always when you're doing system level changes, make sure you have a backup of everything important. Then simply reconfigure the MySQL package:
You should be good to go now...
I'm using MySQL version 5.7
I'm using MySQL version 5.7.20.
Also, I have always used apt-get as my package manager.
Do you still recommend I use the above command modified to
?
Thanks.
That command doesn't work. I
That command doesn't work. I am using 5.7.21 MySQL Community Server.
The three users I have are:
The three users I have are:
mysql.session
mysql.sys
root
Everything works fine with it except for TKLBAM.
I would definitely not have
I would definitely not have removed that account and the debian.cnf file myself.
Hmmm, I have no idea on mysql-5.7
You could try reinstalling it. I.e. 'apt-get install --reinstall <package-name>'. Not sure if that will affect your DB or not (shouldn't do, but I can't be sure).
Also we've never tested TKLBAM thoroughly with MySQL of a later version than 5.5, so perhaps it's incompatible? We've done some preliminary testing with MariaDB (10.1, what TurnKey v15.0 will have - roughly the same as MySQL 5.6) and it appears to work fine so far. TBH, I'd expect it to "just work" but there are significant changes between 5.5 and 5.6, let alone 5.7, so there are way to many unknowns to be sure.
In retrospect, first up I probably should have clarified the version of TurnKey you were running on and what customizations you had done... Hindsight...
I ran
I ran
but still those threee users.
Yes. I had originally
Yes. I had originally installed mySQL 5.7 from a downloaded package on these servers (I needed 5.7 rather than 5.5 for certain reasons.)
I just saw this:
"Nice to know your feedback on our APT repo packages. Unfortunately, you have been hit by an unsupported case where we do not provide both debian-sys-maint user debian.cnf file through our packages. They are provided only by the native repo packages." - https://bugs.mysql.com/bug.php?id=73711
So, to try resolve this, I put in a new debian.cnf file, at /etc/mysql/, copied from elsewhere.
And I ran command in mysql - GRANT ALL PRIVILEGES on *.* TO 'debian-sys-maint'@'localhost' IDENTIFIED BY 'password-in-debian.conf' WITH GRANT OPTION; FLUSH PRIVILEGES;
It seems to be working now. At least I don't get that error, and it appears to restore ok.
So just to repeat, if anyone installs MySQL 5.6 or 5.7 from a downloaded package rather than the native mySQL package, they will be hit with this problem.
Any insights on this case, or experience in it, appreciated. Thanks.
Awesome work!
Like I mentioned, I don't have any experience with MySQL-5.7, so can't offer any guidance, but now you mention it, it makes a fair bit of sense that it wouldn't work as it was. But your workaround should provide everything that TKLBAM requires.
One thing that it probably worth mention though is that you may hit this (or somewhat similar) issue if/when you use TKLBAM to migrate your data to a new server. TKLBAM will reinstall packages, but only packages from default repos (i.e. Debian and TurnKey).
So that will mean you'll probably want to split the restore into multiple operations. First download your backup to the new server. Then make the required tweaks, including adding the 3rd party repo and the new config required for the new version of MySQL. Then restore all the data etc. If you do it that way, it should install the new version of MySQL and everything should just work. Having said that, YMMV, but please do not hesitate to post with any questions or concerns and I'll do my best to help out.
Thanks for your help, Jeremy.
Thanks for your help, Jeremy. You pointed out the problem for me.
Just towards feedback, I only use TKLBAM to back up the databases - using overrides, and the skip packages and skip files tags on restore. - I use whole server snapshots to back up and restore the basic Debian set up, and the two server applications set up on it, and git to manage backup and restore of my main application files.
Also, I know from another server, that one shouldn't have any trouble if one upgraded from the native mySQL 5.5 to 5.6 or 5.7. If you do it that way, the debian-sys-maint user and debian.cnf file are kept. I only encountered this when installing straight from the non-native package.
All the best.
Ah ok, no worries.
Whilst snapshots are a nice easy way of backing up the file system, unless you stop the server when the snapshot is taken, there is a small risk of data corruption. I'm guessing that you are aware of that though, hence why you use TKLBAM for the DB and git for your app. But I thought it was worth saying anyway, just in case! :)
Good luck with it all moving forward.
Add new comment