You are here
Greetings:
I've searched both the forum and documentation for a resolution to the following issue. I'm trying my best to rectify this on my own, but now I don't know what steps I overlooked.
Web site went down last night. I've had to reboot our Micro Lamp Stack weekly for the past few months to keep the site functional. When it went down last night I restored a TKLBAM backup onto a new Small Lamp server, changed the DNS to reflect the new IP, and now it seems MYSQL will not run on the new server .
Is there a simple step I'm overlooking, other than launching the MYSQL page from the hub and clicking "start"?
I'm unfortunately not as technically well-versed as I'd like to be in the functions of this host. But I feel like I'm close, and likely missing a simple obvious step I cannot find.
Any help would be greatly appreciated.
Reis
Have you checked logs?
Hi Reis, Sounds like a major pain...! :(
I would recommend that you check the error logs (/var/log) - particularly MySQL logs (IIRC mysqld.log & mysqld.error) and daemon.log (i.e. services starting/stopping). The main log (messages) probably also contains info that may be relevant...
Just manually starting MySQL (i.e. 'service mysql start') may even give you some clue.
TBH my guess (without anywhere near enough info...) is a corrupt DB although if your DB is using innodb then even just corrupt log files can stop it from starting...
If you still can't work out what's going on I suggest that you post (any MySQL relevant) logs; along with some more info about anything else you've tried or tested.
Check that you are not out of disk space
With a Micro instance you will have a small amount of RAM which is augmented with swap space on your disk. There is a process monitor on linux systems called the OOM Killer. When you run low on memory it kills the biggest RAM user which is usually the database. The RAM hog though is usually apache. If you get a lot of robots accessing your site Apache default settings allow too much memory to be held by Apache...and the OOM killer kills your database. When you reset the appliance the apache RAM allocation is reset to a minimal amount so your server runs again until the robots hit your site again.
If you have logging enabled your logs can get really big. I have noticed a lot of heavy apache traffic logging over the past few months and make a point of going in and deleting old achive logfiles which appear as zipfiles on your disk. If you are low on disk you won't be able to create a sufficient swap space for your MySQL to run.
That's my best bet for a likely culprit based on the limited information you have provided.
Cheers,
Tim (Managing Director - OnePressTech)
To reduce / eliminate reboots - adjust apache defaults
Hi Reis,
I'm glad you sorted out your reboot issue. During my 35 year career, I found that log files filling up disk space is the number one cause of servers failing to operate correctly. The second likely cause of problems is dumb-as-a-stump watchdog software that does things it shouldn't. You have experienced both in one go unfortunately.
Regarding reboots...you will experience fewer reboot situations with a small appliance but they will still happen unless you adjust the apache default settings so that Apache can't exhaust your RAM. See the following blog article I contributed sample settings to: http://www.turnkeylinux.org/forum/support/20150117/replacement-turnkey .
Regarding swapping IPs it depends how you have set things up. I am not familiar with the TKLX Server Hub so not sure how to do it via the hub but if you log into your AWS account and assuming you are using an Elastic IP address, it's one click to dis-associate the public elastic IP address from the old micro instance and one additional click to associate the Elastic IP address with the new small instance. See the following link for a screenshot of the elastic IP associate / dis-associate feature http://media.amazonwebservices.com/articles/nat_monitor_files/ec2_associated_eips.jpg
Jeremy would be able to tell you how to do it via the Hub.
Cheers,
Tim (Managing Director - OnePressTech)
Hi Reis, Sorry for delay
I've been crazy busy...
How is your current DNS setup? Are you using HubDNS and/or an elastic IP?
If you have an elastic IP attached to your current server then you should be able to simply detach it and reattach it to your new small server. However I've just been testing it in the Hub and for some reason it was giving me errors (about my server being in a VPN - correction: VPC!!). But I just used the AWS Management Console to do it. And I'll raise the issue with Alon.
VPN or VPC?
Just checking...was that a typo on the error message...did you mean VPN or VPC. I could see the latter more than the former. Managing ip addresses in a non-default VPC is different then managing it in a default VPC (EC2 Classic replacement a the default). It is possible that the hub is configured as a TKLX VPC or set of VPCs specific to each client. That should be apparant via your AWS console.
Just curious :-)
Cheers,
Tim (Managing Director - OnePressTech)
Oops; right you are! :)
Yeah VPC! :)
Funny thing was though: The elastic IP (that I created via the Hub) couldn't be associated with the Micro instance (that was also launched via the Hub) - hence the error. However in AWS Console I could associate the same IP to the same instance... The Hub also showed that it was successful...
You may have bypassed the VPC indirection
WIth a non-default VPC you can associate the Elastic IP to the instance or you can associate it with the VPC elastic network interface (ENI) and then attach / detach instances from the ENI. It is possible that you are experiencing a collision of old and new worlds.
See http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/vpc-migrate.html
and
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html#VPC_EIPConcepts
and
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html
There are a number of possible ways that the TKLX hub has been designed to configure things but you should be able to reverse engineer it by looking via your AWS console. Do you have an ENI?
NOTE: You will see different behaviour if this is a new instance or an old instance due to the switch from EC2-Classic to EC2 VPC. A new instance only has EC2-VPCs and and old instance has both.
Fun and games :-)
Cheers,
Tim (Managing Director - OnePressTech)
Thanks Tim
You obviously have a much deeper understanding of all this than me! :)
This was a freshly launched instance so I am guessing that the Hub needs a bit of an update in that regard... I'll let Alon know...
Thanks again! :)
Add new comment