Hi Keyturns,

Caveat: I don't work for the Turnkey Linux organisation...I am just a TKLX VM user like you.

When it comes to security and security breaches a TKLX VM is simply a packaged Debian release with some security features enabled (like auto-patch and component password reset on first boot). The TKLX VM is no more or less secure than any other Debian VM.

Regarding how you got hacked...have you checked the WordPress vulnerability databases to ensure you are not using a vulnerable plug-in (in Dec 2017 a number of plug-ins changed ownership and the new owner put in a back door)?

If I were you I would start with a fresh VM, lock it down, and migrate WordPress data over manually.

I hope you get it all sorted.

That really sucks! :( Apologies I'm a little slow responding (I've been on holidays). Thanks Tim for dropping in and sharing your suggestions! :)

A quick bit of googling suggests that Yam miner is a legitimate CPU cryptocurrency mining client. And the yam.ko (kernel module) is a (completely unrelated) legitimate kernel module (for use with Ham radio, as the path suggests). It is provided by the default Debian kernel package.

With all that in mind, it seem likely that your server was hacked and a CPU miner installed. It may have been done manually, but I'm more inclined to imagine that it was a bot attack. So it's possible that besides the miner, there is other software also installed. In fact, it's highly likely that the miner was the payload installed by some other malware. Therefore, Tim's advice (move to a new server) is sound.

Having said that, the fact that Yam is in your /root directory, suggests that the attacker gained root access. So whilst it's still worth checking WordPress itself as per Tim's suggestions (he knows much more about WP than me), it seems unlikely that was the vector they gained access. A brute force attack seems much more likely. So making sure that a new server has a really good root password is super important. Better still, just use keys to access SSH.

If you use Webmin (and therefore need to set a password) you can still disable SSH password access. Please edit the SSH config file (/etc/ssh/ssh_config) and look for the "PasswordAuthentication" option. If it's anything like the server I have in front of me, it has a "#" at the start of the line, remove that first. Then change the "yes" to a "no". I.e change:

#   PasswordAuthentication yes

to

   PasswordAuthentication no

Then restart like this:

service ssh restart

Please keep in mind though, that if you do that and lose your key file, you won't be able to log ion via SSH! We can't help you recover it!

When cleaning up the current server, you could try tools such as ClamAV (apt-get install clamav), Sophos AV (proprietary, but free as in beer), chkrootkit (apt-get install chkrootkit) and rkhunter (apt-get install rkhunter). But personally, I find it hard to ever completely trust a machine that has been compromised. It may still be worth running them on your server before you migrate though to reduce the chances of any nasty following you...

Personally, what I'd be inclined to do would be to clean up your current server as best you can, then migrate the data to a new machine as Tim suggests.

If you were to go the route suggested by Tim, you could still use TKLBAM to migrate the data to a new server, just don't do a full restore! To do that, you'll want a current backup of your existing server. If you don't already have recent full backup, please create one (after you've at least cleaned up WordPress itself) like this:

tklbam-backup --full-backup 1D

Then on a new server, dump the backup to a directory. Assuming that the server is already linked to your Hub account:

mkdir /tklbam-dump
tklbam-restore BACKUP_ID --raw-download=/tklbam-dump

Then all of your backed up files will be in /tklbam-dump. You can then manually restore the files to their proper location if you wish. Although it is also possible to use TKLBAM to restore some stuff if you want?!

E.g. to reinstall all the Debian packages that were installed on your old system (so long as you don't restore /etc/apt then it will only be able to re-install legitimate Debian or TurnKey packages):

tklbam-restore --skip-files --skip-database /tklbam-dump

E.g. to restore the WordPress DB (assuming it's still called 'wordpress'):

tklbam-restore --limits=mysql:wordpress /tklbam-dump

E.g. to restore the WordPress directory:

tklbam-restore --limits=/var/www/wordpress /tklbam-dump

You could also use that method to restore other directories, such as /etc but as they may contain traces of malicious software that was installed, please be careful. I would recommend that you just manually migrate directories that you need, rather than doing any blanket restore.

E.g. to restore Apache config:

tklbam-restore --limits=/etc/apache2 /tklbam-dump

Hopefully that helps you get back up and running on a clean server ASAP.

One other thing that may be worthy of inclusion in your new server (and we're considering including it for v15.0 too) is Tiger (apt-get install tiger). It's more of an audit tool than a specific anti-malware tool, but can be very useful for keeping a track of files that change. If you have it running regularly on a cron job, you can use the logs to work out what changes happened when.

There is also another security checking script that I'm aware of in Debian, it's called checksecurity (apt-get install checksecurity). I personally don't know much about it, but it may be worth a look?

If you find anything that you think we should be doing better for the initial set up, please get in touch. General hardening suggestions can be posted here, or on our GitHub Issue tracker. Hopefully you won't, but if you do find something significant, please contact us privately, via support@turnkeylinux.org.

Hi Keyturns,

If you were hacked via SSH that is something that Jeremy and the TKLX community would like to know more about if you turn over any additional details. Thanks for sharing :-)

FYI - I always put a host firewall block against the non 80 / 443 ports locking them down to my IP address. Fail2Ban is a good secondary but is not a good primary defense since the hackers cycle through pools of hacked IP addresses. You can't block them all on an individual basis. You need to block all IP addresses except for your own. It's a bit of a pain in the ass since ADSL users like me and others get their IP address cycled on a daily basis but it's a good tool in the protection game.

Again Tim is on the money! :)

I'd be interested in learning more about what happened on your server so that perhaps we can look at stronger mitigation for what you experienced in our next release.

WRT your question on /etc/ssh/ssh_config the answer is yes! Changing that setting (and restarting ssh) will mean that only keypair login is supported via SSH. I would recommend making sure that you can log in via Webmin (using your password) before you do anything. That is because if you do that while connected via SSH, when you restart SSH, you will be booted. Worst case scenario, then you can still access your server config (via Wedmin) if something goes completely pearshaped (e.g. accidentally enter something in the SSH config which causes it to fail to start).

Once you have proven that SSH key only log in works fine (after making the above change) then you can also disable Webmin (if you don't use it and/or you want to lock things down). Incidentally both Webmin and Webshell run behind stunnel. So disabling them should be as simple as:

service stunnel4 stop
systemctl disable stunnel4.service

You'll get some warnings about runlevels from the second command, but you can safely ignore them. If you don't plan to use Webmin and Webshell at all, then I'd also recommend disabling them (same as above, but substitute "webmin" and "shellinabox" for where it says "stunnel4").

Tim's suggestion to block access to ports that you don't need public access to is very sound. Even if you don't have a static IP but want to raise the bar a bit without causing yourself too much overhead, AWS Security Groups allow you to block IP address ranges. In my experience, generally your IP address will be assigned within a certain block of IPs (allocated to your ISP). So blocking everything bar that block of IPs should limit the people who could potentially hack your site. If you are happy to do it manually, then you could allow your current IP address only and just update it via AWS console, everytime you want to log in (I assume that's what Tim does).

If your site has a clear geographical user base, then you can even consider completely blocking access from certain countries (generally each country has a specific IP address range assigned). That may be useful if say your site was hacked by a Chinese hacker and your userbase is relatively local (i.e. block all Chinese IPs, or at least from the ISP the attacks originated). Having said that, often these sort of attacks come from previously exploited AWS servers, so it may not be of much value.

Finally, another vector worth exploring is a CDN service such as Cloudflare. They have a free plan so you can test it out and for a smaller site, it's probably enough. For details on the security advantages of using Cloudflare, please see their site. Please don't take that as an endorsement of Cloudflare above all other CDNs. I assume that their competitors (including AWS Cloudfront) probably provide similar security, it's just that I have only had experience with Cloudflare so can only speak to that.