You are here
SamTzu - Mon, 2016/04/25 - 18:26
I thought you should know.
I downloaded the latest version of TurnKey Core started it on Proxmox, updated it and configured the firewall to redirect all queries to port 12320 (DropBear SSH console) to go to SSL port 443. My briliant idea on how to always have a command line available from any device.
Unfortunatelly the new VM was "corked" within an hour by some chinese dude who managed to get SSH connection to it.
Forum:
Sorry don't quite understand...
You created a new v14.1 TurnKey Linux Core container from within the Proxmox UI (i.e. downloaded the template). You then installed DropBear SSH server into Core and configured it to listen on port 12321? FWIW port 12321 is TKL's default port for Webmin. Then you configured your external firewall to forward incoming (i.e. external internet) connections to 443 to be redirected to DropBear on your Core server (i.e. port 12321).
Then someone hacked into your Core server and you think it's a vulnerability in TurnKey Core?
Assuming that the only way to connect to your server is via DropBear (SSH over port 443), why do you think it's a TurnKey vulnerability? I would guess that it's either a vulnerability in DropBear (which is not included in TurnKey by default) or a brute force attack (assuming that you are using password authentication).
FWIW brute force SSH attack against a poor password is a really common cause of Linux servers being "hacked" and it's my "go to" guess when I hear of Linux server's being hacked. It sucks but it's not really a vulnerability; more a configuration shortcoming.
A vulnerability in DropBear would be a bad thing and certainly warrants some response. But seeing as DropBear is not a default component of TurnKey then it's not really a TurnKey vulnerability. It needs to be reported upstream; either to Debian (if installed from the repos) or to DropBear themselves (if installed direct from DropBear developers).
TBH without some further evidence my guess is still with a brute force attack. But perhaps I misunderstood something?
No need 4 flames...
Please don't attack me when I'm trying to report something that seems important and please don't accuse poor passwords. I did not "install" anything (except the normal security updates.) I simply used firewall rule to direct traffic that normally goes to port 12320 so I can access it on normal SSL port. I assumed that port was used by DropBear but after your comment I checked and it seems to be used by similar ShellInABox. Still the problem remains. Somebody got access.
Sam
I deeply apologise if I came across as being "flamey"
However I do find it fairly unlikely. There are a number of reasons why I find it unlikely to be directly TurnKey related. Firstly, TurnKey is based on Debian which has a track record of taking security really seriously, we auto install Debian security updates nightly; plus you said you installed them on firstboot too. Whilst we package Shellinabox ourselves, we have been keeping an eye on the version included in the Debian repos (the version in Wheezy and Jessie is essentially the same as the one we package) and I just checked to make sure that there has not been a security bug recorded against it (there hasn't). Additionally we have hidden Shellinabox behind stunnel to provide additional layer of security.
Finally I would guess that there are at least one hundred thousand TurnKey servers running live on the internet all with Shellinabox (not to mention vanilla Debian servers with it installed). Other than the odd Amazon server that gets brute forced (due to a crappy password) we don't get a lot of support requests for hacked servers. Actually IIRC I think yours is the third in the last few years. Obviously you've done something a little different by providing Shellinabox via port 443 but it seems incredibly weird that all the thousands of other servers using Shellinabox would be fine, but your server (with only Webshell on 443 available to the internet) is hacked inside of an hour. Unless it was a targeted attack at you or your IP is normally a target for hackers (e.g. an AWS IP address) then even with a crappy password that is pretty quick...
As I have never had contact with you before; I don't think it's unrealistic to guess that you may have used a crappy password. Especially considering that is the most common cause of hacked servers by far (and you didn't explicitly state that you used a really good one). TBH I've actually had a server hacked myself because of a crappy password I used (a test server that I forgot to destroy when I was finished) so I well know that it can happen and how quickly; especially in a hostile environment like AWS (where bots are constantly scanning the know IP addresses for exploitable servers)...
So I'm not saying that there isn't any problems, and I'm not saying that TurnKey security is flawless. But I am saying that you haven't yet convinced me! :) Ok so hopefully we've got that out of the way?
So to clarify; you set a good root password when you created the container in Proxmox (i.e. at least 8 chars, with a mix of at least one lowercase, one uppercase and a number)? I'm guessing from your answer that you thought it was a good one, but unless we're explicit how do we know if we mean the same thing? You then went on to initialise the server (either by preseeding or via the interactive inithooks)? You installed security updates then you have done nothing further than set up a firewall rule that redirected traffic from 443 to 12320?
Is there any other way to access the server (from the internet) at all? Is Proxmox itself accessible from the internet? I'm guessing that there must be some sort of content that you are serving from the server? Is that publicly available?
And some more questions relating specifically to the hack: What lead you to believe that your server had been compromised in the first place? What lead you to be so sure that it was Shellinabox (I'm guessing a process of elimination but can't be sure)? You mentioned "some Chinese dude" but didn't elaborate on how you had came to that conclusion. Do you have the records (from your logs) relating to a remote login? Did you see some remote connections to China?
How is your firewall architecture configured?
Hi SamTzu,
Without delving into how you chose to protect your appliance through configuration of internal and external firewall(s), it is reasonably likely for a server to be cracked without doing the following:
1) Configure external firewall to only allow SSH port access from your IP address.
2) Disable server Root login
3) Use 32-64 character randomised server admin account name instead of "admin"
4) Use 64 character randomised server admin AND root account passwords
I haven't had any of my servers cracked yet (knock on wood :-)
NOTE1: Even steps 1-4 are not foolproof if someone cracks your firewall. If you want to go a step further install fail2ban to block brute force attacks with email notification of brute force attacks configured. If they guess your IP address you'll have a second line of defense and some warning.
NOTE2: You will also need to put appropriate protections in place at the application level. My WordPress instances, for example, gets probes and brute force attacks daily primarily from China, Russia, France and U.S. proxies. I have an audit log plugin and brute force protection in place at the WordPress level AND have 32 character randomised WordPress admin account names and 64 character randomised WordPress admin account passwords.
Jeremy...you might want to discuss with the TKLX core team the pros / cons of adding fail2ban as a core component for brute force protection out-of-the-box. You might also want to review with the TKLX core team the default debian accounts and their settings.
Hope this helps.
Cheers,
Tim (Managing Director - OnePressTech)
Thanks TIm
Internally we have also discussed the idea of randomly generating a root password rather than asking for the user to create one but we don't want to make the appliances too hard to use. As you are no doubt well aware, security is (almost) always a compromise between ease of use and how much you lock things down...
WRT to the default Debian user account (i.e. root) Alon & Liraz both feel pretty strongly about "security by obscurity". Personally I get their point but at the same time I think that anything that raises the bar for attackers is surely a good thing...?!
Bottom line, anything that we can do to make the appliances more secure (without making them too hard to use for newbs) has got to be a good thing!
Glad to help :-) Regarding Debian accounts...
I noticed that the MySQL user accounts are tight but the server has a lot of user accounts that could provide an attack entry point. For example:
1) Why is there a games user?
2) Is there a reason we use "mysql" as the account name for mysql access? The name "mysql45xy7". for example, would be readable to the administrator but would not be easy to guess from outside the system.
Just a thought :-)
Cheers,
Tim (Managing Director - OnePressTech)
Add new comment