Although you may wish to harden it up a little such as enable the firewall and disable any services you don't plan to use. Also make sure you set good passwords etc.

Personally I mostly just use commandline these days, but both the Firewall (aka IPTables) and services can be controlled from Webmin.

for instance use iptables to block the webmin interface port and only allow it from a specific ip address

Or only allow the webmin interface if a user is logged in using ssh

Or install knockd  to allow access only after some kind of port knocking sequence.

Also you can do things like denyhosts, which detects failed ssh login attempts, and blocks the host, sent you an email... or whatever

Don't use passwords, use ssh keys, block all login attemps using passwords.

Just be critical in what you have running and if it is really needed.

If not, don't run it.

if only sometimes needed, don't run it, bring it up when needed