Also launced new FS appliance v13, and surprise: Webmin not working either.

Must be something in the HUB. My wild guess is that is has something to do with elastic storage. I needed that for the big restore I was testing. Now I have a feeling it is expecting big storage for every new instance. I got this in the console messages.

Mounting local filesystems...mount: special device /dev/xvda2 does not exist
[?25l[?1c7[1G[[31mFAIL8[?25h[?0c[31mfailed.​

I dont have enought knowledge of Linux or the workings of the HUB/AWS to solve this :(

v13 appliciance is located at: testfs13.tklapp.com 

I had been trying for an hour or so, before posting here. I am sure I used https, not http.

Now I can login to Webmin just fine. Yesterday it would not even give me the login screen, just a server timeout.

Will try again today with a fresh new instance, and let you know.

[edit] Just installed fresh instance from scratch, and it is working perfectly. Only difference is I am working at a different location now. My next guess is a firewall / NAT issue. To test that hypothese I will go back to yesterdays location and see what happens... [to be continued]

[edit2] At my office now... Not working v14. The firewal is not logging anything strange. Can it have anything to do with routing and stunnel mapping 12321 to 10000 and webmin miniserv listening on either 12321 or 10000? Argh, my head is spinning. Could it be something with NAT / masquerading? The restore I did was from a v13 appliance to a v14 appliance. Maybe the webmin configs are overwritten by the restore??

This is gonna look like an involuntary crash course in Linux fundamentals III

[edit3] back home. Firing up my laptop, going to the hub. URL to appliance doesnt work (slow DNS propagation?) but the ip does. https://[ip address]:12321, et voila webmin!

When testing at the office, I have 2 Webmin servers active at the same subnet. Different ip's, but same port 12321. Could that cause problems?

That would explain why I cannot access the new appliance I am testing, but not the appliance deployed on the HUB.

Digging in my router / firewall documentation. It is a Mikrotik router, firewall is very much iptables.

looked at my firewall with a microscope, might have found something. Will have to test to make sure

[to be continued]

[edit] Seems I forgot to add the [in-interface] in my firewall rule. So every connection to a port 12321 would be validated and forwarded by this rule... Now I can access 2nd webmin server on my subnet and no more problems connecting to webmin servers on the HUB.

add action=dst-nat chain=dstnat comment="port forward Webmin on Bridge" \
    dst-port=12321 in-interface=ether1-gateway protocol=tcp to-addresses=\
    192.168.69.7 to-ports=12321

[edit2] After restoring v13 backup to my new v14 Fileserver appliance, I cannot access webmin on the new one anymore. Syslog gives me this:

Oct  3 15:19:27 fileserver stunnel: LOG5[1716]: Service [webmin] accepted connection from 192.168.69.17:64522
Oct  3 15:19:27 fileserver stunnel: LOG3[1716]: s_connect: connect 127.0.0.1:10000: Connection refused (111)
Oct  3 15:19:27 fileserver stunnel: LOG5[1716]: Connection reset: 0 byte(s) sent to SSL, 0 byte(s) sent to socket
Oct  3 15:19:27 fileserver stunnel: LOG5[1717]: Service [webmin] accepted connection from 192.168.69.17:64523
Oct  3 15:19:27 fileserver stunnel: LOG3[1717]: s_connect: connect 127.0.0.1:10000: Connection refused (111)
Oct  3 15:19:27 fileserver stunnel: LOG5[1717]: Connection reset: 0 byte(s) sent to SSL, 0 byte(s) sent to socket

[edit3] /etc/webmin/miniserv.conf files are absolutely different between v13 and v14. Main differences I already spotted are:

• port=12321 and listen=12321 ==> both 10000
• v14 has extra line bind=127.0.0.1
• keyfile=/etc/ssl/certs/cert.pem ==> keyfile=/etc/webmin/miniserv.pem
• some other differences in blockhost and blockuser (probably my specific changes)
• ...

Can I just change the file or copy the v14 config file back?

[edit4] adapting /etc/webmin/miniserv.conf solved the problem, and now I can login to webmin again. Next problem: my Samba users do not exist anymore on the v14 appliance...

The users exist as Linux users, so I can convert them to Samba users again. This does not set the Samba password to be the same as the Linux password. Is the libpam-smbpass package removed from the appliance?

I am thinking about testing the Linux / Samba users, using 4 scenarios creating new users.

So for user1, I will first create it in Linux and then convert to Samba, using Webmin

I hope this will show if Webmin and its mgt scripts are the issue.

[edit] the results are in:

user1, Webmin, Linux 1st
Create user in other modules:true
user1 is created in Samba
smbclient login with -U user1 works

user2, Webmin, Samba 1st
not a Webmin function

user3, cls, Linux 1st
useradd user3 
not yet created in Samba
smbpasswd –a user3
Now created in Samba
smbclient login with -U user3 works

user4, cli, Samba 1st 
smbpasswd -a user4
New SMB password:
Retype new SMB password:
Added user user4.
Linux user4 is created, no home directory or shell, inactive, primary group=users
Also created in Samba
smbclient login with -U user4 works

Everything seems to work as it should. Even with webmin telling libpam-smbpass is no longer installed.

The errors and unexpected behaviour are somehow related to the proces of restoring a v13 backup to new v14 appliance, but where to look?

That libpam-smbpass thing kept nagging me. So I decided to try and re-install the package. That gave an error. Apparently there is two password files with the name passdb.tdb on my system now.

jacob@bridge2 ~$ sudo apt-get install --reinstall libpam-smbpass
[sudo] password for jacob: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
Need to get 112 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://http.debian.net/debian/ jessie/main libpam-smbpass amd64 2:4.1.17+dfsg-2 [112 kB]
Fetched 112 kB in 1s (95.3 kB/s)         
[master f6239d9] saving uncommitted changes in /etc prior to apt run
 9 files changed, 59 insertions(+), 23 deletions(-)
 rewrite webmin/system-status/info (100%)
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 40369 files and directories currently installed.)
Preparing to unpack .../libpam-smbpass_2%3a4.1.17+dfsg-2_amd64.deb ...
passdb.tdb exists in /var/lib/samba and /var/lib/samba/private, aborting libpam-smbpass preinst
rename one of them to allow the install/upgrade to continue
http://bugs.debian.org/726472
/var/lib/samba:
total 4404
drwxr-xr-x  6 root root          4096 Oct  3 15:04 .
drwxr-xr-x 43 root root          4096 Oct  3 14:37 ..
-rw-------  1 root root        421888 Sep  3 07:03 account_policy.tdb
-rw-------  1 root root        110592 Jan  8  2014 group_mapping.ldb.replaced
-rw-------  1 root root        430080 Sep 15 09:58 group_mapping.tdb
-rw-------  1 root root          8192 Jan 29  2013 ntdrivers.tdb.bak
-rw-------  1 root root           696 Jan 29  2013 ntforms.tdb.bak
-rw-------  1 root root          8192 Jan 29  2013 ntprinters.tdb.bak
-rw-------  1 root root         36864 Oct  2 18:06 passdb.tdb
drwxr-xr-x  2 root root          4096 Oct  3 14:53 perfmon
drwxr-xr-x 10 root root          4096 Sep  3 06:57 printers
drwxr-xr-x  2 root root          4096 Sep  3 07:03 private
-rw-------  1 root root       3334144 Oct  3 15:05 registry.tdb
-rw-------  1 root root         45056 Jan 29  2013 secrets.tdb
-rw-------  1 root root         36864 Jun 30  2014 share_info.tdb
-rw-------  1 root root         36864 Jan  6  2014 share_info.tdb.bak
drwxrwx--T  2 root sambashare    4096 Sep  3 07:00 usershares
-rw-r--r--  1 root root           220 Oct  3 15:04 wins.dat
-rw-------  1 root root          8192 Mar 27  2013 wins.tdb

/var/lib/samba/private:
total 840
drwxr-xr-x 2 root root   4096 Sep  3 07:03 .
drwxr-xr-x 6 root root   4096 Oct  3 15:04 ..
-rw------- 1 root root 421888 Oct  8 10:51 passdb.tdb
-rw------- 1 root root 430080 Oct  4 12:28 secrets.tdb
dpkg: error processing archive /var/cache/apt/archives/libpam-smbpass_2%3a4.1.17+dfsg-2_amd64.deb (--unpack):
 subprocess new pre-installation script returned error exit status 1
Errors were encountered while processing:
 /var/cache/apt/archives/libpam-smbpass_2%3a4.1.17+dfsg-2_amd64.deb
Counting objects: 8969, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (2845/2845), done.
Writing objects: 100% (8969/8969), done.
Total 8969 (delta 5420), reused 8935 (delta 5398)
E: Sub-process /usr/bin/dpkg returned an error code (1)
jacob@bridge2 ~$ 

Can this be caused by the upgrade to v14 and restore v13 data process? How should I proceed now? (Im digging deeper in Linux than I've ever done before, so feeling a bit unsure)

[edit] I just verified. My 'old' v13 appliance has the file in /var/lib/samba/passdb.tdb

a fresh v14 appliance uses /var/lib/samba/private/passdb.tdl

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=726472 indicates this may cause the kinds of problems we are running into. But to be honest, I do not understand all the expert talk they are using... Wich one of the passwd files should I use? Delete the other one?? Or do I need to start from fresh, and make some changes to the appliance, before I restore my v13 data??

Does not seem to solve everything. Perhaps I already tested and changed too much.

I will restart with a fresh v14 appliance, do the restore again, and then process the changes needed to get Webmin and Samba working again. (is that something that can be automated for all other fileserver apl. users who still have to upgrade??)

These are the steps I performed to test the fileserver upgrade, and the additional steps needed to get Webmin and Samba behaving.

• I created a fresh v14 fs appliance on AWS and made sure all packages were up to date. Webmin is accessable and there is only 1 Samba user, root
• Made a copy of /etc/webin/miniserv.conf
• Then I restored my v13 data on the new appliance, using tklbam (and went for coffee).
• Reboot. Webmin does not respond, but ssh does
• Changed /etc/webmin/minserv.conf with only these 3 changes: port=10000, listen=10000, bind=127.0.0.1
• Reboot. Webmin still not working, giving the message "running in ssl, try https://localhost:10000"
• Copied the default v14 /etc/webmin/miniserv.conf back (instead of the v13 that got restore in its place)
• Reboot. Webmin working as expected.
• Now getting Samba to work with my old users again: simply copy paste the commands you wrote 3 posts up.

• cd /var/lib/samba/private
  mv secrets.tdb secrets.tdb.orig
  mv passdb.tdb passdb.tdb.orig
  mv ../secrets.tdb .
  mv ../passdb.tdb .​

• Reboot. And Samba knows all my users again. Tested different users with smbclient.

I know I don't need to reboot so often, restarting services should do. But I wanted to be sure I was not missing one or two. 

I also have compared the v13 and v14 miniserv.conf files. There are more diffences than just the 3 lines I changed. Replacing works for me, but you may want to dig deeper. Sorting both files and comparing with diff:

3,6c3,7
< bind=127.0.0.1
< blockhost_failures=5
< blockhost_time=60
< cipher_list_def=1
---
> blockhost_failures=3
> blockhost_time=600
> blocklock=
> blockuser_failures=
> blockuser_time=
11,14c12,13
< inetd_ssl=1
< ipv6=0
< keyfile=/etc/webmin/miniserv.pem
< listen=10000
---
> keyfile=/etc/ssl/certs/cert.pem
> listen=12321
15a15,16
> logclear=0
> logclf=0
16a18,19
> loghost=0
> login_script=/etc/webmin/login.pl
17a21,22
> logout_script=/etc/webmin/logout.pl
> logouttime=
20,22c25,27
< no_resolv_myname=0
< no_ssl2=1
< no_ssl3=1
---
> no_pam=0
> pam_conv=
> pam_end=
31c36
< port=10000
---
> port=12321
35d39
< preroot=theme-stressfree
38c42
< server=MiniServ/1.760
---
> server=MiniServ/1.630
40,41c44
< sockets=
< ssl=
---
> ssl=1
44a48
> utmp=

The new use of stunnel in v14 will break more appliances. I hope this detective work does help a tiny bit in creating an automated fix for all the upgraders out there :D

I took a leap of faith (with enough backups and a fallback scenario) and converted our production fileserver to v14 and the restored the v13 data back to it. Using all the steps I described I got it up and running during the weekend. On monday our users didn't even notice any difference. Smooth migration I would say.

But... Webmin is seriously messed up (and of course the usual nagging log messages I have to get rid of one-by-one).

$sudo apt-cache policy libpam-smbpass 
libpam-smbpass:
  Installed: 2:4.1.17+dfsg-2
  Candidate: 2:4.1.17+dfsg-2
  Version table:
 *** 2:4.1.17+dfsg-2 0
        500 http://http.debian.net/debian/ jessie/main amd64 Packages
        100 /var/lib/dpkg/status

I would say it is installed. But in Webmin => System => Software Packages, Installed Packages, search for package "libpam-smbpass" gives:

Show sidebar » Package is no longer installed Return to previous page

This is not just with libpam-smbpass, also with postfix. Postfix doesnt even show up in the Webmin menu under servers...

When in Webmin I look at the package tree of installed packages, I tried the first 20 or so packages, they all give the message: package is no longer installed.

I have the feeling Webmin is keeping its own list of packages, and it somehow got out of sync with reality. Is there any way to 'rebuild' it?

I found a discussion about Webmin not seeing installed packages here: http://ehc.ac/p/webadmin/discussion/600155/thread/ddaa8ed6/ 

It recommends to execute this command:

dpkg --get-selections \* | awk '{print $1}' | xargs -l1 aptitude reinstall​

But I don't understand 100% what it does. And I prefer not to test on my production server. What would you do?

I will not test on my prod server. So today I decided to create a fresh instance on AWS. My idea was creating a clean instance, and then instead of restoring from TKLBAM, migrating all users and then the data and configs for Samba.

Directly after getting it up, before making any modification or configuration, I checked the installed packages in Webmin. And guess what: Error, Package is no longer installed. 

For every package I checked, including Samba, Postfix and Webmin it self. I expected a clean image to run flawless, but that is not the case.

Perhaps I should downgrade back to TKL v13?

Just restarted the stopped instance. First tried the "upgrade now" from webmin. That did not work.

Then logged in as root and did the "apt-get update" from cli. Webmin still shows basic packages as "no longer installed"

Postfix is installed and visible from the Webmin-Servers menu. (But not as installed package).

I will keep the instance running for 24 hrs now. See if that makes any difference.