Breaking a big "problem" down into smaller chunks is always a good idea. Although in this instance, unfortunately we don't have any other "desktop" type appliances for you to look at.

FWIW we did always intend to provide our own minimalist TurnKey "desktop" distro, but in the end the idea stalled because the interest seemed limited. We had also considered a "security focused" minimalist live distro, but Tails appeared and gained a lot of traction pretty quick (confirming it was a good idea, but we got beaten to the execution). There is clearly interest in BitKey, but we're a small team with a mile long "todo" list related to the headless servers we provide and as BitKey is essentially a "side project" it hasn't received the love that it deserves...

It's also worth noting that a community member, Steven did a fair bit of work maintaining and extending the (default) "community" branch of BitKey. As I think I may have hinted previously, he also started work on a Stretch based (i.e. v15.x) version. I wouldn't necessarily recommend that you try to complete that, but I'm sure that it's at least worth checking out. One part that you won't need to worry about is the python update (to v3.7) - IIRC that was for a particular app that required a newer version than Debian Stretch had at the time (v3.5). Debian Buster (basis of v16.x) has v3.8 so that shouldn't be an issue.

Anyway, back to the matter at hand. If you look a bit closer under the hood, the TKLDev build process essentially has 3 main components, "plans", "overlays" and "conf scripts". The make files are just a nice way of consolidating sets of overlays and conf scripts (plans are handled somewhat separately).

The "plan" provides a list of packages to install. Note that only the listed packages and their hard dependencies (i.e. "depends") are installed. If you want recommended dependencies (i.e. "recommends") then the relevant package names need to be explicitly listed in a plan. In the case of the desktop plan you'll need to ensure that the packages required to make it all work are included in a plan.

As the name suggests, "overlays" are files that are overlaid over the filesystem (relative to '/'). E.g. the desktop overlay.

Again, as the name suggests, conf scripts are executable scripts that run at build time to configure specific components. As this occurs after packages have been installed, technically these scripts could be in any language supported by the OS being built. In practice, they are all shell (sh) or bash scripts (IMO bash scripts are preferable).

It might also be worth having a bit of a look through the TKLDev docs. Unfortunately, they're far from complete; some bits out of date and some bits a bit confusing... However, there is still some gold in there if you can be bothered...

---

So to circle back to a process to get this working...:

I suggest that you start by modifying the desktop make file (i.e. turnkey-desktop.mk to fix the initial issue you noted. I've just had a quick look through and I think that should be all that's required to get that base component "working".

The next thing will be modifying the plan(s). Once the above make file is fixed, then next issues you'll hit IMO will be packages that no longer exist and/or have been renamed.

You could just comment out the package names that cause issues, until it successfully builds. Then worry about what to replace them with (or if they need to be replaced at all) later. Alternatively, you could research each one as you go. Here are some resources which I imagine you will find invaluable (FWIW I managed the v15.x -> v16.0 migration, so have a fair bit of first hand experience).:

• Debian 9/Stretch Releases Notes - Particularly Chapter 2 - What's new and Chapter 5 - Issues to be aware of. (Debian 9/Stretch was the basis of v15.x).
• Debian 10/Buster Releases Notes - Particularly Chapter 2 - What's new and Chapter 5 - Issues to be aware of. (Debian 10/Buster is the basis of v16.x).
• Debian Package Info - Particularly the search function about halfway down the page.

FWIW note that the plans are written in (and parsed as) C language format. So full line comments are a double forward slash at the start of the line (i.e. '//') - inline comments are forward slash, asterisk at the start of the comment and asterisk, forward slash after (i.e. '/* COMMENT */').

Once you have the plans sorted out, then you may find some other issues with conf scripts that break. I generally tend to comment out problematic lines initially. In my experience, overlays rarely cause build time issues, although can obviously cause issues later.

Once you have the build process completing, you should get an ISO. I suggest testing that in a VM and see if you can boot into live mode. If it fails, I find that a screencast of the boot process can be an invaluable tool to troubleshoot the issue (often the lines go by too fast to read them, if you record them, you can play them back slowly to read them).

I can't guarantee how much I'll be able to help out, but I'd love to see an updated BitKey. So please don't hesitate to ask and I'll do my best to help out. It may also be worth posting your interest/intent on the BitKey tracker. Who knows, perhaps there might be others keen to get involved too?!

As you suspected (and note below) the issue is that the Changelog is malformed (it needs to be formatted as per Debian specs.

Out of interest, you if you use dch to edit the changelog (contained within the devscripts package - i.e. install devscripts in your TKLDev).

insserv is a init.d (SysvInit - i.e. the legacy initialisation system) ordering script, which is no longer relevant with systemd (the new/current initialisation system). It's not installed in v16.x appliances. So that line shouldn't be running and you can safely remove it. You'll find it in the turnkey-desktop.mk file.

Removing packages will (obviously) have impact on the software installed in final ISO. Whether or not it will also have an impact on the ISO build process, will depend on whether commands/files from the package are used during build time.

To use the 'insserv' issue as an example: Whilst the inserv package and that command are no longer needed, it doesn't actually do any harm. So instead of removing that line from turnkey-desktop.mk; you could just (re)include the 'insserv' package in one of the plans that are installed. It's because the build process is trying to call a command from a package that is no longer installed, that the build chokes on that line.

I hope that makes sense.

looks like you are getting close to getting an ISO to build! Hopefully removing the insserv line from the turnkey-desktop.mk file might even be enough to get you over the line...

Probably the next tricky part will be getting it to boot. I may have mentioned already, but to assist troubleshooting the boot of the ISO, I suggest testing in a VM and recording the boot process (e.g. with some sort of screen recording software) so that you can review the boot messages. Or perhaps you'll be lucky and it will boot ok?!

It's been a while since I did the transition from casper to live-boot, but IIRC the casper-bottom dir should be renamed live-bottom. That way it should only run when in "live" mode; the init-* dirs should be run on everyboot. Although I could be wrong...

I assume the line you are referring to is this one. In which case it should be changed from:

. /scripts/casper-functions

to:

. /scripts/functions

Regardless, it looks like you are having some solid progress...

I've managed to get the JS offline apps working, together with a working shutdown icon and chromium installed properly. My github buster branch refers.

Great work! :)

What I'd like to do now is make some changes locally within my BitKey instance by downloading stuff from the internet and installing locally to test things work, before then modifying the build accordingly.

Yes, very sensible approach! You can then install stuff in your test bitkey and then add them to the plan.

The problem I have is the internet connection doesn't work on my new BitKey whereas on the old v14 BitKey, it does. Below are 2 screenshots, one from each version. My new BitKey is giving "Network is unreachable" errors. Both are VMs running in VirtualBox with the same Network settings.

Any idea what I'm missing as far as v16 config is concerned to ensure internet connectivity via the VM host? I haven't modified any of the plan/net packages (except iproute -> iproute2).

Ok, so looking at your screenshots, I think that there are a few things going on here. But before I start suggesting fixes/workarounds, it's worth being aware that TurnKey (and therefore BitKey) uses a legacy network config. It hasn't been the recommended way to do networking for some time. We still use it for our headless servers because it's easy and continues to work. However, it may not be the recommended way to do things in a desktop environment anymore. I don't know for sure, but I suspect that desktop UI tools may no longer support the old way of doing things?!

So first thing I note is the interface in the new BitKey is enp0s3 (vs eth0 in the old one). This "new" naming scheme is known as "predictable network interface names". We disable that by default via the conf script: common/conf/turnkey.d/grub-iface-naming.

If you wish to try to transition to the newer networking paradigm, then you almost certainly don't want to disable this as we do. If you want to try to just use the legacy networking (which may or may not work with GUI tools such as Wicd, but should work fine via commandline) then it's recommended to use the legacy interface names.

FWIW you can find more info about the Debian networking config in Debian manual - chapter 5. There is also a Debian wiki page on Networking but I can't say how up to date that might be (Debian wiki pages can be a bit hit and miss re how up to date they are...).

So our default config uses the /etc/network/interfaces file and my guess is that it only explicitly notes the loopback device and eth0. That is likely configuring DHCP for the interface.

At some point we're going to need to decide whether we update networking for BitKey or whether we just persist with the legacy setup. I guess to some degree it depends on whether Wicd can still cope with the legacy set up... For now though, to just get your networking working, you could edit the interfaces file and replace 'eth0' with 'enp0s3'. Then try this:

ifdown enp0s3
ifup enp0s3

A secondary issue is that of the machine name: my build is showing user@tkldev whereas the v14 BitKey shows user@bitkey. How can I change the build to use "bitkey" as machine hostname too?

My guess is that for some reason it's picking up the TKLDev hostname at build time. I vaguely recall that we explcitly do that somewhere to workaround an build time issue. To fix that, just add this line to one of your conf.d script like this:

echo bitkey > /etc/hostname

To answer your questions:

Is there any way of automating this entry in the hot-online option so I don't have to edit the file manually each time it live boots?

Yes, but not without modifying (and/or rebuilding) the ISO. Just add it as an overlay or edit the interfaces file from a conf script.

Is there a way of entering the correct interface in the file depending on what the distro is running on (my VM is as above but for my MacBook Air with WiFi adaptor, ip address show lists wlp2s0b1 instead of enp0s3)? I guess some form of parsing "ip address show" in a script to build /etc/network/interfaces and exec the ifup, but I wouldn't know where to start with that.

Possibly, but I'm not exactly sure how you'd do about that. I assume that there is probably a way to add scripts to get run as part of the network config.

The easy way to do that is to follow our lead from the headless servers as disable the "predictable network interface names" (wlp2s0b1 and enp0s3 are these so-called "predictable network interface names"). On our headless servers, that gives you the legacy 'eth0' rather than 'enp0s3', etc. However, have said that, 'wlp2s0b1' looks like a wifi interface and I suspect that 'eth0' won't cut it...

Besides the likely wifi issue, as I already noted, I'm not 100% sure whether that will work with Wicd (or other UI networking tools). So any/all of my suggestions above are likely only temporary hacks that will allow you to focus on other aspects for now.

Bottom line it seems likely to me that the whole networking issue will need to be resolved properly at some point. So perhaps it's worth doing a bit of reading as per the links I provided in my last post and fixing properly now?!

After having a closer look at the links I provided, it appears that the wiki is no help. However, have a look at Debian manual - sec 5.2.1 - GUI network configuration tools. That has notes regarding Wicd which will hopefully be of value?! It may also be necessary to (re)configure the DHCP client too. You can find info about the config file via the dhclient.conf man page (AFAIK BitKey has isc-dhcp-client installed as a DHCP client).

Is there a way of force-disabling all networking when booting in cold-offline mode? It kind of already is like that but malicious software entries could re-enable networking as above automatically if not careful, so force disabling networking is vital for a truly safe cold-offline experience.

TBH, I'm not particularly familiar with BitKey but by my understanding is that originally when booted as "cold-offline" all networking tools were purged (i.e. uninstalled and config removed) and all networking kernel modules (i.e. drivers) were unloaded (perhaps even blacklisted?).

Note that whilst that would stop any user level malicious software from re-enabling networking, if the malicious software could gain root access (or was running as root) then all bets are off. There is not really any way to stop root from undoing anything that you can do.

However, if the disabling was done as root at boot time (as it should be) and sudo was also removed and as a final step root was disabled, then a limited user would not have any mechanism to re-enable/re-install that stuff. So that would work pretty well.

Actually how you would best go about that, I'm not completely sure though. My suspicion is that previously it was done via the 'casper-scripts'. If so, it should be possbile to re-implement via 'live-scripts'.

I'm sure there are other options too. E.g. I suspect that it'd be possible to leverage systemd and set up custom systemd targets (e.g. "cold-offline" etc) to boot to. Systemd targets are essentially sets of services to run at boot and can be specified as kernel commandline options (which can be included in the boot menu options). Although unless you are familiar with systemd, that may be a convoluted method with a lot of trial and error... (I'm speaking from my experience with the early v16.0 development cycle).

Sorry it didn't occur to me that the live functionality doesn't boot from grub! That's why the grub.cfg changes didn't work and you needed to make the changes in menu.cfg (the ISO and therefore the live functionality, boots via isolinux/syslinux)...

Now that you have disabled sudo access when running in cold-offline mode, I'd be inclined to put that aside for now (i.e. disable it/comment it out/etc). Not having access to sudo will make troubleshooting the iptables issue (and others) much harder.

Also as something of an aside, you can remove 'sudo' and any other packages that you don't want users using in cold-offline as part of the start up process too. E.g.:

apt-get purge -y sudo ANOTHER_PKG ANOTHER_PKG2 ...

Going back to the iptables issue, the message that you note suggests (as you've possibly already realised) that the required iptables kernel module (essentially a driver) isn't loaded. The first thing I'd check is to see if it is loaded after boot (hence why I suggest ensuring that sudo still works for now). It may be that it's just not loaded at that point in the boot cycle (and you'll need to run your command later in the process) or perhaps it's not being loaded at all? (In which case you'll need to explicitly load it)

Also there may be other ways around it. I'm not sure, but perhaps it's possible to blacklist all networking drivers so networking isn't possible at all? I also thought that there may be a kernel commandline option to completely disable networking, but I couldn't find anything other than disabling ipv6... AppAmor may be another option (IIRC it's enabled by default, but isn't configured that strictly). Also as per above note about removing packages, all the required networking tools could also be removed?! And one final note, some methods of disabling networking may have unintended consequences! That is because some software may rely on loopback networking!

Anyway, sorry that I don't have any really solid advice for you here... :(

PS oops... I just looked at your script and I can see that you HAVE commented out the sudo disable line... :)