We have recently had a long time TurnKey user reach out asking how to:

(re)synchronize my running AWS instance [on the Hub] and current TKL server settings/update.

Whilst I could answer directly, I thought it'd be a great opportunity to walk through my recommendation publicly, so others might also benefit. This relates explicitly to our Drupal7 appliance. But (with some tweaking) should generally be relevant to many/most appliances. Note that some particular appliances have radically changed their install method. In those cases, steps such as these are likely not going to work.

If you are trying to achieve something similar but with a different appliance, please start a new thread and link back to this thread. Also note the particular appliance and the version you are coming from and if you aren't planning to migrate to the latest; the version you hope to migrate too.

These instructions describe launching a new Hub server (i.e. "NEW" server), though could be modified for an alternate host (e.g. local VM). The current server location/host (i.e. "OLD" server) is irrelevant, so long as it has outgoing internet access. These instructions also leverage TKLBAM and assume that your server is linked to the Hub.

Overview of process

It is highly recommended that you read through this whole post prior to doing anything. Any steps that appear unclear, or questions you may have, please ask and ensure that you are satisfied before starting. Most steps documented are non-destructive, but it's always better if you have a clear understanding of the requirements and consequences before you start.

Essentially the process is to take a backup of your OLD server, collecting all the required data. Then launch a NEW server and (selectively) migrate your data to it; using TKLBAM. The process that I will detail below, should "just work" and result in an operational copy of your OLD server, but with all the new software and new default configuration options of a NEW server.

The idea is that you can leave your OLD server running, serving your content, while you do a "dry run" of the migration. This means that there is no external/artificial pressure to get it up and running (in case there are any issues you need to work through).

If your content is relatively static (i.e. nothing has changed on your OLD server while you ensure that everything works on your NEW server), then you may not need to re-run through the process. If your content is particularly dynamic, you will likely need to do it in 2 runs; an initial "dry-run" (taking notes along the way), followed by another "final run" through to do the final migration. When you do the final run, you will likely want to put your site in "maintenance mode" prior to taking the final TKLBAM backup.

Then once you are happy with your NEW server and can confirm that everything "just works", then you can adjust the DNS records for your domain to point to your NEW server. At that point, I recommend stopping the OLD server. Once you are completely satisfied that there is no important data remaining on your OLD server, you can destroy it. I recommend leaving your OLD server stopped for about a month - that should be sufficient time for any glaring issues related to missing data to become apparent.

You may wish to reapply relevant config tweaks (e.g. MySQL performance tweaks) that you had applied to your OLD server. As a general rule you are best off doing this manually on a case-by-case basis as options have likely changed since your old server. The bigger the gap between the version of the OLD server and the NEW server, the more generally important it will be to NOT automatically apply changes in global config files (generally found in /etc).

Now to the actual process... Note that all actions assume they are being run from the root account. If you are not running as root, you will likely need to prefix all commands with 'sudo'.

Get data from OLD server

By default, our backup tool, TKLBAM is pre-installed in all TurnKey servers. All versions of TKLBAM since the release of v13.0 should be compatible with these instructions (users of v12.x may need to tweak instructions - but hopefully there aren't many of you still about...). Regardless, if your OLD server is not already linked to the Hub, initialise TKLBAM like this:

tklbam-init YOUR_HUB_API_KEY

(Replace YOUR_HUB_API_KEY with your actual Hub API key - found on the Profile page of your TurnKey Hub account).

Then ensure that you have a full and complete backup. If this is your "final run" (and/or your site is relatively static) you may wish to put it in "maintenance mode" prior to creating a backup. You may also wish to clear caches and run any outstanding updates prior to migration (regardless of whether you are doing a "dry-run" or not). When you are ready, force a full backup:

tklbam-backup --full-backup now

Load data to NEW server

The first step is to launch your NEW server. The latest Drupal7 appliance at the time of writing is v16.0, but I highly recommend always checking and being sure that you are using the latest release when following these steps. Currently we are still transitioning our servers from v15.x to v16.x so some appliances are still only available as v15.x. To be sure that you have v16.x, when launching from the Hub, on the Launch new server page double check that you are selecting the latest version by checking for a text link in the top corner (currently it should be "15.x 16.x", like this). At some point soon 16.x will become the default. Once all appliances have been updated (or deprecated) the 15.x link will be removed.

Assuming that your NEW server is launched from the Hub, then TKLBAM should already be initalized and linked to the Hub. If not, run 'tklbam-init' as per above. Then download the data from your OLD server:

mkdir -p /tklbam-dump
tklbam-restore BACKUP_ID --raw-download=/tklbam-dump

(Where BACKUP_ID is the ID number of the backup from your OLD server).

This should download the complete backup to /tklbam-dump. In our case, the relevant components are the /var/www/drupal7 directory (and the files and directories within it) and the drupal7 database. To restore those:

tklbam-restore /tklbam-dump --skip-packages --limits="/var/www/drupal7 mysql:drupal7"

This should restore the 2 previously mentioned components and skip any old apt packages and other changes throughout the filesystem. Whilst this should include all the required files, the "drupal7" MySQL user account password will be out of sync. This is because we are using the "drupal7" MySQL user account from the NEW server, but the Drupal7 config file (containing the OLD server "drupal7" MySQL user password). So the last major step we'll need to do is fix that. Manually re-running the relevant inithook firstboot script should take care of that for us:

/usr/lib/inithooks/firstboot.d/20regen-drupal7-secrets

Finally a number of services require restart. The easiest way to ensure that happens cleanly and completely, it's easiest to do a reboot. If you'd rather just try to restart services, then please try this:

systemctl restart apache2 mariadb

Final steps

As noted above, if you wish to make further tweaks, then please do so. You will also find all the files from your backup within /tklbam-dump (relative to '/' - i.e. '/etc/config_file' will be found at '/tklbam-dump/etc/config_file').

If your site is dynamic and you need to redo the migration, be sure to document any additional steps that you take to ensure that you can get back to this exact same place.

Once you are happy with your migration, the /tklbam-dump directory can be deleted:

rm -r /tklbam-dump

Once everything is ready to be finalised, you can now stop your OLD server and transfer DNS records so that they point to your NEW server. I strongly suggest that you DO NOT destroy your old server yet, instead just stop it. I recommend that you leave your OLD server stopped for about a month. Under most circumstance, any missing data should become apparent within that period.

Also, if you wish to keep a copy of the final TKLBAM migration backup form the OLD server as an archive, but delete all the prior backup history ensure that you change "max backups" to '1' for the relevant BACKUP_ID (within the Hub) and then run one last full backup. If you are sure that you do not old backups from your OLD server, then you could do this prior to making your final "full backup" above.