Update Gitlab TurnKey GNU/Linux 14.0 / Debian 8.11 Jessie

Reapers59 - Wed, 2020/09/02 - 11:54

Hi,

I have been trying to update a version of Gitlab for several days and I cant find the corresponding procedure. Indeed, when I try to follow a tutorial to do a basic update of Gitlab (in my case from version 8.8 to 8.9) it turns out that the directories are not the same as in the tutorial.

Could it be because I have a TurnKey version?

It is imperative that I update Gitlab because this version is becoming obsolete (at least to switch to Gitlab API v4)

Can you tell me what to do, please?

In my case, I have the Gitlab TurnKey GNU / Linux 14.0 / Debian 8.11 Jessie version which is on a VM.

Thanks!

Forum:

Support

Tags:

gitlab

update

turnkey gnu/linux 14.0

Add new comment

Hi there... I have some (mixed) news for you...

Jeremy Davis - Thu, 2020/09/03 - 02:13

So prior to our GitLab v15.2 release, TurnKey GItLab appliances installed from source code. Initially we did the install via upstream instructions, but there were some changes along the way.

Since v15.2, we now install GitLab via their Omnibus package. You can read about the development of that in a previous forum thread. Whilst we have some reservations about the "all-in-one" nature of the Omnibus package, we also get that is a decision that the GitLab developers have made to make their lives easier. And in turn, using that for the install makes it much easier for our users to maintain their GitLab servers.

So in the big picture, I'd recommend that you upgrade/migrate to the latest v16.0 GitLab appliance. However, you can't just jump from your v14.x appliance with GitLab installed from source, to the latest appliance.

However, lucky for you, some time ago a user update/migrated from a REALLY old TurnKey GitLab server (TurnKey v13.0 GitLab appliance; which included GitLab v5.0.2! He was kind enough to document the whole process; from his old server, to GitLab v11.10 on a v15.x server (with GitLab installed via the Omnibus package). The full GitLab update thread runs through the process. All the details are in the thread, and it may be worth having a quick read through and/or be useful for troubleshooting along the way. But he also kindly documented it all and generated a bundle of PDFs. They're linked in the top post of that thread.

Once you get to the Omnibus package, then updates get much easier...

If you have any problems or specific questions, please do not hesitate to ask. Although please be sure to post enough info to give me the context of any issues you're hitting or questions you have.

Good luck, fungers crossed...

Hi Jeremy,

Reapers59 - Thu, 2020/09/03 - 11:15

Hi Jeremy,

Thank you for your answer.

I was just following this tutorial, but I have the impression that it is not compatible with the version of Gitlab that I currently have: 8.8.2.

Indeed, as there is no indication concerning this version itself, I followed the tutorial to go from version 8.8.9 to version 8.9.11.

But when I try to type even the command to shut down the Gitlab service "sudo service gitlab stop", it displays an error message: "Failed to stop gitlab.service: Unit gitlab.service not loaded."
Do I first have to do a "minor" update to go from version 8.8.2 to version 8.8.9 and then I can follow the tutorial?

There are also some

Reapers59 - Thu, 2020/09/03 - 11:16

There are also some directories that I don't have, like the "home" directory, I only have the "etc" directory.

Oops, sorry missed this post...

Jeremy Davis - Fri, 2020/09/04 - 07:50

Hmmm, so you don't have a /home/git directory?

You can see from the GitLab v14.0 buildcode that GitLab is installed to "$GIT_HOME/gitlab" - where GIT_HOME=/home/git.

So if you have no /home/git directory, then either this isn't/wasn't the v14.0 GitLab appliance and/or somebody has made some significant changes to this appliance since it was originally installed...

Perhaps try searching for a gitlab directory like this:

find / -type d -iname gitlab

That will take a while to run (as it will search the whole filesystem) but it should return every directory which is named "gitlab" (not case sensitive).

Actually pondering a little more... There was no usage of /etc/gitlab in our v14.0 appliance, so perhaps your GitLab is already a "Omnibus" install? Could you please give me the output of this:

apt-cache policy gitlab*

Is this actually a TurnKey GitLab appliance?

Jeremy Davis - Fri, 2020/09/04 - 07:39

[This post is actually a reply to this comment - but for some reason appears as a reply to the OP...?!]

Is this actually a TurnKey GitLab appliance? Or is it a different TurnKey appliance which has had GitLab installed on it by someone else?

You can double check what the appliance is (or at least what it started life as) like this:

turnkey-version

Assuming that is is v14.0 GitLab, then that should return this string (or very similar):

turnkey-gitlab-14.0-jessie-amd64

For what it's worth, the v14.0 appliance shipped with GitLab v7.11, so if you have v8.8, then someone has already done an update at some point. Ideally, that shouldn't make any difference, but ultimately what you have will depend on what was done previously by whoever was maintaining this server (i.e. things may have been moved or configured differently).

It's also worth noting, that if you are logged in as root (as you should be in a TurnKey VM), then the only times you need to use sudo are when you run commands as an alternate user. E.g. as the git user - commands that start 'sudo -u git'. Commands that start with 'sudo', but don't include '-u USERNAME', don't require sudo when running as root ('sudo' with no '-u USERNAME' default to running as root, but as you're already root, they have no value).

But to circle back, the fact that you don't have a gitlab service is a little weird, although if I check the v14.0 buildcode, I can see that there should actually be 2 GitLab related services; gitlab-unicorn.service & gitlab-sidekiq.service. I can only assume that something somewhere along the line that changed, but fingers crossed, you can continue with that for now...

As should have already be noted, be sure that you have a full clone of this VM. I personally recommend taking a copy of the vHDD at the very least (e.g. stop the VM and physically copy the vHDD somewhere safe). Snapshots are also a great tool (but personally I wouldn't just relay on that). IMO you should take a snapshot at any point where you have reached a logical point (e.g. just finished an update) and things appear to be working as they should.

To explicitly answer your question:

Do I first have to do a "minor" update to go from version 8.8.2 to version 8.8.9 and then I can follow the tutorial?

According to the latest GitLab upgrade notes, it is safe to upgrade within a minor version. Although I note that historically there were a few minor versions where that wasn't the case. Having said that, that particular doc page explictly notes "8.9.4 -> 8.12.3" as being ok. So personally, I'd still go to v8.9 before I started following that workflow, just in case they are not accounting for older releases... Regardless, the note about "checking for background migrations" looks pretty important!

No, I don't have a / home /

Reapers59 - Fri, 2020/09/04 - 11:30

No, I don't have a / home / gitlab directory, yet when I type "turnkey-version", I have the version "turnkey-gitlab-14.0-jessie-amd64"

When I type "find / -type d -iname gitlab", I have:

/etc/gitlab
/opt/gitlab
/opt/gitlab/embedded/service/gitlab-rails/spec/tasks/gitlab
/opt/gitlab/embedded/service/gitlab-rails/spec/lib/gitlab
/opt/gitlab/embedded/service/gitlab-rails/app/views/kaminari/gitlab
/opt/gitlab/embedded/service/gitlab-rails/app/views/import/gitlab
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab
/opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab
/opt/gitlab/embedded/cookbooks/gitlab
/var/log/gitlab
/var/opt/gitlab

When I type "apt-cache policy gitlab" I have:

Ah ha! Someone has already migrated to Omnibus package.

Jeremy Davis - Mon, 2020/09/07 - 02:32

It looks like someone has already migrated your appliance to the GitLab Omnibus package. That actually will make it much easier for you! :)

If you look at the output of the apt-cache command that you provided, you can see that you have 'gitlab-ce' (GitLab Community Edition) installed and your currently installed version is v8.8.2. The latest version of GitLab-CE available for Debian Jessie (the basis of TurnKey v14.x) is v12.9.2. This is the bit that shows that:

gitlab-ce:
  Installed: 8.8.2-ce.0
  Candidate: 12.9.2-ce.0

Beneath this, it lists all versions available for your current OS. You can see the version that you have installed highlighted in this list by 3 asterisks ('* * *'):

*** 8.8.2-ce.0 0 500 https://packages.gitlab.com/gitlab/gitlab-ce/debian/ jessie/main amd64 Packages 100 /var/lib/dpkg/status

Before you do anything, unless you've run it very recently, I recommend that you run:

apt-get update

That will update all the package lists available. Then run:

apt-cache policy gitlab-ce

That will give you the (refreshed) information just related to the GitLab CE package (now that we know that's what you have installed).

I'll assume that the info is the same as what you posted above, but please be sure to double check the latest info and adjust if need be.

As I noted before, according to the GitLab upgrade notes you should generally be able to upgrade from one minor version to the next. However, it also notes that major version upgrades can be more involved.

Plus if you read through the whole page (which I strongly encourage - perhaps I missed something?!), you'll note a warning that when updating from versions v8.11.x and earlier, it is recommended to upgrade to v8.12.0 before updating to v8.17.7 (although I note that the latest v8.x release is v8.17.8). There is also a further note that starting v12.0.x there is an extra step required.

So to put all this info together, here is the recommended upgrade path:

8.8.2 -> 8.12.0 -> 8.17.8 -> 9.5.10 -> 10.8.7 -> 11.11.8 -> 12.0.12 -> 12.9.2

To get beyond GitLab-CE v12.9.2 you'll need a newer OS. So you'll need to either migrate your data to a newer TurnKey GitLab release (my personal recommendation) or upgrade your base OS from Debian 8/Jessie to Debian 9/Stretch (the basis of TurnKey v15.x). Seeing as the current TurnKey release (v16.x) is based on Debian 10/Buster, that's probably the one to aim for, but somewhat similar to GitLab, you need to upgrade via the intermediate version. If you just migrate your GitLab data to a new server, you can skip that intermediate step. But we can discuss the details of that when you get there. Let's just get your GitLab version updated to v12.x first!

So the first thing to do is to upgrade to v8.12.0. Then to upgrade to the latest v8.x (v8.17.8). After you've done that, you can look to upgrade to v9.x.

As mentioned previously, make sure that you have full backups of your VM. And I would encourage you to test your backups to ensure that they actually work. E.g. try launching a new VM with your data... Moving forward, I suggest that you make a new backup for every successful version update that you complete, test and confirm is working fine. That way if something goes pear shaped along the way, you won't need to go all the way back to the start...

So first up, I urge you to read through the full upgrade docs. I also recommend reading the relevant version specific changes. Having said that, there don't appear to be any notes related to v9.x, so nothing to read until you've updated to the latest v9.x release (9.5.10).

So update to 8.12.0 via apt like this:

apt-get install gitlab-ce=8.12.0-ce.0

Once you've completed that, I suggest that you reboot the server and double check that all your data is there and everything works as it should.

Once you've confirmed that works, then the next step will be upgrading to the latest v8.x (8.17.8). Do that like this:

apt-get install gitlab-ce=8.17.8-ce.0

Again I recommend a reboot and double check all is well. Create your new backup and then on to the next version; latest v9.x (9.5.10):

apt-get install gitlab-ce=9.5.10-ce.0

Again reboot, test and create a new backup. This time there are notes for v10.x changes. So be sure to read them before going any further and take note of any specific steps. In this case, it looks like you need to upgrade PostgreSQL. Apparently this command should do the trick:

gitlab-ctl pg-upgrade

As noted, you can double check the PostgreSQL version like this:

/opt/gitlab/embedded/bin/psql --version

Check that you have PostgreSQL v9.6

Assuming that's all good, then you can upgrade to v10.x (10.8.7). And so on... Be sure to take lots of backups and name them clearly so you know which is which (if you need to roll back). I also highly recommend that you document what you've done every step of the process. Then even if you need to rerun previous steps, you'll be able to retrace your steps as far back as required. If something goes wrong, you'll also be able to clearly define at wqhich step things broke.

I hope that helps and I'd really love to hear how things go (good or bad). Good luck! :)

Update successful. I followed

Reapers59 - Tue, 2020/09/08 - 14:43

Update successful. I followed the steps you wrote and the updates went right so my version of Gitlab is 12.9.2.

Regarding the PostgreSQL update, it apparently happens by itself over the updates because when I typed "gitlab-ctl pg-upgrade" I was told that I had the latest version.

For the moment I will leave it there because updating Gitlab still has not solved the initial problem that I have been trying to solve for weeks with Mattermost (we use Gitlab SSO to connect to Mattermost and sometimes when trying to connect to Mattermost we have "Error getting token" popping up, I thought updating Gitlab to go from API v3 to API v4 would fix the problem but apparently not.

However, if I am looking to re-update in order to reach the latest versions, I would come back here, either in this topic or then open another one.

Thanks you very much for your time ;)

Great news that the updates went well

Jeremy Davis - Wed, 2020/09/09 - 02:35

Great news that the updates went well. Pity that it didn't resolve your API issue though...

If you haven't already, I suggest checking the GItLab logs to see if there is any issues occurring on GItLab's end when the token issue occurs in Mattermost?

Although it probably should show up in the GitLab logs, if there is no clear cause on GitLab's end, then my next wild guess would be resource availability. Perhaps you could try monitoring resource usage on your GitLab server? E.g. even something simple like a cron job that writes current CPU and RAM usage to a file every few minutes? Then the next time the token failure occurs, check resource usage on your GItLab server at that time.

Good luck.

Please feel free to post back and share what you discover if/when you work out the problem.

Would you have the steps to

Reapers59 - Thu, 2020/09/10 - 14:42

Would you have the steps to see the Gitlab logs with the version I have pls? (or the command, if it's a command)

We need to know what version you have.

Jeremy Davis - Fri, 2020/09/11 - 01:11

I'm more than happy to try to assist, but more info would help. The TurnKey version is always a good start. But as a general rule of thumb, you can almost never provide "too much" info when you are seeking IT support. :)

It's also worth noting that unless you have something "on topic" to add to an existing thread, it's generally better to start your own thread.

Regardless though, in the case of GitLab, as you may have gathered from this thread, there are 2 main ways that it could have been installed:

"from source" - older TurnKey releases; prior to 'v15.2'; or
"Omnibus package" - more recent TurnKey releases, i.e. v15.2+.

In the case of the source install, our install should generally follow upstream instructions, although I note that there may well be some divergence. Having said that, GitLab should be installed to /home/git/gitlab and the log files will likely be a sub-directory of that.

In the case of the Omnibus installer, IIRC GitLab is installed to /opt/gitlab; and again the log directory should be a sub directory of that.

Please note too that the default log location (where most default TurnKey and Debian packages should log to) is within /var/log. In newer TurnKey versions, that include systemd, there is also the "system journal", which can be accessed via the 'journalctl' command.

Add new comment

form.antibot { display: none !important; } You must have JavaScript enabled to use this form.

Main menu

User menu

You are here

Hi there... I have some (mixed) news for you...

Hi Jeremy,

There are also some

Oops, sorry missed this post...

Is this actually a TurnKey GitLab appliance?

No, I don't have a / home /

Ah ha! Someone has already migrated to Omnibus package.

Update successful. I followed

Great news that the updates went well

Would you have the steps to

We need to know what version you have.

Add new comment

Plain text

Search form

You are here

Add new comment

Plain text