You are here
Daniele Lolli (UncleDan) - Sat, 2024/05/04 - 18:04
I just installed Gitlab 18 in a VMware virtual machine. I want to use it only in the LAN, so I specified as domain the IP address that I then statically assigned to the machine, but I can't log in even as root user: the password is correct because if I get it wrong I get invalid login. Does anyone have an idea where I can at least check what is going wrong? Thanks in advance.
Forum:
Argh!
Argh! Historically 500 errors with GitLab have been something of an ongoing issue with GitLab (our appliance in particular, but also other users). However we thought that we had fixed it for v18.0!? FWIW I personally tested v18.0 - using the password 'TurnKey12?' - and it seemed to be working ok!? Although it was VERY slow to start...
In my experience, it takes ages to be ready to login, up to 10 minutes before it's ready to log in. Before it starts properly it will throw a 500. Although your note that it gives an incorrect password error if you use the wrong password, makes me wonder if that's actually the problem here?
As per your note about wrong password errors, I don't think this is the issue. But just to be sure, you are using the username 'root' too, right? Note that's not the Linux root user & password, it's the GitLab root user & password. As
Also, does your VM meet the minimum system requirements, particularly RAM? Perhaps it's running out of resources and crashing when you get the password right and it logs you in? Also, as noted above, did you give it plenty of time to start up?
The GitLab docs note that the general production minimum requirements are 8GB (16GB recommended. Although the "constrained environment" docs (e.g. personal use/small teams) say that it should work with only 2GB & 1GB swap (2.5GB & 1GB swap recommended min). So I'd take that to mean 3-3.5GB minimum without swap - FWIW I used 4GB & 2x vCPUs with no swap in my test env and whilst it was slow to start and sluggish to use, it worked.
After doing a bit of reading online, some users have reported that if they don't set a password with enough complexity, they get 500 errors. AFAIK our GitLab firstboot config script requirements should be enough to allow login, but if your password isn't very complex, perhaps it's worth trying to set it again with something more complex? I.e.:
Keep a look out for any error messages that occur after that script exits too.
If all that checks out, then perhaps there are some hints in the GitLab logs? According to the docs, this should show the most recent log entries:
FYI we use the "omnibus install" but sudo is not required when running as the root Linux user.
There should also be a collection of log files in /var/log/gitlab. I forget which service is most likely responsible, but I'd try 'gitlab-rails' first or perhaps 'puma'. Perhaps it might also be worth checking Nginx logs too? Although my guess is that they will just note that the backend didn't respond appropriately (i.e. 500 error) and nothing particularly useful.
FWIW the inithook (firstboot script), uses the documented rake task:
Although if you run out of things to try/test, then perhaps it's worth trying to run that directly yourself. And/or the interactive rails console. Although I would expect the same result from that as the rake command would give above.
Either way, please let me know how you go. If you continue to have issues, I'll try again myself and see if I can reproduce the issue.
And thanks for your report. Whilst it's frustrating, I'd rather know.
I've now reproduced this latest 500 error on v18.1
I recently tried to reproduce the GitLab 500 error locally again and I can now!? I haven't launched a second one, but on this one I can consistently reproduce it... WTF!?!
TBH I don't understand. As I noted above, I have previously tested and couldn't reproduce it - although perhaps I should have tried multiple times? Beyond my previous manual tests, I am sure that it is an intermittent issue as we now have automated smoke tests. And I still have the automated screenshots of the GitLab v18.1 dashboard page (after you successfully login). So I can only assume that it's some sort of race condition during install or on first boot. It seems that if it occurs, then it will persistently recur.
Regardless, I can verify that Fabbe & Marcos' notes about tweaking the DB works for me too. Although I used the 'gitlab-psql' command. I.e.:
I can log in ok after that, although considering that it wipes the contents of the application_settings DB table I'm not completely clear what (if any) further implications it may have?
We try to make the firstboot scripts idempotent where possible, and if there was any data in a GitLab server and the GitLab firstboot scripts were re-run (including that 'gitlab-psql' command) then it would almost certainly lead to data loss. OTOH because we already recreate the encryption keys (with the exception of one particular one in some circumstance it seems...) then the firstboot script will already cause similar damage to an existing instance.
I hope to do some further testing ASAP and assuming that DB tweak has serious no further implications and I can reliably and repeatedly log into GitLab (after multiple clean installs) we will do an updated release. Fingers crossed that will be a "proper" and log lived fix for this issue...
In the meantime, I would love to hear feedback from anyone who has applied the tweak (removing the 'application_settings' DB table) and what - if any - negative implications and/or other issues they have hit after the DB adjustment.
Same error here
Same error here
Found in logs :
OpenSSL::Cipher::CipherError ():
encryptor (3.0.0) lib/encryptor.rb:98:in `final'
encryptor (3.0.0) lib/encryptor.rb:98:in `crypt'
encryptor (3.0.0) lib/encryptor.rb:49:in `decrypt'
Thanks for confirming the issue & providing additional info
Thanks for confirming the issue and providing additional info.
Did you try the suggestions that I noted above (from the GitLab docs)? If so what was the result? If it throws an error please post it.
I just tried locally and it worked for me!? Although I ran it a little unconventionally; I unpacked the ISO to a podman (docker like) container. The host was a Proxmox (KVM) VM with 2 vCPUs and 7GB RAM. Whilst perhaps the RAM was the fundamental difference, I would expect that if it was going to fail anywhere, then in a container would be the place! I will try again in a "proper" VM just in case the issue is intermittent or there's something else weird going on in the container that makes it work.
Unless you were already using lots of RAM, perhaps it's worth retrying with lots? Also making sure that you wait for ages for it to start.
Anyway, after waiting nearly 10 minutes after boot for it to finish starting/restarting, here's some screenshots I took:
Log in page
Logging in
Admin dashboard
Projects page
New project
Gitaly!?
In my case it seems that it is something concerning Gitaly, that seems to be quite critical in the documentation.
Tomorrow I will rebuild a new machine and my be rebuild updated ISO from the github scripts.
BTW: I gave 4 CPUs and 16GB RAM so it should be pretty ok...
{"error":"open /var/opt/gitlab/gitaly/gitaly.pid: no such file or directory","level":"error","msg":"find process","pid":48822,"time":"2024-04-11T05:59:38.655Z","wrapper":48822} unclean Gitaly shutdown: graceful upgrade: completed unclean Gitaly shutdown: received signal "terminated" {"level":"error","msg":"wrapper for process shutting down","pid":49831,"process":49771,"time":"2024-04-11T06:17:17.246Z","wrapper":49831}Hmm, so a slightly different issue?
Thanks for adding this extra info.
So it seems that while it still has the same result (500 error), your underlying issue is not quite the same!? That suggests to me that it's some sort of race condition... That sucks as they're notoriously hard to reproduce, thus diagnose...
I'm going to ask my colleague who did some work on the v18.0 release to have a look. He's more experienced with GitLab and Ruby on Rails apps in general.
It works!
Don't ask me if it makes any sense, but done like this:
1. rebuilt the ISO of tkldev from GitHub.
2. rebuilt the ISO of gitlab from GitHub
3. did a clean install on standard machine like “Debian 11 64bit” (I'm working on VMware 7.x) setting only 16GB RAM and 4 CPUs
It works like a charm!
Screenshot
Hmm, that's interesting!
That's interesting! As you're likely aware, v18.0 is based on Debian 12/Bookworm so I'm not sure exactly what it means that it works for you on a Debian 11/Bullseye base? But it's definitely interesting extra info.
As I noted in my other post above, I'll ask Anton (the dev who did some work on the v18.0 release) to have a look.
More info
For my testing purpose this is not a problem, but if it can be useful in troubleshooting it seems that my instance works only in HTTP and not in HTTPS:
PS: the reference on debian 11 is for testing: in VMware 7 debian 12 is not present as possible guest OS, so for the vm I chose debian 11even if obviously it is bookworm installed inside.
GitLab only supports HTTP by default
GitLab only supports HTTP by default. To support HTTPS, GitLab needs to be configured.
TBH, I don't 100% recall the specifics, but I'm pretty sure that we had issues making it work with self signed certs (like all the other appliances have). So we just left it as default.
I fixed this issue with the
I fixed this issue with the following commands
Checking encrypted values in the database
For me i had failure on application_settings so i ran this command
After this i am able to change settings in Admin Area.
Thanks for sharing!
Thanks for sharing, it's appreciated.
Some further investigation
The main problem with encryption is generated first if the SSL key generated for the domain is not previously set. This type of behaviour has come from reverse proxys for me.
Another tip i noticed if that after trying to comition the server the database is unmutable. So you kind a have to hard code the parameter using:
After that run again the
And we expect no errorsBe sure that nginx is pointing to the domain and generate a certificate in the reverse proxy
If non RP just restart as said before
Any ways
Wait a couple of minutes (LOL) and hope no short cut is made and you will be able to do it
For some reason kernel and privileged containers can be some pain, i recommend running it on a VM.
Thanks for that Marcos.
Thanks Marcos.
TBH, I thought that the latest TurnKey release (v18.1) had fixed the 50x issue (again). Although it has been an ongoing issue, so perhaps it is an issue again and/or still?!
Regardless, thanks for the pointers. I'll have a closer look.
Agreed re container - IMO VMs are a better option for software like GitLab.
And li little bit more
Another thing to notice is the version 17.3 is already EOL for GitLab team, a simple apt update & and upgrade wont work
Just make a curl to a file listed here compatible (i think 17.5.0 or something)
https://packages.gitlab.com/gitlab/gitlab-ce/packages/debian/bookworm/After that
After that you run
and simply run on edge version.
So keep in mind GITLAB is a Maintainers Heavy Duty VM
You rock man! Thanks for the heads up!
Great info for those already running GitLab, but also super useful for us to know too.
I'll put updating GitLab ASAP on the todo list so we ship with the latest stable version installed OOTB and new users won't need to worry about this.
500 Error version 18.1
Can't reproduce here?!?
Assuming that these are clean installs (sounds like they are) and therefore no modifications or customization, I'm stumped why it isn't working for you. Unfortunately I'm unable to reproduce this issue on a local VM installed from the v18.1 ISO (KVM on Proxmox - 2 vCPUs & 8GB of RAM). :(
My colleague (who has been working on a bug that seems unrelated) has also done a couple of clean v18.1 installs this week. I forgot to ask him what resources or platform he was using, but he said that he didn't hit any 50x errors.
Perhaps there is some race condition that only occurs under some very specific circumstance? The fact that it failed for you twice in a row - and we've been unable to reproduce it on our end makes me think some sort of environmental factor.
Have you double checked that you aren't over allocating resources on your Proxmox host? If you have limited resources on the host, then perhaps your container isn't actually getting the resources you've allocated it. Beyond that or some other host or hardware limitation/issue, the only other thing that I can think of is something specific to LXC and/or your LXC config?
Actually, I just read back through previous posts in this thread and I wonder if the bug I noted above in passing (seemly unrelated) might be related after all? To quote myself:
The bug we're working on is also a log in failure. I thought it was unrelated as the symptom is a log in loop (with error message) - not a crash with a 50x. FWIW it occurs if the password set on firstboot is in GitLab's password blacklist (which we've just discovered). But given my quote above, perhaps it is related?!
FYI as I noted earlier in this thread, the reliable password we use for testing is 'TurnKey12?' (no quotes). So perhaps that's worth a try? I.e. set a new password by rerunning:
Another thing that might be worth a try is installing to a VM?
Beyond that, if you want to try to troubleshoot, check the output of:
It might be worth having that running in a terminal, while you recreate the issue in the browser.
I'd love to hear how you go with it and if any of that makes any difference (even if it doesn't fix it). If it works ok in a VM, then when I get a chance I'll fire one up in LXC and see how that goes.
Sorry I don't have anything more for you. Hopefully something in that helps out?
Installed Gitea Instead
TBH I personally prefer Gitea to GitLab myself
TBH I prefer Gitea to GitLab myself and have a local instance running. Beyond other personal preferences, it uses MUCH fewer server resources. So unless you can give a GitLab server a ton of resources, then Gitea is much snappier.
Admittedly, Gitea doesn't have quite as many bells and whistles as GitLab. That suits me fine as I only use it as a local git repo so other GitLab features are no advantage for my purposes. Other TurnKey users noted that Gitea didn't included integrated CI - which they wanted to use. However Gitea now has "actions" which are quite similar to GitHub actions.
Add new comment