Daniele Lolli (UncleDan)'s picture

I just installed Gitlab 18 in a VMware virtual machine. I want to use it only in the LAN, so I specified as domain the IP address that I then statically assigned to the machine, but I can't log in even as root user: the password is correct because if I get it wrong I get invalid login. Does anyone have an idea where I can at least check what is going wrong? Thanks in advance.

Error500

Forum: 
Jeremy Davis's picture

Argh! Historically 500 errors with GitLab have been something of an ongoing issue with GitLab (our appliance in particular, but also other users). However we thought that we had fixed it for v18.0!? FWIW I personally tested v18.0 - using the password 'TurnKey12?' - and it seemed to be working ok!? Although it was VERY slow to start...

In my experience, it takes ages to be ready to login, up to 10 minutes before it's ready to log in. Before it starts properly it will throw a 500. Although your note that it gives an incorrect password error if you use the wrong password, makes me wonder if that's actually the problem here?

As per your note about wrong password errors, I don't think this is the issue. But just to be sure, you are using the username 'root' too, right? Note that's not the Linux root user & password, it's the GitLab root user & password. As

Also, does your VM meet the minimum system requirements, particularly RAM? Perhaps it's running out of resources and crashing when you get the password right and it logs you in? Also, as noted above, did you give it plenty of time to start up?

The GitLab docs note that the general production minimum requirements are 8GB (16GB recommended. Although the "constrained environment" docs (e.g. personal use/small teams) say that it should work with only 2GB & 1GB swap (2.5GB & 1GB swap recommended min). So I'd take that to mean 3-3.5GB minimum without swap - FWIW I used 4GB & 2x vCPUs with no swap in my test env and whilst it was slow to start and sluggish to use, it worked.

After doing a bit of reading online, some users have reported that if they don't set a password with enough complexity, they get 500 errors. AFAIK our GitLab firstboot config script requirements should be enough to allow login, but if your password isn't very complex, perhaps it's worth trying to set it again with something more complex? I.e.:

/usr/lib/inithooks/bin/gitlab.py

Keep a look out for any error messages that occur after that script exits too.

If all that checks out, then perhaps there are some hints in the GitLab logs? According to the docs, this should show the most recent log entries:

gitlab-ctl tail

FYI we use the "omnibus install" but sudo is not required when running as the root Linux user.

There should also be a collection of log files in /var/log/gitlab. I forget which service is most likely responsible, but I'd try 'gitlab-rails' first or perhaps 'puma'. Perhaps it might also be worth checking Nginx logs too? Although my guess is that they will just note that the backend didn't respond appropriately (i.e. 500 error) and nothing particularly useful.

FWIW the inithook (firstboot script), uses the documented rake task:

gitlab-rake "gitlab:password:reset[root]"

Although if you run out of things to try/test, then perhaps it's worth trying to run that directly yourself. And/or the interactive rails console. Although I would expect the same result from that as the rake command would give above.

Either way, please let me know how you go. If you continue to have issues, I'll try again myself and see if I can reproduce the issue.

And thanks for your report. Whilst it's frustrating, I'd rather know.

Guest's picture

Same error here
Found in logs : 

OpenSSL::Cipher::CipherError ():
  
encryptor (3.0.0) lib/encryptor.rb:98:in `final'
encryptor (3.0.0) lib/encryptor.rb:98:in `crypt'
encryptor (3.0.0) lib/encryptor.rb:49:in `decrypt'

Jeremy Davis's picture

Thanks for confirming the issue and providing additional info.

Did you try the suggestions that I noted above (from the GitLab docs)? If so what was the result? If it throws an error please post it.

I just tried locally and it worked for me!? Although I ran it a little unconventionally; I unpacked the ISO to a podman (docker like) container. The host was a Proxmox (KVM) VM with 2 vCPUs and 7GB RAM. Whilst perhaps the RAM was the fundamental difference, I would expect that if it was going to fail anywhere, then in a container would be the place! I will try again in a "proper" VM just in case the issue is intermittent or there's something else weird going on in the container that makes it work.

Unless you were already using lots of RAM, perhaps it's worth retrying with lots? Also making sure that you wait for ages for it to start.

Anyway, after waiting nearly 10 minutes after boot for it to finish starting/restarting, here's some screenshots I took:

Log in page

Logging in

Admin dashboard

Projects page

New project

Daniele Lolli (UncleDan)'s picture

In my case it seems that it is something concerning Gitaly, that seems to be quite critical in the documentation.

Tomorrow I will rebuild a new machine and my be rebuild updated ISO from the github scripts.

BTW: I gave 4 CPUs and 16GB RAM so it should be pretty ok...

{"error":"open /var/opt/gitlab/gitaly/gitaly.pid: no such file or directory","level":"error","msg":"find process","pid":48822,"time":"2024-04-11T05:59:38.655Z","wrapper":48822}

unclean Gitaly shutdown: graceful upgrade: completed

unclean Gitaly shutdown: received signal "terminated"
{"level":"error","msg":"wrapper for process shutting down","pid":49831,"process":49771,"time":"2024-04-11T06:17:17.246Z","wrapper":49831}
Jeremy Davis's picture

Thanks for adding this extra info.

So it seems that while it still has the same result (500 error), your underlying issue is not quite the same!? That suggests to me that it's some sort of race condition... That sucks as they're notoriously hard to reproduce, thus diagnose...

I'm going to ask my colleague who did some work on the v18.0 release to have a look. He's more experienced with GitLab and Ruby on Rails apps in general.

Daniele Lolli (UncleDan)'s picture

Don't ask me if it makes any sense, but done like this:
1. rebuilt the ISO of tkldev from GitHub.
2. rebuilt the ISO of gitlab from GitHub
3. did a clean install on standard machine like “Debian 11 64bit” (I'm working on VMware 7.x) setting only 16GB RAM and 4 CPUs

It works like a charm!

 

Daniele Lolli (UncleDan)'s picture

Jeremy Davis's picture

That's interesting! As you're likely aware, v18.0 is based on Debian 12/Bookworm so I'm not sure exactly what it means that it works for you on a Debian 11/Bullseye base? But it's definitely interesting extra info.

As I noted in my other post above, I'll ask Anton (the dev who did some work on the v18.0 release) to have a look.

Daniele Lolli (UncleDan)'s picture

For my testing purpose this is not a problem, but if it can be useful in troubleshooting it seems that my instance works only in HTTP and not in HTTPS:
 

PS: the reference on debian 11 is for testing: in VMware 7 debian 12 is not present as possible guest OS, so for the vm I chose debian 11even if obviously it is bookworm installed inside.

Jeremy Davis's picture

GitLab only supports HTTP by default. To support HTTPS, GitLab needs to be configured.

TBH, I don't 100% recall the specifics, but I'm pretty sure that we had issues making it work with self signed certs (like all the other appliances have). So we just left it as default.

Fabbe's picture

I fixed this issue with the following commands

Checking encrypted values in the database

gitlab-rake gitlab:doctor:secrets

For me i had failure on application_settings so i ran this command

gitlab-rails dbconsole --database main DELETE FROM application_settings;

After this i am able to change settings in Admin Area.

Jeremy Davis's picture

Thanks for sharing, it's appreciated.

Add new comment