Glad to hear that you're happy about the move to the Omnibus package. It certainly makes it easier to maintain for all of us! :)

Regarding the backups. That's very interesting. FWIW, I spent a ton of time working out the backup hook scripts to get it all working reliably. I was under the impression that TKLBAM should still do a diff of the file and only sync the bits that had changed. During my testing, the incremental backups were larger than ideal, but were still smaller than full backups. But I was testing with a fairly minimal data set and re-running backups soon after, which may explain why I didn't notice this.

Perhaps it might be better to extract the tarball that the GitLab backup produces and then backup the actual files (instead of a bundled archive) instead? Then an incremental backup should only include changes. Although the downside of that is that we'll end up with 2x data sitting on your server. We could clean up the data after upload I guess, but that may cause users issues which they may find hard to work out (i.e. running out of free space, but when you check it might appear to have plenty of free space).

Pondering this some more, the GitLab backup page notes a couple of things that are probably well worth a try. Namely BACKUP=dump and GZIP_RSYNCABLE=yes. According to the docs, that makes incremental backups work better. So might be the way to improve things? (TKLBAM doesn't use rsync, but I imagine that if it's more friendly for rsync, then it'll also most likely be more friendly for TKLBAM incremental backups). If you have some spare time and/or energy, it'd be great if you get a chance to check it out and let me know if that appears to help.

My only potential concern regarding that path, is that we'd still need to collect the GitLab version (IIRC it currently collects that from the original filename). I'm sure there'd be a relatively easy workaround for that, but I wanted to mention it.

Regarding your explicit question WRT to Postgres, yes unfortunately TKLBAM doesn't recognise the third party install of Postgres that GitLab does. TBH, I'm not really clear exactly how TKLBAM does the Postgres backups (well I know in theory what it does, but not how). I have had a look through the code, but it's Python2 (still) and I'm a bit novice still when it comes to Python (particularly Python2, I'm more comfortable with Python3). Plus I find the TKLBAM code particularly dense. So unfortunately, I don't have any ideas on how to make TKLBAM "see" the GitLab Postgres instance.

However, if you wanted to pursue your current path (i.e. manually collect the bits you want backed up; rather than using "GZIP_RSYNCABLE=yes" as noted above) then I have a few ideas.

The first is to create a custom TKLBAM hook script to dump the DB somewhere (that TKLBAM then backs up). IIRC the GitLab Postgres instance requires usage of a GitLab specific command (perhaps even 'gitlab-postgres' or similar).

The other would be to modify the existing hook script (which I wrote) to just run a GitLab backup of the DB. I.e. use the SKIP=... option to skip everything except the DB. I suspect that this should do the trick:

gitlab-backup create SKIP=uploads,repositories,builds,artifacts,lfs,registry,pages

Regardless, thanks for bringing this to my attention and if you're able to assist, hopefully we can come up with something that works a bit better than the current arrangement... Although please note, I'm currently trying to funnel all my energy into getting a v16.0RC of Core out the door. Once I have that published, then I'll have a bit more time and energy to work on stuff like this.