Ok, this all sounds brilliant. I hope to make a start today, but I have a bit of a backlog, so no promises.
My inclination would be to take a phased approach to this:
Build new GitLab appliance using Omnibus installer - basic testing to ensure all is as it should be (probably the easy part - I have no reason to think that there will be any issues).
Test TKLBAM backup/restore to ensure we're capturing all required data.
Test migration from existing (source install) GitLab server to new (Omnibus install) server (we've also had an offer via email for assistance with that which is cool). I have an idea on how this might be done, but yet to try "in the real world".
Look at optimisations as suggested by you. I really like your suggestions to add value!- these could have a staged rollout if time gets tight (e.g. include some initially, with others added in future releases).
Out of interest, re your swap tests/research; are you using a file or partition? Seeing as most builds have swap by default (it's only AWS AMIs that don't IIRC) my inclination would be to have a confconsole plugin which adds a swap file. There are few advantages to that IMO:
Super easy to implement and modify on the fly (ease of tuning - can be easily adjusted bigger/smaller).
Can easily be disabled/removed if freeing disk space becomes higher priority that providing swap (related to #1).
No low level disk modifications required (safety - reduced risk of inadvertently causing damage due to circumstance outside our control).
Will still be relevant to existing builds (AFAIK you can have both a swap file and a swap partition) - although it could easily be included in AWS only if that is deemed preferable (general applicability).
Alternatively, an additional volume dedicated to swap could be another option? But obviously would require adjustments outside of the server itself. So I think that just documenting that possibility is probably the best way to go.
I'd also be interested in what "swappiness" value you found to be ideal (assuming you played with that?). My inclination would be to set it somewhere between '1' (minimum swap usage - without disabling) and '10' (only swap when RAM usage hits ~90%). FWIW Debian default is '60' (and we don't adjust that). We could make it adjustable by the user (within Confconsole?), but I'm inclined to just set a sane default. We could provide some documentation to point users in the right direction if they want to dig in further.
[update] Actually, I'm not so sure re swappiness now... I've just read a more technical writeup which suggests that setting the ideal value is not as straight forward as I had been lead to believe! It's also worth noting that because unused RAM is used as a disk cache, if the RAM is getting quite full, then that reduces the opportunity for caching. I guess considering that AWS servers have none by default currently anyway, adding swap (even if swappiness is low) will still have value.
Awesome, thanks Tim
Ok, this all sounds brilliant. I hope to make a start today, but I have a bit of a backlog, so no promises.
My inclination would be to take a phased approach to this:
Out of interest, re your swap tests/research; are you using a file or partition? Seeing as most builds have swap by default (it's only AWS AMIs that don't IIRC) my inclination would be to have a confconsole plugin which adds a swap file. There are few advantages to that IMO:
Alternatively, an additional volume dedicated to swap could be another option? But obviously would require adjustments outside of the server itself. So I think that just documenting that possibility is probably the best way to go.
I'd also be interested in what "swappiness" value you found to be ideal (assuming you played with that?). My inclination would be to set it somewhere between '1' (minimum swap usage - without disabling) and '10' (only swap when RAM usage hits ~90%). FWIW Debian default is '60' (and we don't adjust that). We could make it adjustable by the user (within Confconsole?), but I'm inclined to just set a sane default. We could provide some documentation to point users in the right direction if they want to dig in further.
[update] Actually, I'm not so sure re swappiness now... I've just read a more technical writeup which suggests that setting the ideal value is not as straight forward as I had been lead to believe! It's also worth noting that because unused RAM is used as a disk cache, if the RAM is getting quite full, then that reduces the opportunity for caching. I guess considering that AWS servers have none by default currently anyway, adding swap (even if swappiness is low) will still have value.