It sounds like a strange issue!

The fact that you say that it often works eventually, suggests that it may be some sort of race condition?! Intermittent issues can be notoriously hard to resolve in my experience.

Next I was going to say that it's probably also worth double checking free disk space. But you beat me to that and it sounds like you've ruled that out...

Re the StackOverflow Q&A that you refer to, unfortunately the answer order isn't guaranteed to be the same, and many answers include the command you note. FWIW, you're best referring to a particular answer via clicking the "share" button below the relevant answer and then copy/pasting the URL.

Anyway, I'm no Redis expert, but after doing a but more reading, I'm wondering if it's a permissions issue? I.e. Redis can't write to disk? (TBH, seeing as it's intermittent, I'm really not sure but perhaps?). According to what I've read now about the setting you tweaked though, that should suppress the error message, but not necessarily fix the underlying issue. See this answer on the Q&A you linked to. Although I suggest that you keep reading before you race off and try that blidly.

This answer (again on the same Q&A you linked to) sounds like a sensible approach and has quite a few upvotes. I'm not recommending that you necessarily follow the suggestion to modify /etc/sysctl.conf, but starting at the log file (probably /var/log/redis/redis-server.log) or checking via journalctl seems like a really good idea!

I suggest that you recreate the issue, and get the last few lines of the redis log so you can try to see exactly what it's up to. Assuming that the log file is /var/log/redis/redis-server.log then to get just the last 30 lines, do this:

tail -30 /var/log/redis/redis-server.log

Feel free to post here if you want my 2c on it.

Regarding getting further help. Canvas have a support Q&A, sort of like forums, although my experience suggests that it's not so great for technical self-hosting related issues. The Canvas Google group has been quite good and contains some really good info, but getting timely assistance for specific issues can be a bit hit and miss sometimes. Still it may be worth posting there? If you do, please feel free to cross post a link to that here.

I'm really glad to hear that you worked it out and thanks for reporting back to let us know. I'm almost certain that this will be useful for other users.

TBH, I'm not completely surprised that it ended up being system resources. Canvas is a beast of a thing and the hardware requirements seem to just escalate over time!

As for the update, I haven't yet announced it, but the new Canvas release should now be available on AWS via the Hub (and I hope to have it available from the Marketplace really soon - but unfortunately no clear ETA). It's possibly irrelevant to you, but I'll update the appliance page soon with the ISO, OVA & VMDK.

Regardless, there are 2 primary pathways open to you. One is to manually upgrade/update your existing server (both OS and Canvas itself). The other is to migrate your data to a new server. Unfortunately in the case of the Canvas appliance, neither are going to be super clean and easy and both will require you to get your hands a bit dirty and may need some trial and error.

If you choose the upgrade path, then you will need to create a snapshot of your current server. I personally advise that you launch a new server from your snapshot and work within that. I also advise that you document as you go, so worst case, you can trash the snapshot server and start again (and be able to easily recreate the steps that worked). That will also allow you to limit the downtime. You could even leave the current server untouched and do a trial run with a snapshot and then replay your documentation later. Essentially, the OS migration is migrating from Debian 9/Stretch (the basis of TurnKey v15.x) to Debian 10/Buster (the basis of TurnKey v16.x). The Debian docs are quite extensive, plus you'll find plenty of tutorials online. As for Canvas, generally the Canvas upgrade docs will be the primary source of info in that regard. There may be some idiosyncracies (such as our last Canvas release included a fork of the RCE code repo in an effort to work around an issue in the upstream source code). But mostly it should be a case of following their recommendations.

The other option is to migrate your data from your existing instance to a newer one. In theory, this path would be my preferred option as then you just move what you need to and everything else is fresh and shiny (e.g. newer version of Ruby without needing to make any changes) and with no old config and files laying around. Unfortunately, I'm not super familiar with the inner workings of Canvas, so am not aware of everything that might need to be moved. You would obviously need to move the database (i.e. dump your current DB and upload to the new server). Your config files will also need to be copied, plus wherever the uploaded files are stored (a quick google suggests it may be /var/www/canvas/tmp perhaps?). You would then need to update the config files as per the Canvas upgrade docs; as well as running the migrations (to upgrade your database).

Our TKLBAM backup tool may be useful, but unfortunately, in the case of Canvas, it almost certainly won't be a simple case of backing up your current server and restoring your backup to a new TurnKey server...

Whichever way you go, I would urge you to work within a "disposable" server and document as you go. That will relieve some pressure as worst case you can trash it and start again. Also, you won't be getting so much pressure from others as they won't be affected any more than needed i.e. when you actually do the migration or upgrade "for real". If you go the data migration pathway, you could even download the ISO or VM build and do your initial explorations in a local VM.

Anyway, I'm happy to assist as best I can. So please feel free to ask if you have questions. I can also elaborate further if you need more info.

Investigating another issue with another Canvas user, we've discovered a weird intermittent problem. You can read more on that thread and/or on the issue. Essentially it's a race condition between the "Passenger standalone" service and mod_passenger. The "standalone" config does not include serving the API so if it runs under that, the API won't be working. That explains why it worked when I tested it, but others have reported ongoing issues (or not - as the case maybe).

As noted, the fix is to disable the "passenger" service and restart Apache. I.e.:

systemctl stop passenger
systemctl disable passenger
systemctl restart apache2

It's also worth noting that there are 2 other requirements for the RCE API to work properly. You may already be fulfilling these, but for the sake of completeness:

• Your server must be configured to be served and accessible via a "proper" domain name which you own.
• Your server must have a valid TLS (aka SSL) certificate (Let's Encrypt certs are fine, but self signed certs won't work).

We'll be releasing an updated Canvas appliance with the required changes included ASAP. However, following the instructions above should get your current server up and running properly.

If you continue to have issues, please confirm that you have fulfilled the other points as well. If you want further assistance, please share the URL so that I can have a look from my end to double check the DNS config and cert. Further investigation will require you create a log in for me on your Canvas server (it doesn't need to be admin, so long as you can guide me on how to recreate the issue from a non-admin user account).

I've just logged in and had a quick look. On face value everything appears to be working ok. Although I can't see the page that you show in your screenshot and I can't work out how to get to it?

Although, one thing I did notice is that a lot of the content (including cookies) is being served from your server's IP address, rather than the domain name. I'm not sure, but I suspect that this may be part of the issue?! The RCE API is served via HTTPS on port 3000.2 It uses CORS to allow the usage of resources from different sources (different ports are considered different sources). For security, CORS has a same origin policy. So to allow the use of an alternate origin (i.e. port 3000) a special header is provided by the web server allowing the use of DOMAIN_NAME:3000. However, my guess is that instead of providing DOMAIN_NAME:3000, the header is providing IP_ADDRESS:3000 so it doesn't match what Canvas is pre-configured to accept.

I could be wrong, but my guess is that when you first set this up you were using the IP address. But then you switched to using the domain. And somewhere along the line the IP address has been stored somewhere (perhaps in the DB? perhaps in a a config file?) and although it appears to work ok via the domain, it's actually serving a lot of the content directly from the IP address...

Also FWIW (and I'm fairly sure this isn't related, but is worth fixing) a few things (such as images) appear to be being served via vanilla HTTP (even when browsing via an https URL). A dirty but easy way to work around that would be to redirect http to https. The better way to do that would be to fix what is explictly using http, and redirect to https anyway...

Ok so I worked out how to get to where you were. Part of the issue was that I had "Chrome developer tools" enabled (which allows you to see javascript errors which is invaluable for resolving these sort of issues, but was sort of a little confusing in this case). I disabled that and browsed to Courses >> [selected a course] >> Pages >> Add page.

That showed the same issue as per your screenshot. I then re-enabled the dev tools and I can see that your Canvas instance is trying to load the API from localhost:3000 ?! Clearly that's not going to work - unless of course you had installed Canvas locally...!

So I had a bit more of a look. I'm not sure why or how, but my bet is that for some reason, the relevant config file wasn't updated on firstboot. The "dynamic settings" file ships with "localhost:3000" as the rce-api location, but it should be updated when you run the firstboot scripts. Regardless, please try manually adjust that. The file should be at /var/www/canvas/config/dynamic_settings.yml and the line that you'll need to update should currently look like something this:

     app-host: localhost:3000

Change it to this (instead of YOUR_FULL_DOMAIN use your actual full domain name for your Canvas site):

     app-host: YOUR_FULL_DOMAIN:3000

Then restart the relevant services:

systemctl restart canvas_init
systemctl restart apache2

And hopefully you should be all good now! :)

Please let me know regardless...

So I don't think it's necessary to backup Redis (although I could be wrong... I certainly don't consider myself a Canvas expert). Regardless, have you tried troubleshooting the 500 error? That usually refers to a background service that has crashed and/or isn't running.

The bulk of the data is in the PostgreSQL 'canvas' DB. and the rest should be within a particular directory within the Canvas dir (/var/www/canvas). I forget what it's called, but perhaps "files"?

Personally, I think migrating to the newer version is a better path (then you get the updated OS as well), but upgrading your current install should be possible. You note that "it is not pulling from any git repository and the commands all fail". What version of the appliance are you using? TurnKey Canvas appliance has been installing Canvas via git fro some time. Although one v15.x release was using a third party repo for the RCE API because of a bug in the Instructure repo. So my guess is that perhaps you were in the wrong dir (or perhaps it's a really old TurnKey Canvas server)?

I don't know about any other modules/plugins/addons (or even if Canvas has those?) but the DB and the files should be enough to transfer your data from the old server to the new one. Our built in blackup tool; TKLBAM should be able to collect everything you need from your old server and transfer it to your new server. Although I doubt that a vanilla restore across a major version will just work, so I'd recommend what we call a "staged restore". If you're interested in hearing more about that, then let me know.

Regardless of how, if/when you migrate your data, you will still need to run some of the "upgrade" steps from the Canvas docs. Particularly the database migrations, but some of the other steps too perhaps...

Ultimately, they make money out of hosting it themselves. It's open source and I don't think that they go out of their way to make it any harder to self host than they have to. But their business model depends on the fact that it's often easier to pay them than it is to host and maintain yourself. So there is not much motivation for them to make it even easier to self host/self manage as it would likely cannibalise their sales.

I just realised that you raised a question which I hadn't directly addressed:

Shouldn't the Turnkey setup change that URL during the initial setup process?

Yes, it should- and it should have! I'm not at all clear why it didn't in your case but I will investigate as it's possibly a bug (which may have already been addressed by my colleague in the upcoming update and if not it will be)? The new appliance (which I hope to release next week) will certainly be more robust and should "just work".

I am unaware of those modules, but the background task runner runs as a separate server called "canvas_init". So I suggest that you check the status of that. I.e.:

systemctl status cavnas_init