You are here
Hello - I was able to run dehydrated successfully with just 1 domain, but when I added a second, this is what I see in the console:
[2019-09-24 10:46:47] dehydrated-wrapper: INFO: running dehydrated
+ ERROR: An error occurred while sending post-request to https://acme-v01.api.letsencrypt.org/acme/new-authz (Status 400)Details:
{
"type": "urn:acme:error:badNonce",
"detail": "JWS has no anti-replay nonce",
"status": 400
}
After doing some research, and confirming that my DNS is matching, it seems this could be caused by too many certs being issued within a week? Is there anything else I may be missing?
I've modified /etc/dehydrated/confconsole.domains.txt like so:
# please use this file with confconsole or
# alternatively use dehydrated with it's appropriate
# configuration directly
staging.domain1.com staging.domain2.com
I appreciate your input.
Something interesting to note
Something interesting to note - if I add this to my hosts file, it goes through with no errors, but only creates certificate files for just the first domain inside /var/lib/dehydrated/certs/staging.domain1.com:
I believe this problem is
I believe this problem is because they were on the same line.
How to update cronjob?
I'm looking at the cron job inside /etc/cron.daily/confconsole-dehydrated - and I'm unsure how to change the job to update certs inside of /var/lib/dehydrated/certs/sitename.com, as there's only one value defined for CERT=
Any help would be appreciated!
Apologies on slow response
Please ignore most of this post... I have left one paragraph which may still contain some relevant info.
TBH, I haven't tested it for a while, but last time I did, multiple domains worked fine.
If you are using Confconsole, then each domain goes on a separate line (within Confconsole). That should result in the domains all going on the same line (separated by a space) within the domains.txt file. That should result in a single certificate, which covers all configured domains.
If you want separate certificates, then you can manually edit the domains.txt and put each domain on a separate line. [Please see my link to the docs in my new post]
However, a quick google bought up this thread which suggests that this issue is caused by a change that has been made by Let's Encrypt which doesn't play nice with the version of Dehydrated that TurnKey comes with. My reading suggests that if you update to a newer version of dehydrated, that should solve the issue. It's looks like the current version of Dehydrated is in the Stretch backports.
You can find the full instructions online, but here's a quick run through:
Hopefully that fixes the issue...
Trying again...
Ok, so someone else reported this via support today and I ended up having a closer look. It turns out that the fix wasn't anywhere near as easy as I had hoped... Although it's not too hard, just a little mucking around.
I've documented the issue and the workaround on our issue tracker. As noted there, the method I've used; overwriting the default binary (/usr/bin/dehydrated) is poor practice, but it's the only method I've fully tested, so rather than risk something else going wrong, I'll leave it like that for the moment.
If you'd like to try doing it "properly", you could remove the Dehydrated package (apt remove dehydrated) and rather than downloading Dehydrated to /usr/bin/dehydrated, instead download it to /usr/local/bin/dehydrated. If you do give that a go please let me know how you go. If you run into any issues, please post back with the output.
Also FWIW if you want a quick and dirty way to launch it (rather than opening confconsole and clicking through the menu) try this:
Re your other questions, I've realised that I didn't really answer those either. Apologies on not answering explicitly. In the meantime, I suggest that you have a read through the Confconsole Let's Encrypt docs. Hopefully that might give you some insight into how it works. If you still have further questions after reading that, please ask and I'll do my best to answer.
Hello sir - please no need to
Hello sir - please no need to apologize - I realized I was not setting up the domains correctly in domains.txt. This is all working perfectly fine at the moment (provided I put this entry in my hosts file)
Anyhoo - I appreciate your insight on the dehydrated issue. Yeah - I've actually created a symlink so I can just call dehydrated with just dehydrated-wrapper.
Much appreciated! Keep up the good work.
Strange - I followed your
Strange - I followed your steps on the issue tracker, this is what ended up happening:
Oops... This is a bit embarrassing...
Well this is a little embarrassing... In my spam cleaning flurry, I just accidentally removed Theirry's user account, including a post he'd made here which was likely useful! Oops, so sorry Thierry! :( Unfortunately, I didn't realise my mistake until it was too late. :(
Luckily I still have the content open in another tab, so I'll repost here:
Thierry Mailloux
I ran into the same issue today. Following the workaround here:
https://github.com/turnkeylinux/tracker/issues/1359
Everything is working properly,
@JP The first thing i tried before running into this thread was updating dehydrated package from stretch backports. As this https://community.letsencrypt.org/t/trying-to-understand-urnerror-badnonce/102652 imply that it was outdated. Once the backports installed I had exactly the error you ran into.
Package info: https://packages.debian.org/search?keywords=dehydrated
I manually installed the dehydrated 0.6.5 and the fixed hook script.
My certificate has been issued correctly.
I decided to update to 0.6.5 because the changelog imply that APIv1 compatibility was broken
https://github.com/lukas2511/dehydrated/releases
Regards
No worry
Thanks for your understanding Thierry
I was really careful to click the right button this time when I added you to the "contributors" group (so you can skip most of the spam checks)! So we should be good now. Thanks for contribution (and your understanding). :)
JP, there are 3 components to the "fix"
Your most recent post suggests that you missed the 2nd step.
Still getting the error
Hi folks--
I'm still getting this error ("this_hookscript_is_broken..."). I definitely installed the recommended deb file, and it didn't report any errors. But I'm not sure it ran correctly, because I tried to follow JP's fix of replacing the script file directly, and I don't see any /usr/share/letsencrypt directory at all.
Does this mean the installer is not working right? Can I just replace the copy in /etc/dehydrated/confconsole.hook.sh directly? I'd really like to get past this and get a certificate working. Thanks for any insight!
--Clay
I recommend that you carefully read through the release notes
In theory, you should be able to just remove the /etc/dehydrated/confconsole.hook.sh file. If it doesn't exist, running the Confconsole plugin will copy the latest version across.
But having said that, there are 2 different releases which fixed the initial problem, the first included a new bug. So to be 100% sure that all is as it should be, I'd urge you to carefully read through the complete v1.1.2 Release notes. I recommend that you follow all the steps as outlined in "How to install/update" section (it won't matter if you've already run some of them).
Once you've done that, you should have a valid Let's Encrypt cert. But to ensure that everything is as it should be, please reboot. If your server comes back up and everything is working as it should, you're good to go. If it doesn't, then please run these commands (as root, or prefix both with sudo):
Although having said that, the current release (v16.x) should just work. This issue only occurs in our old v15.x appliances. So you are probably best off getting the latest version if possible...
Ba da bing!
Thanks, Jeremy! I had followed the release notes diligently (I thought), and didn't have any luck, but removing the script file from /etc/dehydrated as you suggested seems to have done the trick. No more self-signed certificate warnings now!
Regarding updating to v16, I will look into that. I'm relatively new to TurnkeyLinux, so I'll have to RTFM on how to do that with an active appliance, so I can update without losing my data.
Thanks very much for the help!
--C
Yep - I am all set! I
Yep - I am all set! I appreciate your help.
One thing to note: This command fails, I had to manually download the hook script and replace it. No big deal, just pointing it out.
Also, I had to run apt-mark hold dehydrated rather than:All is well - You guys rock. Thank youThanks JP.
I've applied your fixes to the GitHub issue. Thanks for posting back with the corrections and apologies that I missed those...
Hey guys, so I just added a
Hey guys, so I just added a new domain, and went to run dehydrated - and got this error. Apache would not start, saying there's an address already in use, could not bind to address:
Strange - it worked after a reboot
After a reboot, it worked ;-)
I'm not sure, but looks like our python server was still running
Looking at your log output, it appears that Apache wasn't running when you ran the Dehydrated wrapper and it didn't deal nicely with that scenario. TBH, I'm not 100% sure why as it should work ok, even on Core (which doesn't even have a web server installed by default). It sounds like there may be an edge case bug.
Then in your situation, to compound the issue, the Dehydrated wrapper still started our "add-water" web server (minimalist python webserver which hosts the Let's Encrypt challenges), when it perhaps shouldn't have. Then when you tried to restart Apache, port 80 was already in use (by "add-water"), blocking it from starting.
As "add-water" doesn't run as a service (it's explicitly launched by the Dehydrated wrapper) a reboot would have killed it, and Apache would start as per usual.
I'd really like to think that you've hit a rare edge (corner?) case bug. But regardless, thanks tons for reporting it. I've opened an issue and will aim to have a closer look when I get a chance. Please post back if you have any similar issues.
Do you mean after apply the fix discussed in this thread?
Just to clarify, you are hitting this issue after having applied the fix discussed in this thread?
And is it reproducible? I.e if you kill the "add-water" server (or reboot) and re-run the wrapper script the same issue reoccurs? And when you say "multiple domains" do you mean multiple subdomains (e.g. www.somedomain.com & other.somedomain.com)? Or with multiple completely separate domains (e.g. www.somedomain.com & www.otherdomain.com)? Or both?
Regardless, considering that you are also reporting this same issue, it seems likely that we have a bug in our script!? IIRC I did test with multiple domains when we initially set things up, but perhaps this latest update has broken something?
FWIW, I'm planning to spend a bit of time on this tomorrow in an effort to devise the simplest fix possible. Once I have that, I will publish it in a blog post and send it our via a newsletter as I'm almost certain that this will affect many users (as certificates expire). So any info you can share regarding the steps you've taken will assist me to understand any potential issues.
Ok great, thanks for the info.
Thanks for clarifying.
As you've possibly seen, JP noted that he was using it with domains on separate lines (which you've also noted works). So it seems likely that you're hitting a bug.
I'll try to have a look at this today and see if I can reproduce the issue. Assuming that I can, I'll develop a workaround and post back ASAP.
I think that I should probably write up a blog post and send out a newsletter to let people know. I anticipate that others will hit this as their certificates expire, so it'd be good to be on the front foot as much as possible.
Thanks again for reporting.
Hmm, it looks like it's trying to run them in parallel
Thanks for posting your log. It looks like it's trying to run them in parallel, rather than one by one. AFAIK, that's not how it used to work.
I have tested 2 on the same line and that worked ok, but looking at your log, it appears that is was the 3rd one that it really choked on.
So TBH, I'm not really sure what is going on here... Unfortunately, I haven't yet had a chance to test any further as I've been a bit tied up with other stuff. But I hope to circle back around to this ASAP. Sorry that I haven't been able to assist you to resolve this sooner.
Thank you
Thanks for all your help! I suppose I could have just killed the process on port 80, but it was almost 2am.
Appreciate the insight.
FWIW, in future if you wanted to check what was using port 80
Yeah, that's totally reasonable. As a general rule, applying kernel updates is the only time you should really need to reboot. But OTOH sometimes it's just the quickest/easiest way to clean things up! :)
FWIW, in future if you wanted to check what was using port 80 you could use "netstat". I only have a TKLDev running and handy, so I'll give an example using port 22 (SSH) instead.
So it's "sshd" running and it's PID (process ID) is 914. In this case, I know that SSH is running as a service, so the way to stop it would be via the "systemctl stop" command (otherwise it will just be restarted again). But in your case, all you would have seen is "python", which may not have helped you recognise what it was. Still you could have killed it anyway with "kill -9 PID". To use my above example, like this:
Anyway, glad to hear you're up and running again.
Hey JP, another user has reported the same issue...
You may well have already seen it, but another user has reported the same issue. If you don't mind, could you please manually re-run the wrapper script and see if the issue occurs again? (I.e. the issue where it fails to clean up the "add-water" server after running when more than one domain is configured).
It'd also be useful if you could share whether you are/were using 2 subdomains, or 2 separate domains. Also, whether you added the multiple domains either via confconsole (or manually on the same line of the /etc/confconsole/confconsole.domains.txt) or whether the domains are on separate confconsole.domains.txt lines. FYI if they're on the same line (as when they're added via Confconsole), it will generate a single cert with all domains; if you have multiple lines, a separate cert will be generated for each line (with the last line being the cert that will be used by default by Webmin/Webshell and the relevant webserver).
Hey Jeremy: Hopefully this helps
Hey Jeremy,
I manually re-ran the wrapper script with no issues just now.
I added these entries to my confconsole.domains.txt file manually, not using confconsole. It contains:
Great, thanks JP! :)
Thanks for checking that and posting back JP. That's really useful.
Well, this is interesting....
Well, this is interesting.....I just ran this again this morning to see what would happen. Here's the output:
So, I killed the python
So, I killed the python process on port 80, and ran it again. No issues.
Hmmm, it seems like our hook script is a little fragile...
I'm not quite sure why, but for some reason, it appears that our hook script is a little fragile, particularly the bit that stops the add-water challenge server.
FWIW, the way it works currently (or at least the way that it's meant to work), is that when add-water is launched, the PID (process ID) is written to a file (/var/run/add-water/pid). When it's done serving the Let's Encrypt challenge, the PID is read from the pidfile and the specific PID is then killed (and the pidfile cleaned up). Incidentally, that is a pretty common way of keeping track of a Linux process that runs as a daemon.
However, it appears that under some circumstance, the pidfile contains the wrong PID (and/or doesn't exist). My suspicion is that there is a race condition where the new challenge is launched (and the pidfile updated) before the clean up of the previous instance has completed. TBH I'm not really clear how that could happen, but it's the only way that your log makes sense, especially considering that it's only an intermittent issue (intermittent issues are often caused by race conditions).
I think what I'll need to do is actually check the path of the process which is using port 80, rather than just trying to keep track of add-water's PID (via the pidfile). Actually, looking through all the code, it's a bit messy and could really do with a "proper" clean-up as there is some duplication between our dehydrated-wrapper script and our dehydrated hook script. In the long term, it probably requires a rewrite, but for now, I'll see if I can just tweak the scripts to account for the possibility that the pidfile is wrong and/or doesn't exist. Hopefully that will resolve your issue, as well as the very similar looking one noted higher up in this thread (albeit slightly different circumstance; multiple domains on a single line).
Anyway, thanks for posting back with your log, that's really useful in trying to get this working reliably. I don't want to make any promises on when I will have something for you but I'll aim to get to it ASAP.
New challenge I am trying to solve....
Hello everyone,
I've deployed a new Turnkey LEMP stack, utiliting nginx. I installed and configured Varnish. Varnish is listening on port 80, and nginx is listening on port 8080. Varnish is acting as the proxy (caching server)
When I run dehydrated-wrapper, it sees an existing process running on port 80, and recognizes it as varnish. It will not stop the process - so I've stopped varnish manually, and confirmed port 80 is not in use. After updating my dehydrated environment, similar to how I updated it on my LAMP stack, here are the logs:
Anyone have any insight as far as why I am getting this add-water error again? If I can get this working, I'd try to find a way to manually stop varnish first.
The wrapper script has no knowledge of varnish
We only configure it to be able to deal with the default "webservers" that TurnKey ships with, so whilst it will be able to name the server, it won't try to stop it.
However, assuming that varnish is running as a service and will respond to the "service stop|start varnish" commands, it shouldn't be too hard to add support for it (or any other webserver for that matter). If you look at the wrapper script, then you could just add an additional entry there. I.e. change the existing line:
apache2 | lighttpd | nginx )To be:
apache2 | lighttpd | nginx | varnish )Or alternatively, you could add it on it's own line (especially if you need to do something else with it, rather than just run "service varnish stop"). So add an extra bit like this:
varnish) stop_server $WEBSERVER;;To put it in context:
[...] case $WEBSERVER in apache2 | lighttpd | nginx ) stop_server $WEBSERVER;; varnish) stop_server $WEBSERVER;; java ) TOMCAT=/etc/init.d/tomcat; [...]Either should work fine...
Also apologies that I still haven't swung back around to this. I'm getting really close to a v16.0RC of Core and have really been trying to push that forward...
Thanks!
Thanks!
One more thing to try and figure out - when I try to perform the renewal, the IP of my server doesn't match the cloudflare IP that the domain is pointing to - so the renewal ultimately fails.
I may just scrap using LetsEncrypt since these sites are using Cloudflare, unless anyone has any other ideas ;-)
Cheers
Should be possible to use Cloudflare and Let's Encrypt
TBH, I personally don't have much experience with Cloudflare. But a bit of googling suggests that if you set your Cloudflare config "just right" then it should be possible to use both Cloudflare (CDN and DNS) and Let's Encrypt (SSL/TLS) via our current setup. I can't test it myself, but this post on Let's Encrypt forums looks promising?!
Essentially you just need to make sure that Cloudflare is allowing access the URLs that Dehydrated is serving the challenges on. Otherwise the validation will never complete.
Another option to consider would be to switch to the DNS-01 challenge type. FYI all challenge methods are described here. But in short, to prove ownership of your domain, the DNS-01 challenge type relies on setting tokens via specific DNS TXT records, rather than serving them via a specific URL (HTTP-01). The advantage of that method is that you don't need to mess around stopping/starting services.
Currently TurnKey's Confconsole/Let's Encrypt setup only supports the HTTP-01 challenge type. So if you wanted to use the DNS-01 method, you would need to use Dehydrated directly and you'll also need a DNS-01 hook script. I.e. Dehydrated as currently installed, but launched directly, with a third party hook script and default Dehydrated config file; rather than via Confconsole, our custom config files and our Dehydrated wrapper script.
Assuming that you are using Cloudflare DNS, then there are a few Cloudflare DNS challenge Dehydrated hooks scripts floating about. I haven't tested any of them, but below are 3 I found (in no particular order). They all appear to be maintained (or at least have recent commits). Have at look at these:
Obviously you could also try a completely different Let's Encrypt client (i.e. something other than Dehydrated). Although it's perhaps worth noting that we explicitly chose Dehydrated as it doesn't leak any more info than it has to (as some of the other clients we evaluated did). I haven't looked at other DNS-01 supporting clients, but I'm sure they're about if you want to try your luck.
Perhaps a further option is to just use Cloudflare SSL/TLS certs. I have no experience with that, nor have any idea of costs etc, but AFAIK it's a service that they offer.
Wow - you are just amazing
Wow - you are just amazing man! As always, I appreciate your insight and detailed answers.
The biggest issue I had with varnish / nginx and LetsEncrypt/Dehydrated was the proxying. Varnish was not redirecting properly, and even after modifying the wrapper, it indicated varnish was stopping, but it wasn't. Then, the IP wasn't matching up with the Cloudflare IP for some reason. I suppose an alternative would be to disable Cloudflare temporarily, setting my IP back to the original, allowing me to perform the validation, but I was looking for something more automatic.
I did some digging myself - my cloudflare is setup with full (strict). I did some experimenting with varnish proxying requests to /.well-known and the like - and the funny thing is, I used the regular certbot, and the SSL cert was issued with no problem using this syntax, which doesn't need to stop nginx, it just uses webroot:
And also with the following below, which does need to stop nginx:
At any rate - I decided to just install the Cloudflare root CA origin certs, which don't expire for 15 years. My brain was just fried with spending so much time with nginx / varnish that I figured this was the best for now. However, I am looking forward to v16!
Hope this posts helps anyone using Cloudflare.
Hi all, a bit of a progress report...
I just wanted to post that over the weekend, I did a little work on making add-water (the mini server which serves the Let's Encrypt challenges) more reliable (i.e. to work around the issues that some with multiple domains have been having). I have just written it up in response to a new forums thread.
It's still unfinished (as noted in the post I just made) but I reckon it's quite close. Hopefully my post provides enough info for someone with some bash skills (and patience) to put the final pieces together (and share with us all!). We'll get to it eventually, but I have some competing priorities ATM so I can't give an ETA.
Hi all, there's now a "proper" fix! :)
I've just published Confconsole v1.1.1. It's not yet available from our repos (although will be soon) and still requires some specific steps to install and set up on a v15.x server (although better than instructions published previously).
Please note that users who have already updated via various other means are still recommended to install this update as it includes reliability fixes for add-water; our custom challenge mini-server. Please see the release notes for full step by step setup and further info - instructions cover both new and existing users.
Any issues, please ask. Any feedback (e.g. anything that isn't clear, etc). Please ask.
Heads up on an issue with the update...
It's been bought to my attention that after installing the v1.1.1 Confconsole update, the add-water service is being inadvertently enabled. That means that on reboot, it will start up and will likely block Apache (or other webserver) from starting!
The fix is easy:
Please note that on any server where you have already run the v1.1.1 update, you need to apply the above line and there is no value in updating to the newer package. For any new servers you launch though (or for anyone else who hasn't applied any fixes and stumbles across this thread) the newer v1.1.2 release resolves this issue (it's exactly that same as v1.1.1, but doesn't auto enable the add-water service on install).
Add new comment