BFF's picture

Hi y'all,

When I try to issue a certificate in Lets Encrpyt through confconsole in http-01 manually, everything goes well as follows:

	[2024-06-23 10:13:27] dehydrated-wrapper: INFO: started
	# INFO: Using main config file /etc/dehydrated/confconsole.config
	+ Account already registered!
	[2024-06-23 10:13:29] dehydrated-wrapper: INFO: found apache2 listening on port 80
	[2024-06-23 10:13:29] dehydrated-wrapper: INFO: stopping apache2
	[2024-06-23 10:13:30] dehydrated-wrapper: INFO: running dehydrated
	# INFO: Using main config file /etc/dehydrated/confconsole.config
	Processing myexampledomain.com
	 + Checking domain name(s) of existing cert... unchanged.
	 + Checking expire date of existing cert...
	 + Valid till Jun 23 15:35:05 2024 GMT (Less than 30 days). Renewing!
	 + Signing domains...
	 + Generating private key...
	 + Generating signing request...
	 + Requesting new certificate order from CA...
	 + Received 1 authorizations URLs from the CA
	 + Handling authorization for myexampledomain.com
	 + 1 pending challenge(s)
	 + Deploying challenge tokens...
	[2024-06-23 10:13:36] confconsole.hook.sh: INFO: Deploying challenge for myexampledomain.com
	[2024-06-23 10:13:36] confconsole.hook.sh: INFO: Serving /var/lib/dehydrated/acme-challenges/(PRIVATE) on http://myexampledomain.com/.well-known/acme-challenge/(PRIVATE)
	 + Responding to challenge for myexampledomain.com authorization...
	 + Challenge is valid!
	 + Cleaning challenge tokens...
	[2024-06-23 10:13:40] confconsole.hook.sh: INFO: Clean challenge for myexampledomain.com
	 + Requesting certificate...
	 + Checking certificate...
	 + Done!
	 + Creating fullchain.pem...
	[2024-06-23 10:13:43] confconsole.hook.sh: SUCCESS: Cert request successful. Writing relevant files for myexampledomain.com.
	[2024-06-23 10:13:43] confconsole.hook.sh: INFO: fullchain: /var/lib/dehydrated/certs/myexampledomain.com/fullchain.pem
	[2024-06-23 10:13:43] confconsole.hook.sh: INFO: keyfile: /var/lib/dehydrated/certs/myexampledomain.com/privkey.pem
	[2024-06-23 10:13:43] confconsole.hook.sh: SUCCESS: Files written/created for myexampledomain.com: /usr/local/share/ca-certificates/cert.crt - /etc/ssl/private/cert.key - /etc/ssl/private/cert.pem.
	 + Done!
	[2024-06-23 10:13:43] dehydrated-wrapper: INFO: dehydrated complete
	[2024-06-23 10:13:43] dehydrated-wrapper: WARNING: Python is still listening on port 80
	[2024-06-23 10:13:43] dehydrated-wrapper: INFO: attempting to kill add-water server
	[2024-06-23 10:13:43] dehydrated-wrapper: INFO: Cleaning backup cert & key
	[2024-06-23 10:13:43] dehydrated-wrapper: INFO: (Re)starting apache2
	[2024-06-23 10:13:43] dehydrated-wrapper: INFO: (Re)starting webmin.service
	[2024-06-23 10:13:46] dehydrated-wrapper: INFO: dehydrated-wrapper completed successfully.

But I don't know why i keep getting errors from the cron jobs:

	[2024-06-23 10:14:02] dehydrated-wrapper: INFO: started
	[2024-06-23 10:14:02] dehydrated-wrapper: FATAL: Unexpected challenge type:
	[2024-06-23 10:14:02] dehydrated-wrapper: WARNING: Something went wrong, restoring original cert, key and combined files.
	[2024-06-23 10:14:02] dehydrated-wrapper: INFO: (Re)starting webmin.service
	[2024-06-23 10:14:05] dehydrated-wrapper: WARNING: Check today's previous log entries for details of error.


It's like $CTYPE is null.

	$turnkey-version
	turnkey-nextcloud-18.0-bookworm-amd64
	$apt list confconsole
	Listing... Done
	confconsole/bookworm,now 2.1.5 all [installed]

 

I would post the letsencrypt.log here but it makes no sense since is only showing the same daily errors.

Could someone give me a help here?

Forum: 
Jeremy Davis's picture

It gives the full info when it works, but not when it doesn't!? (as it should).

TBH, I don't know why that's the case but there is a line in your logs that gives me some idea (albeit not real clarity on the issue). This is the line I'm talking about:

[2024-06-23 10:14:02] dehydrated-wrapper: FATAL: Unexpected challenge type:

Note the colon at the end. That's where the full dehydrated error log should appear. Obviously it's not...

As it's only failing when it's run from the cron job, I suspect that it's something related to the cron job itself. FYI cron jobs run in a limited environment so it seems likely that there is something missing from the cron env.

Out of interest, does the Let's Encrypt error occur within 24 hours of manually running it? I ask because beyond the cron env, there is another fundamental difference between running it manually and via cron.

When it's run manually, it will force update of the cert - regardless of when the current cert will expire. Via cron it should only attempt to update the cert if it expires in 30 days or less. New certs are valid for 90 days. Given that log line I quote above, if the cron failure happens within 24 hours of the manual update, then that's another unexpected and weird component of this problem.

TBH, it's a real mystery why no one else has reported the issue you're hitting. FWIW we're dogfooding our LE integration - this server uses the same Let's Encrypt process that you are. Although actually, perhaps your server is a different TurnKey version and/or has a different version of Confconsole? IIRC this site was actually built before I built the publicly released images.

Regardless, can you please confirm the TurnKey version of your server and the version of Confconsole it's running. Here's how to do that respectively:

turnkey-version
apt list confconsole

Re your suspicion that it's a $CTYPE issue, why do you think that? FYI if it's anything to do with the below message when installing/updating packages, then that's not an issue:

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = "en_US.UTF-8",
	LC_ALL = (unset),
	LC_TIME = "en_AU.utf8",
	LC_MONETARY = "en_AU.utf8",
	LC_CTYPE = "C",
	LC_COLLATE = "C",
	LC_MEASUREMENT = "en_AU.utf8",
	LC_NUMERIC = "en_AU.utf8",
	LC_PAPER = "en_AU.utf8",
	LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = "en_US.UTF-8",
	LC_ALL = (unset),
	LC_TIME = "en_AU.utf8",
	LC_CTYPE = "C",
	LC_MONETARY = "en_AU.utf8",
	LC_COLLATE = "C",
	LC_MEASUREMENT = "en_AU.utf8",
	LC_NUMERIC = "en_AU.utf8",
	LC_PAPER = "en_AU.utf8",
	LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").

FYI, I copied that from a local v18.0 LAMP server I have running. I have not changed any of those server side. (Some are propagating via SSH from my local PC,

BFF's picture

Thank you for your reply,

I was analysing the source code here and i found out that the message that you stated only appears in 3 cases. Either is http-01, dns-01 or neither as in lines 262 to 271:

case $CTYPE in
    http-01) cp "$SH_HOOK_HTTP" "$CC_HOOK"
             PORT=80
             sed -i '\|^CHALLENGETYPE=|s|=.*|="http-01"|' "$CONFIG";;
    dns-01)  cp "$SH_HOOK_DNS" "$CC_HOOK"
             PORT=443
             sed -i '\|^CHALLENGETYPE=|s|=.*|="dns-01"|' "$CONFIG";;
    *)       fatal "Unexpected challenge type: $CTYPE"
esac
export CHALLENGETYPE=$CTYPE

Therefore my suspicious of $CTYPE == null. People tend to not report it for lazyness and also because they might not use the resource as often. As I have seen in the foruns, they tend to use dns-01 more often than te basic http-01.

For your question about the cronjob: Yes it's daily at 06:08 am and this behavior persists when triggering manually using:

/usr/lib/confconsole/plugins.d/Lets_Encrypt/dehydrated-wrapper --force

I already post it before but here it's:

$turnkey-version
turnkey-nextcloud-18.0-bookworm-amd64
$apt list confconsole
Listing... Done
confconsole/bookworm,now 2.1.5 all [installed]

Basically i'm just using the nextcloud ISO provided for proxmox. I tried to not tweek as much as possible with this ISO to maintain stability so I'm (almost) sure that no basics distro configurations where not removed or changed (apart for ssh.service which was completly removed and doesn't concern this problem).

Jeremy Davis's picture

My apologies for misunderstanding the context of your comments. It never occurred to me that you were referring to the source code and I didn't recall the 'CTYPE' variable name off the top of my head. Thanks for the clarification, as well as the extra info. Alos apologies that my previous response was so wildly off the mark.

Anyway, you are totally on the money! I've just run a few tests and can (easily) reproduce it. TBH, I'm not sure how that slipped through, it's a pretty serious issue. On face value it appears that it was a regression that I introduced when I fixed the issues we had with the DNS-01 challenge early in the v18.0 release.

I'll develop a fix and get back to you ASAP.

Thanks again, particularly for your persistence!

Jeremy Davis's picture

Actually scratch that. Looks like my initial testing was poor... :( It seems this new version may have introduced a new regression **sigh**

I'll be back soon with another new build.


Thanks again for reporting the issue. I'm pretty confident I've fixed it, but it'd be great if you could test it to 100% confirm.

To download and install it (on your appliance):

# download
wget https://www.turnkeylinux.org/files/tmp/confconsole_2.1.5+2+g40d7d22_all.deb

# for extra security, check the sha512sum against what is posted below
sha512sum confconsole_2.1.5+2+g40d7d22_all.deb

# install
apt install ./confconsole_2.1.5+2+g40d7d22_all.deb

SHA sum:

e5aa2a31aa6e9d60e03a26aed05e260f1969294a7f1f819a81a89037c33318b01f87dc09a869b02887dbd313bf5fa8c99cd96a52883c865a5be8ca820f34423e

Look forward to hearing how you go. Assuming that we confirm it's good, I'll push it to the apt repo.

Jeremy Davis's picture

That'll teach me for getting ahead of myself... Anyway, I've tested this better. So please try this one:

# download
wget https://www.turnkeylinux.org/files/tmp/confconsole_2.1.5+5+g515d28d_all.deb

# for extra security, check the sha512sum against what is posted below
sha512sum confconsole_2.1.5+5+g515d28d_all.deb

# install
apt install ./confconsole_2.1.5+5+g515d28d_all.deb

SHA sum:

70cedcb717ff3eb872ce2e3d6b9c8f56fe4ef7f09abe32e5933fa2ebf4d9631f2058b8059f93df84ea9d4f81261078e1ca6e212cb934883b0aba656f01d69c4e

Look forward to hearing how your go.

BFF's picture

I tried to download but got 404:

	wget https://www.turnkeylinux.org/files/tmp/confconsole_2.1.5+5+g515d28d_all.deb
	https://www.turnkeylinux.org/files/tmp/confconsole_2.1.5+5+g515d28d_all.deb
	Resolving www.turnkeylinux.org (www.turnkeylinux.org)... 2a06:98c1:3120::c, 2a06:98c1:3121::c, 188.114.97.12, ...
	Connecting to www.turnkeylinux.org (www.turnkeylinux.org)|2a06:98c1:3120::c|:443... connected.
	HTTP request sent, awaiting response... 404 Not Found
	2024-06-27 23:31:41 ERROR 404: Not Found.
BFF's picture

I took your commit from github and edited in my server.

Everything seems fine:

[2024-06-28 00:07:13] dehydrated-wrapper: INFO: started
[2024-06-28 00:07:13] dehydrated-wrapper: INFO: found apache2 listening on port 80
[2024-06-28 00:07:13] dehydrated-wrapper: INFO: stopping apache2
[2024-06-28 00:07:13] dehydrated-wrapper: INFO: running dehydrated
[2024-06-28 00:07:24] confconsole.hook.sh: SUCCESS: Cert request successful. Writing relevant files for mydomainexample.com.
[2024-06-28 00:07:25] confconsole.hook.sh: SUCCESS: Files written/created for mydomainexample.com: /usr/local/share/ca-certificates/cert.crt - /etc/ssl/private/cert.key - /etc/ssl/private/cert.pem.
[2024-06-28 00:07:25] dehydrated-wrapper: INFO: dehydrated complete
[2024-06-28 00:07:25] dehydrated-wrapper: WARNING: Python is still listening on port 80
[2024-06-28 00:07:25] dehydrated-wrapper: INFO: attempting to kill add-water server
[2024-06-28 00:07:25] dehydrated-wrapper: INFO: Cleaning backup cert & key
[2024-06-28 00:07:25] dehydrated-wrapper: INFO: (Re)starting apache2
[2024-06-28 00:07:25] dehydrated-wrapper: INFO: (Re)starting webmin.service
[2024-06-28 00:07:28] dehydrated-wrapper: INFO: dehydrated-wrapper completed successfully.

Thank you for your fast response.

Jeremy Davis's picture

Yay, thanks for confirming that the updated script works. Sorry about the broken link, not sure quite what happened there, but it was getting late.

Thanks again for reporting. Given my testing and your confirmation, I'll aim to rebuild and push the new build to our apt repo today.

I aim to post back and if/when you have time it'd be great if you could confirm the apt package works too.

Regardless, take care and I look forward to next time... :)

Jeremy Davis's picture

FYI I finalised the new version of Confconsole (v2.1.6) last month and pushed to the TurnKey v18.x (Debian 12/Bookworm) repo.

Add new comment