You are here
deutrino - Fri, 2022/12/23 - 00:29
My TKLDev instance seems to be downloading OS updates and the like over and over again. Might this have something to do with squid-deb-proxy-client being installed? I typically install it on all my Debian VMs as I have an instance of apt-cacher-ng running for my network. But with this latest TKLDev instance, it seems like neither cache is getting used - not TKLDev's and not my apt-cacher-ng instance.
I didn't realize until searching for answers that squid is now being used in 17.x. The main reason I installed squid-deb-proxy-client on this TKLDev host is because I'm on fairly slow internet and was hoping it might populate its internal cache from the perfectly good one I've already got built up on my dedicated apt caching server.
Forum:
TKLDev should use it's own squid cache
It sounds like you're on to it, but to ensure that we're on the same page, on TKLDev squid is set up as a caching proxy for http and https. We use squid with a more generic proxy config to allow us to cache all sorts of packages and/or files (not just apt). Apt on TKLDev should be using it's own squid cache - configured in /etc/apt/apt.conf.d/00proxy:
Beyond that, TBH, I'm not completely sure of the best way to achieve your ends.
Possibly the quickest/easiest way to go would be to reconfigure your TKLDev and the TKLDev build process to use your existing apt-cacher for fetching packages. Configuring TKLDev itself is easy (i.e. replace /etc/apt/apt.conf.d/00proxy). To adjust the building appliance's apt cache, adjust the common bootstrap_apt conf script (should locally be found at /turnkey/fab/common/conf/bootstrap_apt). You'll need to adjust the following lines that start with 'Aquire' (lines 38 & 39 currently):
Hi Jeremy, thanks for the
Hi Jeremy, thanks for the explanation. I'm actually a bit more tempted than you might think to eventually set up something like Squid as a package cache, apt-cacher-ng works okay but it's got a few bugs and seems a bit worse for wear.
I think for the time being I'm going to uninstall squid-deb-proxy-client and see if that magically brings things back into working order, then decide what to do based on the results of that.
No worries.
Apologies that I didn't explicitly respond to your question about squid-deb-proxy-client. Assuming it's true to it's name, I have a fairly good idea what it likely does, but I have no idea how it does it specifically.
Regardless, worst case scenario, it's likely doing nothing. Worst case it's perhaps the cause of your issue?
Good luck with it. If you do go the Squid route, please feel free to ask if you have any questions. I wouldn't suggest that I'm an expert, but I went for a fairly deep dive when I set up the squid proxy for TKLDev v17.x. So if you have questions soon, hopefully I'll still remember something! :)
Actually, just before I posted, I had a quick look at the squid-deb-proxy-client package page and apparently it scans for network services published as _apt_proxy._tcp and will configure apt to use the proxy that it finds. In theory it probably should work fine (assuming apt-cacher-ng provides a "_apt_proxy._tcp" network service), although it's probably just as easy to manually configure it yourself (unless you plan to have fallback servers, etc).
squid-deb-proxy-client
squid-deb-proxy-client actually works great when it works, but Avahi based discovery is not easy to debug when it decides to stop working, which it occasionally does for mysterious reasons.
I started using it long ago to avoid doing more manual configuration, but such a thing mostly makes sense for laptops (use the proxy if discovered, continue without it if not), not for machines that stay in one place all the time.
Ah ok, that makes sense!
Thanks for the additional info. I wasn't aware that squid-deb-proxy-client works that way (i.e. autoconfigured proxy). That's actually pretty cool and I might have a play with it at some point. TBH, I was wondering what the point of it was and that makes much more sense to me.
Jeremy, I'm curious if you
Jeremy, I'm curious if you know offhand whether Squid can be a MITM caching proxy for downloading packages via HTTPS.
Currently I'm stuck downloading packages over HTTP so apt-cacher-ng can cache them, it'd be great if I could manage to stop downloading in cleartext as part of switching away from apt-cacher-ng.
No worries if you don't know, just figured I'd ask if you have experience (because if so you can warn me if it's a huge can of worms)
Yep, that's how Squid is configured on TKLDev! :)
By default Squid is configured to cache HTTP and HTTPS traffic. As you note, it essentially performs a MITM (man in the middle). It terminates the "real" TLS connection with the external remote, then uses it's own CA (which you need to have preloaded into the system for other apps to trust the connection) to generate new certs on the fly.
FWIW, when I developed the updated caching system using squid for TKLDev v17.x, I post the squid config info on *Nix StackExchange. I can also point you towards the TKLDev code that leverages it if you wish.
Adding: that question is for
Adding: that question is for my personal network, not in relation to use of HTTPS with TKLDev.
The info I posted above is very Debian specific
It is Debian specific although AFAIK, it should work the same on any Debian derived systems (including Ubuntu - although the package name may be different). If you are using an alternate distro, then I'm not sure, although worst case scenario, you could build it from source (with the relevant build option enabled).
Add new comment