Bill Taroli's picture
Bill Taroli - Tue, 2016/01/19 - 10:37

Working on a new custom app based on tomcat (standalone), mongodb, with oracle java bits and our own app code... But I noticed that doing the make was getting stuck with:

...

Setting up casper (1.236-turnkey+12+gc47bd84) ...

Warning: Deferring update-initramfs -u
Setting up ca-certificates-java (20140324) ...

At this point there is a java process eating nearly 100% CPU and quickly shows up as defunct.

I have repeated this on two different host platforms (Mac and Linux) under VBox 5.0.12. Imported TKLDEV OVA, git clone'd tomcat source and did a make. I'm either missing a step or there's a problem.

 

 

Forum: 
Support
Tags: 
tomcat
tkldev
building
Jeremy Davis's picture

Jeremy Davis - Tue, 2016/01/19 - 11:07
But it certainly sounds like something is going wrong. I won't be able to give you much advice until I have a look myself and see if I can reproduce it.

In the meantime you could try downloading the existing ISO (to use as a base) and use TKLPatch to customise and create a new ISO. TKLPatch was the precursor to TKLDev and uses a similar design to allow customisation. I.e. it has a conf script and overlay directory.

Bill Taroli's picture

Bill Taroli - Sat, 2016/01/30 - 00:41

I wondered if you'd had a chance to check on this, being able to use TKLDev to build with tomcat bits enabled...?

I'll try TKLPatch, but I guess I'm worried that I'll wind up having to redo stuff later once things are addressed. This is actually my very first experience with TKL and I am hoping not to paint myself into a corner.

Jeremy Davis's picture

Jeremy Davis - Sat, 2016/01/30 - 06:57
But I just did a build of the v14.0 Tomcat appliance on v14.0 TKLDev and it completed fine for me...

Perhaps it's worth trying to build the standalone Tomcat appliance on it's own first. That should complete successfully (as it just did for me). Then you will know whether there is something wrong with your TKLDev or if it is some of the customisation you've done...

Actually, looking again at your OP, I'm guessing that this is fairly early in the build process. TBH I'm almost certain that it's caused either by a zombie process (from a previous failed or interrupted build) or by something else running in the background (that is conflicting). A reboot should probably fix both of these possibilities... And a make clean wouldn't hurt either...!

So to reiterate:

  • reboot
  • cd tomcat (if it exists; cd products and git clone it if not)
  • make clean && make
  • (assuming above completes successfully) cd custom-app-dir
  • make clean && make
    • Bill Taroli's picture

    Bill Taroli - Sat, 2016/01/30 - 09:35

    Steps I followed:

    1. Re-imported TKLDEV 14.0 via OVA (VBox 5.0.14)
    2. Booted and completed install; skipped api and email, but accepted security updates
    3. Auto rebooted
    4. Quit console app to drop to login, and logged in as root.
    5. cd products
    6. git clone https://github.com/turnkeylinux-apps/tomcat.git
    7. make clean && make

    For giggles, I went back to my previous TKLDEV VM and did the following:

    1. cd products/tomcat
    2. make clean && make

    Both of these got stuck at exactly the same point.

    After getting to the same stuck state on both of these, I tried a fresh re-import of TKLDEV on VMware Fusion (8.0.2) instead. Nope... precisely the same behavior at precisely the same step.

    So finally I tried again on both, but instead of accepting the installation of security patches in step 2, I declined them. Viola! Both builds on fresh deployments succeeded. Um, OK... so why do patches requested during TKLDEV install result in Tomcat Standalone make to fail in this way? And since these images are, as I understand it, supposed to auto-patch regularly, won't these builds just suddenly start failing again anyway?

    Hopefully this helps narrow the cause, but at least I can manage to get clean builds... if I don't accept OS patches.

    Jeremy Davis's picture

    Jeremy Davis - Sun, 2016/01/31 - 11:04
    Great detective work. However I'm not sure that it really gets us much closer as to why yours wasn't working (with the sec updates) and mine is. Mine runs all the time and so has all the security updates installed! They auto install every night...

    To double check, I had a look at my cron-apt logs. The auto update checker last ran Sat Jan 30 21:27:01 AEDT 2016 (about 20 hours ago). And the last time it actually installed updates was Thu Jan 28 21:54:31 AEDT 2016 so it should pretty much be up to date as per the TKLDev VMs you did the auto updates on...

    Still, it's worthy of further investigation. I'll try downloading and installing and see if I can reproduce it. Thanks for providing clear instructions to make that possible! :)

    Also it doesn't make any sense to me what in the base TKLDev OS would be causing the Java CA certs to lock up the install. TKLDev doesn't even have any Java components, of anything else I would imagine could interfere with that.

    A build (on a clean, new TKLDev server) will download fresh packages (from the Debian repositories) for the build. Apt downloads are cached in TKLDev so in theory there is a slim chance that my build was using cached packages (that are good) and your was using newer versions (that are broken).

    But then that wouldn't explain why the TKLDev server that you didn't install updates on, would complete successfully. If the newer packages are broken, why did that one work? All very weird...

    Bill Taroli's picture

    Bill Taroli - Mon, 2016/02/01 - 18:01

    I seem to recall that Tomcat triggers OpenJDK to get installed (based on the scripts I read) and perhaps the CA cert setup is just part of that installation? Didn't track that down. I'm not totally up on how the builds go from make to make so far, but I'm wondering if there's a chance the CA cert creation being done once doesn't get triggered again on subsequent builds? If so, and that's the part that's interacting poorly with updated packages, it might explain how a working build env continues to. But that's pure conjecture at this point.

    I'll be doing some more testing with this same TKLDEV VM today, so I'll let you know how that goes when I try to make the Tomcat app and our custom one again. :)

    Funny part is that I don't even /want/ OpenJDK. We're installing Oracle's JDK 8 for our app. So if I could figure out how to block OpenJDK in the first place, that'd be awesome. :D

    Jeremy Davis's picture

    Jeremy Davis - Tue, 2016/02/02 - 10:19
    There should essentially be no difference between me doing a 'make clean && make" (in the relevant product directory) on my TKLDev server and you doing a 'make' on a clean TKLDev server (in the relevant product directory). Any difference should only be that mine will be a little quicker as it won't need to download all the packages (my will use the cached packages).

    WRT changing the files that are installed by default, you can tweak them by adjusting the plan. Just remove the ones you don't want. If you're not 100% sure and just want to comment them out, preceed the line with double slash (i.e. //).

    Bill Taroli's picture

    Bill Taroli - Wed, 2016/02/03 - 08:37

    Having made no other changes, so that all that's really happened in my TKLDEV is for regular updates applying, the make clean && make in same tomcat folder now stalls at the same ca-certifcates-java step.

    This seems to confirm that something in the package updates applied to TKLDEV is interacting poorly with the tomcat standalone app.

    For now, I think I'll be disabling the apt daily cronjob. This will get exhausting to keep importing a new TKLDEV each day I need to work on this. :(

    Bill Taroli's picture

    Bill Taroli - Tue, 2016/02/09 - 09:02

    I've had good luck with my builds since stopping automatic updates. But in the meantime my partner found a reference to precisely this issue as a kernel bug. It was originally reported as being a Docker problem but was ruled out...

    https://github.com/docker/docker/issues/18180#issuecomment-162866404

    Just FYI

    Jeremy Davis's picture

    Jeremy Davis - Mon, 2016/02/22 - 02:15
    It's really unfortunate that it appears that the regression is packaged with one of the security bug-fixes.

    I have reproduced your issue. So for building Tomcat based appliances we'll need to not update/downgrade the kernel. I've just downgraded a TKLDev server and that tested ok. 

    apt-get install linux-image-3.16.0-4-amd64=3.16.7-ckt20-1+deb8u2
reboot

    I also tested upgrading to the 4.x kernel in Jessie backports but that doesn't support AuFS so it totally breaks TKLDev.

    [update]: Lodged as a bug on the issue tracker: https://github.com/turnkeylinux/tracker/issues/576

    Jeremy Davis's picture

    Jeremy Davis - Fri, 2016/03/04 - 07:59
    FWIW this bug should now be resolved in the latest Jessie kernel release. I haven't tested yet but the new kernel was released today.

    Add new comment