Blog Tags: 
aws
ec2
cloud
hub
cloudtask
batch
parallel

TurnKey Hub: Not just adding random features

Alon Swartz - Fri, 2011/08/19 - 15:50 - 10 comments

I started writing a review for cloudtask as a comment on the announcement, but decided it would be better to address a topic that was raised by Jeremiah when we launched the Hub API:

I'm impressed with the way all the pieces of TurnKey just fit together like puzzle pieces. Even as new functionality is added it never seems like something gets patched onto the existing framework.  Instead, it just integrates smoothly with what's already there.

Liraz's reply:

While the new Hub features are coming out incrementally, they really are part of a bigger unifying vision. I don't like to hype up vaporware, but we have a lot of great ideas we're working on and we like to think we're just getting started. There's a big gap between what the Hub does right now and what we envision it doing in the future.

Also note that we're using the Hub internally to help develop and test TurnKey, so we're probably using it more than the typical user and run into its limits sooner. We're scratching our own itch, and the community benefits from that.

To recap, some of the features Liraz was referring to in his comment, and some that came after his comment are:

More features are coming, but this a great milestone to stop, reflect, and talk a little about a pain point we've had, and how the features we've released to this point are helping us solve it.

Building and maintaining exponentially growing library

TurnKey currently has 45 appliances, available in ISO, VMDK, OVF, EC2 S3 and EBS backed, Xen and Eucalyptus.

To sum up, thats 585 images we need to build and maintain. 

The (long overdue) TKL 11 part 2 will double the appliance library. Additionally we are working on supporting more build targets, as well as the new Tokyo EC2 region. Throw 64bit support into the equasion and the number of images to build and maintain will grow to about 3,300!

Automation and a glimpse into a pain point

We try to automate everything we do, but even executing and managing the stuff thats already automated needs automation when you reach scale.

To give you a little glimps into what I mean. When building the Amazon EC2 S3 backed images (in the past), I couldn't do it locally as my upstream kinda sucks are would take me about 2 weeks to build and upload.

So do it in the cloud, right? Correct. But building and uploading from an EC2 instance still takes a couple of days, mainly due to subsequent builds and uploading images built for different regions is quiet slow.

OK, so lets just spawn more servers in the cloud, in the different regions. Yep, this allows us to perform the relative region builds so upload is much faster, and they run in parallel.

But manually launching each instance from the Hub's web UI (pre- hub-api), updating my /etc/hosts file to keep track of the different instances (pre- hub-domains), logging into each instance and transfering the build infrastructure (pre- restore-on-launch), and finally starting the automated build process while making sure nothing breaks is painful to say the least.

It would also be much faster (and cost the same) if we could split the appliance builds amoungst more instances in each region, but try managing that manually and keep your hair - argh!

To throw some salt on an already bloody wound, remember we also have to build the other target formats (EBS, VMDK, OVF, etc...). Try do that without going bald.

Parallel batch execution with auto-launched cloud servers

This is where integrating the features mentioned above, and cloudtask comes into play.

Instead of pulling my hair out and waiting days, yesterday I threw a couple of tasks at cloudtask, which launched about 50 instances spanning the globe, our build infrastructure was automatically setup on each instance, and cloudtask started dishing out jobs to each of them.

Within about 1 hour, I received reports via email that all the builds completed successfully and were published. Pain point gone.

Now thats automation on steriods! And best of all, I get to keep my hair!
 

As mentioned above, we are just getting started. Lots more to come - it's getting exciting...

Comments

Jeremy Davis's picture

Jeremy Davis - Sat, 2011/08/20 - 00:02

Have a look at this thread or the SourceForge project itself. I haven't uploaded as much there as I'd like because life has got in the way of my plans and hopes for it, but it's a start.

Also I think as a matter of accessability and legacy support, 32 bit builds are important (many report that they reuse old hardware for server tasks). I think that while Ubuntu is available in 32 bit then TKL should support it OOTB.

Jeremy Davis's picture

Jeremy Davis - Sat, 2011/08/20 - 01:05

Nice roundup of features and progress Alon.

I know I have banged on a lot about it in the past but I still stand by my desire to see some sort of roadmap for TKL, even if it is a vague one without clear timelines. :)

Liraz Siri's picture

Liraz Siri - Wed, 2011/09/07 - 02:43
A roadmap could be a highly useful for coordination when a greater community of developers is involved working towards a common goal. We're working on that, we're just not there yet.

The thing about a roadmap is that it would set expectations, and from expectations not met disappointment and disillusionment could naturally follow. If there's one thing I regret is setting up too many expectations, not too few, but that's only because under present circumstances TurnKey development still depends way too much on manual labor by Alon and myself. That's the bottleneck which we've dropped almost everything else to work on removing. Unfortunately there are no off the shelf development chain that would be a good fit for TurnKey, so we're having to develop the pieces ourselves, one by one. That takes considerable time. The general direction is to move all of our development infrastructure into the cloud and then build interfaces that tuck away all of the complexity into something simple developers could use.

Liraz Siri's picture

Liraz Siri - Sun, 2011/10/23 - 17:42
There's a lot of ad-hoc infrastructure behind the scenes powering TurnKey. Unfortunately it wasn't designed to be collaborative. It wasn't much designed at all. It just sort of grew organically to accommodate our needs. The build infrastructure has dozens of components. Documenting all of that to the point where outside people could step in and help isn't something we can do on a whim. We're also not confident that it would really help. The barriers to getting involved would be pretty high. Outside developers wouldn't be able to just jump in before they understood how everything fits together.

Rather than invest in explaining the existing infrastructure, we decided it would be better in the long term to redesign and simplify it, then move the whole shebang to the cloud and build super simple interfaces to it that almost anyone could use. But that's a bit of an ambitious undertaking, especially with everything else we have going on, so it's taking a bit more time than planned.

PS: Sorry for the late reply.

L. Arnold's picture

L. Arnold - Wed, 2011/09/07 - 19:00

Keep up the good work...  I do encourage you to keep 32bit support going for a while, not a small part because my VMServers are 32 bit servers...

I will also encourage more "support" if you call it that for upgrading the aps themselves.  Joomla and Magento both have upgrade paths but I have not found the process to be as easy as I would like.

Magento is now at ver. 1.6, not 1.4.2 as the TKL is built.  Joomla is now 1.7.  I believe that Tracks has a new upgrade.  In liu of a full "switchbox" approach, at least some simple docs as to how to manually upgrade the different systems would be in order...

Perhaps this is about the Hub.  Probably more germain to ecosystem itself.

thanks again.

Liraz Siri's picture

Liraz Siri - Sun, 2011/10/23 - 17:48
We don't have the resources to build safe upgrade paths to all the applications in the TurnKey library. Not even Ubuntu have the resources to do that. Both of us rely on Debian to do most of the heavy lifting. And Debian have decided to prioritize stability at the expense of having the latest package versions. Which kind of makes sense under normal circumstances. It's more important for most production sites that a package upgrade doesn't break their website than for their CMS to have the latest bells and whistles.

If you really want the latest features at the expense of stability and security, I recommend installing the applications you want on top of TurnKey LAMP. But then remember you have to keep a close eye on security issues for the applications you've installed as you'll need to upgrade them by hand.

PS: sorry for the late reply.

Jeremy Davis's picture

Jeremy Davis - Sat, 2012/01/28 - 11:31

Let's hope we get a few committed community members who are willing to do that! :)

We may even be able to have 'stable' versions (which contain standard repo packages with the auto backported updates) and 'testing' versions which have latest upstream software. I would suggest that these 'testing' versions still use mostly standard repo packages, but with the main frontend software being at the latest stable.

Pages

Add new comment