You are here
For those of you about to read this, let me preface it by saying a few things:
-
I hope no-one takes offense to anything I say, I apologize in advance if that is not the case
-
I am a fan of the work being done by TKL, I use several of the appliances for personal use and will continue to do so
-
What I say here are personal opinions regarding what I consider to be the weaknesses of TKL as they exist today. There are, obviously, benefits as well but this post is focused on areas of opportunity.
Community
I've been using TKL for a few years now, 80% of the appliances I have deployed were/are on my personal vSphere instance. The remaining 20% are spread between physical hardware and non-VMware hypervisors. I have built a number of TKLpatches and distributed them here. I do consider myself a semi-active member of the community and I try to improve other users experiences. To that end, I consider the community (the forums) to be one of the best aspects of TKL. Jeremy, in particular, tries to respond to everything and should get kudos for doing so (though I do not think he is a TKL employee).
I don't see a lot of interaction from the 'core devs' here and think their absence is a large detriment to the community. I am sure they have other things to do but see many posts where their input/response would be welcome. I would very much welcome their participation here.
64-bit support
People have been asking for some very specific features for quite a while now. In particular 64-bit support is likely the most requested item and people were really looking forward to it being part of the recent TKL 12.0 releases. Unfortunately, 64-bit was not delivered with this recent release; in this day and age, I consider this a huge hole in the appliance portfolio. This leads me to a somewhat related point, I'm not sure what the TKL model is. Is it open source? Is it some closed source and some open source? If it were all open source, maybe the community could put forth some effort to help port/support 64-bit appliances. As it stands today, I consider the process/tools used to create the official TKL releases to be a black box that I cannot help with. Can we make that more public so others can help deliver features that are being asked for?
Quality and Testing
I have certainly not tried every one of the 100 appliances, but I have tried several, and think they're missing the mark slightly. Take for example, the ejabberd appliance... I installed this in less that 5 minutes and went to create an account.. only to find that I can't create and account due to a configuration issue with either ejabberd or speeqe. There were several forum posts, from 2+ years ago pointing out the same problem yet it was not addressed in this latest release. Sure, I can figure it out. I'm a developer. I've been using Linux for almost 20 years. But, why should I have to? I have had similar experiences with other appliances, where features just don't work like they should. In short, I am not sure these appliances are really tested before being released and don't think that community developed fixes/workaroduns are incorporated into future versions.
Preference of distribution upstream packages instead of up-to date software
If you look at the dokuwiki appliance, it includes dokuwiki version 0.0.20091225c-10+squeeze2. There have been 4 official releases of dokuwiki after that release. I know it's easiest to include the versions of software packaged with Debian or Ubuntu but I would prefer a much more recent version. One could argue that we could use TKLBAM to migrate from the old to the new but often times, the software packages themselves do not offer a clean upgrade path, which leaves us in a situation of starting from scratch or using old software. It is more work to package newer versions, I grant that. Regardless, I want that newer version; without it I have been reverting to taking a clean OS install and putting the packages on myself. Similar to quality and testing, I don't think the dokuwiki appliance is the only one with outdated software.
Breadth of TKL Library
I consider the breadth of the TKL library to be too large. I, personally, do not see the need for 100+ appliances and get the idea that having a large library is more valued than a high quality one. I would rather see 20 appliances that work very well with updated software than 100 that may or may not work properly. I can't see how two people can support 100 different appliances and carry forward much knowledge regarding what the appliances should do.
In short, I think there is a lot of potential with TKL and I am happy that it exists. I do think there are some shortcomings that need to be addressed, possibly with some ability for expanded community involvement and/or a pairing down on the number of supported appliances. I'd be happy to discuss this with anyone who cares to.
TurnKey from a core-dev's viewpoint
I have to admit I agree with pretty much with all of the points you've raised. We haven't been active in the community. Certainly not as active as we'd like to be. The community suffers from this, and if it weren't for Jeremy (who is a volunteer) I'm not even sure there would be a community.
Let's see - lack of 64-bit support - inexcusable in this day and age. Agreed.
TurnKey development still a black box. Still true, and very much to our detriment. We're having to do all the heavy lifting ourselves.
The root issue is behind the scenes we're spread ridiculously thin. TurnKey doesn't have employees. At least not yet. It's an open source project (at least in spirit) that could definitely benefit from having more people involved yet doesn't yet have the infrastructure and processes to support that.
When we broke TurnKey free from my former employer (a security startup) we had to make a tough decision regarding where we wanted to take the project.
We could have made things happen a whole faster if we had a bigger team. But that requires funding and we've decided, perhaps naively, not to seek outside investment. Then you have more money and resources to get stuff done, but you also have to maximize "shareholder value", which means making TurnKey more of a commercial project - beholden to financial interests - a concept which we have so far resisted.
So on the upside we don't have guys in suits threatening to bring in adult supervision because we're not making them enough of a return on their investment. On the downside, we have to make hard choices with regards to where we spend what precious time and energy we have and recently that's meant prioritizing development over being involved in the community.
64-bit and a truly open source collaborative development infrastructure so we can get more people involved in the open source model - these are our top priorities along with developing more open source friendly commercial offerings/revenue models so that we have more resources to work with and can get stuff done faster.
Currently we're in this crazy jack of all trades role where we're trying to develop all this new stuff we have on our roadmap while keeping everything we've already put in place from breaking at the seams. Just maintaining a project the size of TurnKey is quite a bit of work: over a hundred products, thousands of images in many formats, a complex Drupal website, the Hub. Let's just say we're spread way too thin. Something has to give.
One of the most negative ways this manifests is that we "core devs" tend to fall into a black hole during development cycles. We're getting a lot of stuff done behind the event horizon, it just isn't visible yet to the rest of the world.
Our main challenge is breaking free of this model, making the project transparent and open source not only in spirit but in practice as well. We think that's the key to unleashing TurnKey's potential.
Hopefully, once we do that we'll also have more time and opportunity to communicate with the community. I'd really enjoy that. Alright, back to that script I was working on...
Thanks for the reply
Liraz,
Thanks for the reply. I know that you're spread really thin and completely understand why things are the way they are. I really appreciate the work that has been done and where TKL has gotten to. Whenever I need a machine for something, the first distro I think about is TKL. Only when I have ruled TKL out do I turn to Ubuntu, OpenSuSE, RedHat, or [god forbid] Windows. I think that's a great testament to what has been delivered.
I really hope TKL can break "free out" of the current model, as i think it's crucial to success. To that end, I am going to try and be around more often... posting on the forums, building patches, writing tutorials, or whatever it takes. if I'm not around and you need me, contact me and I'll jump in anywhere to tackle anything. If I'm around too much, tell me to go away :)
Eric
I too agree with most of what you wrote
And I certainly won't be telling you to go away, no matter how active or involved you are. I may not always agree with what you do or say but that's life and the reality of being part of any community (and experience suggests that I mostly will anyway - that it if I have any relevant knowledge or experience :D). And we're all adults here so working through disagreements and/or agreeing to disagree shouldn't be a drama.
Personally I really appreciate your (and anyone/everyone else's) input into the community. I've invested much into the TKL community and I don't intend to go anywhere anytime soon. TKL is my home now! But having said that I must admit that I sometimes feel relieved when someone beats me in responding to a forum thread with a useful post. I often learn something too.
There are other things I would like to do around here but often the time I set aside for TKL gets chewed up on the forums (and then some...) and some of the other (dev type) stuff I'd like to do I just never have time for. Don't get me wrong, I certainly don't grudge TKL for the time and energy my involvement takes. It hasn't been a one way street at all. I have been heartily acknowledged (in word and action); especially by Alon and Liraz but also by yourself and many, many others. And I have learned so so much. When I stumbled across TKL ~4-5 yrs ago I was a complete Linux newb and although I still don't consider myself an expert, many have disagreed. And I'm certainly no Linux newb anymore that's for sure...
Some thoughts
I read your post yesterday Eric and have been pondering on it overnight and more this morning...
Being a very solutions focussed sort of a guy I have some thoughts...
TKL 'testing' repo?
One thing that has occurred to me (more than once over the years) is that one way to go could be to leverage the fantasic Linux/Debian package management system!?
I recall way back when that the TKL guys started off packaging their own debs (at least they packaged Joomla and Webmin too). Obviously that has somewhat fallen by the wayside - with the exception of Webmin. I am not sure of the context of that decision, perhaps too much effort!? Or just another job for the TKL dynamic duo to add to their never ending list!? Regardless I think that with the prospect of community involvement that could be harnessed perhaps this needs a rethink? For example I know that Rik (Goldman) has in the past suggested/offered that he and his students would potentially be interested in maintaining a package or 2 (given some guidance around packaging) and if I knew more about packaging debs then I'd be keen on that too. Perhaps Eric, Adrian and others would be interested in getting involved there too!?
If the guts of packages were hosted somewhere like GitHub (or even a TKL GitLab instance) where people could easily contribute to the packages. And a couple of trusted TKL community members could build and move these into a TKL 'experimental' and/or 'testing' repo (or probably better still a TKL 'squeeze-experimental' and/or 'squeeze-testing' repo) then once thoroughly tested and demonstrated as stable - the devs could move these into the stable TKL repo. The added bonus of going this way may also ease issues around migrating from old appliances to new ones (using TKLBAM)?
Perhaps some of these updated packages (and their dependacies if needed) could just be simply drawn straight from the Debian testing repo?
And now that TKL is Debian based maybe TKL community developed debs could even be pushed back up into the Debian repos!? Having a 'TKL team' creating and potentially being able to give back (up) to the wider Debian community seems like a really cool possibility to me. I guess that needs more thought though as I am well aware supporting a Debian package is a lot of work. Backporting security fixes could grow to be a pain if the general thrust of the TKL packages is to get newer versions of software working on TKL.
Regardless, I think that even just having a repo with updated packages (and/or debs that don't exist elsewhere) may draw in new community members and developers. We may even get some upstream devs wanting to work with us around packages for their software!? Obviously there may be some users that just add the TKL repos to get app XYZ installed easily on their Debian/Ubuntu server, but I'm sure there'd be others that may be interested in getting more deeply involved.
It also strikes me as an easy way to get the TKL community engaged and allow TKL to be a little more responsive to the community and share some of the dev's load without requiring too much extra effort or initial setup.
Or maybe GitHub (or similar)?
I guess another possibility is using GitHub (or similar) itself as an update mechanism (and leveraging TKLPatch). I have noticed that many upstream projects now use that for developing and publishing their software (with a stable git branch as well as a testing one). Perhaps that is something that we could do already Eric?! By leveraging GitHub and TKLPatch maybe we could make a concerted effort to make some appliances more easily updatable? Perhaps just pick a few to start with and see how we go? I guess it is something that I could start myself (and it has crossed my mind) but I am loathe to go too far off in my own direction without having some others on board.
More generally...
Similar to the vision of Alon and Liraz; in my mind getting community members to 'own' appliances or the software building blocks of appliances (as maintainers) would reduce their load and help build the community, with the added benefit of TKL being more up-to-date and useful for people.
I know that the core devs have plans for opening up their build infrastructure, but perhaps instead of needing to wait for the devs to create the tools to do this, we could use the tools that already exist? I suspect that their vison for opening up the 'black box' is still a worthy aim and would have benefits, but maybe the more immediate need for this could be mitigated in the meantime?
IMO the bottom line is that it is the core devs' responsiveness (probably more correctly speed of reponsiveness) to community input is the major bottleneck for community growth. Without wanting to be too harsh on Alon and Liraz, there have been a few passionate community members that have fallen by the wayside because of (in my perception) lack of (timely) results from their hard work. Adrian is one that immediately springs to mind. I know that he is still marginally involved and I am sure that there are other factors as well (we all have lives and shifting priorities outside of TKL). And I also know that there is a lot of context to the current TKL situation, but still it seems a shame to see people frustrated (again my perception) to the point of moving their energies elsewhere.
To me, going some path like this would also make TKL stay more relevant for users. The appliances are great when they first come out, but over time many become less and less relevant. Many appliances that rely on Debian repo packages are quite acceptable IMO (such as Core, LAMP, MySQL etc) but the ones that are built using upstream software do not have newb friendly upgrade/update paths and as Eric pointed out, many of the Debian repo packages are seriously out of date. As the software that is included dates, the appliances become less and less useful for newbs and devs alike (defeating the whole point of TKL as I understand it). This is especially the case with appliances that include upstream software. Not only do they become dated, but they become potential security risks in the hands of newbs who don't fully understand the consequences of the (quite reasonable IMO) choices made by devs. It doesn't matter how many times I post about it or how much I write in the wiki, reality is human nature is going to mean that many users are just going to use the appliances without researching and/or really understanding the implications.
Take for example the ownCloud appliance - great appliance and I'm happy to see it there (especially as I developed the initial patch). But already the included software is somewhat outdated seeing as it is under quite heavy development. OTTOMH the lastest version of ownCloud now includes file versioning which AFAIK the TKL version does not. It is all well and good to say "check upstream to see how to upgrade and please report back your findings for the community so thers may follow in your footsteps" (as I often find myself saying) but wouldn't it be great if I could instead say "please enable the TKL testing repo and run apt-get update && apt-get install xyz" (the meta-package that depends on the latest version of app xyz from the repo) or run these git commands!?
Finally...
What do you guys think??
On package management based upgrade paths
As usual there are some major challenges involved and a great deal of labor resources would be required to maintain package management based upgrade paths for even a tiny subset of the appliances in the TurnKey library. That's pretty much the reason those packages are not in Debian - and Debian has thousands of package maintainers on board!
My personal view on this is that the only really credible way to pull this off would be to get the application authors involved and convince them it's in their best interests to play by the Debian playbook -- backporting security fixes for stable versions, and providing automatic upgrade paths for new versions.
Financial incentives would be the surest way of doing that and we have some ideas on that but I think TurnKey still needs to have a lot more users for us to be able to present a compelling argument for them to do this.
No worries!
I'd be very interested to hear more details on your thoughts when you have time.
In the meantime, I know that you guys want to do things the 'proper' Debian way (ie backport security updates to packages) but why does it have to be that way? And would that actually address all the issues? It would definately mitigate them to a degree, but as you guys are well aware sometime packages in a freshly released Debian stable are already 6mths+ old. Some apps under heavy development move so far in 6mths. Having newer packages (with back ported security fixes) only solves the issue in the short term - the other part of the issue remains. A lot of the time it seems like users actually want later versions of software, not just security fixes.
And like you say backporting security fixes takes a lot of work. I would imagine that for many devs of rapidly developing software even with some enticement that is not something they'd be interested in taking on. And even if we could convince them to do it, wouldn't they be looking more towards Debian than a smaller distro like TKL (not that there's anything wrong with TKL obviously)? And if they were to do that wouldn't we be back where we started!? Albeit with a Debian package for something that may not have existed previously...
So my thought is why not backport the latest versions of some software to the Debian base? Then the updated package includes not just the security fix but also a newer version of the software. As you would be aware there are many PPAs on LaunchPad that do exactly that and it seems to work ok in my experience. As you would also be aware I am an avid user of Bodhi Linux and that is the theory behind their whole distro ... A stable Ubuntu LTS core with a myriad of backported 'latest' packages in their own repo.
Obviously a Server distro is a completely different beast to a Desktop one but I have had no issues with any of the custom Bodhi packages - beyond a few E17 bugs that generally get fixed pretty quick (and I suspect that they are upstream bugs rather than ones specifically introduced by Bodhi). In many respects I would suggest that a distro of purpose built (and used) appliances that rely on the Debian repos by default but with a sprinkling of updated apps would be more stable than a desktop one running a huge different array of software.
That would be even moreso the case if appliances that had software in the Debian repos used that by default and end users needed to enable the TKL 'extras' repo to get access to these updated packages. Also as I said a TKL 'testing' repo could hopefully pickup most of the bugs before they even made it into the other repo.
Why Debian does things the Debian way
Debian don't backport security fixes because they're trying to make life difficult for package maintainers. They do that because it's actually easier to backport a security fix to an older version than to upgrade the package version and still make sure it doesn't break anything.
Of course, Debian also do actual package version upgrades. Otherwise Debian users would still be using the same package versions from their first Debian installation. But they only do package version upgrades for new Debian releases and that involves a complex testing process that can take months - hence the 6 month old package in the stable repository.
Some Debian users actually do continually update their packages to the latest versions. They're running Debian unstable - sid, the kid who will break your toys. Other distributions that don't have the Debian package policies are actually much more like Debian unstable than Debian in terms of stability, even if they don't call themselves that.
Years ago, I used to run Gentoo on my workstation. Gentoo doesn't have releases. It doesn't have binary packages. It builds everything from source. You're continually getting the latest packages of everything from upstream every time you update.
I too liked the idea of always being on the cutting edge and being able to super-customize my system at the package compilation level to fit my needs. Initially it wasn't so bad and I didn't understand why everyone wasn't doing this. Usually things worked out ok. But every so often they didn't, something would break and I found myself spending increasingly larger chunks of my time figuring out why. The more time passed, the more packages had major upgrades, the more things broke.
Eventually I realized that while I was learning a lot about Linux internals and troubleshooting, the time I was spending wrestling with my operating system was eating into time I had to actually get stuff done.
I initially reacted by reducing the frequency of whole-system updates, but then when I actually did need a new package, which depended on another new package, I would end up having to do a massive system upgrade whether I liked it or not.
Of course, updating a lot of components at the same time increased the likelyhood that things would break and I finally gave up on Gentoo in frustration one day when my system basically melted down during a small update that turned into a big update that went terribly wrong.
The timing couldn't have been worse, I needed to get something done a tight schedule and I no longer had a functioning workstation and very little patience to spend hours trying to figure out what was wrong.
Mind you, this was my experience at the other extreme for a desktop system. Server systems are much more sensitive to changes:
For example, this site runs on the Drupal CMS. We're using over 60 extension modules. We're currently only doing minor upgrades where required for security fixes and such and every upgrade is still a potential nightmare. It's like playing Russian roulette. Sometimes you pull the trigger and it only goes click. Other times it's going to blow your brains out.
So I have to usually stage upgrades on a production system, test and fix (or workaround) the inevitable bugs, and then reproduce that on the production system.
It can be a pain even for minor upgrades and major upgrades are a much bigger undertaking. It took several weeks to upgrade from Drupal 5 to Drupal 6. Extension modules that didn't have Drupal 6 versions had to be swapped out. Certain Drupal 5-isms had been deprecated and we had to learn how to do them the official Drupal 6 way. Customizations had to be rewritten for the new API.
It's very easy for a user to say I want newer package versions that I can just upgrade to automatically or semi-automatically without understanding the implications and why fulfilling their seemingly simple wish requires not just more work but different tradeoffs.
In other words, even if we had infinite resources you'd often have to optimize the user experience for one audience at the expense of another audience.
There's a reason a distribution with the stability of Debian and the up-to-dateness of Gentoo doesn't exist and will never exist.
TurnKey simply aims to provide a better starting point than the default empty server on top of which you customize everything. If users want new versions from source code and an automatic upgrade path doesn't exist then they have to do with TurnKey what they would have to do with any other distribution - install a newer package from source (or from a PPA), and deal with any problems that creates. TurnKey doesn't make this any more difficult than any other distribution and I would argue that having an older version of the application working as the default starting point usually makes things a whole lot easier.
Thanks Liraz
I really appreciate you taking the time to post that extensive response. You raise a lot of points that I hadn't considered. And more than anything it reminds me that my experience as a somewhat skilled hobbiest, mostly on desktop Linux; with very little 'real world' server experience is perhaps not the greatest vantage point to see the bigger picture. It's good to remember that just because I see things a certain way that doesn't make it the way they are...
It also reminds me that beyond testing (playing) and my own personal use, the only servers that I manage that are 'in production' I have not updated for a long time! I'm still running Proxmox 1.9 at work because it still works as I need it to and I don't have time to deal with the issues it would cause if something were to break during upgrade. And the Win Server is even more out of date (for the same reasons and a few extra as well).
It is also a good reminder to me that TKL isn't (and never has been) about trying to be everything to everyone. Your reminder that it has always been meant a good starting point, not neccessarily the end point is good grounding.
So bottom line is that I get you. And in some respects I agree in as much as core components (such as Apache, MySQL, etc) that the 'Debian way' is sound. I certainly don't think that replacing any of that core software with bleeding edge is going to win any TKL friends (as I recall you saying once before they don't call it 'bleeding' edge for nothing...)
I still think that there has to be a better way for some of the non-core software, espcially the ones where there is no Debain packages available in the repos. I guess what that is is still somewhat elusive. Perhaps for some appliances, using the upstream git 'stable' stream may be the way to go (software such as GitLab, Canvas and ownCloud spring to mind). And perhaps that's something I can do; work out and document how to align those appliances back to their GitHub source so they can be eaily updated using git.
So perhaps leveraging upstream mechanisms for updates is more the way to go? Many upstream apps now have inbuilt means to update themselves, or third party means; such as Drush with Drupal (as an aside I really like the Drupal 7 appliance in that regard - Drush is way cool!)
I guess the other thing is that there is nothing stopping me creating my own repo (PPA?) so I could test the package approach that I have suggested with some software too if I was keen enough. And/or leveraging TKLPatch to upgrade appliances is still a valid option I beleive. My only feature request in that regard would be that it would be great to be able to launch appliances from the Hub and have them apply a patch and then apply TKLBAM backup/migration data all configured from the Hub interface prior to launch.
But more than anything it is good to be reminded that whilst end users and TKL enthusiasts alike may not always like or agree with the way things are done, you don't generally do them just for the hell of it; there is usually a reason whether it is obvious or not.
An additional use case to consider
Let me preface this by saying that I second tssgery's post in full, including the fanboyism. ;)
While I have only registered here a few days ago, I've been using TKL appliances for a while and regularly encountered the forums in search for help.
That being said, I think the commercial/business sector is being a bit neglected by this project: The company I work for is primarily a Windows-company, and has no desire to hire Linux admins; Turnkey allows us to use Linux machines because my colleagues don't have to deal with SSH, package management, etc., because everything is either automated or abstracted away.
In other words, Turnkey appliances are really the perfect "distro" for us, and I'd happily deploy them for everything we need.
A simple online/offline setting for TKLBAM would help a lot here.
I realize that these are mostly points that have already been said, but I think it's important to emphasize that they affect business users differently than "normal" ones.
Yes, as a normal end user my colleague could've gone in, hacked the appliance and manually upgraded OTRS. As a business user, he can't justify his use of time and the added future administrative workload for that. It either works as needed, or it's not suitable.
I am pointing this out because I think putting more focus on business users and emphasizing the advantages of Turnkey appliances compared to custom Linux server setups is your easiest way to alleviating some of the problems acknowledged in this thread.
I am not saying you should "sell out", go the way of Red Hat or anything like that. I am just saying that the project is already very attractive to non-Linux companies, and if you emphasized these existing strengths and mitigated some of the weaknesses, you could probably reach more business users, and, in turn, gain advantages from that.
To put it bluntly: Businesses are more likely to invest money than private people. More money would help this project a lot. So while I'm sure we'd all be happy if TKL turned into an open source wonderland carried by private effort, the reality is that the vast majority of successful open source projects just plain don't work that way. The Linux kernel is largely developed by commercial developers, Red Hat (now Enterprise) Linux is largely commercialized, Ubuntu has a commercial entity and a millionaire behind it, Wordpress has a company behind it, etc., etc.
I'm not suggesting to turn TKL into a business per se. I'm just suggesting you change your "marketing" and focus a tiny bit to encourage other businesses to establish a reason to support you.
Just my two cents, anyway.
Carl, I agree
Carl, I agree with where you're going around business users. I work for a software company and am responsible for a few different projects, all which deliver as virtual appliances so I'm pretty familiar with what business users would want from appliances. They want up to date software that works, is secure, is easily updateable, easy to debug in case of failure, and easy restorable in case of failure. TKL falls pretty short in this area. I took would love TKLBAM to be able to use storage services other than Amazon in a reliable, easy to use fashion (like my 8TB NAS system sitting 3 feet from my vSphere servers). I know I can go in and modify a few configuration files to enable this, but supporting one 'user controlled' target would be nice.
I, too, am not sure that TKL wants to turn into a full fledged business but I agree with your points that if TKL were to serve business needs, it would greatly benefit them. Consultants would be tempted to utilize and support TKL, businesses might be willing to pay for features and support which could extend the ecosystem, etc.
Regarding 64-bit support, it's been stated here many times but without it... companies won't take TKL seriosuly. As a hobbiest playing with TKL, 32 bit is just 'ok'.
Jeremy, thanks for the additional comments. If this were my business or project, I would try to do something similar to what you suggest. Take a few of the more popular appliances (core, lamp, and wordpress as example) and:
Only when we can do this with a few of the core appliances, would I tackle additional ones. The ones that have gone through this process would be considered supported at a different level than ones that have not. Maybe mark these in some fashion as special. I know this seems like a pain and a lot of work, but when I take an appliance, I want to know that it works and that it'll be kept up to date.
When I mentioned above that I think 100+ appliances is too many, I really meant it. Let's do 5 (arbitrary but small number) appliances really well first.
The long comment that turned into a blog post
For great justice: all your criticism are belong to us!
Add new comment