You are here
MacNB - Tue, 2012/02/28 - 17:16
Hi,
I just came across Turnkey and looks very interesting for what I wish to do.
That is to create a light weight NAS for my new HP Microserver (N40L)
I have tried FreeNAS and OpenMediaVault.
FreeNAS is BSD based -- I prefer Linux as I am a NB and got a better chance to learn *ix with Linux than BSD. Boots from USB Flash Drive but runs in RAM.
OpenMediaVault (OMV) is like FreeNAS but based on Debian.
It is very good. Had it up and running in 10 minutes.
The issue with is that when installed on a USB Flash drive, it is writing to the flash drive all the time. There feedback from users stating that this kills the flash after a few months. They have no plans to run it from Ramdisk after booting.
As I only have 5 AHCI SATA ports, I want to use all of them for HDD's so looking for an embedded Linux server for CIFS, NFS, FTP and SSH booting from USB Flash but running from ram with no constant writes to the USB Flash (except to save configs).
Can the Turnkey solution be installed to USB Flash, boot off it but run from Ram ?
Thanks,
Mac
Forum:
Tags:
A couple of ideas
I can think of 2 ways to skin this cat... (And a couple of other thoughts too).
The first (and probably the most straightforward; would be to install to the USB stick (as you would a normal install to HDD) but then tweak it a bit to reduce writes to the USB. Things like not having a swap file (virtual memory in Win speak), using a non-journalling files sytem (such as ext2) and mounting directories which are often written to such as /tmp and /var/log as tmpfs (ie they stay in RAM). You could have a look at the 'TKL optimised for SSD' blueprint for some ideas there (SSDs are obviously not USBs but many of the issues are the same). Some won't be relevant and I have some misgivings about log files being destroyed on reboot (makes it really hard to diagnose issues if the logs dissapear after every reboot) but if it is simply a home fileserver then you are perhaps willing to go that path? A workaround wold be to write those files to HDD (ie run most of the OS from USB, but any dirs that are written to lots mounted from dirs on an HDD).
My second idea would be to customise and rebuild a TKL ISO (using TKLPatch?) to include your desired settings. Then use something like UNetbootin to 'burn' the ISO to USB. That way you could run it as a LiveUSB and the whole OS would reside in RAM. The downside of this is that it would be more work initially and any customisations made post initial ISO config would be lost on reboot. You could again work around that by having some of the specific config folders mounted from one of the HDDs. This would allow you to run your OS totally from RAM, but also maintain some settings and/or config and/or logs between boots.
I guess a third idea is to just use a cheap USB stick. I have seen 4GB sticks (which would be more adequite for a TKL install) for less than $20, so who cares if it only lasts a year or 2? I have a persistent USB install (which I use for troubleshooting, repairs and runnign Linux on my desktop at work) although I don't use for extended periods of time (as I'm guessing you will be) but it has had heavy use and is now 3 or 4 years old and still going strong.
My fourth and final thought (for now anyway), is that in this day and age of multi-terabyte harddrives a few hundred MB is not a lot to give up for a OS install...
Sound advice
Hi Jeremy,
Many thanks for your time and the suggested ideas.
Many thanks for the ideas.
Mac
Further musings...
AFAIK with Linux this is really quite easy (easy being a relative term - but in comparison to Windows) as Linux has built in knowledge of a 'RAMdisk' type set up called tmpfs (= temporary filesystem ie stays in RAM). Generally the /tmp folder is configured to utilise this OOTB. So it would just be a case of editing your fstab file (the file that contains the config for what is mounted where on boot) to set the desired directories to be mounted as tmpfs rather than from HDD. I haven't personally tested this but have read about it.
For sure! :) But unless you use some sort of OOTB solution (which seems unlikely unless you go with option #3) you are going to need to learn this stuff to bend Linux to your will regardless. So why not do it right from the start?
You don't necessarily need to use TKLPatch, but in my mind it's your best bet of the first 2 options (assuming you go with TKL). The usage scenario I am suggesting is not the intended usage of TKLPatch but would work well IMO (I have used it in that manner myself). If you wish to run a totally live environment, a TKLPatch is the only way to go IMO. The initial development will require a learning curve, but it will be easy to add tweaks and/or update (inc underlaying OS) once created. The best part is that it won't rely on you remembering what you did last time (because it's there in the patch)! If you choose to go down this path I am more than happy to give you some pointers.
Easiest in the first instance but over time will require more maintenance. And reality is that unless you have an automated backup (such as TKLBAM) you may well find your backup regime getting streched out (we all get slack over time...) - to the point that it results in a headache when you inevitably have to replace the USB.
Not neccessarily. Your data has to go somewhere and I would assume that the reason they take over the whole drive is because they are intended to store the data internally (ie on the same HDD as the OS). So whilst the OS partition takes the whole drive there is still tons of room in there for data (the OS will still take the same space whether it is a small partition or a large one...). I don't know about FreeNAS, but seeing as OMV is Debian based you could use LVM (sort of like software RAID, but not...) to include the other HDDs. So in effect you could have OMV running on an LVM volume of 10TB (assuming 5x2TB HDDs). OS & storage all together... You could do the same with TKL (and added advantage of this is that LVM is configured OOTB during install) so you'd just need to add the other 4 HDDs to the main TKL volume.
OTOH If you prefer to keep data separate to OS even if FreeNAS/OMV uses the whole HDD initially, there would be nothing stopping you from resizing the partition afterwards ('taking over' the whole HDD means just one big partition). Partimage is a great Linux based LiveCD/USB partition moving/resizing tool that you could use to shrink the partition to the desired size afterwards - don't forget to leave a bit of room though!
TBH not an option I'd prefer. IMO if you wish to keep the OS completely separate to storage, then one of the other options (using a USB stick would be better). My main reason for thinking this is that the thing that kills HDDs quickest is heat. So the more space you have in your case the better the airflow, thus lower the heat.. And with 5 HDDs already in there spinning away, another HDD plus the USB->SATA adaptor will create more heat and reduce airflow (thus creating more heat buildup...)
Time to play
OK Jeremy - it's a challenge so I downloaded the TKL File Server build last night (both the VM & the ISO). Installed VirtualBox and followed the tutorial to generate the VM with the virtual HDD image of TKL.
Not bad for 20 minutes effort (had a a bit of an issue with setting a static IP address for the server which it would not accept so I gave up for the time being and let it get it via DHCP).
I was surprised that a pre-built TKL File Server does not include NFS. I downloaded and added it but not had a chance to play with it yet - it was getting way past my bedtime :). I particularly need NFS as it seems to handle streaming HD movies better than CIFS with my XBMC HTPC.
So now I know how to install a VM and TKL. If I get some pointers and inspiration :), I'll try TKL Patch. I read and read about Patch but it seems quite involved for a noobee.
Did not quite understand how to develop it on a running VM.
My requirements are:
So, where do I start customising the stock TKL Fileserver ?
Mac
PS. I did also try installing Ubunto Server 11.10 on the USB HDD. Took much longer to install and configure all the bits (also used Webmin but sometimes it was easier to use the SSH) but it's up and running.
So which tack are you taking?
Are you planning on creating a LiveUSB (ie whole non persistent OS run from RAM)? Or a customised install (Install system to USB and tweak to minimise USB usage)?
Perhaps it's useful to you if I give some explation of TKLPatch first. The term 'TKLPatch' is used in 2 contexts: The .deb package of scripts (installable with 'apt-get install tklpatch') that the TKL devs created to allow (relativly) easy customisation of TKL appliances. And the TKLPatch that end users create, which can be applied to a TKL ISO to create a customised appliance (using the command 'tklpatch appliance.iso patch.tar.gz')
TKLPatch (the package/scripts) may seem involved to start with but the theory is quite simple. It is basically a set of scripts which unpack the ISO, make whatever changes you wish (as contained within the user created TKLPatch - eg install/remove apps, tweak config settings, etc) then repack back into a (modified) ISO.
The user created TKLPatch itself is a tar.gz archive which consists of an executable script (the conf file) and overlay files (contained within the overlay folder). So for example if you want NFS installed on your custom ISO then you would include the commands to install it in your patch conf file. If you want NFS config customised in a certain way then you can include a pre-configured NFS config file in the overlay folder. The only thing to keep in mind is that by default the TKLPatch overlay is applied prior to the TKLPatch conf being executed. Generally this is not an issue for apps installed with apt-get as by default they will respect an existing customised config file but for some things your custom file may be overwritten. There are work arounds to resolve this but I won't bore you with the details here... We can deal with that if/when we need to.
Anyway if you plan to actually install to your USB and tweak it to minimise USB usage then creating a TKLPatch is totally optional. However I would still encourage you to consider using it for 2 reasons. Firstly it will allow you to easily share your work with others (by way of the TKLPatch you create). And secondly it will allow you to create an updated installation ISO easily when it comes time to update TKL to v12.x (you may need to tweak your patch a little, but fundamentally it should remain the same).
If you go the LiveUSB route then you will need to make a TKLPatch. Actually that's not strictly true because you could use the component TKLPatch scripts to manaully create your customised LiveISO, but it will make updating your ISO in future much more difficult (because you will need to go through the whole process again and the likelihood of errors creeping in is much higher).
So if you are going LiveUSB route, then this thread may be of asistance. Obviously the purpose is different, but the theory of getting the system to boot straight into live mode is the same.
Whichever way you go, to create your TKLPatch you will need to setup your VM as you desire your end system to be. But make sure to clearly document every step along the way. All config steps will need to be either done from the command line (so you can transfer them into your TKLPatch conf file) or be written into config files (which you can copy into your TKLPatch overlay folder). It's a pity that I didn't follow my own advice when I created the LiveCD (forum thread linked to above) because then I could've shared my exact steps with you and that would've given you a head start. Anyway...
If you go the LiveUSB route then you will have to preconfigure lots more stuff (and include that in your patch), such as assigning a static IP, assigning Samba/NFS users, shares, mounting of HDDs (you can do this in your VM by creating 5 new vHDDs) and other stuff that I probably haven't thought of yet. If you go the tweaked install to USB route then a lot of this stuff can be configured post install. So you'll be looking more to include the tweaks to reduce writing to USB and installing NFS etc.
As for NFS inclusion in the Fileserver appliance, it has been discussed and I think it is intended to be included in the next release. Just to make sure it doesn't get forgotten I have created a Blueprint.
More ideas (and a bit of a rethink)...
To further muddy the waters, a little research suggests that OMV can theoretically be installed to an existing Debian install (despite the fact that the OMV devs say that it is 'unsupported'). If that takes your fancy you could install a minimal Debian Squeeze and install OMV over the top. You could perhaps even have the best of both worlds and install OMV on top of the freshly released TKL Debian based Core. I also found some suggestions re tweaking OMV to write less to disk (ie extend the life of your USB when installed). From what I gather these are generic Debian tweaks that probably relate to Ubuntu too (so may be useful whether you use TKL or not).
I am obviously a big fan of TKL, and encourage it's use whenever I can! But after re-reading this thread and thinking further about your desired usage/outcome (and your stated preference), the more I think that perhaps you would be better off going with OMV and tweaking it to write less to USB? It sounds like you already have a preference for OMV and from what I gather it was the USB writes issue that bought you to TKL. Unless you create a custom TKL LiveISO (which is non-trivial) then the same issues apply equally to TKL (ie writes to USB). So really TKL is not bringing anything to you, other than the option to tweak it to run as a LiveCD, which as I say will require a fair bit of effort.
So rehashing my thoughts, here are the preferred options as I see them (in no particular order):
Let's Keep It Simple
Hi Jeremy,
Many thanks for the time to think it through for me.
I think you are right with the last three options.
I just spent a couple of hours this weekend - experimenting with OMV and minimising the writes to Flash. It partially works but the solution is not elegent - not easy to reproduce if a new install is required; not fool-proof in that one needs to know what services are going to be writing to Flash (and not all write to just /var/log but other places too); and it's an error prone.
I could look at TKL LiveISO route but I fear it will be the same set of issues as OMV (i.e. finding all the services/apps that regulary write to the Flash) which means it's an iterative and imperical method. And if new updates or apps are added, then fine tuning/reconfig will be required. That is, seems like high maintenance.
Also, OMV has other issues in the way it allows File Shares are created -- it's extremly limited (e.g. you cannot simply browse to a folder in a hierarchy choose it for sharing). But with TKL (other distro) using Webmin, this is not an issue.
So I think I will take your 2nd suggestion and install on one of the HDD's and partition it for data too.
I will also attempt the TKL LiveISO route too - so look of for me begging for help :)
Mac
Installing to HDD and resizing partition will be easiest
But if you go with TKL then that isn't even necessary as TKL Fileserver is designed to store data internally (ie on the same partition as OS). Also as I think I may have mentioned previously, by default TKL uses LVM (Wikipedia) and if you want you can add your other HDDs to the TKL LV so for all intents and purposes you have a single 10TB (5x2TB) volume (which includes OS and data).
If you wish to keep data and OS separate then that's fine, but you don't need to. If you do want to keep then separate then you can just install TKL, resize it (or perhaps you can specify how much HDD to use during install? I don't recall...) and then create a new LV for data from the remaining disks + whatever is left of the HDD you install OS to (so a data LV of 10GB less however much you set aside for OS).
If you were hell bent on using USB then installing a customised live TKL system (ie LiveISO running from USB) would be the go because then the whole OS will run from RAM (USB would be read only - load OS on boot). But as I suggested earlier that will be much more involved.
Why not use USB HDD?
I might have missed this in the above conversation but have you considered using a small USB HDD instead of a Flash device for your server boot disk? I am using a small USB-powered USB HDD with the TKL File Server (added CUPS for printing) on an old HP thin client that has no internal drives. Ultra low power usage ~40 watts was my key as I like to stay as environmentally friendly as possible.
The USB HDD will run the TKL appliance fine without any of the degradation problems associated with the Flash drive.
Mike
USB Stick install
I installed everything, but my computer keeps booting from my hard drive!
You'll have to adjust your BIOS
So that it boots off the USB not the harddrive. In my experience some PCs (especially older ones) are a bit fussy (works with some sticks but not others). Many I have come across refuse to boot off USBs over 4GB. Some really old ones won't boot off USB at all (in which case you'll need to create a bootCD that will bootstrap the USB).
Unfortunately I can't give you clear instructions on what you'll need to adjust in your BIOS because they are all a little different. I'd probably find out your motherboard and have a good google about it. Failing that you could just try chaning the BIOS boot options and see if you can get it to work.
Probably need to diagnose issue
You really need to find out what is causing your installs to break after a few days. Seeing as you have had the same experience with a number of distros suggests to me that something is not quite right with your hardware. Perhaps the corruption being caused on your USB (assuming that's what is happening) is due to some other hardware issue (and perhaps that is even the cause of your HDD woes too?) Bad RAM can sometimes corrupt filesystems... Although I'm completely guessing...
I suggest that you reinstall one of the distros you've tried (or try another) and then when it fails post on their forums and try to get some assistance from the respective community of the OS you are/were using.
Add new comment