Anyone using Open Solaris & ZFS?

[RaiderJ]

I just picked up an Intel Atom board, and plan on running a file server using ZFS - curious if the board will be able to handle a 4 drive ZFS RAID. Anyone done something like this? How does it work?

 

[QuixoticOne]

Give it a try. My guess is that it probably may not have the device drivers needed for your
ethernet or SATA ports( if that's what you'll use). I'm still waiting for it to support some
very mainstream Intel *desktop* chipsets and major desktop NICs. It'd be rather amazing serendipity if
it "just works" on the brand new ATOM chip now. There are also some ACPI / USB related problems
that apparently block successful operations for many people, though these seem easier to work around
than the lack of chipset drivers for NET / storage / etc.

You might try FreeBSD 7.0 + patches as being possibly more likely to support the ATOM chipset sooner
than Solaris may. There's experimental ZFS support in FreeBSD 7.0 which apparently works sort of well
as long as one doesn't reach some of the many possible limiting cases of its operation. YMMV if you run out
of RAM memory as to how well it'll handle that. I've heard that it's STRONGLY advised to run ZFS with
a 64 bit OS and LOTS of RAM (4GB or more) if you're building a ZFS of any appreciable size so that the various
cache / buffer / data structure operations won't fail potentially catastrophically. q.v. the mailing list traffic on
the state of ZFS under FreeBSD for more info. Anecdotally apparently it's working well "in practice" for
some testers.

I imagine if you get a silicon labs cheap ($25-$50) multi port SATA PCI card you may be in better luck as to
storage support if the atom's built in SATA doesn't do it for you, maybe something like a Rosewill RC-209,
if that's supported (I haven't checked; I just know it's a cheap multi-port SATA card that works for LINUX).
If you're willing to spend $100-$300 on a RAID / multi-port storage card for the task I'm sure there are better
commended models with more features / performance that you can find that are well documented as being well supported by Solaris' hardware compatibility list. Of course this will possibly cost comparable or far more money than the rest of the ATOM and consume non-trivial amounts of power compared to the rest of the system, somewhat defeating the purpose of using an ATOM vs. a desktop PC.

I don't know if the graphics will work either, but hopefully they would do so in VESA driver mode at least,
and even if not you can always (hopefully) install in text mode.

SXCE B95 should be out within a couple of days; that's probably your best bet to play with, though AFAICT
there hasn't been that much in the way of new driver / package support added yet since the 2008.05 Indiana
mainstream release.

The upcoming 2008.11 release may be closer to "significantly better" but I'm thinking
it'll still be pretty far behind the state of LINUX and various BSDs for hardware support.

I'd expect it to be pretty slow in performance given all the checksumming and memory I/O solaris and ZFS
will do and the limited CPU / RAM bandwidth / IO bandwidth of the ATOM, though it'd still be useful for many
purposes (hopefully reliable / redundant but slow) if it works.

There are other projects like nexenta's free OS distribution for a solaris based RAID/NAS, and
FreeNAS http://www.freenas.org/ and so on that may be alternative / interim things to look at.
AFAICT the nexenta solaris distribution isn't really any better than the opensolaris SXCE system in terms
of driver support though; YMMV.

Another option that might work in theory is to run VMWARE ESX or something similar and run
Solaris + ZFS as a guest under that with emulated drivers for storage / video / network to insulate the
possible incompatibilities of the actual system hardware from the OS. Unfortunately I'd say there's a pretty
poor chance of good VM support for the ATOM anytime soon (probably a worse speed of support than
Solaris itself), so this may be infeasible in practice; YMMV.

Of course there's experimental EXT4 support for LINUX which lacks several of the nice points of ZFS,
but might be more quickly deployable on your actual hardware. Even so, I'd expect it'll be another 9 months
or more before EXT4 is particularly commended for non-experimental uses; YMMV.

You could probably (eventually) set up some ATOMs (with LINUX or BSD or whatever) to export emulated
iSCSI devices (created from one or more directly attached local discs) over ethernet, and then use another
system capable of successfully running Solaris/ZFS to create a zpool out of those remote storage devices,

Let me know if you get OpenSolaris running on the ATOM; it'd be neat if it works.

Originally posted by: RaiderJ
I just picked up an Intel Atom board, and plan on running a file server using ZFS - curious if the board will be able to handle a 4 drive ZFS RAID. Anyone done something like this? How does it work?

 

[RaiderJ]

You are my new best friend! Thank you for all that info!

I expected that the Atom platform might not be the best for Solaris - but my main interest is in ZFS, so other OS's aren't out of the question. I did a search, but found very little talking about the Atom & Solaris. My hope was that since it used an older chipset, that there would be less compatibility issues. As for SATA ports, I do have an older Intel PCI RAID card, so I might be able to use that and avoid problems too.

Space-wise I'm ideally going for a 4 drive RAID, but the Atom might only support a couple of drives in a standard pool. The VIA Nano however might be a better platform for a full on 4-drive array. But, since it's brand-new, it might have compatibility issues as well. Still, I think either of these boards would be almost perfect for a low power file/FTP/HTTP server.

 

[QuixoticOne]

http://bugs.opensolaris.org/

Search under keyword atom and also various related ones like the model of your motherboard
or the NIC / whatever chip manufacturer / model you have, et. al.

I see there are several filed under atom which is overall a good sign since at least it means people are
using them with varying levels of success.

It seems there are a few driver support weaknesses for various NIC / other chips that are used, as expected,
and the 32 bit vs. 64 bit boot issue for one kind of Atom chip anyway.

I just got the current OpenNevada (ON) B94 build version to install and seem to work on an Intel desktop
platform that had blocking bugs in the previous OpenSolaris ON/SXCE/SXDE releases, so they're making good progress. Sometime in early August they'll have another new set of ISO images for a re-spin of the main
distribution to replace 2008.05 with some of the newer changes integrated from the B9x builds and the
2008.11 alpha development tree. In the meanwhile, the ON B94/(upcoming) B95 or the
genunix 2008.11 alpha development ISO distributions are likely good to start with.

http://opensolaris.org/jive/index.jspa?categoryID=1
http://opensolaris.org/os/downloads/
http://opensolaris.org/os/downloads/sol_ex_dvd_1/
http://genunix.org/

 

[QuixoticOne]

BTW in case you care, the latest roadmap for ZFS crypto support indicated that it was planned to be merged
at/around B100, so I'm guessing that'll make it into the 2008.11 release given that they tend to release another
build every 2 weeks or so and we're coming up on B95 now.

I noticed last night a blurb claiming they've fixed (some of?) the major memory related danger issues relating to
ZFS, so maybe it's not so necessary to run it in 64 bit with lots of RAM for safety / reliability / performance as it once was.
I didn't see much detail in the post, though, indicating exactly how complete the changes were or exactly when they may
have been (or plan to be) merged into the release builds. I'd certainly research that a bit before running 32 bit mode
and/or with less than 4GB RAM if you're going to be setting up terabyte level RAID-Zs.

http://www.mail-archive.com/zf...aris.org/msg05591.html

http://en.wikipedia.org/wiki/ZFS
...
FreeBSD
Pawel Jakub Dawidek has ported and committed ZFS to FreeBSD in experimental capacity for inclusion in FreeBSD 7.0, released on February 28, 2008.[29] The current recommendation is to use it only on amd64 platforms with sufficient memory but there is a newer port (uncommited yet) which fixes the memory issue.
As a part of the 2007 Google Summer of Code a ZFS port was started for NetBSD.[30]
...

http://kerneltrap.org/FreeBSD/ZFS_Stability

 

[RaiderJ]

Crypto support would be a great feature - especially if it could somehow make use of VIA's onboard hardware (but those two might be totally unrelated). I had planned to run Solaris in 64-bit mode, even if the board itself doesn't support >2GB RAM. However, maybe due to the memory limitation it doesn't really make a difference?

My board should arrive next week, so I'll start some testing then. As for ZRAID, I'll probably start testing just using thumbdrives over a USB hub - makes it easy to plug and unplug "drives" that way 🙂. If all goes well then I'll upgrade to real hard drives.

 

[QuixoticOne]

Which ATOM board did you get? I've been looking at a couple of them like the $75 one on Newegg, though I don't think
I'll get one quite yet.

I don't see why Solaris shouldn't use the VIA crypto hardware; VIA has recently published the details of how to use
their padlock / RNG / crypto hardware for open source operating systems IIRC. I'm guessing that Solaris would quickly
integrate drivers for those things to speed up AES and random number generation and so on. I don't recall what crypto
alrgorithms ZFS can be configured to use, but I wouldn't be surprised if among them would be AES of the sorts that the VIA
chip can speed up.

There's a couple of interesting articles about ZFS/Flash for flash memory storage as a level 1 memory subsystem in front
of a conventional disk storage pool; the combination is said to be quite effective at increasing throughput. Your comment
about the USB drives made me recall it, though I expect they're mainly thinking about the case where the flash is
integrated into a hybrid flash+rotational disk drive, or when you have a large high performance flash based SSD
with a SATA/SCSI/whatever interface. Even so, it isn't inconceivable that having a few 8GB USB flash drives independently of your rotational discs might be a big improvement if used with ZFS/Flash.

Sun?s ZFS/Flash initiative, ZDNet
http://blogs.zdnet.com/Murphy/?p=1191

Flash Storage Memory, ACM
http://mags.acm.org/communicat...m=adam+leventhal&pg=49

 

[RaiderJ]

I picked up this board plus a 2GB stick of RAM. I haven't seen any other boards floating around, although I've heard there is a new chipset coming out for the Atom boards. Hopefully to reduce overall power consumption.

Nice read on the flash memory... would be very interesting to try out with different amounts any type of flash memory. I don't really see USB drives as useful, mostly due to the limits on the USB interface - but maybe for a bunch of small random reads it wouldn't hurt.

I forget the website, but they did a test of newer SATA II SSD drives, 4 in a RAID0 I believe, and the read/write speeds were incredible. Run a set of those in front of a large multi-terabyte system and you might very well have a very impressive system for not a whole lot of money (relatively speaking).

If crypto-ZFS does support the VIA hardware, the Nano boards would make great little storage nodes. Cluster a bunch of them over GBe or fiber and you'd have an awesome storage network/SAN.

 

[QuixoticOne]

The thing that bugs me about many of the ITX and smaller boards is that they don't have on-board power
conversion and require only a 9-15V DC input, but rather usually (with few exceptions) require what is
basically a full ATX compatible power supply. If they just ran off of DC input then adding a UPS for them
would effectively be as simple as adding a 12V battery, whereas with the ATX power supplies they often
cost similar cash to the whole ITX motherboard if you want a smaller / more power efficient one that can
run off of a DC input. The other thing is that many don't have Gb ethernet and can't take all that
much memory; 8GB or more support would be nice on at least a better selection of them given today's
RAM prices; it'd certainly be optimum for a storage appliance like we're talking about.

I'll have to keep an eye on the newer generation of VIA boards to see what looks good as well as the
ATOM ones. The VIA crypt. engine is nice, certainly; I don't recall seeing anything about the ATOM having
a similar thing, though I wouldn't be surprised if it did.

Looks like the ZFS crypto should probably work at least in part with the VIA Padlock hardware AFAICT,
assuming the CCM used in the ZFS is the same as used in 802.11i as below.

http://www.opensolaris.org/os/project/zfs-crypto/
http://opensolaris.org/os/proj.../zfs-crypto-design.pdf
..."2.4 Alternate cryptographic algorithms
Initially only AES will be supported with key lengths of 128 and 256 using the
CCM and CBC modes. The design and implementation must allow for alternate
algorithms."

http://www.via.com.tw/en/initi...s/padlock/hardware.jsp
...."The VIA PadLock ACE directly supports all three AES key sizes (128-bits, 196-bits, and 256-bits) in hardware, and with the same performance."..."Cipher Engine...EBC...CBC...CFB...OFB..."

http://www.via.com.tw/en/downl.../programming_guide.pdf
..."REP XCRYPTCTR: Counter Mode"...
"There are many possible counter modes. ACE implements a counter where the rst 14 bytes of the counter
(in normal ascending address order) are the nonce, a random value specic to the encryption of a particular
message and which should never be re-used. The last two bytes are considered to be the 16-bit integer counter
in big-endian format.
This counter mode is compatible with the CCM mode as specied by wireless protocol 802.11i and was chosen
specically by Centaur Technology and VIA to be compatible with that protocol."

Originally posted by: RaiderJ
I picked up this board plus a 2GB stick of RAM. I haven't seen any other boards floating around, although I've heard there is a new chipset coming out for the Atom boards. Hopefully to reduce overall power consumption.

Nice read on the flash memory... would be very interesting to try out with different amounts any type of flash memory. I don't really see USB drives as useful, mostly due to the limits on the USB interface - but maybe for a bunch of small random reads it wouldn't hurt.

I forget the website, but they did a test of newer SATA II SSD drives, 4 in a RAID0 I believe, and the read/write speeds were incredible. Run a set of those in front of a large multi-terabyte system and you might very well have a very impressive system for not a whole lot of money (relatively speaking).

If crypto-ZFS does support the VIA hardware, the Nano boards would make great little storage nodes. Cluster a bunch of them over GBe or fiber and you'd have an awesome storage network/SAN.

 

[RaiderJ]

The picoPSU is fairly cheap, ~$40 for 90watts, a bit more for the 120W version. That would work connected to any 12V power source like you're talking about.

VIA is releasing a dual-core version of the Nano at some point in the future - which would justify at least one board that has 4 RAM slots and 1 or 2 Gbe ports. Might be tough to physically fit all that onto a mini-ITX board however. SO-DIMM or 4GB sticks maybe?

Installed Solaris on the Atom last night. Only problem so far seems to be it's not picking up the audio controller, but even if there is an updated driver, my internet is down ATM and I can't check. I have 4x250GB drives coming in, and aside from not being quite sure how I'm going to connect them, I think the build should go just fine. Looks like the older chipset on the Atom board is actually a benefit in this case!

 

[QuixoticOne]

Thanks for the tip on the picoPSU. It's good to hear they'll have a dual core Nano, hopefully with the 4 RAM slots.
Given the low present cost of DDR2 and drives / flash that makes a lot of little appliances / servers a very attractive
proposition. SO-DIMMS would be OK too they're cheap enough for DDR2 though they seem to be more volatile in price
and still not quite as consistently low cost as DDR2 full size DIMMs. Some projects I've used ITX boxes on have used
SO DIMM units.

I'm glad to hear Solaris is working on that Atom board. I suspect they'll have the audio covered at some point before long,
and if you have to be missing a driver that's not a bad one to be waiting for for most applications.

I've noticed the SXCE/ON download site for the B95 CDs/DVDs has been non-functional for a couple of days.
I'm waiting to upgrade my server box to that version. ZFS/B94 has been working well on the P35 chipset and
my setup, though SMB CIFS file sharing seems to have some issues that I'm still sorting out; maybe I just need
a master browser or domain instead of a workgroup.

I've used this 4-port SATA controller under LINUX successfully for a JBOD mode interface providing the individual discs non-aggregated to LINUX's software RAID-5 driver, and a similar thing (JBOD mode) would be ideal for ZFS RAIDZ use:
http://www.newegg.com/Product/...16816132013&Tpk=rc-209

The 3114/3124 chips are a bit dated (which might be good from the aspect of driver compatibility), though I see there are some other Rosewill models on newegg RAID/disc controller pages that have the Silicon Labs 3132 and others; RC-214/215/216/217/218/219 IIRC were some of them, the numbers are just from memory and I'm probably off a bit.

Some of those newer models support 5 drives but only have one or two physical ports which operate in
SATA multilink mode, so you'd have to buy an adapter to expand the multi-link SATA link into five distinct connectors
going to the individual drives. That might not be too disadvantageous especially if one wanted to use one of the
multi-drive external enclosures which has an integrated or optional SATA multi-link interface which it splits out into
SATA links to the individual drives.

I imagine SIIG/Addonice/et.al. have other 4 or 5 port SATA controllers that are worth checking out.

 

[RaiderJ]

Hmm... problems with OpenSolaris - apparently the NIC drivers don't work! And even better, when trying to get working drivers put onto the system, it won't recognize USB sticks! Throws an error about how it can't mount the device, although it recognizes all the USB drives I throw at it.

Might try FreeBSD to see if its hardware support is better.

 

[QuixoticOne]

Ouch. Sorry to hear that. Yeah they only *just* added ethernet support for my pretty common desktop
Marvell chip to the rge driver in B94, I've been waiting for months. That and working ICH9R RAID chip support
or whatever was making it not recognize the drives.

I just finished downloading B95, they finally fixed the download site. I suppose that might be better for you,
though I'd plan on potentially waiting until november / december timeframe since they're going to take at least
that long (2008.11 release) to integrate / update a lot of the packages and features they're already working on.

It'd be a pleasant surprise if they fix enough to make it totally workable on any given platform before that, though obviously
they've been making progress as I found out in B94.

You can always file a bug report for adding the chipset support. Sometimes it is very simple like just adding a
PCI or USB device ID to a table of "supported" devices that all use a similar chipset but which all have variously
different ID numbers depending on the product's vendor.

Another way to get drivers on to the system would be to burn them to CD/DVD.

It can also be handy to have some PCI or USB NICs with very common chipsets around in case one needs those
for driver support if the mainboard one doesn't have it. In this case, though, I'm guessing maybe the
USB controller driver itself might not work in which case no USB peripheral may work. It'd be odd for a generic
"mass storage" class fiash drive to fail if the chipset / USB controller drivers are working.

I'll test with a couple of models of flash drive on B94/B95 and see if it works on my desktop later today.

FreeBSD does have pretty excellent hardware support in comparison to OpenSolaris at the moment.
BSD lags a bit behind some of the major LINUX distributions for things like USB ethernet / wireless NIC / TV card
type of support but BSD has made great strides over the past few years. BSD probably has better core
platform compatibility than LINUX / Solaris, it runs on all kinds of powerpcs, sparc, many kinds of x86, etc. etc.

 

[RaiderJ]

I'm pretty sure it's the chipset having compatibility issues, which of course would explain the USB problems. Had a friend recommend trying SXCE for some better hardware support. Either way, I figured the Atom would have its quirks, so none of this is really unexpected. As for trying other hardware, that would be one way to work around the issues, but then kinda defeats the purpose of the integrated board to begin with.

Maybe a virtual machine would be worth trying - would certainly make testing different configurations easier!

 

[QuixoticOne]

Indeed. Well give it a few months and I bet it'll be much better in supporting the hardware.
File some bug reports with a list of vendor / device IDs from the PCI and USB buses so they can see
which network chips and USB chipset they need to improve support for.

There's a new SXCE Open Nevada build release ISO made every 2 weeks or so, so if the changes are easy they
may become available fairly soon.

As for a VM, that's a good idea. Check out sun's free VirtualBox as well as the free VMware Server and
free VMware ESXi 3.5.something. ESXi runs hosted on the bare metal so it doesn't need a host OS but it does need
hardware chipsets / devices that it has drivers for, so YMMV on compatibility with that. VMWare server could probably
run underneath FreeBSD or CentOS LINUX or something like that. In any case you might well be able to get an
OpenSolaris guest running under the VM. KVM/QEMU are other VM manager choices for LINUX hosts.
VirtualBox and ESXi are supposed to run pretty fast though. With 2GB of installed RAM, though, you'd probably get
1GB-1.5GB max. available to the guest more or less depending on your configuration. I'd guess a console mode
BSD + Vmware or ESXi would be the most memory conserving options.