I'm not streaming hd movies from my ZFS storage (just a few self-made
ones, which can be streamed over the local network just fine), nor do I
do any video editing - so this might be somewhat off-topic -, but I
wanted to comment on the port multiplier issue (how to connect the drives).
I have a zpool with 10 drives, 5 drives per hot-swap enclosure, each
enclosure has a SATA port multiplier, so they're each connected to a
single eSATA port (using no more than 2 eSATA ports on the host
computer). Those are the kind of affordable (home-use) RAID enclosures,
but I've disabled their hardware RAID functionality, passing all the
drives through to the host, because I want ZFS to be able to fix errors
(I've tried the hardware RAID feature, good thing I had a backup when
the first drive started failing and the enclosure just went off).
This setup works for many months, until one drive starts
failing/misbehaving. In most cases, that caused read/write errors on all
5 drives in the affected enclosure, even though the remaining 4 drives
were actually fine. Of course, r/w errors on 5 drives (in a RAIDZ-3
pool) froze the pool to protect it from further damage. After a reboot,
everything was working again (ZFS could fix errors thanks to sufficient
redundancy) and after replacing the bad drive, the system would be
running for another year until the next drive would go bad.
It might be just a bad/cheap port multiplier in the enclosures that I'm
using, but at least in my case, I've come to the conclusion that I'd
rather avoid one (sort of a single point of failure) and try to connect
all the drives individually.
Lately, I've been experimenting with SAS cards (connecting the SATA
drives directly, which are in a different kind of enclosure with one
SATA connector for each drive). I've connected bad drives that I've
taken out of my old system previously (these drives caused crashes, as
described above), put the test system under load and I've not yet had
anything unexpected happen to me so far (no crashes). Single drives
(with smart errors, for example) would suddenly be listed as "UNAVAIL"
or "too many errors" (and positive error counts), but not affecting any
of the good drives. This is eventually better, because a single bad
drive (which is failing, corrupting data), should obviously not crash a
RAID system (with enough redundancy).
smartctl works fine too. And the /dev/disk/by-id/ata* symlinks also have
the exact same names as with the port multiplier setup (they usually
contain the drive's serial, which makes identifying/replacing drives
(Also, someone mentioned fixing the drives with foam tape, but I still
prefer hot-swap enclosures - just in this case, ones with one data port
So, again, maybe my old enclosures are just too cheap and this doesn't
happen with other port multipliers. But I wanted to mention that it
might be a problem to use them sometimes, rather than connecting the
Post by Gordan Bobic
With a 5:1 PMP and a 4-port card you need 1 PCIe slot for 20
disks. You do need somewhere to put the PMPs, of course, but I
find that using thick double-sided foam tape works very well for
just sticking them to the side of the chassis where they are out
of the way and don't use up any slot space.
Actually this looks pretty cost effective (thanks for the hint),
but also means a lot of extra cabling...
It is a little cheaper per port than, for example, a 2nd hand 3ware 16
or 24 port SAS card. The advantage, however, is that unlike on the
3ware card all the usual tools like hdparm and smartctl "just work".
On the 3ware card you can just about manage the most basic of things
that hdparm lets you do using sdparm (useful for little more than
enabling write cache on the disks), and you need vendor provided
utilities, often binary-only, which also provide the necessary devic
nodes to make smartctl work with the disks attached to the card. And
replacing disks without rebooting is just as problematic - you have to
use 3ware utilities to rescan the buses, then manually get udev to
create the /dev/disk/* symlinks. And you never get wwn-* nodes.
All this compared to a SATA+PMP solution that is cheaper to begin with
and on which all of the above "just works".
To unsubscribe from this group and stop receiving emails from it, send
To unsubscribe from this group and stop receiving emails from it, send an email to email@example.com.