35

I work in film production and need VERY fast access to very large raw footage files. I am presently using eSATA 6Gbps docks for internal hard drives, which go as fast as the drive will go.

If I were to use a server and connect to my workstations via network, what (relatively affordable) technology would allow me to get anywhere near to, or exceed the drive speed?

  • 12
    10Gbps ethernet is well into "affordable if you have budget" range these days. However, 8 or 16Gbps SAN (storage area network - aka fiberchannel) might be better suited to the particular use? – Ecnerwal Mar 8 '17 at 2:00
  • 3
    It's specifically designed/optimized for storage networks. – Ecnerwal Mar 8 '17 at 2:21
  • 3
    Just want to check -- are you using SSD's? If not, that is an essential upgrade. – trpt4him Mar 8 '17 at 2:33
  • 1
    I see no added value of a SAN compared to a simpler and cheaper NAS here. – jiggunjer Mar 8 '17 at 3:01
  • 3
    To properly answer this, more info is needed: which amount of data must be stored, how many concurrent users, where are the data accessed from (what kind of machine)? A 100Gbs network will not help if accessed from a 2 or 3 machine with 1Gbs network adapter. – JFL Mar 8 '17 at 13:05
31

Here in early 2017, judging from information readily available online, it looks like the fastest SATA HDDs max out at around 220 MegaBytes/sec, which is 1.760 Gigabits/sec.

So if you're just trying to beat the speed of a single drive, and you're limited to HDDs for cost-per-terabyte considerations for huge video files then, 10 Gigabit Ethernet is plenty.


As an aside, note that Thunderbolt Networking is 10 Gigabit/sec as well, so if you already have Thunderbolt ports, you can experiment with that. It could conceivably beat your 6 Gigabit eSATA 3 ports, although I'm not sure about that because eSATA is very storage-specific, whereas doing storage over Ethernet has more overhead. Also note that Thunderbolt is a desktop bus; it only reaches a few meters, not the 100m that 10 Gigabit Ethernet can handle. So while Thunderbolt may be interesting for experimenting and prototyping while you weigh your options, it's probably not the right long-term solution for you unless you want to keep all your workstations and disks connected back-to-back around a large table.


So that was for single HDDs. But if you RAID those drives together so that every read or write gets spread out across multiple drives, you can get much better performance than the single drive performance. Also, depending on your budget, you could put PCIe/M.2 NVME SSDs into a PC to act as a server/NAS, and you can get blazing fast storage performance (around 3.4 GigaByte/sec == 27 gigabit/sec) per drive.

In that case you might want to look at something faster than 10 Gigabit Ethernet, but glancing around online, it looks like prices jump quite dramatically beyond 10 Gigabit Ethernet. So you might want to look at doing link aggregation across multiple 10Gigabit links. I've also seen some anecdotes online that used network equipment, such as used 40Gbps InfiniBand stuff, can be bought on eBay for dirt cheap, if you don't mind the hassles that come with buying used gear on eBay.

  • 1
    What about multiple 10Gbit NICs + bonding? – André Borie Mar 8 '17 at 10:22
  • 2
    bonding allow to have separate flows use different physical links, but a single flow will use only one NIC. So this will be useful when several files are served to different people at the same time, but will not increase bandwidth for a single file access. – JFL Mar 8 '17 at 10:39
  • 4
    @JohnU No, Ethernet is 94% efficient with standard 1500 byte frames, and even better with jumbo frames. I routinely see 930+ Mbps of throughput without even trying. Gigabit Ethernet does not allow hubs, only switches, so it is always full duplex and thus never has collisions. – Spiff Mar 8 '17 at 17:05
  • 1
    @JFL Ethernet bonding (IEEE 802.3ad link aggregation) distributes load packet-by-packet, not per-flow. Because networking technologies are generally designed in discrete layers, Ethernet (layer 2, the data link layer) has no idea about the protocols above it such as IP or TCP, so it has no idea of flows. – Spiff Mar 8 '17 at 17:29
  • 2
    @JFL Linux aggregation certainly can used some layer3+ information (Linux can do lots of unusual things), please remember you really need both sides to support the same bonding method to be useful. If you are connecting to a switch then you can really on use what the switch supports. – Zoredache Mar 8 '17 at 18:42
5

If you stick to SATA disks, implementing 10Gb ethernet and building a reasonably sized RAID10 on the server will net you a noticeable performance increase beyond that of a single SATA disk. This will be a worthy investment because you can share the server among as many workstations as you need, and add in the future by adding switches. You'll need to run Cat-6 ethernet cable, as Cat-5E won't cut it - don't forget to add this expense to your calculations. You could also add SSDs as cache to speed the system up even further; since you're working with video footage, I assume you need vast amounts of storage space, which would be extremely expensive to build purely with SSDs.

You could buy a pre-made rackmount server from Dell or HP and use the hardware RAID card, or if you're more of a hardware person, you could buy a cheaper chassis from Supermicro and build the storage machine yourself, using software RAID in either Windows or Linux. Hardware RAID is often faster when a RAID1 is involved, since software must write to each disk in turn and wait for the write to complete before moving onto the next operation; a RAID card can generally write to both disks in parallel, and cache the write operation, returning control to the OS immediately. Do note, however, that although a RAID0 would be even faster, you have no redundancy and a single drive failure will cause complete data loss; never use a RAID0 when you have data that you want to keep. I recommend contacting Dell or HP or another big name to see if they can help you spec out a system to meet your needs.

At the high end of the scale you have Storage Area Networks (SANs) but these are designed to allow many operations from vast numbers of separate clients in parallel; a benefit of this is that the throughput is very high for a small number of connected machines, but is likely overkill for your needs, and very expensive. At the low end, you have Network Attached Storage (NAS) devices that others have mentioned, but although much simpler than a full server, I don't recommend them since a NAS is frequently a black box; they're designed to be plug & play for most users, and as a result you have little control over the OS - I have just had to send back a small NAS I bought for a client's office as it became unstable after a day's use.

Another advantage to building a server is that you're concentrating all your footage in one place, which makes it practical and relatively easy to back it up regularly. Never neglect your backup strategy; one day you'll need to depend on it!

  • 3
    "software must write to each disk in turn and wait for the write to complete before moving onto the next operation". That hasn't been true for several decades; even non-server disks nowadays support command queuing. And strictly speaking, with DMA software didn't need to wait either. That's been supported in desktops since 1998 or so. – MSalters Mar 8 '17 at 10:25
  • It's probably worth noting that while your first sentence is true for a single user, the 10Gb being shared bandwidth means that with multiple users trying to access the share simultaneously the bandwidth each gets will eventually drop below single user SATA levels. – Dan Neely Mar 8 '17 at 16:59
  • 2
    a NAS isn't some special device by definition, you can have a normal Linux PC with samba or NFS and call it a NAS. The key difference is file-level sharing vs block-level sharing. A block device can only be used by one client at a time. Also SANs are more expensive and take longer/more knowledge to configure. – jiggunjer Mar 9 '17 at 2:40
  • @jiggunjer Very true, but most who would build a PC with Samba would call it a file server, not a NAS. I'm just playing on the technicality to mean the special-purpose device. Also very true about SANs, which is why I classed them as overkill. – Gargravarr Mar 9 '17 at 9:39
  • @DanNeely yes, as more clients are added with simultaneous access, speed to each workstation will decrease, but the total rate of data coming out of the server ought to remain very high. Augmenting with SSD cache could compensate for most of this. – Gargravarr Mar 9 '17 at 9:41
3

Nothing beats 10gbe's flexibility and ease of configuration, but SAS is surprisingly networkable all on its own:

For small numbers of workstations (n<8) where volumes do not need to be writable from multiple computers at once, SAS works great. With a Tyan JBOD ($1,500) and an LSI HBA ($400), we get 3,400 MB/s (27 Gbps) sustained transfers to SSD. The JBOD has an internal switch with 3 uplinks to HBA's, but SAS switches are available for higher port counts.

Here is a speed test of one of our volumes:

CDM5 showing 2,862MB/s sequential read with 32 iops in queue.  4k single threaded random read 29MB/s

Internally, we use this solution with Clustered Windows Server running Storage Spaces that are distributed to clients with 10gbe.

Clustered Windows Servers

2

https://www.newegg.com/Product/Product.aspx?Item=N82E16820147593&cm_re=samsung_m.2--20-147-593--Product

First up, M.2 SSD which given the correct motherboard can reach 2gb-4gb/s, and possibly higher. RAID a few of these together, and your speeds are even higher.

Teaming multiple 10gb NIC could get you close to native speeds.

Anything more than 10g is a lot more money.

40 gigabit NIC https://www.serversupply.com/products/part_search/pid_lookup.asp?pid=263133&gclid=Cj0KEQiA9P7FBRCtoO33_LGUtPQBEiQAU_tBgIU2jZrJKf0kXFy96roOllcRkp7j-VoubG_n7xb0_pgaAnaT8P8HAQ

  • Going above 10gb is also likely to put local bus speeds on the scene as a limiting factor :) A PCI-express 2.0 x16 slot is theoretically 64GBit/s - but overheads on all sides (bus to memory, driver software to memory, cpu load...) will gnaw that down pretty close to where you can be glad if your 10GBe runs at line speed ... – rackandboneman Mar 8 '17 at 8:33
  • 2
    @rackandboneman: That's why you have smart NIC's. The 10Gb/s Ethernet is raw speed. The Ethernet, IP and TCP headers can be handled by the NIC, leaving the CPU to deal with just the real data. – MSalters Mar 8 '17 at 10:20
  • IF these offloading mechanisms work in combination with your application scenario - and while they cut down on CPU usage, they won't reduce data volume THAT significantly unless a lot of that traffic is garbage in the first place... – rackandboneman Mar 8 '17 at 13:14
  • @rackandboneman PCIe 3.0 x16 exists so speeds can be double that. – cybernard Mar 8 '17 at 13:29
  • 10gbe can be far cheaper than that: Mellanox nic for $180 and 25gbe comes close Mellanox 25gbe for $300. What kills you is port cost on switches, but eBay or a full mesh makes that problem go away. – Mitch Mar 8 '17 at 20:38
0

Use a fiber ethernet card with a fiber switch, ideally multimode or single mode with pads. If you've got the budget you can make the whole thing 10gbps, if you use link aggregation with a dual port fiber card you can get 20gbps. Since the term "relatively affordable" changes depending on what your budget is, actual numbers as to what you're willing to spend would be helpful.

0

The cheapest way would be to use link aggregation (IE run N cables, typically 2) instead of one.

This way a dirt cheap 1Gb link will become a 2Gb link.

Obviously it means you need at least 4 ports for a single client server.

  • and the client needs 2 ports... Do simple switches work with this? or should it be a direct link? – jiggunjer Mar 9 '17 at 2:28
  • @jiggunjer simple switch can be used for that but you need a special configuration on all computers/servers that use this scheme – Antzi Mar 9 '17 at 2:38
  • is this aggregation the same as bonding mentioned in the comments of the answer by @Spiff? In which case bandwidth isn't increased for single file access... – jiggunjer Mar 9 '17 at 3:07
  • @jiggunjer This really comes down to how you do the setup, so I can't really answer. – Antzi Mar 9 '17 at 3:21
  • 1
    @FlorianCastellane latest cheap one I bought wasn't. Depends of the specs – Antzi Mar 10 '17 at 0:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.