7

It seems to me that most network "file sharing" protocols are one or more of old, slow, and insecure. The most used protocols seem to be SMB, NFS, and WebDAV.

I'm sitting here, looking at iTunes trying to scan a media library over SMB, and it's going at about 2 MB per second. It's hooked up through wired gigabit networking, and the share lives on a RAID array that can do 50x as much throughput, even when searching. This is ludicrous!

Some file sharing protocols from the past might include:

  • Andrew File System
  • 9P/Styx protocol from Plan9 / Inferno
  • RFS from old System V
  • AppleShare protocol
  • Novell Netware protocol

My requirements are reasonably simple:

  • Modern security -- ideally, uses public/private keys like SSH. Tunneling over TLS would be great.
  • High performance -- scanning a file system or reading bulk data should run at the speed that the server and network can support.
  • Native clients for Windows and Linux -- others would be gravy but not interesting to me.
  • File locking compatible with Linux and Windows.
  • Change notification compatible with Linux and Windows.
  • Ideally, high quality open source implementations are available, but I'd be OK with a small license fee. (And, no, not enterprise-wide "small" -- I'm a regular guy with a family and a mortgage)

Am I missing something?

4
  • 2
    Two things: (1) this isn't a programming question, and (2) you have competing requirements. Security is always at odds with performance. You want fast, use NFS. Dec 18, 2011 at 22:51
  • Have you tried tuning the transport layer's performance first? (note: the text that looks like broken links [more][less] etc is not broken, they just don't know how to style links properly)
    – darvids0n
    Dec 18, 2011 at 23:37
  • Does Kerberos count as "modern security"? If so, Samba with SMB2 enabled might work. Dec 19, 2011 at 15:48
  • 1
    Given how much faster CPUs are than long-haul wires, security doesn't need to be opposed to performance.
    – Jon Watte
    Dec 20, 2011 at 6:27

3 Answers 3

5

What is wrong with NFS 4?

Modern security -- ideally, uses public/private keys like SSH. Tunneling over TLS would be great.

NFS 4 offers modern authentication. Data security should be done via a VPN.

High performance -- scanning a file system or reading bulk data should run at the speed that the server and network can support.

NFS 4 is reasonable high performance for the general case. Lots of caching, aggregation of commands, etc.

Native clients for Windows and Linux -- others would be gravy but not interesting to me.

I am not aware of anything here. Any reasonable wide-spread solution comes from one of the corners and is only supported as secondary citizien in the other world. That said: I think it is fair to say that e.g. SMB is better supported in the Unix world (via Samba) than e.g. NFS is in the Windows world.

File locking compatible with Linux and Windows.

NFS gives you this.

Change notification compatible with Linux and Windows.

Good question. I don't think that NFS supports this in a good way. At least I am not aware of any command that e.g. would call a callback if something changes on the (meta data) server.

To your 2 MB/s with SMB? There should be something misconfigured. SMB is certainly capable of more than that. 100 MB/s (GB wirespeed) should be no problem in a reasonable configuration.

1
  • The 2 MB/s is probably because of iTunes, rather than the inherent set-up. That being said -- the answer to my question seems to be "no." Perhaps enough extensions can be added to NFS 4 to support change notifications and perhaps a Windows client could be written. But -- why hasn't anyone done this already? I want it all, right now, for free :-)
    – Jon Watte
    Dec 20, 2011 at 6:46
5

The fastest file sharing protocol in the world isn't going to be able to make up for the fact that some apps aren't written with remote filesystems in mind, so they dispatch lots of small reads or writes, and wait for one operation to complete before starting the next one. This is fine on the relatively low latency of a local disk, but it's painful over a network, where the latency is much higher.

If you want scanning of a remote media library to go fast, you need to have code running on the server do the scanning and report the results back to the client process in one big lump.

2
  • I agree that iTunes acts like a P.O.S. It's version 10.5, and it's still locking up the UI to scan a folder! That being said: you can solve this problem with judicious caching and pre-fetching. Which a high-performance remote file system would do.
    – Jon Watte
    Dec 20, 2011 at 6:44
  • I agree that iTunes acts like a P.O.S. It's version 10.5, and it's still locking up the UI to scan a folder!
    – Jon Watte
    Dec 20, 2011 at 6:46
0

Multipath TCP is supported on Linux and on Windows. I am not sure of compatibility. If you have more than 1 n/w paths then you can use it to improve performance. More than security, look for reliability. Most of the caching systems with write caching enabled, trade here.

1
  • This has nothing to do with a local area file sharing protocol. MPTCP is still at the "Transport" layer and is lacking all the higher-layer pieces needed for remote file access and file system usage.
    – Jon Watte
    Nov 20, 2019 at 22:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .