0

When designing our next-gen airborne LIDAR system, we intended to store our data on storage drives that were formatted with NTFS. This is the way our previous-gen systems work. Our systems run Linux (CentOS) but using NTFS hasn't caused us any issues. It also allows our customers to remove the storage drives and connect them directly to a Windows PC for processing.

As our system collects data, it is divided up and written to multiple SATA drives (we do plan to move to removable NVME drives once the motherboard manufactures catch up to the times). Two thirds of the way through development we encountered a massive problem. Although storing the data to NTFS worked great for our last-gen systems, CentOS couldn't write to NTFS fast enough to handle the increased data flow of our new system. The only solution we could come up with at the time was to reformat the storage drives to a native Linux file system. We chose ext4 and got the write speeds we needed.

Problem 1: Our customer now has several terabytes of data on ext4 drives while the processing software runs on a Windows PC.

Our Current Solution: We have the customers install "Linux Reader" software that allows them to copy the data from the ext4 drives.

Other Solutions We've Tried: Having the customers use ext2fs software. Also, having them boot the processing PCs into Ubuntu, transferring the data, then rebooting back into Windows.

Problem 2: The solutions for problem 1 feel unprofessional. They also each have their own set of problems. For example, the Linux Reader software sometimes doesn't see all of the files on the drives.

I've had a few additional ideas that I haven't tried yet (such as running Ubuntu on a VM on the processing PCs to read the drives and copy the data). Its hard to justify continuing down this rabbit hole when we already have a solution that "works" (mostly). It's just that when the customer pays such a large sum for professional equipment, they shouldn't have to deal with shoddy "workarounds" just to process their own data.

Does anyone here have any ideas or suggestions for a better solution?

Edit 1: I wanted to clarify that I'm not asking this question on behalf of my company. They are fine with the current solution for now. I personally want a better solution for the customers as I am unsatisfied with what we currently have.

3
  • I see one close vote already. This site often focuses a bit more on end users; you seem to be trying to apply enterprise values/criteria. I suspect another site might be a closer match: maybe software engineering (unless this question is considered to not defined "best" well enough -- meta.stackexchange.com/questions/124867/… ) or maybe one of: softwareengineering.meta.stackexchange.com/questions/8066/…
    – TOOGAM
    Sep 13, 2021 at 22:08
  • Why not using native windows linux subsystem, as far as I know it can read and write ext4 partitions docs.microsoft.com/it-it/windows/wsl/wsl2-mount-disk
    – DDS
    Sep 15, 2021 at 14:49
  • @DDS I may actually try that in the future. I've been trying to keep up with Microsoft's newfound embrace of Linux. However the Microsoft documentation states that this feature will only be available on Windows 11. It will likely be a year or two before the processing PCs are updated to Windows 11. Thanks for the suggestion though. I'll keep an eye on it.
    – Blackwood
    Sep 15, 2021 at 16:11

3 Answers 3

1

If you really want to use a native Windows filesystem then currently there are only a few solutions:

So IMO a Linux driver with great Windows support should be used. Currently there's no good ext4 driver on Windows so you should use Btrfs or ZFS instead. They're both more superior than ext4 with far better Windows drivers:

If the client uses Windows 11 Build 22000 or later, or some insider build of Windows 10, then you can even mount any Linux filesystems on wsl2

11
  • exfat has been available to download and build for a good few years and works with v4 kernels. You would have to ask though, why use Centos for such a specific application. There are better candidates if looked for.
    – Bib
    Sep 14, 2021 at 11:49
  • @Bib but previously exFAT drivers are userspace drivers (FUSE) so they'll be extremely slow and not suitable for speed or latency-sensitive applications
    – phuclv
    Sep 14, 2021 at 12:01
  • 1
    github.com/arter97/exfat-linux, github.com/bkuhn/exfat I am sure there are others. These are kernel modules, not FUSE userspace. That last one is dated from 8 years ago. As for zfs, the less said about that the better. As the op has explained, getting the customer to use additional software does not present a good image, so Btrfs is also out.
    – Bib
    Sep 14, 2021 at 13:06
  • @Bib if the OP sells this device then they should also provide some Windows software to work with it, no? Then the FS driver is one of the thing included in the Windows tools
    – phuclv
    Sep 14, 2021 at 13:14
  • 1
    I've done some initial testing with Btrfs and I'm impressed with the read/write speeds. This would basically be the same solution we currently have but (hopefully) more reliable than LinuxReader. Unfortunately I can't call this a winner because I keep running into this issue: giters.com/maharmstone/btrfs/issues/375, github.com/maharmstone/btrfs/issues/321 If they can't get these resolved then I may have to wait for Windows 11 and your wsl2 solution.
    – Blackwood
    Sep 22, 2021 at 19:44
1

Why not use exFAT? It's small, has huge capacity, Linux can write to it, there are no issues with user permissions and there is reduced overhead.

3
  • It didn't occur to me to try different Microsoft file systems. Even though it would be slower than ext4, I can still test to see if it is "fast enough" for my purposes. It may be a while before a system is available for testing but I'll comment back with results when I can.
    – Blackwood
    Sep 14, 2021 at 18:16
  • 1
    Just make sure you use a kernel modules and it does not use FUSE. It should be faster than ext4.
    – Bib
    Sep 14, 2021 at 18:25
  • I like this solution the best. Unfortunately I won't be able to use it for my current situation. CentOS 7 uses kernel v3.10 by default. To use an exFAT kernel module I would have to upgrade the kernel to v5.x. There is a lot of custom software running on the system so it would be too big of a risk to update the kernel on production systems. (The irony here being that way back in the design phase we actually considered updating the kernel to v5.x but decided against it) On a side note, it looks like Linux may get a kernel level NTFS driver (from Paragon) sometime in the next year.
    – Blackwood
    Sep 22, 2021 at 19:27
0

Possible solution #3...

well, first, let's re-cap some of the details: the first two solutions mentioned in the question:

#1: Move hard drive to Processing PCs, then Use extfs-reading software in Windows

#2: Move hard drive to Processing PCs, then use Ubuntu to read data (and presumably write it to Windows, and then boot Windows)

How about:

#3: Move hard drive to dedicated file servers that use Ubuntu to read data. They share data: Windows computers can either check for data (or get triggered by an outgoing connection by the Ubuntu servers) and automatically copy data when it becomes available, or the Ubuntu servers can set up a network file share which can be accessed remotely from Windows computers. The network file share would look like a drive letter and feels rather natural to most people who would interact with the Windows PC).

1
  • I had an idea very similar to this at one point. Thank you for the suggestion. Unfortunately in this situation this idea would require additional hardware and development time to implement it the right way. I might could try it out as a side project at some point.
    – Blackwood
    Sep 22, 2021 at 19:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .