4

This is very similar to Question 326211, but in this case, the LAN is an unstable Wi-Fi connection.

I need to transfer about 11 GiB of files between two computers, both running Linux (although one may be rebooted into Windows.) Their connection is both slow and unstable (due to Linux's awful Wi-Fi support,) but removable media (such as a flash drive or external hard drive) is not an option at this time.

Right now, I'm slowly transferring the files, one by one, across SFTP, but I have to reconnect each computer approximately every 90 seconds, and the computers are not very close to each other, so this is not feasible.

This is not a duplicate of Question 30186; that one specifically concerns Windows 7, and all the proposed solutions involve closed-source, Windows-only programs (which are all spyware IMHO, and are all off the table even if I trusted them - one of the computers is Linux-only.)

5
  • 5
    Since when is Linux's wi-fi support "awful"? I've never had any problems.
    – snapshoe
    Jun 23, 2013 at 22:00
  • 1
    @snapshoe Since I tried 4 different Wi-Fi adapters, and the most stable one overheats every 90 seconds during heavy usage! And that's AFTER updating the driver... I could barely download the new version, and even with my lame-O 10 Mbit DSL, my Wi-Fi is usually a chokepoint. I know everyone always says Linux has great Wi-Fi support, but that has not been my experience in the slightest. For the few Wi-Fi adapters Ubuntu supports out-of-the-box, none of them even come close to rivaling the speed and stability they achieve on Windows. Jun 23, 2013 at 22:07
  • 2
    why is sneakernet not an option? en.wikipedia.org/wiki/Sneakernet it's awesome! use a 16gb usb flashdrive. or use the open source filezilla with the auto re-connect (retry) option. Jun 24, 2013 at 11:45
  • Memory Stick, HTH :)
    – Ardesco
    Jun 24, 2013 at 12:00
  • @snapshoe: Many many laptops have wireless cards that need proprietary drivers which are not installed with the OS.
    – shieldfoss
    Jun 24, 2013 at 13:44

5 Answers 5

8

What about using something like 7zip or winrar to create split archives of, say 50MB each. That way you can transfer files until connection drops but you don't need to start from scratch each time.

# -mx0 indicates no compression to be used (quickest method)
7z a -mx0 -v50m video.7z myVideo.mp4

Once all files transferred (using scp or equivalent) just recombine and extract? You'll probably also get better speeds from transferring multiple files at once.

--

EDIT: I know you said removable media was out of the question, but could you remove the hard disk from computer 1 and just plug into computer 2 at all to perform the transfer?

9
  • Close... but no dice. The files are MP4 videos, and each about 30 minutes long (~190 MiB,) so every compression method ADDS about 2 MiB.. +1 for ingenuity, though! Jun 23, 2013 at 21:59
  • 1
    I'm not saying to compress it - just to split it to prevent having to restart the whole transfer when connection drops out. Think you're missing the point.
    – James
    Jun 23, 2013 at 22:14
  • 2
    Oh, I see. Sorry, yes, I did miss the point there. But then it would be much faster to simply use split and cat, and possibly tar beforehand to get all files in one large chunk.
    – LSerni
    Jun 23, 2013 at 22:18
  • 1
    Tell me more about this split command.. it sounds like it may be the droid I'm looking for... Jun 23, 2013 at 22:20
  • 1
    @JamesTheAwesomeDude google man split Jun 24, 2013 at 12:16
11

First of all I'd try and see whether it's possible to make the connection more stable. File transfer apart, I don't think it's healthy to work with a connection that drops every 90 seconds.

A simple USB WiFi dongle could work miracles there (first, investigate which device is actually causing the disconnection: host A, host B, or the access point?

Possibly, Linux has nothing at all to do with the signal loss, but's rather a matter of WiFi instability per se. Try locking the access point to a lower speed. Even a paltry 11 Mbps, if rock-solid, will outperform a flaky 54 Mbps connection. And very often, cards will try for the higher speed even if, given the situation, they can't possibly deliver.

If you're stuck with a connection that drops and renegotiates every 90 seconds, your best bet might be either rsync or chunk the files into small enough pieces to filter between renegotiations:

split -b 10000000 file.mp4

will split the file into N chunks of (say) 10 MiB each, called xaa, xab, xac... Once the copy is complete, you can reassemble the chunks on the other side using cat.

Another possibility could be to abandon TCP altogether and use an UDP based protocol, for example UDT or UFTP.

Finally, you could set up a simple Web server on one machine, and use wget or any HTTP client supporting retrieval integrity, byte-range and auto-retry to download the whole set of files (which, if they aren't already, it would be better to compress).

This is longer to set up, but then it ought to go on automatically and uses available and tested & proven software (apache, wget).

Come again - gone again - Finnegan

Granted that you need to be sure the scripts are synced, or you'll waste a lot of connection time, you can probably play with either iwconfig or ifup/ifdown in order to force the access point to behave:

while true; do ifdown wlan0; ifup wlan0; sleep 90; done

or if you want to run it as a script (it will require root access either way, to run if* commands):

#!/bin/sh
while true; do
    echo "Going down..."
    ifdown wlan0
    echo "Coming up..."
    ifup wlan0
    echo "Ready"
    sleep 90
done

A (hopefully) better way

The problem with the "sleep of 90 seconds" is that unless something is really weirdly flaming wrong in the router, the time it stays up won't be exactly 90 seconds. So we're wasting a lot of time either staying up when the WiFi carrier is down, or going down when the WiFi carrier might have stayed up.

So let's say that the router has IP 192.168.0.1, on both Linux hosts we run

#!/bin/sh

IP=192.168.0.1

while true; do
    if ( ping -f -c 3 -w 1 $IP | grep "0 received" > /dev/null ); then
        ifdown wlan0
        ifup wlan0
        # Extra time: we MUST be sure the network is up!
        sleep 5
    fi
    sleep 2
done

This script will shoot three packets towards the router. The packets are quite small and won't greatly affect the bandwidth. One dropped packet might be happenstance, two a coincidence -- three dropped packets trigger a WLAN power cycle. Hopefully, this should waste no more than at most two seconds' time per cycle (instead of up to 90 seconds), and never trigger a power cycle unless there's a need.

It goes without saying that the PING functionality must be verified on the router - if the router doesn't answer pings, or drops ICMP when under TCP/UDP load (e.g. because of protocol prioritization), then the second version of the script could do far more harm than good.

But now that I think of it, did you try and fiddle with iwconfig parameters? For example the sensitivity threshold:

sens Set the sensitivity threshold. This define how sensitive is the card to poor operating conditions (low signal, interference). Positive values are assumed to be the raw value used by the hardware or a percentage, negative values are assumed to be dBm.

Depending on the hardware implementation, this parameter may control various functions.

On modern cards, this parameter usually control handover/roaming threshold, the lowest signal level for which the hardware remains associated with the current Access Point. When the signal level goes below this threshold the card starts looking for a new/better Access Point. Some cards may use the number of missed beacons to trigger this. For high density of Access Points, a higher threshold make sure the card is always associated with the best AP, for low density of APs, a lower threshold minimise the number of failed handoffs. On more ancient card this parameter usually controls the defer threshold, the lowest signal level for which the hardware considers the channel busy. Signal levels above this threshold make the hardware inhibits its own transmission whereas signals weaker than this are ignored and the hardware is free to transmit. This is usually strongly linked to the receive threshold, the lowest signal level for which the hardware attempts packet reception. Proper setting of these thresholds prevent the card to waste time on background noise while still receiving weak transmissions. Modern designs seems to control those thresholds automatically.

Example :

   iwconfig eth0 sens -80
   iwconfig eth0 sens 2
2
  • The connections for both devices are unstable on a regular basis. The router I'm using is very old, as well. An in-progress transfer will survive a Wi-Fi reconnect with the files fully intact (fully = checksums match,) but when the Wi-Fi dies, there's no actual error (on either host); it just stops working. So a simple shell script to disconnect+reconnect the Wi-Fi every 90 seconds would do. Jun 23, 2013 at 22:19
  • I've never met such a situation, but try with this simple script and see how it goes. I think there should be a way of making that network behave properly, or at least not so abysmally, though.
    – LSerni
    Jun 23, 2013 at 22:29
5
while ! rsync \
  --bwlimit <KB/s value> \
  -rP \
  /path/to/directory_that_contain_the_data_to_be_transferred \
  [email protected]:/path/to/target_directory ; \
  do sleep 90;
done

taken from https://patrick-nagel.net/blog/archives/434

You'd want to put this in a bash shell script. something like the following

#!/bin/bash

while ! rsync \
  --bwlimit <KB/s value> \
  -rP \
  /path/to/directory_that_contain_the_data_to_be_transferred \
  [email protected]:/path/to/target_directory; \
  do sleep 90;
done

you need to setup ssh keys which is another question if you don't know how to do that but it's basically:

ssh-keygen -t rsa

that will create your public/private key pair. Copy your public key to the destination machine so you don't have to use password credentials each time the connection drops.

2
  • o.0 So... do I run that as a shell script? Do I need to install an extra program for that? Will it work if the connection is too unstable to transfer a single file in one go? That was about as useful as a link-only answer... I really need more information. Jun 23, 2013 at 21:54
  • if you're on linux then you probably don't need to install any additional programs. All you'll need is rsync, ssh, ssh-keygen which are all usually installed by default.
    – ekeyser
    Jun 23, 2013 at 22:00
5

A lot of the other answers involve splitting the file into pieces and adding integrity checks - this is rather similar to how torrents work. Could you use something like rtorrent?

http://libtorrent.rakshasa.no/

2
1

Bittorrent Sync works across LAN. Current requirements (per manual) are:

Linux with kernel 2.6.16 (glibs 2.4) or newer on ARM/PPC/i386/x86_64

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .