What are some options for transferring large files without using the Internet? [closed]

Question

My employer's Internet connection is extremely slow. It's $6,000 a month, 9 Mb, bonded T1, shared by over 150 users. It's not untypical to see regular file transfer speeds of 20-100 KBps. And that was after I convinced upper management to upgrade - before it was $3,000 a month for 3Mb. There are no other carriers in the area with better prices or speeds.

When I need to download a large file (which is frequently), I drive 40 miles back home, use my $60 a month 20 Mb FiOS connection, download the file in ten minutes, and drive back in less time than it would take me to just download the file at work (this is efficient for files greater than 500 megabytes, depending on the congestion of the network).

After more than a year doing this and more and more frequent downloads, I'm at my wits' end.

I am seriously contemplating raising carrier pigeons to fly back and forth between my home and office to transfer the files. Assuming a pigeon is carrying a 16 GB MicroSD card and can fly at a speed of about 50 miles per hour, the equivalent throughput for file transfer using a trained pigeon would be about 48 Mbps. Given the price of gas where I live and wear and tear on the car, that would probably be the cheapest option too.

What are some good, viable, alternative methods of transferring large files, rather than using my employer's Internet connection?

Answer 1

Here's a crazy idea.

If you're within 100 km or so of a populated area, use an ISP in town, set up a wireless network, and make a point-to-point link using a pair of woks. More generally, see long range Wi-Fi; apparently some guys have set up a 300 km, 12-to-48 Mb/s link, using 2 satellite dishes with no special amplification.

You could check if more conventional wireless broadband or power-line internet is available in your area.

Edit: Your user profile says you're in California, so wi-fi will probably only work if you're in the plains, or if at least one site is at high altitude (so that you have line-of-sight between the endpoints).

Answer 2

I'm surprised nobody here mentioned of using rsync. Yes, it still requires Internet access, but considering that the files are weekly software build then there must be lots of unchanged files that do not need to be downloaded and in the files that do change there might be lots of structural similarity where rdiff can do its magic. Your software vendor might even have rsync download server option, or they might be willing to set up one for you; or if they don't then it is relatively easy to setup rsync server at home.

Depending on the changes, rsync can easily reduce a 1GB download to 50MB download.

Answer 3

• Get reimbursed for the gas, or report the mileage on your taxes.

OR

• Are any "consumer" level internet connections available? They tend to be highly asymmetric in terms of transfer rates, so it might be worthwhile to have an inexpensive line for downloads.

Answer 4

Given the relative flight velocity between the two birds I would recommend the obvious upgrade to swallows and coconuts rather than pigeons and MicroSD cards. (How had nobody made that joke yet? Honestly, you can't softball it in like that and not be expected to have someone crank the sucker).

But seriously, an unlimited bandwidth 4g solution is probably the most obvious and convenient method. Correct me if I’m wrong but the only problems with that are:

1) No carrier presently offers such a plan (unless you happened to have an unlimited plan prior to Verizon or AT&T's elimination of those offerings);

2) Your post seems to indicate that you want a sans-internet solution, in which case 4g is not for you.

soandos has some helpful ideas, especially with starting the download via remote access so it is finished and ready for transfer by the time you get home. However, all of this seems remarkably after the though and reactive as opposed to proactive.

If you are aware of such large downloads, why don’t you queue them up at night before nappy time and then just bring them into work with you in the morning? It seems pretty unreasonable to think your company would be so impatient to insist on random +500MB downloads but not be willing to cooperate with you working on the files the following day (i.e. have your manager tell you what you need to be working on for the week/day/month BEFORE you come into work any given day)

That solution may seem kind of old school and certainly not as cool as a 4g wireless tether; but hey, capitalists are known for making due with high efficiency and the resources available. Want a promotion? Save your company $3000 a month in upgraded internet costs (what manager wouldn’t love that) and also help your company get a better grip on its POM (process and operations management), and I can almost give you my Cajun guarantee you will receive high marks.

Answer 5

Looking at you question, it seems to me that the problem isn't you having to drive home, but the fact that your T1 bonding isn't working correctly (I'm going to skip over the fact that you being reamed on the pricing) or that you have a QoS policy in place that is limiting your HTTP download speed.

100 KBps is 800 kbps which as a max speed on a line shared with 150 uses is actually quite good, for a single T1 (~1.5 mbps). Most providers/equipment only allow you to bond 4 T1's at once giving you a theoretical maximum bandwidth of 6 mbps (1.5*4), although if they sold you a 9 mbps line then you probably are lucky enough to have a provider that allows up to 8 bonds with a theoretical max of 12 mbps (1.5*8).

My suggestion would be to figure out why your bonding isn't working and fix that. (If you need help feel free to ask over at Server Fault).

Answer 6

Have you explored satellite and wireless mesh options - maybe there's one in your area if the local xDSL services are not too hot.

Answer 7

Obligatory xkcd:

Answer 8

I don't know what you are hoping for, but you have the two methods of getting the files: Download them, no matter how slow, or transport them in.

All I can say is be clairvoyant, and know what files you might need to download before you go to work, or start downloading common large files you use, like service packs and media, and burn them to disk. Make a list.

Also, if you are using Microsoft products, consider a Software Update Server for Windows Updates, and a proxy server to cache Internet traffic, to minimize your Internet access.

Lastly, make sure your firewall is locked down to block major streaming music sites and Facebook-type bandwidth wasters. And because streaming over HTTP is a bit harder block without a monitoring software, have management implement a strong policy regarding the Internet and bandwidth.

Answer 9

Is 4G service available around your work? That may be an easy way to add a relatively high-bandwidth internet-only connection.

If your bosses are antsy about having an unfiltered internet connection, use a sneakernet isolation system where the computer connected to the wireless is not on the domain and you have to use thumbdrives to get the files from that computer to your own. The costs of such a system could be easily balanced by the alternative of you driving home and back, especially if you start speaking forcefully to your bosses about their assumption you'll use your own economic resources to accomplish what is necessarily their responsibility to make sure you're capable of.

Answer 10

if it's for business purposes, and to/from specific places (customers, suppliers, etc), I've known companies to buy rugged external drives and use FedEx/UPS to ship data back & forth. You may be able to run some numbers and show your management that's actually cheaper than tying up your computer (and the rest of your co-worker's network).

Answer 11

As other have suggested, you have several options to set up a long range connection from an area with better rates to your area. You can do it using satellite dishes or microwave connections or even cable connections, you should really check the cost for those options.

While checking those options, consider this: what if you work together with other companies in the same district to invest in those options? They are probably facing the same issues your as company.
If everything works out you could become the "BRINGER OF INTERNET AT OK PRICES" (a title worse than "the bringer of doom" but a good title nonetheless).

Legal issues for sharing internet connections among multiple companies may apply.

Answer 12

Expanding on what @Zypher answered, if the problem is network congestion from the 150 users, and you can't schedule the download for off hours, then you should definitely look into QoS. You could prioritize your download traffic over all other network traffic.

You should look into exactly what is causing the problem. You may be able to fix the problem with some simple reconfiguration of your network.

Answer 13

I don't know of the pricing involve, but a microwave link could span the distance between your home (where internet access is cheap and fast) and to where your employer is (slow and expensive.) It's all line-of-sight, but you can get full T-1 speeds easily.

I'm not suggesting connecting your home to your business, but you may be able to set up a microwave link between a ISP in the area in which you live to your business.

At $6,000 a month, you would not have to go very long before reaching break-even. A quick search of "microwave link pricing" yielded this. Google shopping (not definitive, I know) shows the most expensive pricing to be about $28K, which has a break-even in about five months.

Let me know if you want more information, and I can contact some folks I know who maintain a rather extensive microwave link relay (secondary emergency data communications for first responders and such)

Answer 14

In a documentary on some SE Asian mudhole, motorcycle riders delivered email by having a wifi-equipped PC in the motorcycle trunk with a mobile whip wifi antenna and driving back and forth between urban and rural areas.

Computers along the route would recognize the availability of the signal to get off a few packets as the cyclist drives by. For short messages, this could be sufficient but big files require some waiting time.

At $10/hour ~ $1600/month, your company could enhance its internet service with a wireless N motorcyclist. For the cost of an extra used motorcycle he can switch cycles so that one is always exchanging data with the work site and one is always enroute.

Answer 15

There are really two problems here:

1. How do you get the file from the Internet to a device you own?
2. How do you get it to your employer?

It would be faster to start downloading it via remote access, so that it is ready to go when you get home. This still leaves the time it takes as total round trip time though, but seems to be the fastest you can do it without spending more money.

If you are willing to spend some money, there might be a 3G or 4G solution out there that you could use for this. Beware that there are generally caps, slow downs after a certain point, terms of use etc that are meant to stop people from using that much bandwidth.

Answer 16

You could download the file to a computer that you use remotely, via Windows Terminal Services, or VNC, or some other screen sharing technology. All you need is the bandwidth to remotely access the computer to do your work.

Multiple people in your office could log into the remote computer at the same time. In other words, figure out how much you could move to a remote server; just because you are in your office doesn't mean your computers need to be.

It could be a co-located machine at an ISP, or maybe even a virtual machine on Amazon's EC2.

Answer 17

Perhaps part of the problem isn't available bandwidth, but latency over long distance connections to India. I ran a studio in the US, using a post production facility in Mumbai, and while we had a fat pipe in our facility, and they had nearly as much bandwidth in theirs, FTP connections were horidly slow. After looking at a number of options - and paying through the nose to UPS in the meantime, killing our schedules - we switched from FTP to a file transfer accelerator product - we used FileCatalyst (don't mean to shill for one product), but there are several others and all were significantly faster than FTP on the same pipe, under the same conditions. All are UDP-based, have built in compression, validation, and reliable pause/resume.

Answer 18

Are there any areas nearby that have better internet options? Perhaps you could come to some arrangement with a business in that area to download your files for you, and they would be closer (and cheaper) than going home?

Answer 19

Plenty of ideas here, but I'll throw in my .02.

Write up an agreement with your employer and see if you can resell your FIOS connection (or make an arrangement to do so as a business customer.) Have fiber connected from your home to your employer.

Or find another business that will do this (share their higher-speed connection to your company).

I work for an organization that is spread out over the county in our state. To connect all the buildings we pay for fiber connections that run to each of the buildings, so we have one Internet connection and share it to all of our buildings bridged via fiber. High speed, and all we pay for is renting the previously dark fiber and the poles on which it's run. Depending on how far away you are, it may be cheaper than what you're paying and you'll get a significantly faster speed, but you'll have to run the numbers and check what it'll cost for you to get a business class connection that can be resold and you'll have to deal with the phone company...not usually a fun proposition. This may be a drastic option (and you'd need to take some precautions to separate this from your work life, since you don't know what the future will hold) but you'll have really fast speeds and you may be able to monitor systems at work from home. If that's a bonus. I don't know.

Answer 20

Can you get such a $60 a month 20 Mb FiOS connection at work too? For 6000$ you should get 100 of them.

Answer 21

Big question. So many votes! But, I'll chip in as well!

My solution is simply: remote desktop/other remote access application to your home computer. You could work with the files, read the files, and even print the files if you needed to. If you needed to have the files on the network for whatever reason, you could just do it at the end of the day, keep a log of what files need to be transferred and throw them on a portable storage. The only problems in that are:

1. It may be choppy working remotely working with your computer at home. The image might not be the resolution you are hoping for.
2. If you need the file on the network right now.

If that works for you, then awesome! Good luck. =)

Answer 22

What I would do is get one of those Verizon cards that gets internet through the cell network, and plug it into your LAN.
NS about price, but it's definetly cheaper than $6k

Answer 23

You can install a second asymmetric 20 Mbps connection only for you.

Can you use a download manager ?

Answer 24

1 . Ask your admin to make tomorrow downloads at night ( when bandwidth is free )

2 . if files are necessary for your job then stop working and ask for dedicated bandwidth

3 . if the bandwidth is shared ( not limited per user ) use download managers with as many possible connections

4 . Send link of this page to your boss

5 . i want FIOS too