191

My employer's Internet connection is extremely slow. It's $6,000 a month, 9 Mb, bonded T1, shared by over 150 users. It's not untypical to see regular file transfer speeds of 20-100 KBps. And that was after I convinced upper management to upgrade - before it was $3,000 a month for 3Mb. There are no other carriers in the area with better prices or speeds.

When I need to download a large file (which is frequently), I drive 40 miles back home, use my $60 a month 20 Mb FiOS connection, download the file in ten minutes, and drive back in less time than it would take me to just download the file at work (this is efficient for files greater than 500 megabytes, depending on the congestion of the network).

After more than a year doing this and more and more frequent downloads, I'm at my wits' end.

I am seriously contemplating raising carrier pigeons to fly back and forth between my home and office to transfer the files. Assuming a pigeon is carrying a 16 GB MicroSD card and can fly at a speed of about 50 miles per hour, the equivalent throughput for file transfer using a trained pigeon would be about 48 Mbps. Given the price of gas where I live and wear and tear on the car, that would probably be the cheapest option too.

What are some good, viable, alternative methods of transferring large files, rather than using my employer's Internet connection?

26
  • 123
    +1 for pigeons. IPoAC is the way to go. Aug 3, 2011 at 20:14
  • 23
    More seriously, $6000?! Are you in some remote location? I'm surprised that that's really the only ISP in your area. Aug 3, 2011 at 20:16
  • 10
    Have you spoken to your software vendor about getting the latest builds shipped to you overnight on a flash drive? A lot of vendors and resellers are prepared to make special arrangements, you'll probably get billed for it but dependant on distance and build frequency you may well find it comparable to your current costs.
    – Robb
    Aug 3, 2011 at 22:22
  • 76
    Your speed estimate is for African or European pigeons?
    – Diadistis
    Aug 4, 2011 at 0:15
  • 17
    You can easily parallelize pigeons without line installation costs!
    – moala
    Aug 4, 2011 at 9:09

24 Answers 24

130

Here's a crazy idea.

If you're within 100 km or so of a populated area, use an ISP in town, set up a wireless network, and make a point-to-point link using a pair of woks. More generally, see long range Wi-Fi; apparently some guys have set up a 300 km, 12-to-48 Mb/s link, using 2 satellite dishes with no special amplification.

You could check if more conventional wireless broadband or power-line internet is available in your area.

Edit: Your user profile says you're in California, so wi-fi will probably only work if you're in the plains, or if at least one site is at high altitude (so that you have line-of-sight between the endpoints).

10
  • 66
    This is badass.
    – benzado
    Aug 3, 2011 at 22:02
  • 2
    @snail: Huh? What does that (Chuck Norris router virus) have to do with it?
    – benzado
    Aug 3, 2011 at 22:13
  • 4
    This is a applicable in the real world and fairly common. Don't forget about the curvature of the earth in an implementation like this though. This is actually where a lot of the cost in this solution lays is the site survey.
    – Mike Soule
    Aug 4, 2011 at 4:16
  • 3
    I'll have to go next door to Boeing and ask if I can borrow a couple of their satellite dishes.
    – Kyle Lowry
    Aug 4, 2011 at 18:52
  • 1
    @Kyle Lowry: Boeing's next door? Rent a 747 and fill it with disks. It'll be cheaper than your current plan... Aug 4, 2011 at 19:24
44

I'm surprised nobody here mentioned of using rsync. Yes, it still requires Internet access, but considering that the files are weekly software build then there must be lots of unchanged files that do not need to be downloaded and in the files that do change there might be lots of structural similarity where rdiff can do its magic. Your software vendor might even have rsync download server option, or they might be willing to set up one for you; or if they don't then it is relatively easy to setup rsync server at home.

Depending on the changes, rsync can easily reduce a 1GB download to 50MB download.

5
  • This is an interesting option that someone mentioned to me before. I'm not sure if the vendor can offer us anything like that, but it's something I'll look into.
    – Kyle Lowry
    Aug 4, 2011 at 0:52
  • 23
    @Kyle ssh to your home computer, download the files and rsync from there. Aug 4, 2011 at 10:04
  • 1
    @Kyle rsync shouldn't be complicated to set up in any instalation. Aug 4, 2011 at 14:48
  • 3
    @wol perhaps it shouldn't be, but setting up an rsync server on Windows is complicated by any measure.
    – RomanSt
    Aug 4, 2011 at 19:37
  • 5
    Get an old machine/laptop and put linux on it then. Ubuntu server and rsync is super easy. Aug 4, 2011 at 21:41
39
  • Get reimbursed for the gas, or report the mileage on your taxes.

OR

  • Are any "consumer" level internet connections available? They tend to be highly asymmetric in terms of transfer rates, so it might be worthwhile to have an inexpensive line for downloads.
3
  • 3
    +1 We found a consumer option to addd a second link and cost was much lower than our 10Mbps link. Works great for downloads though.
    – Dave M
    Aug 3, 2011 at 20:55
  • 5
    Unless you're relying on a WAN and need super high availability, I'd even downgrade back to the 3Mbps on the business-class link and use a few consumer services with a load balancer to keep things moving and to handle connection faults. Net savings in the $thousand each month plus thicker pipe? What's not to like? Aug 3, 2011 at 21:32
  • 3
    I've often wondered why more people don't do this. Sure, load balancing makes things much more complicated, but the money saving is almost already someone's salary. Better to just hire another guy to manage your multi-ISP network. Many companies operating using a proxy server for "security" reasons. If you already have this in place, the multi-ISP stuff is easy as pie.
    – user606723
    Aug 4, 2011 at 15:02
28

Given the relative flight velocity between the two birds I would recommend the obvious upgrade to swallows and coconuts rather than pigeons and MicroSD cards. (How had nobody made that joke yet? Honestly, you can't softball it in like that and not be expected to have someone crank the sucker).

But seriously, an unlimited bandwidth 4g solution is probably the most obvious and convenient method. Correct me if I’m wrong but the only problems with that are:

1) No carrier presently offers such a plan (unless you happened to have an unlimited plan prior to Verizon or AT&T's elimination of those offerings);

2) Your post seems to indicate that you want a sans-internet solution, in which case 4g is not for you.

soandos has some helpful ideas, especially with starting the download via remote access so it is finished and ready for transfer by the time you get home. However, all of this seems remarkably after the though and reactive as opposed to proactive.

If you are aware of such large downloads, why don’t you queue them up at night before nappy time and then just bring them into work with you in the morning? It seems pretty unreasonable to think your company would be so impatient to insist on random +500MB downloads but not be willing to cooperate with you working on the files the following day (i.e. have your manager tell you what you need to be working on for the week/day/month BEFORE you come into work any given day)

That solution may seem kind of old school and certainly not as cool as a 4g wireless tether; but hey, capitalists are known for making due with high efficiency and the resources available. Want a promotion? Save your company $3000 a month in upgraded internet costs (what manager wouldn’t love that) and also help your company get a better grip on its POM (process and operations management), and I can almost give you my Cajun guarantee you will receive high marks.

4
  • The reason I can't queue up the files ahead of time is because they are mostly weekly builds from a software development vendor. They are frequently late, which means by the time the builds are ready for testing, we're already behind schedule and I have to rush to get them transferred to the office server. I'll look into the 4G option and other alternatives. Thanks for the suggestions. Swallows might be a bit out of my price range, though.
    – Kyle Lowry
    Aug 3, 2011 at 21:08
  • 11
    I hear African swallows are cheaper than many other types. Aug 3, 2011 at 21:33
  • "No carrier presently offers such a plan": This is true for consumer-level plans, but might they offer unlimited data to businesses? Aug 3, 2011 at 22:05
  • 4
    There was a well publicized stunt here in South Africa a few years ago where a carrier pigeon was pitted against an ADSL connection provided by the incumbent telco. The pigeon won, by far. It was of course just a publicity stunt, but nevertheless it goes to show that TCP-over-carrier-pigeon is faster here than standard DSL. Link to article: news24.com/SciTech/News/Pigeon-beats-Telkom-20090909 Aug 3, 2011 at 23:01
27

Looking at you question, it seems to me that the problem isn't you having to drive home, but the fact that your T1 bonding isn't working correctly (I'm going to skip over the fact that you being reamed on the pricing) or that you have a QoS policy in place that is limiting your HTTP download speed.

100 KBps is 800 kbps which as a max speed on a line shared with 150 uses is actually quite good, for a single T1 (~1.5 mbps). Most providers/equipment only allow you to bond 4 T1's at once giving you a theoretical maximum bandwidth of 6 mbps (1.5*4), although if they sold you a 9 mbps line then you probably are lucky enough to have a provider that allows up to 8 bonds with a theoretical max of 12 mbps (1.5*8).

My suggestion would be to figure out why your bonding isn't working and fix that. (If you need help feel free to ask over at Server Fault).

3
  • 10
    ... and if the answer is that the problem is everyone else in the office listening to Pandora, etc., then the solution is to set up QoS—prioritize business-critical traffic.
    – derobert
    Aug 4, 2011 at 17:50
  • 1
    that's exactly the answer: some traffic shaping / prioritization. Dead easy to do with squid. You're using a proxy cache, right?
    – Javier
    Aug 7, 2011 at 0:56
  • Alternatively: Use IPoAC with QoS. Sep 6, 2011 at 3:55
19

Have you explored satellite and wireless mesh options - maybe there's one in your area if the local xDSL services are not too hot.

1
  • 9
    +1 for satellite: horrible latency but decent throughput is fine for large files. Aug 3, 2011 at 20:23
14

Obligatory xkcd:

Every time you email a file to yourself so you can pull it up on your friend's laptop, Tim Berners-Lee sheds a single tear.

12

I don't know what you are hoping for, but you have the two methods of getting the files: Download them, no matter how slow, or transport them in.

All I can say is be clairvoyant, and know what files you might need to download before you go to work, or start downloading common large files you use, like service packs and media, and burn them to disk. Make a list.

Also, if you are using Microsoft products, consider a Software Update Server for Windows Updates, and a proxy server to cache Internet traffic, to minimize your Internet access.

Lastly, make sure your firewall is locked down to block major streaming music sites and Facebook-type bandwidth wasters. And because streaming over HTTP is a bit harder block without a monitoring software, have management implement a strong policy regarding the Internet and bandwidth.

3
  • The difficulty for me is that a lot of these files are from software development service vendors - when they finish the latest build, they upload it to their FTP, and I need to download it. If they stuck to their schedule, then I would be able to predict things ahead of time, and come in to work late that day so I could download the file at home and bring it in with me. So it's hard to predict when I'm going to need to download something.
    – Kyle Lowry
    Aug 3, 2011 at 20:17
  • @Kyle Lowry I am just not sure how you can get around that: Download or drive it (no need to consider mail: You could get it yourself that night anyway)
    – KCotreau
    Aug 3, 2011 at 20:39
  • 3
    +1 for blocking bandwidth-wasting sites - if the work internet connection is struggling, it should be restricted to work-only purposes.
    – jwaddell
    Aug 3, 2011 at 23:13
12

Is 4G service available around your work? That may be an easy way to add a relatively high-bandwidth internet-only connection.

If your bosses are antsy about having an unfiltered internet connection, use a sneakernet isolation system where the computer connected to the wireless is not on the domain and you have to use thumbdrives to get the files from that computer to your own. The costs of such a system could be easily balanced by the alternative of you driving home and back, especially if you start speaking forcefully to your bosses about their assumption you'll use your own economic resources to accomplish what is necessarily their responsibility to make sure you're capable of.

1
  • +1 One of our team found this great when at a client site. They gave him Internet but slowed to a crawl. IT on site apprve the 4G device and he is fully connected. If there is service it would work. There may even be laod balancers that would let you use teh 4G device all the time
    – Dave M
    Aug 3, 2011 at 20:59
6

if it's for business purposes, and to/from specific places (customers, suppliers, etc), I've known companies to buy rugged external drives and use FedEx/UPS to ship data back & forth. You may be able to run some numbers and show your management that's actually cheaper than tying up your computer (and the rest of your co-worker's network).

3
  • 1
    Yeah, but unless the OP has someone else at home to get the file onto some sort of portable media, who's going to hand the drive to the FedEx/UPS at his house?
    – afrazier
    Aug 3, 2011 at 20:10
  • 1
    I was meaning to take the trip home completely out of the process. The companies I was referring to, for example, had to transfer large video files to/from customers. The files were large enough that, no matter how fast their network was, shipping a drive from their office to their clients' was always faster.
    – StevenV
    Aug 3, 2011 at 20:28
  • We've done that for huge chunks of highly sensitive data. Use an encrypted drive that requires a seperate USB key for access. Mail the USB key and drive in two seperate packages, or keep an access key at both ends. Aug 3, 2011 at 21:35
6

As other have suggested, you have several options to set up a long range connection from an area with better rates to your area. You can do it using satellite dishes or microwave connections or even cable connections, you should really check the cost for those options.

While checking those options, consider this: what if you work together with other companies in the same district to invest in those options? They are probably facing the same issues your as company.
If everything works out you could become the "BRINGER OF INTERNET AT OK PRICES" (a title worse than "the bringer of doom" but a good title nonetheless).

Legal issues for sharing internet connections among multiple companies may apply.

5

Expanding on what @Zypher answered, if the problem is network congestion from the 150 users, and you can't schedule the download for off hours, then you should definitely look into QoS. You could prioritize your download traffic over all other network traffic.

You should look into exactly what is causing the problem. You may be able to fix the problem with some simple reconfiguration of your network.

4

I don't know of the pricing involve, but a microwave link could span the distance between your home (where internet access is cheap and fast) and to where your employer is (slow and expensive.) It's all line-of-sight, but you can get full T-1 speeds easily.

I'm not suggesting connecting your home to your business, but you may be able to set up a microwave link between a ISP in the area in which you live to your business.

At $6,000 a month, you would not have to go very long before reaching break-even. A quick search of "microwave link pricing" yielded this. Google shopping (not definitive, I know) shows the most expensive pricing to be about $28K, which has a break-even in about five months.

Let me know if you want more information, and I can contact some folks I know who maintain a rather extensive microwave link relay (secondary emergency data communications for first responders and such)

4

In a documentary on some SE Asian mudhole, motorcycle riders delivered email by having a wifi-equipped PC in the motorcycle trunk with a mobile whip wifi antenna and driving back and forth between urban and rural areas.

Computers along the route would recognize the availability of the signal to get off a few packets as the cyclist drives by. For short messages, this could be sufficient but big files require some waiting time.

At $10/hour ~ $1600/month, your company could enhance its internet service with a wireless N motorcyclist. For the cost of an extra used motorcycle he can switch cycles so that one is always exchanging data with the work site and one is always enroute.

3
  • 1
    Hehe... the "SE Asian mudhole" you are referring to is Cambodia. They did it here some years ago.
    – snap
    Aug 4, 2011 at 14:28
  • Well, I don't recall the country, but the documentary featured lots of mud. I worked in Hong Kong for a while and never saw a dirt road. Even the hiking trails were paved, and some had nice brick staircases (e.g. east of Sai Kung).
    – Paul
    Aug 5, 2011 at 3:38
  • Yeah, many muddy dirt trails here. Great fun to drive with a dirt bike! Not boring like sterile places such as Hong Kong and the west. :)
    – snap
    Aug 5, 2011 at 4:21
4

There are really two problems here:

  1. How do you get the file from the Internet to a device you own?
  2. How do you get it to your employer?

It would be faster to start downloading it via remote access, so that it is ready to go when you get home. This still leaves the time it takes as total round trip time though, but seems to be the fastest you can do it without spending more money.

If you are willing to spend some money, there might be a 3G or 4G solution out there that you could use for this. Beware that there are generally caps, slow downs after a certain point, terms of use etc that are meant to stop people from using that much bandwidth.

7
  • 1
    Yeah - I have remote access set up. So I remote into a computer at home, and start the download before I leave work. In most cases, the file is done downloading before I even get past the surface streets and onto the freeway.
    – Kyle Lowry
    Aug 3, 2011 at 20:09
  • 1
    @Kyle, So then rsync from home to work.
    – TRiG
    Aug 4, 2011 at 10:03
  • 1
    @TRiG - That's such an obvious solution, I never even considered it before. I'll do some tests and see how that works out. fingers crossed
    – Kyle Lowry
    Aug 4, 2011 at 18:58
  • @TRiG - I've just tested it, and the transfer speeds are still too slow. So it looks like rsync isn't viable.
    – Kyle Lowry
    Aug 4, 2011 at 20:05
  • @Kyle. Ah well. Worth trying.
    – TRiG
    Aug 4, 2011 at 20:33
3

You could download the file to a computer that you use remotely, via Windows Terminal Services, or VNC, or some other screen sharing technology. All you need is the bandwidth to remotely access the computer to do your work.

Multiple people in your office could log into the remote computer at the same time. In other words, figure out how much you could move to a remote server; just because you are in your office doesn't mean your computers need to be.

It could be a co-located machine at an ISP, or maybe even a virtual machine on Amazon's EC2.

3

Perhaps part of the problem isn't available bandwidth, but latency over long distance connections to India. I ran a studio in the US, using a post production facility in Mumbai, and while we had a fat pipe in our facility, and they had nearly as much bandwidth in theirs, FTP connections were horidly slow. After looking at a number of options - and paying through the nose to UPS in the meantime, killing our schedules - we switched from FTP to a file transfer accelerator product - we used FileCatalyst (don't mean to shill for one product), but there are several others and all were significantly faster than FTP on the same pipe, under the same conditions. All are UDP-based, have built in compression, validation, and reliable pause/resume.

2
  • I think there are a number of factors involved, latency being one of them. I can try downloading these files at work, and it would take 15 hours. I do it at home, and they are done in 20-30 minutes. Often downloads at the office would just fail, drop down to 1 KBps, or end up corrupted. I have never had any problem like that with my home connection.
    – Kyle Lowry
    Aug 4, 2011 at 0:55
  • Sounds not that different from the problems I had - again, don't want to shill for any one product, but demo a few of the file transfer accelerators. I've used File Catalyst - reasonably cheap - and Aspera and both made a huge difference over long haul, high latency links, especially compared with ftp and http. Can you get high speeds off your office connection when downloading from a nearby high speed file repository - like for instance the closest Sourceforge mirror? Aug 5, 2011 at 0:04
3

Are there any areas nearby that have better internet options? Perhaps you could come to some arrangement with a business in that area to download your files for you, and they would be closer (and cheaper) than going home?

3

Plenty of ideas here, but I'll throw in my .02.

Write up an agreement with your employer and see if you can resell your FIOS connection (or make an arrangement to do so as a business customer.) Have fiber connected from your home to your employer.

Or find another business that will do this (share their higher-speed connection to your company).

I work for an organization that is spread out over the county in our state. To connect all the buildings we pay for fiber connections that run to each of the buildings, so we have one Internet connection and share it to all of our buildings bridged via fiber. High speed, and all we pay for is renting the previously dark fiber and the poles on which it's run. Depending on how far away you are, it may be cheaper than what you're paying and you'll get a significantly faster speed, but you'll have to run the numbers and check what it'll cost for you to get a business class connection that can be resold and you'll have to deal with the phone company...not usually a fun proposition. This may be a drastic option (and you'd need to take some precautions to separate this from your work life, since you don't know what the future will hold) but you'll have really fast speeds and you may be able to monitor systems at work from home. If that's a bonus. I don't know.

1

Can you get such a $60 a month 20 Mb FiOS connection at work too? For 6000$ you should get 100 of them.

2
  • 1
    Unfortunately Verizon doesn't service this area.
    – Kyle Lowry
    Aug 4, 2011 at 18:56
  • 1
    Just use 3g/4g hotspot device. Or if you have a smart phone, use that (Then write it off as a work related expense). Aug 5, 2011 at 23:23
1

Big question. So many votes! But, I'll chip in as well!

My solution is simply: remote desktop/other remote access application to your home computer. You could work with the files, read the files, and even print the files if you needed to. If you needed to have the files on the network for whatever reason, you could just do it at the end of the day, keep a log of what files need to be transferred and throw them on a portable storage. The only problems in that are:

  1. It may be choppy working remotely working with your computer at home. The image might not be the resolution you are hoping for.
  2. If you need the file on the network right now.

If that works for you, then awesome! Good luck. =)

0

What I would do is get one of those Verizon cards that gets internet through the cell network, and plug it into your LAN.
NS about price, but it's definetly cheaper than $6k

0

You can install a second asymmetric 20 Mbps connection only for you.

Can you use a download manager ?

0

1 . Ask your admin to make tomorrow downloads at night ( when bandwidth is free )

2 . if files are necessary for your job then stop working and ask for dedicated bandwidth

3 . if the bandwidth is shared ( not limited per user ) use download managers with as many possible connections

4 . Send link of this page to your boss

5 . i want FIOS too

Not the answer you're looking for? Browse other questions tagged .