Wireless by nature is very prone to interference. The fact that your downloads speed up when using chrome is likely a red herring.
Because wireless is so fluid, you should only consider speed tests reliable when connected over ethernet and the wireless is turned off.
Here is why wireless fluctuates so much:
Dynamic Rate Switching (DRS)
Wireless routers embody a technology called Dynamic Rate Switching. In the simplest sense, your laptop and access point 'negotiate' a safe speed they can both communicate at, depending on how strong the signal is.
You can see what rate your Macbook is transmitting at by holding the 'option' key while clicking on the wireless icon in your menu bar.
In my example, I have an RSSI of -55db (-45 to -70 or so are considered good, -80 to -90 are considered weak).
I have a transmit rate of 54 Mbps. You can see the full list of possible wireless speeds in the MCS Charts
It is possible that you are also in an environment where you are seeing a lot of interference from other wireless networks. The ISM band only has 3 non overlapping channels. If you see overlap in a program like wifi explorer, then it is likely your wireless will fluctuate.
In other words, never count on wireless to give you consistent speeds. You should plug directly into the ethernet. My money says that the problem will go away.