If you do not have this while transferring locally, then it sounds like you are experiencing throttling by your ISP to keep your speeds at what you paid for. What is likely happening is that it only happens when you have a really fast and sustained connection; the transfer gets faster but then your ISP kicks in to prevent you from getting faster than the service you subscribed to allows.
I’d bet that if you averaged the peaks and troughs out, you would find that the overall average is right around what your Internet connection speed is supposed to be. In other words, instead of keep the speed constant, the ISP is letting it spike, but then accommodates by dropping it. If this is how it is at all times of day and all days of the week, then it means that your ISP has poor network management (though ultimately you are still getting the speed you paid for). If it only happens during “peak hours”, then what might be happening is that your ISP is providing you with a “boost” where they actually let you go higher than than your service provides so long as the network has spare bandwidth.