Is there anyone else sharing your connection in the house? This problem is famously apparent when someone piece of crap in another room is downloading and hogging bandwidth, watching Netflix/streaming, etc.
If you play wirelessly, you can bet the problem will be less apparent with an ethernet cable.
https://www.youtube.com/watch?v=_oUp_JjPJok