Have dedicated stockpiles near to where stuffs NEED to be, to shorten active hauling time for work. Turn hauling off for the crafters that you need to be cracking and churning, to keep them from wasting time. Use cages for invaders, instead, less mess that way, and you can selectively dump things, less hauling work, and you can quantum-dump right next to trading deport.
Otherwise, it's pretty much matter of sheer number of haulers. If you don't have enough, then it'll take forever to do anything anyway

But the first thing is to have a team of dedicated haulers. People with useful skills shouldn't haul anything. Leave that for the peasants.
Is there an option to turn off the Bring Item to Depot, though? It seems every time I have a ton of items to be brought, everyone right down to the armorsmith in the basement comes up to bring XX(troll fur socks)XX to the depot.
Nope, not that I can see, that's what doors to workshops and it's stockpile are for, lock the doors while deport hauling is going on. I have a dedicated workshop area with it's own food and drink stockpile, and doors or hatches that can be locked down to keep the workers working while pond zone, animal caging/restrainting or trade deport hauling is ongoing.