when the caravan arrives, I turn off all the jobs in my broker, that includes also the hauling jobs, and even cleaning. That leaves him to only do "bring item to depot" jobs, so I make sure I don't order my whole fortress to be carried to the trade depot, just enough for everybody to get to work, so my broker hauls a thing or two to the depot on his way there, after he's at the depot I order everything else I need to be hauled.
You can also try burrowing the broker in the depot.
They still need to eat, drink, sleep and take breaks, so it might be wise to have a replacement broker just in case.
If your broker is also your bookkeeper you may need to temporarily de-assign him from his study room, if he has nowhere to work at he'll cancel the "updating stockpile records" job immediately and do something else, bookkeepers seem to get stuck record keeping even when there's nothing to update, only stopping the job to eat, drink, sleep, and go on breaks.