My worker always had the best defensive dueling gear he should get, even when he was still a pure builder, and it helped a lot. I've also almost always gotten the best weapon for my level as soon as I got there or the level before, the one exception being a sharp foil, because my town didn't have it and I didn't feel like going shopping. As Azuk said, weapon alone won't really help you: buy the gear too.
Switch to defensive dueling gear as soon as your job is scheduled whenever you can, and you won't be completely defenseless.