I don't quite understand this to be honest. I started out as a builder and I know some builders are also mean vicious fighters.
It all depends on how you spend points, if you pump up pure strength + construction, then don't blame others when they come and duel you. I know one town's builder was dueled by a Miesta player, he went back, KO miesta player in retaliation, that mieta player then cried and Miesta declared war over the incident. That war is ongoing for several months now and the builder in question is still alive, kicking, doing quite well, building up towns, making money, and perhaps even dueling (I haven't spoken with him or that town's leadership about if he's still dueling)..
I have stayed in towns where they avoid dueling "workers" and towns that does not really care. During war, I KO at will, if not at war, I will try to rob people for their cash and/or lowering my motivation.
Dueling without KO, cash, exp, or motivation is kinda pointless really, it cost you energy and you gain exp from it, why waste it.