First, let me state that I would never duel a worker without a good reason for it.
But I'm wondering what is the actual problem of attacking a builder? If he loses the duel, what will happen? Nothing! He's only losing maybe a few dollars, but since he's member of a town he can store it in the bank, for example everyday before he will visit his hotel, so I don't see the problem here.
What else is the matter with attacking a builder? Will he lose else than a few dollars? As far as I can see not.
So what about morals? When you attack a worker it is worse than attacking an adventurer? Isn't this whole game about the Wild West, and isn't it normal that bandits will actually attack the workers for easy money?
Yeah, of course in the Wild West you have sheriffs who will defend the hard working people and try to catch the bandits and put them in jail, but for some reason the game developers didn't invented neither any sheriffs nor jails.
By the way, who says a worker can't defend himself? We can not see from the outside if he's a good dueler or not, can we?
Who's saying attacking a worker is weak...? Is chosing the character worker and putting all your skill points into dueling skills also weak?
Is chosing an adventurer and putting all your points into dueling skills weak?
What if I choose a soldier and put all my points into construction skills. Is everyone attacking me now very weak and deserves to be punished? Of course not; as said you can't see from the outside who/what's inside.