You know what would stop workers/building from whining? It's not that they get attacked, although it is a bit cowardly for a gunslinger to attack a worker, even in the wild west there was a certain code of consuct, unless you were a coward, but it's the fact that a duel is meant to be a duel and therefore the defender is notified they're duelled before the duel.
If there was a feature whichb allowed you to allocate your defending clothes and get up and when a duel was instigated you automatically changed into these clothes to at least give you a fighting chance, then that would be fairer and stop the complaining.
Of course, the duelling element could have it's name changed to 'shooting fish in a barrel' and simply stay as it is, because at the moment it isn't really a duel.
After the fight, the worker puts his previous clothes back on.
Basically, without this fature then I, with my worker character, will call every soldier/dueller with a duel level above me who shoots me in the ars* whilst i'm banging nails into a plank of wood, a coward, and they are