Why should dueling, or showing no interest in dueling, have any affect on fort fights or adventures? Would you vote if it was reversed, and those who don't play adventures get penalized in forts and duels and get lower drop rates on jobs?
why? simply because that's the game design that I like. take risk to make an extra buck, or settle for less without taking risks.
if you decide that you don't want to take any risks (turn off duels), you'd be penalized in other parts of the game. kind of like giving a part of your income to pay for insurance or paying for protection
if you decide not to participate in adventures, you're not avoiding risks (you don't lose HP or energy or cash if you lose an adventure), you're just not trying to get the rewards connected with adventures and that's a penalty in itself IMHO.
FFs are similar to duels (because you can lose energy, hp, cash), but also different. I would vote for something like this - if you don't participate in FFs (e.g. you need to join a certain percent of FFs in the last 3 months), you cannot sleep in them, your town loses points etc. or some other penalties I can't think of right now
I've chosen not to participate in adventure - I don't get the items, achievements, xp, no extra event currency (flowers/hearts/rockets/etc.) = I am penalized in a way.
I don't do FFs - I get no experience, bonds, event currency (but I use alliance forts that others fought for, which I don't really like) = I am penalized in a way
I don't do duels (situation today) - I need to put in an extra effort to get KOed and earn 72h of peace or I go townless = I am penalized for not participating.
I don't do duels (if it was enabled to turn them off) - I walk around without worries, using max clothes bonus for cash/experience/luck/product drop, I don't need to worry about losing HP before a FF etc. = I am actually rewarded for not participating.
if turning duels off was implemented, it would be the only category in this game where you would actually get rewarded for not participating. the only downside I can think of right now - you wouldn't get the achievements (or you would do that, you'd duel people, and then turn the dueling option off).