I am an adventurer on more than one world, and gets duelled more on some of the worlds than one others. I don't have a problem with that, and thinks it is fair. I chose what I wanted to do, with the risks associated with it. Duellers chose their profession, and lose out on the other benefits I chose to have (they don't even get any benefit in terms of winning a duel). Regardless of what class you choose, you can choose to be a dueller in your own right by increasing the correct skills. It is a thing of choice - nothing stopped anybody from making the same choice.
Regarding the argument that people want a "balanced" game - from the number of people moaning about being duelled, I would say there are enough people that still chooses to spend their skills on other items, everybody has not decided to be duellers now. To be balanced, we need the duellers as well, and good for them for focusing their skills on being good at duelling. A lot of people complain that the game is boring - can you think how boring it would be if there was no chance of losing your hard-earned cash? I think it is a fairly well balanced game (except I think builders could get more XP points for construction, which probably belongs in a different topic).
The only change that I would like to see, is that the low limit of level difference for somebody to challenge me, be raised somewhat. Sometimes I get clobbered by guys that are WAY stronger and higher level than I am, sometimes so much that I am not even allowed to duel them back (since it appears it is calculated by percentage differences from your own level).
PS:
By the way - allowing duellers to only duel other duellers would really have no point, would it? There would be virtually no cash in duelling / no monetary reward, since none of them would be able to get money from other types initially... that idea is dead.