I tried that briefly; being a top builder with 300+ construction doesn't help though, so I switched back.
Ideally if you want to really be an effective worker converting to a dueler, I'd talk to johnnygunn on world 3. He was a pure builder, but after he found the piece of note 1 (for Hernando's Sword), he's transformed into an effective dueler. Considering he was at the bottom of the dueling rankings and is now 14-15, I think he must've done something right.
Another person you might want to ask is mekolp, on world 3 again; but mekolp has superhuman amounts of patience and vision. He started switching to a dueler several months ago, pretty much right after our second town maxed.
He did it one point a day, which, well, takes a lot more patience than I know I could ever have