I defeated him on the second try. I'm a builder (not pure, I have a few points in other skills), used the feather hat, yellow checkered shirt, pilgrim shoes, bone necklace and crowbar. Looked where Mr Brown aims, and dodged accordingly.
As a pure builder who has not even the skill points to do the Wappee grandfather quests for respecing... What are my options? Start leveling to get skills into dueling?
found out now to answer my question for others.... he doesnt appear Automatically.....u don't have to actually DO anything but read the quest info and all is explained.
Drunk vs Sober is determined by what time you visit the questgiver and is explained in the in game instructions.
After accepting the quest you'll only be able to duel him during the same time of day each day (2h window)
If you go for the Sober one he is very very tough to beat as a pure builder. It appears he uses a ranged weapon (i.e. you need high reflexes to minimize the damages -- I couldn't beat him until I was able to equip a conquistador's sword) He has very high aim so he'll hit you a lot no matter what way you dodge until you equip a lot of dodging equipment.