I checked your profile and then I saw that you actually are in one of my towns.
You go and attack when I or any of the other "hats" tell you to do so. It's that simple. As a citizen of the Odd towns you also are a member of the Mutual Protection Association, so when we plan fort battles, attacks and defenses, our people go and do the job we ask them to do. No matter of their personal game preferences.
It's that simple. And yes, attacking is fun too, specially when we give a good battle and even win it.
Losing is no shame, you get a fast transportation home, and you can go back on jobs as soon as you have energy enough for it. No bad records anywhere. And it gives XP's as well...