I played for a very long time without premium, and did quite well. Had a great dueling record, built up a strong town, and had a great time playing this game. To be honest, now that I have premium services, the game is a bit more fun, but I would hardly call it unfair, or even unbalancing.
There are games where paying for services does pose an unfair advantage, like games where you can purchase all your stats with straight out cash. What's the point in playing a game like that if the person who has the deepest pockets will always be kicking everyone's ass? Anyway, this game is not one of them, at least not yet. And, if it ever does go that route, which I truly hope it does not, I'll be one of the first to find the exit, mainly because it will be an indication the developers are trying to get their last bucks before they close the servers.
As it is, the game is well balanced and premium provides nice advantages, but at a pocket price that is reasonable, and with advantages that do not imbalance gameplay for the non-paying clients.
So, in answer to the OP's question, no... it is not unfair.