Once you have a fief, there are a few things to remember.
Owning a fief allows you to collect taxes from the populace every week (taxes accumulate, so you don't have to visit every week). In Warband, you automatically receive taxes and do not have to visit to collect them. Towns earn the most base taxes, villages second, and castles the least. The prosperity of each fief also affects the amount of taxes they produce. The prosperity of a castle is affected by the prosperity of the village that is geographically attached to that castle, even if the village is not owned. Usually it is the case that the the player will not own a connected castle and village, unless they have large numbers of both villages and castles.
You can raise the prosperity of a town by making sure that its caravans reach their destinations, and by completing quests from its Guildmaster. For villages, stop it from being raided, kill bandits if they invade, build improvements, and get quests from its Village Elder. Also, when repeatedly purchasing imported goods from a town or village, the prosperity will eventually drop due to the lack of these goods, and they will no longer be available until trade has returned them, which can take a rather long time. For castles, the same methods can be employed on adjoining villages, but improvements cannot be built unless the adjoining village is owned. The profitability (net income) of a castle can be improved by reducing the size of the garrison.
Once a village belongs to you or to your faction you can't loot and burn it, although you can still force the peasants to give you supplies. If you are already disliked by a village that becomes your fief, the villagers will remember and hate you, but you can still collect your taxes as usual. However, you will be unable to get recruits from this village. You can see your reputation in brackets in the description at the top along with a word describing how much they like or hate you, for example "acceptive", "resentful", "hate you with a passion", etc.
If you intend to take part in sieges to earn further fiefs for yourself, you should avoid raiding nearby villages so that they will still like you when you own them. Burning villages also decreases their prosperity, which affects the taxes and recruits you can collect, so it is a good idea to make sure any villages you may come to own are in as good a condition as possible.
The taxes a fief generates are linked to its wealth, ranging from very poor to very rich. Improving a fief's wealth increases taxes and the amount of recruits you can obtain there. Improving a fief's wealth is no easy task; once it is looted, the wealth will drop back to very poor for a rather long time. Improving the wealth is done by a range of actions.
Build a Mill: this is the most important thing you can do. A mill will improve productivity in your villages and their cities.
Increasing reputation: this represents how happy the residents are with your rule. When your people hate you they will leave first chance they get and the village will not grow beyond average, so you must improve the reputation to 0 or higher to gain any real wealth in your fiefs.
Patrolling: villagers need to go to a nearby town to obtain new resources and to sell their own surplus. Patrolling your terrain to make sure your villagers are not attacked and enslaved along the route is needed if you wish to improve the fief's wealth, which can pose a problem to lords who have a wide range of fiefs.
Trade: if the town the villagers trade in is poor, prices will be high, resulting in less affordable goods. Your villagers notice this too, as the wealth of the town largely influences the wealth of the villages and vice versa. A town with two poor villages and one average will tend to become poor after some time, due to the lack of production in the villages, and therefore lack of production in the town itself. The town will increase back to average if the villages near it are at average too, as this allows the town to produce more trade goods which will attract traders who will also bring new goods. After some time this will increase the wealth of the town and give your village a chance to increase its own wealth, as it is directly linked to the town.
To successfully improve the wealth of your fiefs, you should do all of the above, and for an extended amount of time. Depending on your own goals it might not be worth the effort to increase the wealth, as it takes a long time to actually do so and if you stop patrolling the region for a short amount of time the bandits will return and rob your villagers blind.
(from The Ultimate Warband Guide, http://steamcommunity.com/sharedfiles/filedetails/?id=121687291)