Hovels are fragile, but crucial buildings, which increase maximum housing capacity.
As long as there is free capacity, peasants will slowly inpopulate the castle, based on popularity. If overall popularity is above 50, peasants will keep coming, but under that, they will leave the castle.
If not all peasants can be accomodated, overpopulation occurs, which dampens recruitment and peasant reinforcement until enough housing is provided (sometimes accompanied with negative popularity). The starting keep has a limited capacity of 8 to 12 peasants.
Hovels are a priority to be protected from enemy attacks and fire.