Taking fortified houses.

[Bloody Bill]

Hey gents just need a little help on understanding something.  I am playing scenario "The First Battle Of Castiglione".   I am trying to take the Chateau objective for 50pts.  I have been hitting it with my battery of 7 8lbers and storming the place with the 5th,6th and 7th Converged Grenadiers...roughly 800 men.  They are fresh with fatigue of 6.  With shooting and assaulting the Chateau I have reduced the two skirmish units holding the Chateau are under 200 men in strength(180 men)  One of them is Routed and the other is Disordered.  I send in 400 Grenadiers and I lose 32 men the defenders lose 0????  I try this multiple times with the best result being my Grenadiers lose 12 and the defender 2.  I know have the defenders both routed and still cannot take the Chateau. 

My question is am I doing something wrong?  Can those places not fall?  Do I need to completely surround it and not just block the entry point?

Any help would be appreciated.

Logan