Politics is just a fancy word for causing problems after all. But if you raid the castles i don't believer anything will happen to you which seriously affects the course of the game. However when you choose a side (dark or light) if you choose dark then everyone in the castles will h8 you but if you choose light nothing much changes I THINK. Hoowever it could b when you return someone to the other team e.g. Loren steel to queen Catherine the otherside will fall out with you in this case the elves. I personally do both quests and the Castles are the only places where the people do turn gainst you so it doesnt matter as they are unecessary later in the game (except for the Castel Navan which is needed for a certain dark path promotion.)
I just bite the bullet and get on with it. Their loss really as soon as they turn against me i make sure that every last one of them dies.