In the Exile series, clerics could cast priest spells while wearing heavy armor because their spells were spoken or chanted, whereas sorcerors could not wear armor because it would get in the way of their mage spells, which are done with arm motions and not chants. Maybe this is a common theme in games to balance out the two magic users?