It happened to me just once and I sat and thought why?? what did I do different...as I have played this game...SEVERAL times..The occasion I hit this problem I went off to check a few places around Frosgard before I entered the city..
I even poked nose into the castle...but had second thoughts and after clearing the entrance..went back and wandered into the City..did the ice clearing..and went off somewhere else by boat.
They next time I tried to enter the city...CRASH...delete mini saves..
If I was to throw up a guess I would say it is something to do with the teleportal into the castle. Doing it before the Game expects one to do it..