Although your advice is a little late now. I think I confused the game by taking both options when some guy in the prison asked me to rescue his brother. First I said no, and he called the guards, so then I said yes.But the guards seem to have gotten a bit confused. Most of them after that attacked me, but only once, landing a blow and then leaving me alone. But the three guys at the entrance, who I originally got a pass from, wanted to fight to the death when I went back there. So I had to kill them, and I guess I'm not in the guards' good books after doing that 
Nice to know it won't affect the game outcome tho, thanks
