When you - your party - leaves by foot and returns to Harmondale next time - the only path to get in is to fight Goblins through.

I see it as follows - (I quote you and adjust your citation a LIttle) "they arrived in Erathia (the nearest port to Harmondale), inquired at the stables there for transportation and, finding that there was not service to Harmondale, the party hired a pathfinder who charged an exorbitant price - some 1000 or 1500 (because of robbers) - for a safe and secret passage to Harmondale town.

That explains why you cannot return to Harmondale without passing through goblins.

You need not change anything - only to insert a line about a greedy pathfinder - and IMHO it is enough. Oh, as a result you balance will be 1,000 or 1,500 gp less.

and, at the very first arrival