If you are trying to do the quest "clear castle Harmondale of goblins" then it refers to the large building which was the first thing you saw when you first entered Harmondale.

There is also a "fort" out in the open that has been occupied by goblins and they are shooting fireballs, sparks and a few other things at anybody who is approaching, but there is no quest connected to it.