The Supreme Temple of Baa was easier than I had expected. And that was without just speedrunning directly for the memory crystal.
But wow, once you know the password, and find the way to the memory crystal in Castle Alamos, it was a lot easier than I remembered it. I remember having to clear out rooms full of warlocks... but I guess I will save them for later. I just killed a few low-level enemies standing in my way, and finding my way quickly (taking the left corridor, and from the circular room, just straight to the first path to the right, then left).
But it seems Castle Darkmoor will be quite a challenge.
Though I'm pretty sure you can speedrun it as well, I'm still clearing some eyes, one-by-one inside of the huge cave that you get to by the first cube-thingy.