The only thing that greatly affects pixellation, in my experience, is the "Use D3D" key. If its value is 1, the game uses hardware acceleration, which allows blood splats, colored lights and tinting of creatures. It also blurs the pixels in a sprite together, when you're close enough for it to look blocky otherwise. (This may be the effect you're looking for.) It makes some of your spells (and blaster shots) have pretty lighting effects, and accurately draws the area of a fireball.

Unfortunately, enabling D3D screws with targeting. When it is turned on, you can't target a creature through the gaps in a sprite (e.g., between branches of a tree, or above a closer monster's arm). This is a big pain when you're trying to land an area effect spell in the middle of a group of monsters, or attack a particular, partly concealed creature.

Also annoying is that the game's normal color coding of similar types of creatures doesn't happen when D3D is enabled. Tinting is an ugly, poor substitute in my opinion.

Despite all this, I like the "colored lights" option so much that I sometimes leave D3D on.

I haven't managed to otherwise improve the graphics. (I got the same error, fiddling with the "resolution" key in Windows 98.) But others here have more experience than I; they likely know something I don't. (Assuming they're still around. I almost never check in here anymore.)

BV