Years ago, I was making a stealth game in Unreal that I've since had to cancel amid circumstances beyond my control. At one point I was trying to make my characters' eyes nice, and the gold standard for that was (and arguably still is) Half-Life 2.
HL2 takes a novel approach: the eyes are not rotating sphere meshes with bones, they’re more-or-less flat planes with a shader on ‘em that makes 'em look like balls and points the iris/pupil where you tell it. The eye "plane" can be stretched as the eyelids open or close without affecting the visual, and you don't get any mesh intersection issues (which is why you've never seen the gman's eyeball push through his eyelid even though Gmod exists) or the uncanny appearance of rotating "with the head".
To get this right, I asked Valve's Ken Birdwell about how they got such a good sense of eye contact with this shader back in 2003. Here's the scoop:
In HL2 and other games, there are three main textures.
There’s the sclera texture which is textured and shaded independently. It has some hand painted darkening around the edges to simulate self shadowing from the eyelids, as well a shader to do simulated subsurface scattering of light and a clamping cone to reject light sources that would pass through the skull. There’s better ways to do this now, but in olden days there weren’t.
The second texture is the iris texture, which is literally just a just a planer projection onto a sphere. The underlying visible “eyeball” vertices will need to be spherical, and you’ll need to know the original size of the eyeball, and the size of the iris texture, or more accurately the area of the iris texture that actually contains the pixels for the iris. Once you do that, you’re half way done.
The next tricky bit is to simulate the cornea bulge with a third texture. We originally did this by creating a simple run-time texture that mapped each light source into a point on a simulated “sclera ball” and cornea (partial sphere about half the radius of the eye, offset a bit) using a super simple ray tracer. This is about 90% of what makes “eyes” work. You don’t cue off the reflections on the sphere of the eye or even the iris, you mostly cue off the reflections off the cornea bulge to judge view direction, and their subtle differences between the ball and cornea, and differences between both eyes.
Do all that, and make sure after placement the iris/cornea is about 4-5% wall-eyed (the fovea is offset from centerline of the cornea axis), and suddenly you’ll make “eye contact”.
All the numbers for this can be found in any basic eye anatomy book, and don’t worry about eye twist (your eyes slightly spin when you look around due to how the muscles are attached, but it’s not human perceivable)
I think an example for this code might still exist in the SDK, maybe in hlmv? I know eventually it all got replaced with a fancy shader that does it all on one pass, but the HL2 era version didn’t and the code might all still be there.If you really want to get fancy, then you’ll want to do a geometry shader and let the cornea bulge deform the eyelids. That, and play with pupil size and they’ll be alive! It all just depends on how close you want to get to the character, and how much CPU/GPU you’re willing to spend.
On the Unreal end, I made an EyeComponent that any actor with this style of eye uses to manage their gaze and convergence, and it all worked out pretty well, although I never shipped anything using this method. I did also end up simulating pupil dilation and cornea bump!