What games have you read that handled fear and insanity well? How do you integrate these elements into your own designs?
The cardinal rule for most games, electronic and non, is to avoid taking control away from the players. We want to play, not watch our tokens sit around or do stupid things. This is why I tend to avoid war games with Command and Control elements - I want to see tanks bravely storming the Prokhorovka, not have a bunch of cardboard cut-outs ignore me due to "radio malfunction". If we wanted non-interactive story-telling, we could watch a movie.
Going back to my earliest days of role-playing, the Horror Factor rules from Rifts were pretty much universally ignored. Sure, by all means these we some of the simplest fear rules I've seen (roll to save, if not, then miss your first turn, and act normally from there-after.) but it didn't seem right. Our characters were a collection of Demon Hunters, Power Armor equipped mercenaries, drugged-up UberSoldats, dimension hopping mages, and the like - being spooked just didn't really fit the profiles.
As such, I really don't have rules that force actions on players.
Of course, a different option would be to rather than prevent actions, to apply modifiers to actions in adverse conditions. But of course, if the player's are in a tense moment, you don't want to break the flow by starting to argue over -5 vs -4 and doing math.
I'd argue we could try positive reinforcement, and give players a reward for acting scared/running - but what is a sufficient motivation to partake such actions?
In D&B at least, I try to include less about horror in the encounter - adrenaline counts for a lot at that point. However, recovery of expendable resources requires finding safety and creature comforts, so the after affects are more prominent.
Insanity is hard to handle well, and all to often gets used as an excuse for daffy-duck style behavior rather than Hannibal Lector. Despite the fact that it should be prevelant in some of my settings, I don't include rules for mental illness in my games.. HyperFoil Gates in XenoExodous have odd effects, and in D&B, you go from a Post-Cyberpunk all the world is wired to barely any electricity while surrounded by flesh-eating horrors and aliens.