Ok Hear me out on this..... I love many Genre of games. I read over and over again in the Sci-fi section the complantes about Reality of the science. So I was wondering, what about a little reality to fantasy. Namely what about adding so true thought out details to magic and how it works. Having done some true resurch into Easter theory of the body I can tell you to them "magic" its not a simple thing. I think often we game writer gloss of this WAY too much, after all How a energy cannon work is as relivent as why the fire ball is a BALL when fires on earth never burn like that.....oh yeah its MAGiC!!!
Part of it might be that clinical details of an energy cannon are ok - but details about demon summoning and magic can incite a moral panic. Admittedly, this isn't the 80s anymore, but the second edition D&D monster manual changed from names of specific demon types to Type I-V for this sort of reason.
Furthermore isn't that the point - tech is explained, but magic is well, magical, and thus defies classification.
Leaving that behind, we have the anime Full Metal Alchemist where magic follows certain rules, and normally requires a bit of prep-time (creating a summoning circle) and is studied very much like a science. It even has a pseudo law of thermodynamics - the requirement for "equivalent exchange". A means to circumvent the normal rules - the philosophers stone - is the source of great conflict throughout the work.
Ars Magica has a notable focus on the science of magic and study. Indeed, each player makes multiple characters so that when their main mage is off in his keep studying, they still can play the assistants and guards going on errands.
Mage has some elements about gestalt observation and quantum theory.
Rifts has techno-wizardly which can apparently mass produce magical ray-guns and other such tools.
There is also a whole series about making magic systems for games over on rpg.net.
Most magic systems in roleplaying games are already way too scientific as it is. There is no mystery, no sense of the arcane or otherworldly, no uncertainty--no magic--in a typical rpg. Fill the spell slots, spend the manna, and instantly get a spell effect. It works the same way everytime, just like clockwork, and as simple as pulling a lever.
Unless your game is intentionally treating magic as a new science--'sufficiently advanced' or otherwise--it is important to remember that magic and science are not siblings. There is a lot of psychology behind this, but the gist of it is that familiarity and reliability work one way in your brain--as science, while the unknown, the 'other', mystery, uncertainty, and more emotional responses work another--as magic.
Look up 'magical thinking' sometime for more insight here.
Many games, as Chainsaw Aardvark related, have at least a token attempt to explain magic. Even D&D does this. This urge to explain everything is a result of our Meat World having such a strong emphasis on science over magic. None of that stops people from using magical thinking in everyday life, however.
A good theory of magic for roleplaying purposes would try to keep an air of mystery and magic, even if it describes important rules, or explains why, for example, a fireball is spherical. Internal consistency is good in any theory, of course, but you do have an extra bit of leeway with magical theories.
Another thing to consider is that the theory may be wrong, or simply incomplete. That is, you can have something that seems to violate your own rules. Just as scientists in our world are constantly revising their theories, so too may the thaumaturges in your setting.
Making thorough rules as to how magic works for most settings would just create inconsistencies, working to stay within the lines of some random "laws of magic" would be too difficult for most RPGs.
That said, I am/was actually working on more specific ways for magic to work in my own RPG, though these details are too convoluted to be part of the main rules, such that they will be pushed back into some lesser book for the "series" which I'll probably never actually get around to writing.
I do think that being fairly specific about how magic is done is important. In D&D you just say "I cast a spell" and that's it, there's no "I wave my hands in the complicated gestures of the Haz'barr, bringing my mind solely into line with the teachings of the Old Ways, reciting the Draconic words of power in a strong and clear voice." there's just "I cast a spell" with no idea what that entails. I was (and am) particularly irked by the material components of spells in D&D being so unspecific, it's very difficult to maintain any kind of picture of whats going on if all you know is that the spell is being cast and that it's ritual contains "verbal, somatic, and material components".
D&D also has problems with enchanting items, it involves all kinds of rituals, including putting Experience Points into the item. The more a magic ritual in D&D entails, the more often I'm just stuck going "what???". I mean, as a GM I might just invent stuff that has to be done individually for every item enchantment, but that's rather hard to just invent on the spot.
In my particular case I was researching magic for a game based in reality (in the 20th century actually; so nearly modern day) so I actually tried researching real world magic, wicca and voodoo and black magic and demons, and there is a sizeable amount of information available online, unfortunately I didn't really find the stuff useful. Actually the most useful thing I found was a pamphlet for a candle-shop on Google Books which was about different sorts of incense, it had some interesting ideas about how some of the candles would have properties like being magnetic and whatnot; but it was only three or four pages long. My main purpose in the search was actually to find weakness materials like how werewolves are weak against silver and faeries are weak against iron. My searches weren't fruitful however.
If you are going to make a detailed information about how magic works, there are really two areas you could (or would have to) cover: The Specifics of what things cause what (this hand gesture evokes anarchism & danger, this other evokes size and must be followed by a quantity) and the Laws that explain Why (in 1901 Leon Czolgosz made this hand gesture before shooting President McKinley. The movements in the gesture were discovered by Merlin to be exactly matching a "vibration" in the "magical atmosphere" which vibration when used makes the "thread" that sustains it manipulatable to quantities expressed thaumaturgically such that the vibration can be modulated to specific wants). The "Why" is I think what you would refer to as the "science" of magic (which should probably be called Thaumaturgy), the other is what existed in reality, a simple list of what things cause what effects, where no one really knows why (though of course there would be theories... and these might well be close enough to create new magics and spells; though if extended too far their falsehood might be made dangerous; it'd be much like creating a spacecraft without Einstein's Realitivity; working with only Newton's Laws of Motion wouldn't work out well in some cases, particularly the case of how much fuel?).
BTW if you ever do write out some specifics for a "science" of magic I'd be very interested to read it. I'd also suggest reading GURPS Fantasy, which contains a lot of the "standard" ideas on the subject (Glamors, drawing symbols on the ground, true names, etc) and is a good introduction to creating a magic system on a basic level (the level that most RPGs stop at).
A random idea for a reasonably deep magic system: In GURPS the magic system works by having Prerequisites, certain spells that need to be known before others. This probably represents the need to know certain "concepts" or basic knowhow before you can move to more advanced combinations of the concepts. A more advanced system then GURPS might have Concepts within the game, and have each spell require them; this could easily probably allow for a system of how to create new spells (which would indeed be a boon for any system).
The details for any magic system based on reality with any depth to it (I.E. Not D&D's) will be a fine balance between what's needed and what's heresy. I actually read a few pages of a book on rituals to summon demons that was written in the renaissance and is (apparently) considered a preeminent work on the subject by modern-day practitioners. I was very uncomfortable. I imagine writing the bibliography for any work like this would be a stressful and generally not fun for anyone with a strong respect for religions or much of a conscience for what their giving their readers (the vagueness with which the book I read used the words "knife" and "sacrifice" were almost frightening; fortunately the editor had nearby notes clarifying that the sacrifice was metaphorical not literal).
I'd like to suggest that RPGs which are based mostly in their own mythology would be safer from these references, and could more easily invent magical properties and usefulnesses of innocuous substances (like iron & candle wax). Though I don't think I've ever seen a fantasy world not based partially in real-world mythology.
TV Tropes is Aristotle's "Poetics" for the digital age.
That might be rather fun to design. Though the end result is ostensibly the core of magical theory, I would design this from the bottom up, and haphazardly at that. The gestures mentioned by Modred11 are a good start. Focus on the How first, and do the Why much later. Descriptions could be short: "The Fourth Sign of the Haz'barr is a series of zig-zag-y lines made with the pinky extended and the index finger at a right angle. It represents solidarity of purpose and is used to strengthen binding rituals."
Make a half-dozen or so of these at first. Stop if it is no longer fun. The 'theory' doesn't have to be very complete at this point. I would also take a multimedia approach: words of power, symbolic substances, runes to trace, names of entities to call upon, phrases to chant, stances to maintain, colours of pointy hat to wear, ritual elements such as drawing a circle, &c. Again, a handful of each will work for now. The pieces don't yet have to fit into place. It's probably best if it's all a rough fit.
Armed with your new bag of tricks, start assembling spell effects. Look at how some of the pieces might could go together, and start building new spell effects. Then, look at some of the end-result spells you want (or have already crafted prior to this) and see what components would work. If there are missing elements, add them. Use those new elements to go back and again look for new combinations.
The real fun comes when you show your work. You will have a fair selection of spells, and pieces to build new ones. Showing what elements are at play in a given spell gives players the 'source code' for your magic system, allowing them to craft their own spells. Once they start doing that--or even inventing new base elements--your game's total spell base can grow rapidly.
Ever study Eastern Energy theory, the base of eastern medicine? I believe that detailed magic theory would be a nice change of pace. I've been toying with the idea of writing a game where magic is set up alot more like Martial Arts. Describing magic in a more Chi manner then "ooga Booga its done!"
Time Fly's like an arrow! Fruit Fly's like a banana!
The game Alpha Omega has a very interesting take on magic. They treat it as a new science. There is no spell list, Rather you choose a source and desired effect i.e. Elemental+Agony = fire ball. There are no "mage levels" any person can attempt any magical technique, your skill determins if it kills you, or your enemy! http://www.alphaomegathegame.com/
Palladium Books' magic system seems influenced by Eastern Energy Theory, as does Shadowrun's. From a metagame aspect at the level of the game table, the former system's early fantasy roots give it more of an 'ooga Booga' feel; whereas in Shadowrun the associated mechanics--contested rolls and Drain--give more of a sense that the character is actually doing something instead of simply invoking the end effect.
To totally ruin any street-cred that I might have, the Inheritance Series (also sometimes called the Eragon series) of books takes an interesting approach to magic. It takes as much energy from the user to perform the action with magic as without (that is, the wizard will feel no different having lifted a 50 kg rock with magic as compared to without), though energy can be stored in certain items. Additionally, magic is tied to specific words but it also dependant on the user's intent. But that is just the magic that people can use; "natural" magic, though the same magic, is chaotic.
This seems to keep the magic is mysterious aspect while still allowing for rules of magic (well, rules of magic use, I suppose). As long as you like constructed languages, it should also be fairly easy to implement in game. Don't like spoken languages? Same concept can apply but with writing in the air instead (and cuniform has a nice "alphabet" just ready to be adopted for your game).
One of the underlying problems with magic in many RPGs is that it is, essentially, weaponized magic. It takes seconds to swing a sword; making magic take longer (a necessary component of magic it more ritualistic) is thus undesirable. Mechanically speaking, there is no real difference in a game between damage resulting from a fireball and damage resulting from a sword; it is just how we color that damage. Thus, combat magic that takes longer to use than a normal weapon is undesirable. The computer game "Morrowind" takes an interesting approach to this; there are several schools of magic, only one of which is really something you'd want to use in the midst of battle. Pen and Paper rpgs can use a similar system in which battle magic is a "dumbed down, crude" incarnation of real magic. Real wizards might know how to do it, but are above such vulgarities.
Then there is Raymond E. Fiest's D&D derived magic system (which always struck me as being more interesting than the real D&D system). "Magic" as it is practiced is the safe way, achieved through much sacrifice and death. There are those few "safe" rules for magic which no one really understands but knows to obey if they want to keep living. Players could try something new, but they need a lot of understanding and a lot of luck to not burn their eyebrows off, and the rest of their head with it.
The thing that RPGs tend to miss when constructing a Magic system is the "outlook" associated with types of magic. Many games treat magic like basic areas of study like math or foreign languages. The characters "know" how to create a magical effect or they don't. Illusionists and elementalists are not all that different from warriors. They just have a different skill set. I think magic would be much more life altering.
Several of you have mentioned eastern medicine as a reference to something magic-like in the world we live in. People who study eastern medicine tend to be changed by their studies. Role playing games and fantasy fiction often leave out the best part. What does the study of magic do to the student? The Earthsea books and Full Metal Alchemist are great because they depict magic-users as people who are products of their craft. They aren't just adventurers with bags of tricks. Their ability to wield power is directly related to the degree to which they've given over their lives to it.
Something else separating players from a "real" experience with magic in role playing games is the failure to describe it. All too often, the use of a spell is left to a roll of the dice. GMs should bring the senses into play. Do the characters feel all their hair standing on end just before a lightning bolt is cast? Is there a sulfurous smell in the air after a wall of fire finally dies away? Does the wizard who uses mind control have nightmares from the minds of the creatures he/she has been toying with?