as for the reason the have it rare, to avoid mundaness; Gandalf in LotR has awsome powers that you dont know anything about. hes a mystical being, with the looks of an old man, but can beat a balrog in a fist fight.
And this is exactly what's wrong with most people's attempts at magic:
They want it to be like some sort of narrative story's magic.
Those stories could rewrite the magic system to be whatever they wanted it to be for whatever purpose the plot needed it to be at any time they needed it.
We aren't making a story, we're making a simulation. Simulations can't change the rules halfway through just because it might make a better story because it's not trying to tell a story. It needs a consistent set of rules.
Trying to model magic off of LotR is the wrong way to look at anything, because it will always make an inferior simulation to something made specifically to make a good simulation.
This is why I keep arguing that starting from a position of what magic should "feel like" is entirely the wrong starting point. What a magic feels like is the cherry on top. The true thing that should guide the decision is understanding of how the mechanics will change the simulation, and how that simulation will react to the changes.
As for randomness, there's only so much randomness that can occur from a misfired rocket. A fumble chart is not the same thing as making a whole system out of the Rod of Wonder. Besides which, there are a great number of players who very much do not enjoy fumble charts. Throwing everything up to randomness like that indicates the sort of person who doesn't want to take the time to think things through or plan, and wants to rely upon dumb luck and possibly save-scumming to get through a game. That's really not the kind of play we want to encourage more of in DF than it already does, and I honestly would prefer most of the already-random systems be replaced with more deterministic ones.
If I try to read this whole thread, my head is going to explode. That being said I would like to point out the magic mechanics created by the author Mercedes Lackey. I read a 3 book series that's part of a larger narrative a one point, this series being the "Storm" series. These 3 books focused on the ways in which sources of magic interact. In this universe in order to use magic you must have a pocket dimension connected to you known as a "Reservoir". This "Reservoir" contains raw magical energies. A larger "Reservoir" results in the ability to use more magic, but a larger "Reservoir" also takes considerably more time to fill and once the energy is spent, it needs to be replaced. There is also another type of magic user known as a Channeler. These people function like Bishop of the X-men, absorbing and redirecting magical energies. The story in question deals with a massive build of magical energy which builds into a tsunami of cataclysmic proportions. In the end, a young Channeler sacrifices his vision in order to protect the world from the "Storm".
The underlying mechanics of this world are innately magic, but as Kohaku has talked about, this magic functions through the interactions between a series of logical rules that serve as the "Laws of Magic" even as the "Laws of Physics" continue to exist. This universe is defined as ours is, by the concepts of "Cause and Effect" and "Decision and Consequence". Middle Earth contrary to what I saw someone else post, is NOT limited to 7 mages. "The Council of 7" is a group of ultra powerful mages that each represent one of the natural forces of the world of LotR. Sauron, Sauron's mentor, Elven royalty, Ghosts, Barrow Wights, and Nazghul can ALL use magic. Anyone with an enchanted item can use magic, and NONE of these are mages and this leads me to believe that a mage wouldn't need to be one of the 7 in order to
be a mage.
Of course having said all of that, I know need to talk about concepts such as "Wild Magic". DnD there is a form of magic that doesn't confirm to the rules set by Deneir and other such Gods. This magic is highly unpredictable and will much more likely destroy the user than have the desired effect. In order for a simulation like DF to function as intended, you need both the controllable and the uncontrollable to exist. In reality, all of reality boils down to binary code as postulated in stories such as that of the Digital Devil Saga series on PS2. This means that either something will happen, or it will
not happen, creating the concept of
possibility. When a series of possibilities collide, the concept of
probability is born. What I'm trying to say is that if everything is understandable and controllable, then life becomes boring and stagnation begins to set in. This is why Toady created the Titans, Forgotten Beasts, and the current Demons, because they a totally unpredictable they create a limitless source of concern and strife.(well that's his theory anyway, in the games current state, veteran players see all of these as avoidable and exploitable annoyances)
Ideally I'd like to see BOTH forms of magic exist in the game as well as having other random events of a non-magical nature. There would be small events i.e. cave-ins, thieves/snatchers, a sudden local increase/decrease in raw magic etc.; medium events i.e. epidemics, forest fires, "bad air" aka methane pockets etc.; major events i.e. pandemics, small demonic incursions, massive storms etc.; and cataclysms i.e. systemic failure(this would be something like the wind and currents dieing, fire ceasing to exist etc.) zombie pandemics, tears in space/time etc. This would stop the game from ever being boring. A simple example would be someone walling their fortress in and having a massive siege start. Just as they shrug, move everyone inside, and raise the
gate bridge, and earthquake starts. Not only do they end up with a serious mess on there hands
inside, but their wall crumbles to dust letting the enemy in. Fortunately they have the favor of the Gods, who in their endless benevolence and wisdom,
mind hack enslave the besiegers who then suddenly aid you with the natural disaster, before turning into fluffy wramblers who then become your Dwarves pets easing psychological issues. Such a series of both negative and positive miracles would be extremely rare, but would give you that "Oh, my fucking God, what in the name of Armok just happened!!!" moments that would define the history of a DF generated world.
This post has ballooned to truly epic proportions, and for this I apologize, but I have one more topic to discuss which I'll place in a spoiler tag.
The Difference Between Enchanted Objects and Artifacts: We really need to have this distinction in order to have the "magic feels magical" element in the game. An enchanted object is simple, a special crafter would engrave an object in such a way as to allow it to contain a "dweomer" which could have limited charges(wand, rod etc.), limited charges in a span of time, or a passive effect(ring of protection etc.). These would be valuable, expensive to manufacture, but not necessarily rare items. Artifacts on the other hand would be one-offs, and for very good reason.
For Dwarves, an Artifact would function like it does now, only with some radical changes to the rules. I would know like to direct your attention to "The Crystal Shard" by R.A Salvatore. In this book, the Dwarf Lord Bruenor Battlemammer enters what is likely the the inspiration for the moods system. Over a period a few weeks leading up to a Full moon this Dwarf with no talent for magic, gathered the materials for and then forged a mithral war-hammer that was to large for any Dwarf to wield. On the night of the Full moon he engraved the hammer with the symbols of his triumvirate of gods as well as arcane runes on the head of the hammer. This hammer that he then gifted to the young Barbarian that he had adopted, is known as Aegis Fang. The hammer hit's like a stampede and always returns to it's masters hands when called causing it to be immensely useful as a projectile. One of the first things that ways done with the hammer was the slaying of a White Dragon which led to a series of events that began the legend of "The Companions of the Hall" one of the greatest adventuring parties in all of the fantasy genre. After forging Aegis Fang. Bruenor lost all interest in crafting more than the occasional wooden figurine.
My proposal is as follows. Make it so that only a pious Dwarf of Legendary or even a new rare Mythic level of skill can create an artifact. These artifacts would be game changers on the entire world stage affecting the history of a generated world such as a certain Elven King of the Dwarves did. After the artifact is finished the Dwarf will enter a period of depression after which he/she will never craft again, freeing them up for the military or a noble title. This would by no means eliminate the Fell and Macabre moods. On the contrary some would see these as the most desirable as this is how you would get a One Ring, Crenshinibon, or a vampiric dagger that fully heals the user effectively making them immortal at the cost of their victims lives. This would once again create the feeling that you just saw something truly momentous occur and give people on the forums immense bragging rights.
On a final note, when it comes to the nature of magic itself here's a famous quote, "To an observer, any technology or phenomena that his completely beyond their abbilty to comprehend will seem like, and there for be classified as, magic.". Betazeoid and Vulcan telepathy as well as a phaser appear to be magic to the uninitiated, hell the "Vulcan Nerve Pinch" does. However, to any Starfleet officer they are just a part of every day life. In short it all comes down to perception. Just because you do or do not perceive it, does not mean that it does or does not exist. Also in relation to morality just because you do or do not like, does not mean that is right or wrong. Or in other words "Nothing is true, everything is permitted.".