The nature of The Island and The Smoke Monster by Marko Manev and Simon Kalajdziev

This is a general theory about everything paranormal on the show. We don't expect this to be further explained in the finale, it is just a matter of how one can look at Lost. Most certainly it's the way we like to look at it.

It is certain that all the paranormal properties of the Island are generated by one and single source: namely, the scattered deposits, or pockets of some sort of exotic matter. This matter is radiating with intensive electromagnetic waves, bright light and heat. It creates a strong field around the Island and a barrier around its perimeter. This field is creating various physical and psychic anomalies. It also causes the Island and its residents to travel through time and space. This matter can have some healing effects as well, and even raise people from the dead. It is labelled exotic mater by the scientists of the Dharma Initiative, which means that beside the measurable properties it shares with other known substances, it also possesses some unknown properties. It may possibly radiate with some unregistered waves, create unknown physical fields, and display strange behaviour in contradiction with the known laws of physics. (We will stick to electromagnetic waves for the purpose of simplifying this theory, even though we believe other forms of influence may be at work as well.)

What's most important is the ability of this matter to interact with living organisms and people's minds.
It is known to science that the biological functions of each individual living organism are creating a unique electromagnetic field around the creature, so distinctive that it can be tracked with present day technologies. The special geostructure of the Island gives it the unique paranormal property to register the presence of these “bio-fields”, and record not just mere thoughts, but rather the personalities of people! It acts as a sort of data-bank of people's minds, a hard disc constantly installing psyche “programs“. Further, this complex psyche-sensitive system is also able to project human psyche, much like canyons would echo sound and return people's voices! This exotic natural structure does not have a will of its own, neither consciousness, but it is delicate enough to react so accurately and accordingly, effectively mimicking actual will as if the person were there to react as he would.
Electromagnetic waves are probably functioning as a medium for this phenomenon, somehow being recorded on the exotic matter, creating on it an interactive copy of the original psyche, which is in turn triggered, accessed and interpreted by the electric impulses of a new recipient, resulting in a projection of the original psyche in the mind of someone living. The interaction is a sort of telepathy between the “system“ and the recipient, and depending on its strength it may manifest through strong sense of intuition, dreams, visions, voices, whispers, and even illusions of physical presence.
Having different neural structures all people respond differently to it, and some people are more sensitive than others, thus being “special”, and feeling as if in a spiritual communion with the Island. For example, when Isabella appears to Hurley the Island's “system” is “reading” Richard's memory of Isabella and is projecting a response to Hurley, who is more sensitive to the medium. Hurley is so acute to the projections of the system that he can easily be reached by it on any spot of the globe, considering that the radiance of the exotic matter interacts with the planet's magnetic field and is linked to many other places on the surface of the Earth.
Furthermore, this sort of “transceiving” can work through time as well as through space. If anyone at any given time knows something exclusive, that knowledge can be transmitted to anyone at any other moment in time, future or past. (A good metaphor for this is when Sayid and Hurley get a signal of a radio station: Sayid: 'Radio waves at this frequency bounce off the ionosphere. They can travel thousands of miles. It could be coming from anywhere.' Hurley: 'Or, anytime...'.) Whether confusing, comforting, or instructive, the “system” is most often very resourceful revealing to the recipient something he would like to know and/or the recorded psyche would like to share. Finally, it is not a “council of the ancestors“, the recorded psyches do not have collective counselling, but act independently, mostly one at a time and only when triggered. If there are no recipients the system will remain completely silent. The souls of the dead are not actually stuck on the Island, though it might seem so to the living. In the end, the way we see it, Lost is about fringe science, not voodoo magic.

(To simplify, in the terms of Star Wars the Island is a Force imbued place, and some of the characters of Lost are simply put, more Force sensitive than others.)

No one on the show has showed any clear sign of understanding how any of this actually works, except, of course, the Monster.
When Jacob throws his brother into the light the Monster is created, and the man's dead body is cast away. The Monster is a special form of psyche created as a side effect of this incident. When Jacob’s brother's psyche hits the exotic matter it triggers the creation of a new psyche that has no human body. It is a smoke of black particles that can (just like the Island's psyche-sensitive system) scan, record and interpret the minds of the living, among its other abilities.
Jacob’s brother is dead, and the Monster is only assuming his form afterwards. It is not a human ghost, and it is not Jacob’s brother any more then it is Locke. That is why Jacob never calls it brother, but rather “friend” instead. (We honestly hoped to hear from Jacob’s mouth the thrilling expression: “You are not my brother!”.) It acts like it has consciousness and a will of its own, but lacks a developed intrinsic personality. It is clear that it has access to the Island's psychic data bank system and can take on the form of the dead to impersonate them, as well as their character, and when it does it even feels like the real person.
Having such access to the “souls” of the dead, in a sense it is the Island's Cerberus. Sometimes it is even surprised by the strength of the impersonation, like when it shouts out “Don't ever tell me what I can't do!“. Also when it is saying “I'm not a what, I'm a who!“ that is probably John Locke talking. But when it is in the Smoke form it says nothing at all, and does not appear to feel human in any sense. Still, since it was created in direct contact with Jacob’s brother psyche, this specific personality is naturally the dominant force that drives its actions, guided by the strong, inherited obsession to leave the Island and “go home”.