Ever since we saw Widmore being banished in "Dead is Dead" I got to thinking exactly why he was being banished. Ben did list off leaving the island and having a family with an outsider, but these are things that we already know. I got to thinking that maybe Widmore made a deal with Dharma.
In "The Incident" we learn that Eloise is the real leader at the time. We also know that there can only be one leader (interesting how Richard said both). The incident likely started the fertility issue. It's likely that Eloise knew about the pregnancy problem, maybe it's written somewhere in Daniels book, and thus her leaving the island. Maybe Widmore wanted to abort the baby and Eloise was opposed to it, so it caused the split between them, but in 2007 it seems like they're on the same page.
After Eloise leaves, Widmore likely assumes control because they probably ran the Others like a king and queen. So he held Ben's place (maybe this is why he was prone to dislike him from the start) until Ben was old enough to lead. Widmore probably came to an agreement with Horace to stop attacking them if he had there technology at his disposal, like the submarine, to leave and profit from the island. He also has information about the stations that Keamy had.
During this time Ben integrates Ethan into the Others (maybe he steals him as a baby) and carries a relationship with Annie. Ben gets Annie pregnant and Widmore neglects to tell Ben about the pregnancy issue leading to her death. This sparks there rivalry. Widmore probably didn't have any Others get pregnant and since Ben was double agent for them and Dharma it was probably easy to keep him out of the loop. A few years later Ben steals Alex from Rousseau, and adds on to the conflict as we've seen.
Ben forms a coup within the others, has those loyal to him spy on Widmore and those loyal to him (Matthew Abaddon). They find out about Widmore's deal with Dharma and his off island family and decide to banish him, but along with ending Widmore's reign they kill off the rest of Dharma and take over there facilities, just like the military in 1954. They arrest Widmore before and send him away on the sub . It's likely that they move the island by turning the FDW, but who does that could be anyone's guess.