Whispers by Todd

I believe the whispers are the characters voices as they're transported thru time. To try and simplify it; The island is the constant base of operations, that travels thru time and the characters or the time jumpers, are the variables traveling thru that base of operations. For example as the jumpers are stationary in one time their remnants touch all the time periods, as the island itself also parlays all time periods.

It's also possible that in previous episodes the characters were in a consistant time flux without realizing it, and experiencing these time remnants as the lived on the island.

As for the island actually disappearing at the end of season four, it disappeared because the island was actually gone or destroyed at some point in time. Still working on that one.