You should really read more of the books lying around in Skyrim.
The journals 1-4 of Cicero state how Cicero became the Keeper, how the Night Mother was moved, how the old Listener died (in 4E 188, 12 years prior to TES V) and so on.
So clearly the last listener isn't THAT long gone. Not...