It depends on a case by case basis. Usually, it's a case of story, but also a case of what kind of story that is. For instance, if a series employs loose continuity (or no continuity), continuation doesn't become an issue for me - to use examples from this thread, Final Fantasy, Legend of Zelda, Mario, and Fire Emblem are examples of this, where each installment is relatively self-contained. However, if a series has overarching continuity, I think it behoves itself to eventually end. To use more examples from this thread, Gears of War, StarCraft, Dead Space, Resistance, Dragon Age, and Diablo are examples. To throw in another example, Halo - IMO, it should have stopped at Reach. But since 343 took over...yeah. And to be fair, while I think Halo 4 is sub-standard in both gameplay and story, a lot of my disgruntlement stems from my feeling that the series didn't need continuation beyond H3/Reach from a story standpoint. Likewise, as much as I love StarCraft II for both story and gameplay (yes, I did play SC1/BW, since the flames have started, loved them too), I think Legacy of the Void should keep up the promise of being the overall conclusion to the saga. That, and the statement that any SC3 will effectively begin a new story.
That said, there is an exception to this, of what's listed here - Mass Effect. I've only played the first game (bar some of two), but I'm aware of the ME3 ending SNAFU. Personally, I'm content with the series ending as per the extended cut, but I'm genuinely curious about Andromeda, considering it's tangental to the series. It's content to use what came off as a starting point, but (in theory) do its own thing. A series branching off like this is something I can get behind. A more recent example for me is Terminator: Genisys - controversial opinion, I know, but to me, it was the best film in the series since the first two. And a key difference from 3 is that while 3 carried on from 2 and ended up diluting what came before as a result (in T3's mind, fate actually IS a thing, for instance), 5 uses 1 and 2 as a jumping off point to do its own thing - different timeline, and all that, less aping, and more homage, while also going in its own direction. So whatever you think of T5, in my mind, it can't harm what came before in the same way that 3 did.
Certainly there's many series that I think should have ended before they did, but I think series can survive indefinately if they employ loose continuity and/or branch off significantly enough to stand alone.