But weren't there people who lost interested in MLP as most of them loosing interest in the show was around after Season 4?
No, not at all. Not at all. Sure there will be people leaving all the time, but people join too. The first feelings of the community shifting happened around season 8. That is when we started to get rumblings of the show possibly ending.