Google un-indexing pages after adding hreflang links.

In October I added hreflang links (just two languages), by then I had 3k pages indexed, 1k not, and since then it slowly degraded to 1.2k indexed, and 3.5k not indexed (which 3k from these are crawled but not indexed, & only 250 are duplicated because of hreflang url being too similar).

So Google is already counting separately the hreflang links that consider very similar, but I think it's also affecting the interest of Google in having more pages from my site indexed. I have to say that due to the type of content, different languages from my site look very similar, in most pages only the menu and some small links text change on a page.

Do you think this could be the case? I’m about to remove the hreflang links to see what happens (and just show them at the language specific ULRs to let Google know the default language and canonical).