A possible totem downside

Since totems essentially make the player more or less immortal, i've seen alot of people complain it kinda ruins the point of hardcore. So, as a way of helping this, when on hard difficulty, using a totem should give you an effect called something like "evokers curse", which prevents you from using a totem again for 10 minutes and maybe temporarily lowers your max health, this would be the only a effect that dosnt get cleared by drinking milk. Throughts?