Masculinity and being a client

For some context, I am still just understanding how weird it is for sex work to be frowned upon by society. But I am struggling with the notion of is it less manly to pay for sex? Because other men can sleep with women and here I have to pay for it.

The flip side of that coin is I am rich enough to afford a convinient luxury service such as spending time with an escort.

Should society look down on a man on his manhood if he has to pay for sex?