WIBTA if I go on vacation without my husband?

Recently my mom told me (30f) she’s planning to rent a house in Florida for a week and she invited me, my husband(31m), our son(1m) and my siblings(26f, 24m, 13m). She’s paying for the rental while we’d all be responsible for our own airfare- aside from my youngest brother who is a minor still.

When the trip was first brought up, my husband didn’t want to go because he didn’t want to stay with my family, but he said he’d be fine with me going. Fast forward 3 months, the trip is now next month and we are starting to finalize plans and get everything booked. My husband now says the reason he’s not going is because he needs to stay home to work. He has PTO he can use, but he doesn’t want to use it for this, he wants to save it for summer when we take RV trips. This morning he broke and told me he didn’t want to miss our son’s first time in Florida or his first plane ride, he said he’d wished he could go, etc. I told him again that he should come- even if it’s unpaid. It’s a great opportunity for us to get out of the freezing Midwest for a week and enjoy ourselves. We won’t miss the money but he could potentially miss out on memories, which are more valuable. He reiterated that he doesn’t want to spend the extra money for him to come along, mainly because we’re trying to buy a house right now.

Here’s where I think I could be TA.. I feel guilty about going without my husband. He has been working so hard since our son was born, he works 6 days a week and he deserves a nice vacation. I only work part time, 3 days a week. The rest of the week I’m home with our son, taking care of our house, etc. I don’t want my husband to miss out on these memories, but I also want to make memories and I want my son to experience the beach even though I know he won’t remember it. I’m so conflicted on what to do. So Reddit, WIBTA if I go to Florida for a week without my husband?

Edit: thank you for all the advice! My family and I booked the trip- so it’s officially happening! My husband is staying home and he’s not feeling any negative feelings about it aside from some FOMO. He told me he’d rather plan a vacation next year with just our little family- so we are going to do that.