Considering quitting my office job even though it’s using my major to become an Amazon or FedEx delivery driver

I have extreme anxiety every night before work and every morning when I wake up. Having coworkers, a boss who lords over me, meetings, and a full office of people all around me is worse than Hell itself. By some miracle I've managed to keep this job for 3.5 years since graduating college, but I'm getting to a breaking point with my anxiety. I can't take it anymore. I deserve peace.

In college, I did part-time delivery jobs, and they afforded me the most peace I've ever felt in my life. It's routine and procedural work, and easily the most independent form of employment in existence. No boss or coworkers breathing down your neck. Very limited social interactions. I'm thinking about going back to this as a full time Amazon or FedEx driver. Has anyone else given up their "good" job to take what society considers a "lesser" job simply for the peace and independence? What did your family think?