Is it safe to cruise without a passport?

We are thinking about doing a cruise on Legend of the Seas. I have never been on a cruise. The cruise stated they don’t require a passport. My party stated I will be fine without a passport and that most people on these cruises don’t have a passport. However I’m a little nervous as I saw in cases of medical emergencies etc it would be difficult to renter the US.

What is your honest opinion? Am I overthinking this and will I be fine or is it horribly stupid to travel without a passport?

Edit: I should note since everyone is commenting about money that the reason it will be a struggle to get a passport is not financial related.