I've used my Samurai as a DD for going on 12 years now. I guess the saying, "know before you go" applies here. As others have mentioned, a Samurai is a no frills vehicle. That's part of the charm though, not much to go wrong and not much to worry about. And yes, they are spartan, but do you really need cup holders?
I will advise that if you are not an aggressive driver, then you either need to become one or look at other vehicles. To get around on the freeway with a Samurai, you've got to be willing to stick your neck out a bit or you'll get crushed pretty quick. Also, the skinny pedel on the right... yeah it goes all the way to floor and stays there. A good running Samurai with someone who has no issues with wrapping it out holds 75-80mph in normal freeway driving no problem. As for around town, you'll be finding 'Samurai size' parking spots all over the place and they are more than peppy enough off the line.
Also I always laugh when people talk about how they are so cramped. I'm 6'3" and it feels spacious in there (of course my summer car is a Miata, so I might have some strange standards). Hell, I've slept in mine on more than one occasion on a rainy night rolling in to camp late and not wanting to set up the tent. Sure, it's no G-Class, but it's big enough for two people and all the gear you'll need for a week in the desert.
Finally, if you want the ultimate testament for using a Samurai as a DD, my Dad drove his for 21 years as his commuter/trail rig! At the age of 61 he finally decided it was time to get something more comfortable and picked up a Cherokee, but he still drive the Samurai on occasion because, "it just goes places the Cherokee can't. Both on and off road."