You still need an SD card since the Rpi has to boot first from the SD card. You can put the storage on a USB drive, which might have some speed benefits compared to slower sd cards, but minimal benefits if compared to the faster SD cards. The Rpi is really dependent on fast SD/USB speeds, so don't cheap out on this.
Yeah probably easier to just stick to the SD card for boot/OS then you can add a USB drive (or even external HDD) if you want more storage. I believe the USB and Ethernet (possibly wifi too) is shared though, so something to keep in mind.