<< but do you think it will run good >>
That completely depends on the hosting provider. Whos is your hosting provider, btw?
<< or should I setup a server at home through my cable modem, >>
That's probably not a good idea. Most, if not all, cable internet access providers prohibit servers of any nature. Runing one could result in account termination. Check provider's TOS/AUP. Who is your cable provider, btw?
What kind of database is required for vBulletin? Your hosting provider will have to support it in order for you to use vBulletin.