My humble opinion.
There is no such thing as a do it all bike when you play at the extremes. This applies dirt bikes, street bikes or even mountain bikes. I'm assuming cars too but my hobbies involve things with 2 wheels. If you compromise and just get one item you will be disappointed unless you end up only doing one of the things you intended to do. I've made the same mistake multiple times now where I'm trying to turn something into 2 things, it just doesn't work.
Personally, I've owned tons of different street bikes, track bikes and dirt bikes. If you want to try dirt then budget 2-3k and buy something from the mid 2000s or newer to try out and see how you like it. An actual dirt bike not a dual sport or a supermoto by the way. Dirt isn't the same as street and even though they're both motorcycles the skills don't transfer. It's much harder than riding street and some people just don't like it. If you buy something cheap then you can dump it if it isn't your thing. Riding a dirt bike is more similar to mountain bike then a street bike.
Dirt bikes also require drastically more maintenance then street bikes. Unless you're burning through tires at the track, a street bike maintenance is minimal compared to any dirt bike. I guess that is assuming you ride your bike and put actual hours on it.
The reward is that they're much more fun then riding on the street. Sometimes I don't even have a street bike...but I always have a dirt bike in the garage. If I were to rank how "fun" the activities are then I'd say:
Motocross --- Supermoto track --- Street bike track / offroad riding or racing ---------- Street bike
Nothing compares to flying through the air. Not backing in your sumo around a 90 or even flying down the straight doing 160+.
