Depends on the software, lol.
I'm by no means an expert on this; I'm just speaking from what I've heard and what I know from experience.
I believe the windows audio stack has some built-in buffer, and I'm not sure that can be changed. If you use ASIO or WASAPI, then you can change the audio buffers on the devices or their control panels. (Or you should be able to anyway.) I have no idea about video, though.
For playback, the only real difference is that the audio begins playing a fraction of a second sooner, since there's less time needed to fill a smaller buffer. However, if the buffer is too small, then you'll hear audible skips and pauses while the system runs out of buffered data and needs to stream. Similar things happen with recording, except that if the delay between your input and the PC's output is too great, it gets very distracting.