Thanks for all the answers!
Nightowl, that's what I started to do. Time to finish the cabling...
Woodie, that's how I have it now, but I see increasing need for audio from some of the other PC's.
This seems to me like a common need in the low number of ports KVM's. Sure, I don't need it in my server rooms, but for the seriously eqipped home office...