I don't use USB. The two interfaces I have used over the past ten years or so have been PCI, and, more recently, Firewire.
But, in general, does buffering cure all evils? One would hope so, because that is what it is there for --- to smooth out the supply and demand at the cost of higher latency, and, as you suggest, unless doing studio recording stuff, that latency, be it a microsecond or a whole second, really does not matter, except that it might give a feeling of sluggishness when pressing the play button.
Unfortunately, buffering does not cure all ills. One thing is this horrible thing called
DPC latency, which I once had to do a lot of reading about, and would now rather forget

This can cause anything from crackles and pops to complete dropouts in the sound. In short, it can render a PC useless for audio playback!
I don't think it is that common, but it is not that rare either. Google will tell. Don't ask me to try to explain it: it still hurts too much!
The
other form of timing inconsistency in digital audio is
jitter and I'm sure that everyone here must have been involved in at least one jitter conversation. Depending on whom you read, it is either the ever-lurking H1N1 influenza of digital sound --- or something that engineers pretty-much designed out years ago.