And as for limits, why must physical addresses still point to bytes?
As long as we use an x86 architecture? Deviating from this would require an extra layer of redirection for any legacy programs.
Does the CPU ever retrieve a 'byte' from memory?
Actually, memory interfaces these days are almost always much wider (DDR SDRAM is almost always accessed in 64-byte chunks), but yes, programs frequently create and access one-byte variables and one-byte pieces of larger data structures. As noted above, the OS works in terms of much larger units when managing virtual memory.
Is converting a character pointer in software to a n-byte pointer on the hardware really more complex than deciding if that data is in cache or RAM and shifitng it all around. Data movement in a computer already looks more like a packet network than a store-and-fetch machine.
I suppose you could switch to, say, 40-bit addresses, and split it into a 32-bit address (where you are addressing 256-byte blocks of memory) and an 8-bit byte offset within that block.
But then you couldn't fit a pointer into a single register. Managing that would be a nightmare in terms of an x86 CPU core and instruction set.
I guess now with 64-bit addresses, having n-times as much adress space is not very important, but thats the way we felt when we switched to 32 bits. How much longer will hardware continue to name every single byte in the memeory? And I had to walk five miles to school uphill both ways in chest-deep snow every day....
...with no shoes.
Memory ultimately has to be addressable at least down to the granularity of a CPU's registers. It is more convenient in terms of writing programs to have byte-addressable memory pointers, and this way you don't have to change your programs, compilers, and memory access schemes for architectures with differing register sizes.