It's unlikely, if that is all that happened. The inputs/outputs are transformer coupled and current limited.
External sources of voltage (or even PoE), or excessive ground loop currents, however, could potentially be conducted back to the port, and damage the transformer, or if they have fast edges, be coupled through and damage the electronics.