Neutral and hot do matter. Neutral is tied to ground so you can't get shocked by it. Switches and breakers switch on the hot side. (at least they're suppose to). Light bulb exposed part (the screw) is neutral side, etc.
So while it does not matter for functionality of most items it does matter for safety.
Note, for 240v circuits it's a bit different, there are two hots and a neutral. Ex: for a stove or clothes dryer. The two prongs are both hots and middle one is neutral, then there is the ground pin, making 4 prongs total. In your electrical panel each breaker space is staggered between both hots. Connecting something to both hots together gives you 240v while any hot to neutral gives 120. So to get 240 a double pole breaker is used to bring both hots to the outlet.
The way this works is the 7,200v (typical) line is brought down to 240v by the transformer, and the centre tap is the neutral.