It matters a tiny bit, as much as some people decide to care about it.
There is a part of society that cares. If you care that they care, that matters to you.
Sometimes children like to hear their parents were married when they were born. If you care about that, it matters to you.
But the negative impact of it has nearly entirely disappeared for most in the US.
I was watching an older 1950's type movie, and IIRC the plot had a wealthy family saying their only option was to kill someone or it would come out the children were bastards, which would 'destroy the family', so it was not an option they could do. Pregnant girls used to be shipped out of state routinely to protect the family from shame (which should sound familiar to stories from rural fundamentalist societies in other countries today who treat 'family shame' by the female family members very seriously, sometimes killing them).
It used to matter in older times quite a bit in terms of family succession, property rights and so on. If the father knew...
Today in the US, even those who have moral condemnation of it mostly take at least a 'don't blame the child for the sins of the parent' approach, and don't seem to really have any actual punishments they want to do to anyone about it. And of course most aren't aware of the 'bastard' status if someone has it - and the percent of children outside the 'born into married parents' families' has now become quite large, which can be a big help in reducing discrimination.
Note no divorced person was ever elected President, it was considered a negative on them, until the rule was broken... by Ronald Reagan, and now isn't too much an issue.