But validity is socially defined. Even our words which are intended to mean something explicitly concrete and physical have significant shortcomings when looked at through a fine lens. We make approximations because we have to but because they are also useful. For things which are poorly concrete, language is destined to be inadequate. Even for concrete things which arise little conflict internally for a basic understanding such as, say "cancer", are impossible to precisely define. What makes some changes in a cell grouping cancer and another pre-cancerous? And "cancer" connotes a category of extreme complexity, each individual case being truly unique, but still we subdivide as much as possible because doing so allows us to do way better than random at treating the illness. Even the idea of life and death contain a line that is extremely difficult for which to define a boundary.
So true with biological "sex". There is quite a range of genetic and physiologic presentations including many which are not easily classified in a binary way.
For gender, even the most socially progressive terminology being promoted involves extremely rudimentary categorizations. But they are much more useful than nothing, and there is a point where trying to be more specific makes things horribly impractical.
But I think you see the basic principles at play. I believe I understand what you are arguing should be. But I don't understand why you believe that. Finding fault in an alternative does not do it for me. There is, by definition, no language that may be used without serious fault here. Definitely a rigid classification based on biology leads to far fewer miscategorizations, yet it also means a large part of experience goes entirely uncategorized. I don't understand why we should do that to such a degree.