For sure prevalence matters a lot with these tests, but higher specificity will help a lot with lower prevalence populations too due to a false positive rate that can be an order of magnitude lower.
For illustrative purposes, hypothetically, if you had no incidences of covid, a 95% specificity test will report 50 false positives in 1000 people compared to only 5 with a test with 99.5% specificity.
Edit:
I'd further add though, policy shouldn't be set solely on antibody test results. I think it can provide policy makers some useful information, but it needs to be combined with information from existing viral tests, maybe ratio of positive viral tests to hospitalizations (to provide some rough baseline of overall infections), and also err on the side of 'positive for antibodies' != immune to reinfection.
I know the math well, I lecture on biostatistics (from a clinical perspective). You're absolutely right that the better specificy the less false positives, but it's not necessarily the absolute number of false positives that is the issue, it's the ratio of false positives to true positives. Even if the absolute number of false positives is low, if the number of true positives is even lower, it's more likely you're positive test is really negative and thus the problem - and we can't know that without accurate data about prevalence. No of course if we're talking absolute numbers as small as a 99.5% specificity would give us, the false positives probably don't matter a huge deal, probably.
Of course all this depends on a few unknowns as well - how many folks develop antibodies, how long do we retain antibodies (ie is prevalence of positive antibody status increasing indefinitely?)? Probably the vast majority of people and probably indefinitely - but that may not be the case for sure.
You're also certainly right that it shouldn't be the sole determiner of policy.
My initial statement is simply in regards to the increasingly common sentiment of, "Well, everyone may as well just antibody testing because why not?" which is a lot more complex than it seems on it's face and not necessarily a good thing. I'm being pretty pedantic about it, but what can I say, I lecture on biostats, I can be pedantic.