Eh, normal. Don't worry about how you feel about it, those type of lunches always give me a stroke as well. And while it could be about trust, the problem is...how it is construed when viewed by a third party in public which someone can't quite understand.
I mean, for me:
1. I'm a red-blooded male
2. Going out to dinner alone with a woman is called a date where I come from
3. I don't want to give anyone the wrong idea about what I'm doing - like you said, things can be misconstrued
4. I don't want to open the door to things potentially happening that shouldn't - emotional, physical, or otherwise
I'm old-fashioned & I don't like to go near what I consider my boundary lines. I'm not saying anyone else's way is good or bad & I'm not trying to force my opinion on anyone else. I also don't necessarily think there's a "one size fits all" answer because everyone's situation is different. But I do live in a world where people have feelings & have physical appetites, and where things can get legal real quick. Like, my buddy just got fired for sexual harassment for quoting a line from Family Guy to a new girl at his work, I kid you not.
I'm also not saying I have any cheating intentions with anyone, and in reality, it'd be perfectly harmless for me to go have a solo lunch date or drive somewhere with another woman. I'd just rather not open that door...like, at all, ever, even just a little bit. And it's never been a problem in my life because it's simply not something I do. I can't fall down a slippery slope if I don't get near it! I think you can value other people in a variety of ways, but for me, as a married man, having dinner alone with another woman who is not my wife isn't one of those ways. If that's acceptable to someone else, then more power to them!