Are *you* vetting the code personally, then compiling the binary personally? If not, you're getting no more assurance that the software is kosher than a closed source application.
It's been illustrated time and time again that Open Source is not a magic guarantee that an application is more safe or secure than a closed source solution. Even big things like linux releases, you're still relying on the
volunteered expertise of other people who truly understand the complex code in front of them to sift through it with a fine toothed comb every time there's a change. Being able to see the code myself doesn't help me one bit if I don't understand a word of it.
I'm sure we all remember Heartbleed, that massive bug in OpenSSL that caused all sorts of privacy breaches?:
http://money.cnn.com/2014/04/18/technology/security/heartbleed-volunteers/index.html
"The bug wasn't caught until recently, because the OpenSSL Software Foundation doesn't have the resources to properly check every change to the software, which is now nearly half a million lines of code long. And yet that program guards a vast portion of our commerce and government..."