Anything can be programmed with bugs in it that forces a program to change its behaviour. You have to trust the Ledger folks who've written the crypto currency plugins or apps for the Ledger Nano S which are loaded via the Ledger Manager application. There is binary signing (AFAIK) between the apps and app manager to ensure the Ledger isn't running some modified Ether plugin (as an example), but if the bad actors are able to tamper the binary before it's signed by Ledger, there's not much you can do.
There are other protections in place (like the private keys being held on a separate tamper proof chip normally referred to as a "secure enclave") but without a deep dive of the binary behaviour with the plugins it's tough to say for certain what's actually happening. Monitoring all traffic between all devices (not just network traffic) is also the type of secure audits you want performed on these types of devices.
I believe some security expert at Consensys looked at both the Ledger Nano S and the Trezor (another popular hardware wallet) and came to the conclusion the Ledger was superior in almost every way from a security perspective but still had one issue needing to be resolved.
That being said these types of devices need continuous security audits as supply chain management, sourcing of identical parts over long periods of time, secure patch management of known vulnerabilities, and having a trusted staff will always be a challenge.
