It's not annoying enough to keep from buying it. But it is annoying enough to keep you from playing certain games.
The GBA is the only real portable choice in America right now if you want to play games on the go. Only the darker games like Castlevania really hurt you. The brighter games like Advanced Wars and Puyo Pop don't bother many at all. If you do have a major problem with the darker games, you can always modify the GBA to be backlit, but that drains the batteries. The choice is up to you.
Personally I like the Wonderswan, GP32 and the NGPC. I only wish they would come over to America (or in the NGPC's case, I wish the parent company hadn't died out.