The harm would be to the baby birds likely in the nest. Not sure if the little ones have flown off at this point or not. It shouldn't do any harm, except for the work in cleaning the bird poop off the windows and such. You need to wait until later in the summer after the little ones have flown off to remove the nest (unless you hate birds and don't care about killing the little ones).