It's based off other culture's customs and pagan rituals like most christmas traditions. Because an egg represents birth, someone thought it would fit with easter.
Edit: Many cultures had springtime festivals and rituals to celebrate Spring. Spring was viewed as a time of birth or rebirth, depending on your perspective. Everything about Spring speaks of life returning to a barren land. A rabbit is a well known symbol of fertility, even today. It was one of the chief mascots of these springtime festivals. When Christianity decided to celebrate the rebirth of Jesus during this time, they absorbed some of the Springtime celebrations.
Info on the Saxon goddess Eostre from which we obtained the name "Easter".