I currently live in Madison. It's not a bad town. I'm originally from Oklahoma, so madison is an upgrade (except for the winters)
Like others said, living is all over the board. West side of Madison is pretty pricey to live and downtown is the same. The east and north sides are older so they are a little lower.. still sort of pricey though.
the south side of madison has a couple "bad" areas, so do your research
There is alot to do in Madison... and it has three lakes surrounding the town.. pretty nice.
There is not alot of diversity in Madison at a ALL. you have your pockets of diversity but overall, there isn't much.