If you're sure it's what you want and you plan on living in it until you die. Seriously. Otherwise, you are looking at a very poor resale value. Look at the homes around the property. If you build a comparable home then you're looking at a pretty good investment. A mobile home is just going to depreciate. And while there are several sound structural designs outside the ordinary (yurts, domes, etc.), you'll find that when it comes time to sell you'll have a hard time. Both real estate agents and potential buyers will shy away from 'different'.
However, with that being said: If you are not concerned with resale value, go for it. Like I mentioned, if this is a retirement or lifelong home then it doesn't matter. If the land deal is really killer, you could look at this as a startup home to be used as a rental in the future when you build a larger home somewhere else on the property when you have more capital.