In the case of the home, you have nothing to worry about, due to adverse posession. In the case of the fence, you probably stiull have technical possession of the land, but they probably have the legal right to keep the fence up. I dunno about use on thier side of the fence, but you will probably have problems dealing with that. Win or lose, it won't be worth the legal fees. Probably better to work with your neighbour for a mutually agreeable resolution. Unless you've already been an asshole about it. Can happen to the best of us.