I think you are looking at the family job through rose coloured glasses. I made an almost identical switch 20 years ago, and if I knew then what I know now there is not a chance in hell that I would make the same decision.
You have no idea what family drama will come up ... including the fact that who knows if you will be staying with your GF.
I have never heard of any construction PM being able to take "unlimited" time off. It is an extremely demanding and hands on position, with most projects the PM has to be there to deal with the client, manage the crew, manage material orders, check invoicing (both incoming and outgoing). I currently work a minimum of 11 hours a day.
I now own the business myself, and business ownership although rewarding, can be extremely challenging. Dealing with bankers, lawsuits, customers, unions, employees, tenders, changing market conditions and competition means lots of hours and sleepless nights.
Do not make this decision lightly as it could be a life changing decision, either for good or for the worse.