You wanted some perspective, so here are a few new ones:

I have known several families who went the “buy land, live in an RV/cabin/shack, then build as we have time/money/energy” road, and suffice to say it’s a helluva lot harder to do than anyone expects. Figure out how long you think it’ll take to build your house with the materials/time/energy you expect to have, then multiply by four (at least). For instance, if you think it’ll take a year, expect that four years later you’ll still be trying to finish. I’ve seen more than one family never actually finish, and/or move or disintegrate before they could finish. Just from the handful of folks we know who went that route, only 1 in 5 actually succeeded in having a house they enjoyed when it was all said and done. Sobering statistics.
I would be curious to know what the ‘defect’ is, and what you are being told it would take to ‘fix’ it. That defect could explain why the house is ‘embarrassingly’ inexpensive. Many folks don’t want the hassle of such hoops to jump through, so this might be like a fixit car that just needs some moderate repair and then would be good for another 50,000 miles. Or, it could be a serious problem that can’t actually be fixed. We came very close to buying a nice parcel of farmland back when we were in the market, and the seller mentioned something about “needing to clean up the title first” prior to sale. We started to investigate, and that rabbit hole just went deeper and deeper. Turned out the seller had no fewer than six active lawsuits pending, all of which had liens on the land. It would have cost us a fortune to get out of that, and the seller was apparently about to get into another lawsuit even while we were negotiating. Apparently he was the argumentative type, and we found out later he was selling the land to pay his legal fees for these arguments he just couldn’t walk away from. Yet the legal issues kept him from selling. Talk about a Catch-22. Some problems need effort to fix and then they’re done. Others need effort to clean up and will never be done. I’d do careful research on this one before choosing.
You do have another option here; keep looking. You don’t have to limit yourself to just the above two choices. I know it’s hard when you’re just desperate to get out of the situation you’re in. But if at all possible, make sure you’re not going from the frying pan into the fire. DR does point out that a mortgage is one of the few things which most folks legitimately cannot afford to avoid except after they’re already down the financial freedom trail quite a ways. Holding out for a mortgage-free life may cost you in other ways. If the above two options are both too much hassle to realistically choose, keep looking. We found this place after walking away from Mr Six Lawsuits, and we’ve been here ever since. So there’s no reason to settle for Plan #1 or Plan #2 if neither of them really work.