This problem is more common than you think, and it's typically found in the newer asian machines because the quality of the machining performed, as well as the actual cast iron used, is lacking. They rush to production these castings, machining them before they have properly cycled and cured. The curing process on cast iron can take years, and the vintage American and European makers of old knew this. A company like Northfield would let their castings sit outside for as long as 10 years before machining them, and selling them as finished machines. Big difference between a $1500 Grizzly jointer and a $18K jointer from Northfield.
It's quite possible that fence you looked at was straight when it was machined, but stresses relieved in the castings, over time, have released and are causing the castings to move.
I restore a lot of vintage machinery. My entire shop is filled with them. I've had machine surfaces reground, and can tell you that when machining/grinding is not done correctly, or if it is rushed, the castings can move, and they can move a lot.
There's a good chance that fence is not through moving, and you would also need to see if it's moving in other directions, as well. I've seen several fences for jointers that had twist in them, as well, and that would make it very, very difficult to edge joint boards for glue-ups.
You were wise to walk away. Plenty of machines out there, in all sizes, that were made with quality castings. No need to chase gremlins in a machine that may be a problem child for some time to come.