Put a high quality straightedge against the fence and look for gaps. That should end any debate about the fence being the culprit. Hardwood is not the best choice for jigs and fences. I prefer finish grade plywood with 9 or more layers for the best chance of keeping things straight and square. Wiping a shellac finish on both sides helps to keep jigs flat as well.
Good luck with it.
Good luck with it.