I figured it out.
It's not a software problem, it's hardware. The gantry wasn't square to the frame, so if you cut a rectangle you got a rhombus. That translated to an oval skewed on the diagonal.
The mounting plates for the gantry were 1/4" plexiglas. I had to replace one of the stepper motors at one point, and I think it got overheated and warped the plexiglas mounting plate. So I replaced them with 1/4" aluminum. See attached photo.
That almost solved the problem.
While it was much better, I still wasn't getting a consistent right angle when I'd cut a square. I added .002" shims under one side of the ends of the gantry pieces, and that got it closer. But still not consistently right.
Here's the trouble: I think there's enough slop in the tracks that it depends on where exactly the gantry happens to be sitting when the stepper motors are engaged. I found the I could cajole it into cutting a pretty good right angle by nudging one end of the gantry a bit one way or the other before engaging the stepper motors.
Not quite the precision I was hoping for, but I understand this is a hobbyist machine from which I'm trying to get pro CNC results. This small inaccuracy wouldn't be a problem for most users.
