I downloaded and installed the latest update for MaxIm DL, version 6.30. With setup using the AO on Sensor 2 and a self-guiding filter wheel, I get a window for calibrating both the AO and the ordinary guider. In both, I get very strange calibration lines, which look like a representation of a meteor radiant to the calibration star. For what I mean, see the attached graphic. Suspecting a hot pixel, I used a master dark for calibration of the guider. No difference. If I turn off the AO guider and use the normal guider calibration, I get the familiar "L" shape. Looks like something is wrong.
Did you watch the calibration progress, i.e. did the line follow the star? Also that looks to be a very long exposure. Over-saturating the star can impact the star detection algorithm.
I don't think the exposure length is going to be the answer. I started out with one second exposures of fainter stars. I moved to increase exposure time and star brightness in anticipation that I was competing with a hot pixel. That may be faulty logic, but it will be interesting to see what Adam suggests.