AOX star fade

Still have guide star fade issue for me too. Nothing appears to be adverse to the new build.

 

I ran the new build last night and Maxim threw a dialog saying something about an CCD error - unfortunately I couldn't read the code on the tiny screen I was using. This caused ACP to hang.

The loss of the guiding still happens as before - the new version does not seem to improve that. In fact, it seems worse than when I was running with the older driver and 6.08. I say this because now the fade issue seems to reoccur when the scope moves to a new target.

 

Let's set up an online session. Mike Miller's AO is running great with ACP. He has the orphan window buildup issue but that is not related to guiding. Call me this week.

 

Hi Adam,
Should I make this upgrade if my AO-X is working well?
Thanks,
Steve

 

If this helps, I was watching Mike Miller's system last night and saw the problem Geoff is reporting, however it did not result in tracking loss. As soon as ACP calls Guider.Track() it makes checks of the x/y errors for "a bit". The guider trackbox shows a perfectly clean guide star jumping around at the 6Hz rate. Then ACP starts the main exposure via CCDCamera.Expose(). At that time, the AO track box turns to noise for maybe a half second, then it returns to its clean guiding condition.

I believe I have seen this in the past, maybe for a long time over many versions, but I never attached much significance to it since the AO ran fine. I do have a lot of time with the AO devices (the-7' -8 and the -L) except the -X.

I wonder if Geoff's system has an unusual sensitivity to the brief interruption in the signal to the AO sensor. I plan to have a look this coming Monday if possible.

 

I did some testing last night and have discovered that I can influence the ability to recover from the AO 'guide start fade' problem by changing the main camera readout mode. If I use 'normal with RBI pre-flash', it fails to recover the guide star nearly every time an exposure starts. If I switch to 'normal' (no pre-flash) it can usually recover after the exposure starts. I ran using 'normal' mode last night and it ran OK for several hours and several targets but did eventually start not recovering the guide star. It seems that once this happens the only way to get it working properly again is to manually stop and start the AO which seems to work every time (no need to do a locate, just hit stop and then start). In all cases I've been running with the 'use separate thread for camera' on the guider (trying this on the main camera causes problems for me).

One other tidbit is that the failures tend to happen more after moving to a new target. With the exception of the solid failure on the first exposure of the night, I've not see the failure in the middle of a sequence on the same target once the first frame is successful. I assume this is because the guider continues to run between subs.

So, as Bob suggests above, I'm guessing that the RBI pre-flash causes more work for the camera (and possibly the driver) so that the AO is not getting serviced for a longer period which may exacerbate whatever is happening. I also notice that even when doing non-AO guiding it is very clear that the guider is not getting serviced very often between exposures even with the 'separate thread' selected. You can see the guider error readout 'stuttering' while the prior frame is being processed by Maxim (at the behest of ACP) - it slows to a small fraction of the update rate seen during exposures. This also happens during startup when ACP is watching the errors to wait for the guider to settle.

Another item is when using RBI pre-flash readout I have had Maxim crash both when using the AO and also without the AO (SBIG universal mode). I was also getting sporadic 'CCD error (17)' hangs. I have the windows log information details for the crashes. Let me know if this info would be helpful and the best way to get you the information.

The bottom line is that the newer driver is less stable than before. I've never had Maxim crash until this update. I am still seeing the same issue with the AO. Hopefully the readout mode information is a good clue.

 

More experimenting last night and a couple more discoveries. Ensuring pre-flash is NEVER used for any kind of exposure (pointing updates, guide star selection, etc.) seems to have improved the situation. With that change, I notice that the recovery from 'guide star fade' is much quicker and more reliable. If would seem there is something wrong with RBI pre-flash (perhaps related to using the API). It also seems like the recovery of the guide star is sensitive to the tracking exposure time - it seems to work much better with short exposures (0.08 sec). I'm guessing there is some timing issue with the AO tracking logic.

I've also noticed that the tracking rate reported by the Maxim UI will occasionally get very slow - I normally see about 8 Hz but is drops down to 2-3 hz every few seconds during the main camera exposure. Guide camera multi-threading is enabled.

Maxim still eventually gets into a state where it never returns to proper tracking of the guide star. When in this state, I see the guide star in the track box but none of the statistics (counts, tilt, etc) are being updated and the mount stops getting bumped when needed.

I also think I have found the cause of the Maxim crashes. For whatever reason the change to the SBIG driver has resulted in image windows being left open in Maxim. I'm assuming this eventually causes a memory shortage that is not handled gracefully. I discovered that if I close the ACP web UI it resolves the issue of the image windows being left open in Maxim and hence no more crashes. Since I've started running without the ACP web UI I've not had any more crashes. I have no idea what is happening since all the processing that creates the image windows used by the ACP web UI is still being done even though the web browser is not in use. No idea if this is a Maxim issue, an ACP issue or some combination.

BTW, if you would prefer this to be a new thread in the Maxim area, please let me know - I'm happy to create a new thread and summarize what I've learned.

p.s. I think that the guide image in the tracking box turing to noise is a big clue - the telescope is tracking fine, the seeing is excellent and there are no large corrections going on so the image from the guider should be good. That is suddenly becomes noise is troubling as it says something is reading garbage and supplying it as good data to the tracking software.

 

Please do create a new thread for the stuck API managed windows. I have several different cases of windows not being closed, under different conditions, and inconsistently across different customers' systems. Yours sounds like it is the Autoguider Image window which is periodically saved to a file to support the dynamic guider trackbox display on the web. If you don't use the web UI then that process doesn't happen.

 

More experimenting found a couple more tidbits about the AO-X.

I finally thought to turn off all multi-threading (I had been running with guide camera multi-threading enabled as suggested elsewhere). When I did this, I stopped seeing the noisy frames period right after the main camera start there was a very short blip but it recovered nearly instantly - so quickly that there may well be a noise frame or two - but it recovers so fast I can't be certain. I still see an occasional case where it does not recover tracking but it's much less frequently, perhaps once every 20 times.

There is still the issue with not recovering from the 'guide star fade' and having to manually re-start the AO-X tracking at the beginning of the night (and seemingly after re-starting Maxim). It appears to be a one-time thing and the system will run through multiple exposures and targets with no further need for manual intervention beyond that first time.

I also did some testing about the CCD error (17) shutter problem mentioned earlier. Bill Lynch suggested how to go about testing the shutter (thanks Bill!). There is information in the Maxim forum thread about that problem but the exec summary version is that rebooting the PC seems to clear the problem, at least for some yet-to-be-determined time.

 

Good info Geoff. Looking forward to seeing you again in a few days.

 

Is there any progress on this AO-X problem?

I'm running the latest version of Maxim and still have the same problem.

 

Since this is affecting a few of my customers and resulting in traffic over on our forum, I want to cross link this with another thread here

Guide star fade using AOX

Just so it doesn't get lost for being in a different section.

 

Any word on this one? The new SBIG driver update seems to have somehow made this worse.

 

Before the latest driver release I was generally able to get my system to operate most of a night. They way I did this was to observer that the 'fade' issue only seemed to happen in the first exposure after connecting to the camera. My 'solution' was to manually re-start the AO just when the first exposure of the night started. After that, it would usually operate the rest of the night (aside from the false 'shutter error' bug hang). After updating, it fails on nearly every exposure. There are also two OTHER problems - one new one and one that has gotten much worse.

The 'shutter error' bug I referred to is really nasty as it throws a dialog box (which should not happen as this is all using the API via ACP) and hangs the whole system for the rest of the night. Even a weather event will not break it out! This has become much worse with the new driver.

Talking to Doug about the 'shutter error' he insisted that it was a hardware problem (even though its appearance coincided with a prior driver update...) After discussions with Bill Lynch I did some bench testing and have a video demonstrating that the shutter error is false and showing strange behavior of the shutter software. There is discussion of that in another thread. This has also been made worse by the new driver. Prior to the latest update I had discovered that this problem only happened when using Ethernet to communicate with the camera so I avoided the problem by sticking to USB. Since the update it now happens both on Ethernet and USB. As a result, I can no longer let the system run overnight using automation as its just too risky. This problem was introduced with the driver update that preceded this latest release and made much worse with the latest update. (BTW, I'd LOVE to be able to go back a couple of versions as at least I could operate all night as long as I intervened on the first exposure)

A new issue is that I am seeing guide camera exposures simply not start with this new driver release. ACP asks for a guide camera exposure, it says its starting the exposure and nothing happens. The system hangs forever when this happens and a weather event will not intervene. Again, I cannot let the system operate unattended with this problem occurring since a weather event will not break it out since ACP is hung waiting on Maxim.

Another observation that may be of help is that the AO window header says something about 'not responding' when the exposure starts. For that matter, both the AO window and the Maxim camera window both hang for about 8-10 seconds when the main camera exposure starts. What you see is ACP saying its starting the exposure. At that moment the AO window stops updating and you see the 'not responding' in the window header. Then, 8-10 seconds later the Maxim camera window that shows the exposure timing suddenly updates showing the exposure already going to some number of seconds. At the same time the AO guide star box turns to hash, it says 'guide star fade' and then a moment later you see a much smaller version of the guide star showing but it never regains tracking. If I then click 'stop' and then 'start' on the AO window controls it will then gather darks and regain tracking.

To summarize:

The 'guide star fad issue' is easily demonstrated as it is completely repeatable. It does not matter how bright the guide star is, it does not matter what the exposure is, it does not matter

The 'shutter error' problems are also repeatable although it sometimes takes a few attempts.

The 'guide camera exposure hang' problem is new and I don't know how to reliably repeat it but it does occur very frequently. I would estimate it happens every 10-ish exposures. I suspect it is easily repeatably by simply taking lots of guide camera exposures.


As should be obvious these are now very serious problems. They render they system useless in the case of the 'guide star fade' problem and worse expose the system to the possibility of both weather-related damage and crashing the scope into the pier.

I've been trying to remain patient (the AO problem has been going on for six months, getting steadily worse) but this latest release has really made that impossible as I can no longer safely run the system unattended.

My system and myself are at your disposal to help get these resolved. The system is located at a premier remote site and the weather is good nearly every night. I live close by and am able to be on-site on short notice if necessary.

My question is - what can I do to help this along, what can I expect in terms of getting these problems resolved and how can I monitor progress along the way?

 

Hi guys, I have a further observation and I think a development to this problem. Tonight I am imaging on a very bright star at 7.1hz. It does not guide star fade after the main camera starts. Correction it briefly does it for fraction of a second and then it just goes to tracking with AO. Clearly this must be something to do with how stars are perceived by the software.