STXL Timeout Errors

I am posting this issue to both Software Bisque Forum and Cyanogen/Public (SBIG
Products)/STT/STXL Forum.

The problem:
Normally I image in an automated fashion with CCDAP 5 and The Sky X. Over the last 4 months
about 70% of automated sessions my sessions ends with one of these errors in CCDAP:

1) CCDSoft2XAdaptor.CCDSoft5Camera.1: A Camera command is already in progress. Error = 120 Timeout error
2) CCDSoft2XAdaptor.CCDSoft5Camera.1: SBIG driver: Receive timeout. Error = 30008.

I have posted this information to the CCDAP forum and John Smith verified that the issue is
not caused by CCDAP. When these error occur I find that the camera might show "Exposing" or "Ready" as status in the Sky X Camera Add On. If I happen to be at the PC when the Timeout error occurs the
camera appears to be running but is actually unresponsive in The Sky X. When I choose
"Disconnect" or almost any other mouse click I immediately get another Sky X error that
reads: error: Device: Mount Error, poor communication, connection automatically terminated. Error=213

At this point the mount has lost it's connection but still shows up in Devie Manager and
essentially I need to reboot the PC and Power Cycle equipment to get my ports back.
Sometimes I am required to unplug the camera or mount or both and replug to get ports back.
I did reproduce the SBIG Timeout Error while running the SBIG error logging and it's
included with the files. This was a bench test running a series of exposure in The Sky X
with only the camera running.

Another symptom was observaed last night. I Powered on my MX connected to The Sky X and
homed the mount. I left the mount tracking for about 20 minutes and then Powered on the
SBIG STXL. Immediately upon Powering the camera The Sky X returned:
error: Device: Mount Error, poor communication, connection automatically terminated. Error=213

It is worth noting that most of my sessions with these errros do not involve autoguiding so
only the imaging camera is running. These errors have occured as far back as Sky X Build
8909 and with up to three sets of SBIG driver updates.

My Setup:
Permanent Setup in observatory
Mount: Paramount MX
Scope: Officina Stellare RiFast 300 f3.8
Camera: Self guiding SBIG STXL 11002 less than 1 year old
Guider: Camera's built in guider
Rotator: Optec Pyxis 3"
Software: The sky X Pro with V. 10.3.0 Build 9334
Automation: CCDAutopilot 5
SBIG Drivers up to Date as of 1/12/2016
Mount and camera use factory supplied 15 foot USB cables connected directly with no hubs to
WIN 7 Pro 64 PC, 8 GB RAM, Solid State Hardrive I have plugged the camera into multiple USB ports at back of PC and have tried other cables with same errors I have uploaded all relavent documents/logs/screen shots to DropBox:

https://www.dropbox.com/sh/3bh26ztdmc53kms/AADzCgYlwKalAhmauChT4wdQa?dl=0

Any help will be appreciated.

Robert Mueller

 

Thanks Doug....
I'm using the SBIG supplied and fairly new USB cable plugged directly to the back of the PC. I can duplicate the error using more than one USB port. I have not used the PC front USB ports. Having clarified this .... any other ideas?

Also I'm struggling to understand why when the camera timeout occurs, the next thing that happens after any mouse click is that my Paramount connection stops. Somehow the camera timeout is interacting with the mount connection.

Robert

 

Doug,
I did as you suggest and am running a bench test with CCDOPS and logging on. I will report back.
Thanks,
Robert

 

Update:

So today after making the suggested Sky X changes (from SB Forum), changing which USB port the camera is plugged into the PC by selecting a different internal root hub from the MX mount I ran 5 hours of continuous exposures using CCDOPS only. No Transmission Timeout Error.

Next I ran 4 hours of continuous exposure with the STXL connected to The Sky X only. No Transmission Timeout Error. The key was nothing else but the camera was running except CCDOPS or Sky X software. There were no Transmission Timeout errors.

Tonight I launched a normal automated imaging run and within 3 minutes the STXL camera froze with Sky X showing "Exposing". It was trying to complete an autofocus run. CCDAutoPilot returned the familiar error:
CCDSoft2XAdaptor.CCDSoft5Camera.1: SBIG driver: Receive timeout. Error = 30008
reported in my first post. I check Windows device manager and the SBIG drive was still loaded. I tried to gracefully disconnect hardware but The Sky X locked up.

I cycled power on camera and mount, then disabled the drivers and enabled the drivers in Device Manager, launched Sky X and reconnected all hardware successfully without having to reboot.

This testing would suggest the USB connection is not the problem because I had no Timeouts in 9 hours, then after trying to run an imaging session it Timesout in less than 5 minutes.

Update: Ran another 4 hour continuous imaging bench test with only STXL connected to Sky X. No Timeout error.

Still hoping for any suggestions...
Robert

 

Robert,
sbig driver itself is a collection of the lowest level routines which control individual parts of the camera like ccd chip, shutter, ao, cfw, etc. Each routine is tuned in such a way to do its task without any possible error. For example Start Exposure is very complicated especially because it has to process all camera's models, ie. different ccds and shutters. On the other hand, there are high level applications like CCDOps, TSX, Maxim, etc. which call those individual routines to process some higher function, say Grab Image. First of all, the proper sequence of individual driver's routines has to be called, which is described inside the SBIG's API, of course. The CCDOps is a native "simple" application probably best tuned to its cameras, so, that's why it is used as referenced application. When works it doesn't mean other aplications have to work too, especially those which call some functions of different applications.
I control my cameras from Java application, so I know I can easily and repeatedly hang them when I improperly control their shutter, for example. When camera is frozen due to the Rx/Tx timeout errors, you have mentioned above, the USB is blocked in such a way that only power cycling helps. I did a lot of software tests trying to flush, clear, reinitialize the USB stack, but nothing helped. Furthermore, other possible problems were mentioned by Doug above, especially longer USB cable.
But my experience is that improper calling of some driver's functions is the source of those problems. Previously, we had timeout errors every night, now we are testing our software continuosly rougly week - 24 hours a day and no timeouts...hope it will continue such a way...
Q: This is probably difficult to answer, but did your timeouts happen during capturing of calibrated images, when shutter was moving ?
Regards,
Jan

 

Jan,
To my knowledge a Timeout error has not occurred during capture of calibration images. The most common situation is during a plate solve. It also occurs frequently during a light frame exposure but I'm not sure if that is during a shutter movement or not. I have three SBIG cameras on three different computers/setups and all have experienced Timeouts at one point or another. This would seem to me to rule out straight USB problems but I'm no expert!

At present I'm chasing one problem at a time.
Robert

 

Robert,
that's very strange. I would suppose this timeout problems has to happen only during communication with the camera.
Jan

 

I've two more Timeout Errors last two nights. In both cases the camera stopped responding during a Plate Solve so the camera was taking short exposures. These errors have occurred since I made sure my Paramount MX and SBIG camera were on separate internal USB Root hubs.

So today I replaced both USB cables with new quality 15 foot cables. I will try another automated imaging session tonight.

Robert

 

With both USB cables (one for the Paramount and one for STXL) replaced with new ones I had one camera timeout error similar to the others. I also ran one session all night no errors. So it appears that changing root internal hubs to separate mount and camera as well as using new cables still produces "random" timeout errors.

I will be doing some more testing soon. I also saw today on the Software Bisque forum a reported STT timeout error being reported. Although I did not focus on this issue here I also have an STT 8300 with self guiding filterwheel and it produces random timeout errors on a different computer setup running similar software.

Robert

 

I have a similar problem - with similar hardware and software:

My Setup:

• Permanent Setup in observatory
• Mount: Paramount ME
• Scope: Officina Stellare RC360
• Camera: SBIG STXL 11002 less than 6 months old
• Guider: SBIG FW8G
• SBIG AO-X
• Rotator: Optec Pyxis 3"
• FLI Atlas focusser
• Software: TheSky X Pro with V. 10.3.0 Build 9334
• Automation: CCDAutopilot 5
• SBIG Drivers up to Date as of 1/12/2016
• Mount and camera use factory supplied 15 foot USB cables connected directly with no hubs (have also tried an active USB cable to camera)
• WIN 8.1 Pro 64 laptop, 16 GB RAM, Solid State Hard drive. (USB sleep disabled!)

Symptoms:

1. Always, repeat ALWAYS occurs after a guider exposure before the shutter closes (?? I assume this is the case because there is no reassuring "click" after the guider exposure ??)
2. The autoguider display stays on "Exposure completed" in TSX
3. TheSkyX freezes, communication is lost with camera.
4. I receive no on-screen error messages
5. Can happen during imaging or calibration but always after a guider exposure. Not sure but I think guider exposures are always less than a second (I use and AO-X)
6. It has nothing to do with CCDAP because it also occurs during manual calibration in TSX
7. Frequency is random and not repeatable but can happen a couple of times a night. Note that I can go for hours without a problem

Recovery process (typically):

1. 1. disconnect everything in TSX and exit TSX (sometimes a forced exit because it freezes)
   2. unplug the camera USB (I do NOT cut power to the camera)
   3. Wait for 10 seconds or so
   4. plug camera USB back in
   5. start TSX
   6. Connect to the camera (I've heard the camera "click" after doing this - the guider shutter??)
   7. Reconnect mount, focuser and rotator

Regards,
Marcus

 

Marcus,
Your issues sound similar. When you run a CCDAP session and the problem occurs what do you see in the CCDAP log?

Update for my situation:

1) Problem not solved after replacing USB cables with new ones (USB cables likely not the problem)
2) I took the STXL camera off to send in for a blooming issue and installed a QSI 683WS (Kodak 8300 chip)
3) I ran two long 2 target sessions in CCDAP with all the same software I have reported in the mix.
4) I have seen no Transmission Timeout errors but I am seeing an issue while collecting Bias frames. The camera produces an error in Sky X as if it quit responding. The error however is not "fatal" as occurs with the STXL camera. The QSI remains responsive and connected. My conclusion is that this is a different problem related to Bias only (zero second exposure) and has not occurred during acquisition of light frames or darks after many hundreds of exposures.

So I believe this testing verifies the USB cables are not the issue and there is some sort of issue with the STXL camera or it's interaction with the Sky X software as camera control.

Still hoping for someone from SBIG/Cyanogen to weigh in or work with Software Bisque on a solution!
Thanks,
Robert

 

Hi Robert,

Alas, last night's issues were both during a calibration (one in TSX and one during CCDAP initialisation) so there were no CCDAP logs. This has been happening for a while but I'd rather not risk confusing the issue by posting earlier errors (from a month ago) because they happened before I changed the cable configuration to be a single cable to the camera - no hub. If I get errors tonight during a CCDAP run I'll post them.

The most telling (I think) symptom is that it always occurs after a guide star exposure as per my original post. This is an SBIG driver or, less likely, a TSX issue. Perhaps it's a combination of both(?) If it were a USB hardware issue my mount and my focuser would also be failing intermittently and they are not. (PS: last night one of the failures did NOT cause the mount communication to fail - I was still able to slew)

Resolution will require one party or another to actively attempt reproduce the problem by assembling and testing a similar system. In fairness, and being an IT person, I am completely aware that intermittent failures can be very hard to track down.

Cheers, Marcus

 

I had another communication failure with the camera last night while starting up a CCDAP automated run. Same symptoms as described in my post above, same place in the process - a guider exposure:

Extract from CCDAP log:
....
23:28:04 Getting guide star location...
23:28:07 Guide star measurement 1
23:28:11 Peak guide star level: 40660 ADU per sec., measured at 0.05 sec.
23:28:11 Exposure adjusted to: 0.15 sec.
23:28:11 Guider x: 74.0, y: 190.0, Array: 324x244
23:28:11 Guide star measurement 2
23:28:16 Peak guide star level: 34019 ADU per sec., measured at 0.15 sec.
23:28:16 Exposure adjusted to: 0.08 sec.
23:28:16 Guider x: 74.0, y: 191.0, Array: 324x244
23:28:16 Guide star measurement 3
23:29:37 SGC1 CCDSoft2XAdaptor.CCDSoft5Camera.1:, COleException: The remote procedure call failed. (?presumably to the camera itself?)
(800706BE) SCODE: 800706be.
23:29:37 I:FAM1:, CCDSoft2XAdaptor.CCDSoft5Image.1: COleException: The RPC server is unavailable.
(800706BA) SCODE: 800706ba.
23:29:37 SG12: CCDSoft2XAdaptor.CCDSoft5Image.1: COleException: The RPC server is unavailable.
(800706BA) SCODE: 800706ba.
23:29:37 Peak guide star level: 0 ADU per sec., measured at 0.08 sec.
23:29:37 Peak signal too low at: 0, increasing exposure to 0.16
23:29:37 I:SG2, CCDSoft2XAdaptor.CCDSoft5Camera.1: COleException: The RPC server is unavailable.
(800706BA) SCODE: 800706ba.
23:29:37 Start Guider Failed.

Please note this is NOT a CCDAP issue because as the same thing happens during manual autoguider calibration in TheSkyX. It is also NOT a USB hardware issue as both the mount and the focuser are USB and communication with them never fails. Also, if this were a TSX issue there would be many people complaining on their support forum - there isn't. It's logical, therefore, that this is most likely an SBIG driver / firmware issue.

Recovery process (last night):

1. Disconnected camera in TSX. This prompted Windows to say TSX is not responding so I forced TSX to terminate.
2. Waited for a bit (30 sec or so - just for the heck of it)
3. Restarted TSX. Interestingly, in this case I did NOT have to disconnect my camera USB cable from the computer
   
4. Connect camera in TSX. At this point I heard the camera "click". Not sure, but this may be the shutter closing???
5. Reset temperature control
6. Connected everything else (I didn't notice initially whether the mount disconnected before I shut down TSX but in past events connectivity was still OK)
7. Continued imaging after wasting 20 minutes waiting for the camera temp to restabilize ...

Regards,
Marcus

 

Appreciate the input Colin and I have not used USBView. But if this were a bus issue then the failure would likely occur at different times during the imaging process, wouldn't it? Not at exactly the same point every time (Guider measurements during calibration and guide star acquisition). As I also said, the camera is on it's own port and all ports on the Laptop are USB 3.

 

I'm actually imaging now - 6.1Hz using the AO-X, now up to image 54,000 odd . 3 calibrations tonight (2 bump calibrations in CCDAP) and 2 in TSX (mount plus AO-X)), plus a number of exposures requiring guide star measurements between each exposure. So far no failures tonight (knock on wood!). Assuming you mean doing full frame guider images as a bench test, no I haven't done that, but sounds like a good idea.

No, I am not running any scripts - just CCDAP. I don't see the error code you suggest.

Also, my apologies to the OP (Robert) as I seem to be hijacking his thread!! I don't know whether our root causes are the same but IMHO I think there is a possibility.