STXL Timeout Errors

Discussion in 'STX and STXL Series Cameras' started by Robert Mueller, Feb 3, 2016.

  1. Robert Mueller

    Robert Mueller Standard User

    Joined:
    Jun 22, 2015
    Messages:
    79
    Location:
    Sedona, Arizona
    I am posting this issue to both Software Bisque Forum and Cyanogen/Public (SBIG
    Products)/STT/STXL Forum.

    The problem:
    Normally I image in an automated fashion with CCDAP 5 and The Sky X. Over the last 4 months
    about 70% of automated sessions my sessions ends with one of these errors in CCDAP:

    1) CCDSoft2XAdaptor.CCDSoft5Camera.1: A Camera command is already in progress. Error = 120 Timeout error
    2) CCDSoft2XAdaptor.CCDSoft5Camera.1: SBIG driver: Receive timeout. Error = 30008.

    I have posted this information to the CCDAP forum and John Smith verified that the issue is
    not caused by CCDAP. When these error occur I find that the camera might show "Exposing" or "Ready" as status in the Sky X Camera Add On. If I happen to be at the PC when the Timeout error occurs the
    camera appears to be running but is actually unresponsive in The Sky X. When I choose
    "Disconnect" or almost any other mouse click I immediately get another Sky X error that
    reads: error: Device: Mount Error, poor communication, connection automatically terminated. Error=213

    At this point the mount has lost it's connection but still shows up in Devie Manager and
    essentially I need to reboot the PC and Power Cycle equipment to get my ports back.
    Sometimes I am required to unplug the camera or mount or both and replug to get ports back.
    I did reproduce the SBIG Timeout Error while running the SBIG error logging and it's
    included with the files. This was a bench test running a series of exposure in The Sky X
    with only the camera running.

    Another symptom was observaed last night. I Powered on my MX connected to The Sky X and
    homed the mount. I left the mount tracking for about 20 minutes and then Powered on the
    SBIG STXL. Immediately upon Powering the camera The Sky X returned:
    error: Device: Mount Error, poor communication, connection automatically terminated. Error=213

    It is worth noting that most of my sessions with these errros do not involve autoguiding so
    only the imaging camera is running. These errors have occured as far back as Sky X Build
    8909 and with up to three sets of SBIG driver updates.

    My Setup:
    Permanent Setup in observatory
    Mount: Paramount MX
    Scope: Officina Stellare RiFast 300 f3.8
    Camera: Self guiding SBIG STXL 11002 less than 1 year old
    Guider: Camera's built in guider
    Rotator: Optec Pyxis 3"
    Software: The sky X Pro with V. 10.3.0 Build 9334
    Automation: CCDAutopilot 5
    SBIG Drivers up to Date as of 1/12/2016
    Mount and camera use factory supplied 15 foot USB cables connected directly with no hubs to
    WIN 7 Pro 64 PC, 8 GB RAM, Solid State Hardrive I have plugged the camera into multiple USB ports at back of PC and have tried other cables with same errors I have uploaded all relavent documents/logs/screen shots to DropBox:

    https://www.dropbox.com/sh/3bh26ztdmc53kms/AADzCgYlwKalAhmauChT4wdQa?dl=0

    Any help will be appreciated.

    Robert Mueller
     
  2. Doug

    Doug Staff Member

    Joined:
    Sep 25, 2014
    Messages:
    8,350
    Error 8 (ignore the 3000 added by Bisque) is "Recieve Timeout", which suggests a USB bus error. Common causes: using a USB hub not rated for cold conditions, poor quality USB hub, poor quality USB cable, sharing a hub with other high-volume traffic such as an autoguider, using a USB port on the front panel of a desktop PC (they are compromised by the internal cabling), etc.
     
  3. Robert Mueller

    Robert Mueller Standard User

    Joined:
    Jun 22, 2015
    Messages:
    79
    Location:
    Sedona, Arizona
    Thanks Doug....
    I'm using the SBIG supplied and fairly new USB cable plugged directly to the back of the PC. I can duplicate the error using more than one USB port. I have not used the PC front USB ports. Having clarified this .... any other ideas?

    Also I'm struggling to understand why when the camera timeout occurs, the next thing that happens after any mouse click is that my Paramount connection stops. Somehow the camera timeout is interacting with the mount connection.

    Robert
     
  4. Doug

    Doug Staff Member

    Joined:
    Sep 25, 2014
    Messages:
    8,350
    That suggests something is up with your USB ports. Try moving the camera to a different root hub (they usually come in pairs on a PC - each is pair is on its own root hub).
     
  5. Robert Mueller

    Robert Mueller Standard User

    Joined:
    Jun 22, 2015
    Messages:
    79
    Location:
    Sedona, Arizona
    Doug,
    I did as you suggest and am running a bench test with CCDOPS and logging on. I will report back.
    Thanks,
    Robert
     
  6. Robert Mueller

    Robert Mueller Standard User

    Joined:
    Jun 22, 2015
    Messages:
    79
    Location:
    Sedona, Arizona
    Update:

    So today after making the suggested Sky X changes (from SB Forum), changing which USB port the camera is plugged into the PC by selecting a different internal root hub from the MX mount I ran 5 hours of continuous exposures using CCDOPS only. No Transmission Timeout Error.

    Next I ran 4 hours of continuous exposure with the STXL connected to The Sky X only. No Transmission Timeout Error. The key was nothing else but the camera was running except CCDOPS or Sky X software. There were no Transmission Timeout errors.

    Tonight I launched a normal automated imaging run and within 3 minutes the STXL camera froze with Sky X showing "Exposing". It was trying to complete an autofocus run. CCDAutoPilot returned the familiar error:
    CCDSoft2XAdaptor.CCDSoft5Camera.1: SBIG driver: Receive timeout. Error = 30008
    reported in my first post. I check Windows device manager and the SBIG drive was still loaded. I tried to gracefully disconnect hardware but The Sky X locked up.

    I cycled power on camera and mount, then disabled the drivers and enabled the drivers in Device Manager, launched Sky X and reconnected all hardware successfully without having to reboot.

    This testing would suggest the USB connection is not the problem because I had no Timeouts in 9 hours, then after trying to run an imaging session it Timesout in less than 5 minutes.

    Update: Ran another 4 hour continuous imaging bench test with only STXL connected to Sky X. No Timeout error.

    Still hoping for any suggestions...
    Robert
     
    Last edited: Feb 6, 2016
  7. Jan Soldan

    Jan Soldan Cyanogen Customer

    Joined:
    Oct 11, 2014
    Messages:
    239
    Location:
    Czech Republic
    Robert,
    sbig driver itself is a collection of the lowest level routines which control individual parts of the camera like ccd chip, shutter, ao, cfw, etc. Each routine is tuned in such a way to do its task without any possible error. For example Start Exposure is very complicated especially because it has to process all camera's models, ie. different ccds and shutters. On the other hand, there are high level applications like CCDOps, TSX, Maxim, etc. which call those individual routines to process some higher function, say Grab Image. First of all, the proper sequence of individual driver's routines has to be called, which is described inside the SBIG's API, of course. The CCDOps is a native "simple" application probably best tuned to its cameras, so, that's why it is used as referenced application. When works it doesn't mean other aplications have to work too, especially those which call some functions of different applications.
    I control my cameras from Java application, so I know I can easily and repeatedly hang them when I improperly control their shutter, for example. When camera is frozen due to the Rx/Tx timeout errors, you have mentioned above, the USB is blocked in such a way that only power cycling helps. I did a lot of software tests trying to flush, clear, reinitialize the USB stack, but nothing helped. Furthermore, other possible problems were mentioned by Doug above, especially longer USB cable.
    But my experience is that improper calling of some driver's functions is the source of those problems. Previously, we had timeout errors every night, now we are testing our software continuosly rougly week - 24 hours a day and no timeouts...hope it will continue such a way...
    Q: This is probably difficult to answer, but did your timeouts happen during capturing of calibrated images, when shutter was moving ?
    Regards,
    Jan
     
  8. Robert Mueller

    Robert Mueller Standard User

    Joined:
    Jun 22, 2015
    Messages:
    79
    Location:
    Sedona, Arizona
    Jan,
    To my knowledge a Timeout error has not occurred during capture of calibration images. The most common situation is during a plate solve. It also occurs frequently during a light frame exposure but I'm not sure if that is during a shutter movement or not. I have three SBIG cameras on three different computers/setups and all have experienced Timeouts at one point or another. This would seem to me to rule out straight USB problems but I'm no expert!

    At present I'm chasing one problem at a time.
    Robert
     
  9. Jan Soldan

    Jan Soldan Cyanogen Customer

    Joined:
    Oct 11, 2014
    Messages:
    239
    Location:
    Czech Republic
    Robert,
    that's very strange. I would suppose this timeout problems has to happen only during communication with the camera.
    Jan
     
  10. Robert Mueller

    Robert Mueller Standard User

    Joined:
    Jun 22, 2015
    Messages:
    79
    Location:
    Sedona, Arizona
    I've two more Timeout Errors last two nights. In both cases the camera stopped responding during a Plate Solve so the camera was taking short exposures. These errors have occurred since I made sure my Paramount MX and SBIG camera were on separate internal USB Root hubs.

    So today I replaced both USB cables with new quality 15 foot cables. I will try another automated imaging session tonight.

    Robert
     
  11. Robert Mueller

    Robert Mueller Standard User

    Joined:
    Jun 22, 2015
    Messages:
    79
    Location:
    Sedona, Arizona
    With both USB cables (one for the Paramount and one for STXL) replaced with new ones I had one camera timeout error similar to the others. I also ran one session all night no errors. So it appears that changing root internal hubs to separate mount and camera as well as using new cables still produces "random" timeout errors.

    I will be doing some more testing soon. I also saw today on the Software Bisque forum a reported STT timeout error being reported. Although I did not focus on this issue here I also have an STT 8300 with self guiding filterwheel and it produces random timeout errors on a different computer setup running similar software.

    Robert
     
  12. marc4darkskies

    marc4darkskies Standard User

    Joined:
    Oct 13, 2014
    Messages:
    40
    I have a similar problem - with similar hardware and software:

    My Setup:
    • Permanent Setup in observatory
    • Mount: Paramount ME
    • Scope: Officina Stellare RC360
    • Camera: SBIG STXL 11002 less than 6 months old
    • Guider: SBIG FW8G
    • SBIG AO-X
    • Rotator: Optec Pyxis 3"
    • FLI Atlas focusser
    • Software: TheSky X Pro with V. 10.3.0 Build 9334
    • Automation: CCDAutopilot 5
    • SBIG Drivers up to Date as of 1/12/2016
    • Mount and camera use factory supplied 15 foot USB cables connected directly with no hubs (have also tried an active USB cable to camera)
    • WIN 8.1 Pro 64 laptop, 16 GB RAM, Solid State Hard drive. (USB sleep disabled!)
    Symptoms:
    1. Always, repeat ALWAYS occurs after a guider exposure before the shutter closes (?? I assume this is the case because there is no reassuring "click" after the guider exposure ??)
    2. The autoguider display stays on "Exposure completed" in TSX
    3. TheSkyX freezes, communication is lost with camera.
    4. I receive no on-screen error messages
    5. Can happen during imaging or calibration but always after a guider exposure. Not sure but I think guider exposures are always less than a second (I use and AO-X)
    6. It has nothing to do with CCDAP because it also occurs during manual calibration in TSX
    7. Frequency is random and not repeatable but can happen a couple of times a night. Note that I can go for hours without a problem
    Recovery process (typically):
      1. disconnect everything in TSX and exit TSX (sometimes a forced exit because it freezes)
      2. unplug the camera USB (I do NOT cut power to the camera)
      3. Wait for 10 seconds or so
      4. plug camera USB back in
      5. start TSX
      6. Connect to the camera (I've heard the camera "click" after doing this - the guider shutter??)
      7. Reconnect mount, focuser and rotator
    Regards,
    Marcus
     
  13. Robert Mueller

    Robert Mueller Standard User

    Joined:
    Jun 22, 2015
    Messages:
    79
    Location:
    Sedona, Arizona
    Marcus,
    Your issues sound similar. When you run a CCDAP session and the problem occurs what do you see in the CCDAP log?

    Update for my situation:

    1) Problem not solved after replacing USB cables with new ones (USB cables likely not the problem)
    2) I took the STXL camera off to send in for a blooming issue and installed a QSI 683WS (Kodak 8300 chip)
    3) I ran two long 2 target sessions in CCDAP with all the same software I have reported in the mix.
    4) I have seen no Transmission Timeout errors but I am seeing an issue while collecting Bias frames. The camera produces an error in Sky X as if it quit responding. The error however is not "fatal" as occurs with the STXL camera. The QSI remains responsive and connected. My conclusion is that this is a different problem related to Bias only (zero second exposure) and has not occurred during acquisition of light frames or darks after many hundreds of exposures.

    So I believe this testing verifies the USB cables are not the issue and there is some sort of issue with the STXL camera or it's interaction with the Sky X software as camera control.

    Still hoping for someone from SBIG/Cyanogen to weigh in or work with Software Bisque on a solution!
    Thanks,
    Robert
     
  14. marc4darkskies

    marc4darkskies Standard User

    Joined:
    Oct 13, 2014
    Messages:
    40
    Hi Robert,

    Alas, last night's issues were both during a calibration (one in TSX and one during CCDAP initialisation) so there were no CCDAP logs. This has been happening for a while but I'd rather not risk confusing the issue by posting earlier errors (from a month ago) because they happened before I changed the cable configuration to be a single cable to the camera - no hub. If I get errors tonight during a CCDAP run I'll post them.

    The most telling (I think) symptom is that it always occurs after a guide star exposure as per my original post. This is an SBIG driver or, less likely, a TSX issue. Perhaps it's a combination of both(?) If it were a USB hardware issue my mount and my focuser would also be failing intermittently and they are not. (PS: last night one of the failures did NOT cause the mount communication to fail - I was still able to slew)

    Resolution will require one party or another to actively attempt reproduce the problem by assembling and testing a similar system. In fairness, and being an IT person, I am completely aware that intermittent failures can be very hard to track down.

    Cheers, Marcus
     
  15. marc4darkskies

    marc4darkskies Standard User

    Joined:
    Oct 13, 2014
    Messages:
    40
    I had another communication failure with the camera last night while starting up a CCDAP automated run. Same symptoms as described in my post above, same place in the process - a guider exposure:

    Extract from CCDAP log:
    ....
    23:28:04 Getting guide star location...
    23:28:07 Guide star measurement 1
    23:28:11 Peak guide star level: 40660 ADU per sec., measured at 0.05 sec.
    23:28:11 Exposure adjusted to: 0.15 sec.
    23:28:11 Guider x: 74.0, y: 190.0, Array: 324x244
    23:28:11 Guide star measurement 2
    23:28:16 Peak guide star level: 34019 ADU per sec., measured at 0.15 sec.
    23:28:16 Exposure adjusted to: 0.08 sec.
    23:28:16 Guider x: 74.0, y: 191.0, Array: 324x244
    23:28:16 Guide star measurement 3
    23:29:37 SGC1 CCDSoft2XAdaptor.CCDSoft5Camera.1:, COleException: The remote procedure call failed. (?presumably to the camera itself?)

    (800706BE) SCODE: 800706be.
    23:29:37 I:FAM1:, CCDSoft2XAdaptor.CCDSoft5Image.1: COleException: The RPC server is unavailable.
    (800706BA) SCODE: 800706ba.
    23:29:37 SG12: CCDSoft2XAdaptor.CCDSoft5Image.1: COleException: The RPC server is unavailable.
    (800706BA) SCODE: 800706ba.
    23:29:37 Peak guide star level: 0 ADU per sec., measured at 0.08 sec.
    23:29:37 Peak signal too low at: 0, increasing exposure to 0.16
    23:29:37 I:SG2, CCDSoft2XAdaptor.CCDSoft5Camera.1: COleException: The RPC server is unavailable.
    (800706BA) SCODE: 800706ba.
    23:29:37 Start Guider Failed.

    Please note this is NOT a CCDAP issue because as the same thing happens during manual autoguider calibration in TheSkyX. It is also NOT a USB hardware issue as both the mount and the focuser are USB and communication with them never fails. Also, if this were a TSX issue there would be many people complaining on their support forum - there isn't. It's logical, therefore, that this is most likely an SBIG driver / firmware issue.

    Recovery process (last night):
    1. Disconnected camera in TSX. This prompted Windows to say TSX is not responding so I forced TSX to terminate.
    2. Waited for a bit (30 sec or so - just for the heck of it)
    3. Restarted TSX. Interestingly, in this case I did NOT have to disconnect my camera USB cable from the computer
    4. Connect camera in TSX. At this point I heard the camera "click". Not sure, but this may be the shutter closing???
    5. Reset temperature control
    6. Connected everything else (I didn't notice initially whether the mount disconnected before I shut down TSX but in past events connectivity was still OK)
    7. Continued imaging after wasting 20 minutes waiting for the camera temp to restabilize ...
    Regards,
    Marcus
     
  16. Colin Haig

    Colin Haig Staff Member

    Joined:
    Oct 27, 2014
    Messages:
    5,207
    Location:
    Earth
    Am not sure that you can entirely eliminate USB as a contributor to the problems experienced.
    Have you tried using USBView to make sure you haven't got a mix of different speed devices on the same usb bus? It could very well be a USB issue, as in 480mbps for camera vs 12mbps for mount and focuser. The mount and focuser are not running at the same data rate, nor are the moving big blocks of data continuously like the camera does.
    You'll want to have the camera on its own bus, and put lowspeed devices on their own bus.
     
  17. marc4darkskies

    marc4darkskies Standard User

    Joined:
    Oct 13, 2014
    Messages:
    40
    Appreciate the input Colin and I have not used USBView. But if this were a bus issue then the failure would likely occur at different times during the imaging process, wouldn't it? Not at exactly the same point every time (Guider measurements during calibration and guide star acquisition). As I also said, the camera is on it's own port and all ports on the Laptop are USB 3.
     
  18. Colin Haig

    Colin Haig Staff Member

    Joined:
    Oct 27, 2014
    Messages:
    5,207
    Location:
    Earth
    Hmmm.... I hear what you're saying. If it is failing at the same point, the bus issue could be when the "bulk" transfer of the guider image data is being sent down. That said, sounds like you have a decent setup. Have you tried doing say 1000 images off the guider chip, not doing any actual guiding? Just bench test the setup, just the camera to the same usb port and same usb cable, no other tasks running. Might help eliminate something.
     
  19. Colin Haig

    Colin Haig Staff Member

    Joined:
    Oct 27, 2014
    Messages:
    5,207
    Location:
    Earth
    The "23:29:37 SGC1 CCDSoft2XAdaptor.CCDSoft5Camera.1:, COleException: The remote procedure call failed."
    Are you running some script? If so, can you post? There is some operation failing when it goes to access the camera code, not necessarily accessing the camera itself. The message is originating in Microsoft Object Linking and Embedding. There should be an error code number like x81234567 that goes with the error, but maybe its not exposed in the log mechanism.
     
  20. marc4darkskies

    marc4darkskies Standard User

    Joined:
    Oct 13, 2014
    Messages:
    40
    I'm actually imaging now - 6.1Hz using the AO-X, now up to image 54,000 odd :). 3 calibrations tonight (2 bump calibrations in CCDAP) and 2 in TSX (mount plus AO-X)), plus a number of exposures requiring guide star measurements between each exposure. So far no failures tonight (knock on wood!). Assuming you mean doing full frame guider images as a bench test, no I haven't done that, but sounds like a good idea.

    No, I am not running any scripts - just CCDAP. I don't see the error code you suggest.

    Also, my apologies to the OP (Robert) as I seem to be hijacking his thread!! I don't know whether our root causes are the same but IMHO I think there is a possibility.
     

Share This Page