Resolved All Sky 340M crashing ~ daily.

Discussion in 'Legacy Models - Community Support' started by James Pierce, May 20, 2019.

Page 1 of 2
  1. James Pierce

    James Pierce Cyanogen Customer

    Joined:
    May 20, 2019
    Messages:
    66
    We've had our 340M up and running for a couple of months now. For the first month it was 100% reliable. Then the camera 'crashed' and had to be power cycled to come back online... It worked fine for around a week... Then it 'crashed' and had to be power cycled to come back online... It worked fine for a couple of days... and so-on ... now it seems to crash roughly once a day. There isn't an obvious pattern.

    Sometimes it happens in the middle of the day when the heater is turned off, sometimes it happens at night. There isn't an error message, just the software stops. Power cycling the camera brings it back on line every time but it's extremely frustrating.

    All and any advice of where to start appreciated please.
     
    James Pierce, May 20, 2019
    #1
  2. Doug

    Doug Staff Member

    Joined:
    Sep 25, 2014
    Messages:
    10,316
    That’s pretty much unheard-of.

    First thing - check your power.
     
    Doug, May 20, 2019
    #2
  3. James Pierce

    James Pierce Cyanogen Customer

    Joined:
    May 20, 2019
    Messages:
    66
    Pretty sure I'm ok on that front. I did have to cut and extend the cable when I first installed it (approx 5m of heavy gauge cable). The whole observatory is powered via a UPS so I'm fairly confident that's not a problem. The camera has alternatively be powered directly from the SBIG power supply and my eagle computer (which has regulated and controllable 12v outputs). The benefit of powering it via the eagle computer is I can remotely reboot without having to get someone out to the observatory (which is remote). The behaviour has been the same regardless of the power supply used.

    I'd have thought if it was a current / voltage delivery issue that it would happen when the heater kicks in, but that hasn't been the pattern.
     
    Last edited: May 20, 2019
    James Pierce, May 20, 2019
    #3
  4. Doug

    Doug Staff Member

    Joined:
    Sep 25, 2014
    Messages:
    10,316
    So you have to actually power-cycle the camera? That suggests a hardware issue. If it isn’t the power then it may have to come in for service. Please confirm that the symptoms are the same powered both ways.

    I would suggest letting it run for a bit longer; if it’s a rare intermittent problem that is very hard to diagnose. If it gets worse or fails completely the problem will be more easily identified and fixed.
     
    Doug, May 20, 2019
    #4
  5. William B

    William B Cyanogen Customer

    Joined:
    Jan 8, 2015
    Messages:
    641
    Location:
    Christchurch, Dorset UK
    Though not an Eagle owner/user myself I have been following reports on another astronomy forum of flaky USB port control with random disconnects and other issues and that these were fixed by updating the Eagle manager software (Eagle 2 v2.7, Eagle 3 v3.2 respectively).

    Probably not the issue this time but might be worth checking that you are running the latest Eagle manager software:

    https://www.primalucelab.com/astronomy/downloads

    I see that you are running a UPS, If this is an offline/standby type have you cycled it lately?

    I have had many issues over the years with offline/standby UPS devices in critical medical systems where the fast change-over relay thats switches between mains input and battery backup suffers from contact tarnishing and becomes electrically very noisy, causing disturbance to all devices downstream of the UPS. As part of our three monthly test and inspection visits we were required to deliberately cycle the UPS by removing the power input, a) to check the battery backup time capacity and b) to clean the contacts of the changeover relay. If you haven't cycled the UPS recently this is something to try, disconnect and reconnect the mains input to the UPS a few times while it has a reasonable load connected on the output and the arcing of the relay contacts as they change over will clean any tarnishing.

    If your UPS is an on-line / line-interactive type then this does not apply since there is no changeover relay and the UPS is always active.

    Lastly, if any part of your All Sky camera is physically grounded, and thus providing a path to ground on the 0V power supply side, check for leakage currents flowing along the USB cable from your Eagle computer, via the USB cable and down to ground via the camera. I am recently aware of issues with an Eagle and a MESU mount that kept randomly disconnecting the USB and this proved to be due to leakage currents from the Eagle to ground via the MESU mount and earthed pier, once a USB optical isolator was installed between the Eagle and the MESU the problems were solved.

    HTH

    William.
     
    William B, May 21, 2019
    #5
  6. James Pierce

    James Pierce Cyanogen Customer

    Joined:
    May 20, 2019
    Messages:
    66
    Update: The last few days the camera has been freezing about 2-3 times a day. The observatory is remote so I will have to wait until I can make a trip to try a few things. One obvious one is to go back to the standard power supply - With my scope-dome I can remotely switch the 240v side so that will still give me a get out of jail path if it is still crashing. The camera is indeed installed on a steel post so that could be providing a ground. The UPS is fairly new though so I doubt that is the issue.
     
    James Pierce, May 23, 2019
    #6
  7. James Pierce

    James Pierce Cyanogen Customer

    Joined:
    May 20, 2019
    Messages:
    66
    Update: I visited the observatory on the weekend. I took the chance to run a new heavier gauge power cable from the observatory to the camera without any joins and soldering on a locking 2.1mm connector. Despite using two lots of double wall, glue seal heat shrink I wonder if some moisture had gotten into a join somewhere? Regardless so far the camera has run for 3.5 days without issue. I won't claim victory yet, but full credit to you Doug - it does indeed look as if it was a power problem. Fingers crossed.
     
    James Pierce, Jun 11, 2019
    #7
  8. Doug

    Doug Staff Member

    Joined:
    Sep 25, 2014
    Messages:
    10,316
    Crossing fingers here too.
     
    Doug, Jun 11, 2019
    #8
  9. James Pierce

    James Pierce Cyanogen Customer

    Joined:
    May 20, 2019
    Messages:
    66
    Spoke too soon ... has 'crashed' at least once every 24hrs for the last 3 days. Restarting the software, and re-setting the USB port doesn't work, have to do a power cycle again. I'll keep monitoring and see how it goes. FYI I switched back to the SBIG power supply on the weekend to rule out some quirk of the Eagle computer too.
     
    Last edited: Jun 13, 2019
    James Pierce, Jun 11, 2019
    #9
  10. James Pierce

    James Pierce Cyanogen Customer

    Joined:
    May 20, 2019
    Messages:
    66
    Yup. Crashing atleast once a day. Where to from here?
     
    James Pierce, Jun 16, 2019
    #10
  11. Doug

    Doug Staff Member

    Joined:
    Sep 25, 2014
    Messages:
    10,316
    Eliminate environmental factors. Bring the unit indoors. Hook it up on the bench with the included power supply and connect directly to the computer. Run continuously and see if it crashes.
     
    Doug, Jun 16, 2019
    #11
  12. James Pierce

    James Pierce Cyanogen Customer

    Joined:
    May 20, 2019
    Messages:
    66
    Will do. Going to have to wait until I can next make a trip to the observatory unfortunately. If I wasn't on the other side of the world I'd just send it in for you guys to take a look! Oh well... the joys of our chosen passion.
     
    James Pierce, Jun 16, 2019
    #12
  13. Doug

    Doug Staff Member

    Joined:
    Sep 25, 2014
    Messages:
    10,316
    It is a real head-scratcher for sure. It's possible there's a problem with the camera, but we've never seen a symptom like this before. So if you ship it back there's a good chance Bill won't find anything, and then he'll send it back and you'll still have the problem. It's worth doing every experiment we can think of first.
     
    Doug, Jun 16, 2019
    #13
  14. James Pierce

    James Pierce Cyanogen Customer

    Joined:
    May 20, 2019
    Messages:
    66
    Stupid question but there are no logs to check, or enable to try and get some more diagnostics on what is going on?
     
    James Pierce, Jun 16, 2019
    #14
  15. James Pierce

    James Pierce Cyanogen Customer

    Joined:
    May 20, 2019
    Messages:
    66
    Just adding the log from this morning for about ~3hrs . You can see it flaked out at around 6:10am this morning. There are a few other weird errors earlier than that - does that shed any light?
     

    Attached Files:

    James Pierce, Jun 17, 2019
    #15
  16. Colin Haig

    Colin Haig Staff Member

    Joined:
    Oct 27, 2014
    Messages:
    7,962
    Location:
    Earth
    James, this looks almost exactly like a Windows Power Management problem with USB Selective Suspend powering off the USB to serial device on you.
    There's a few parts to fixing this.
    1) Find out what USB to Serial adapter you have. Get the latest driver from FTDIchip.
    2) Turn off Windows Power Management - USB Selective Suspend
    selectivesuspend.png
    3) Run devmgmt.msc (The good old device manager). For each USB device, USB hub, USB root hub, COM Port, make sure that the option to Allow Windows to Power Off the device is OFF (disabled). You may also have to do this for PCIe devices.

    Give it a shot.
     
    Colin Haig, Jun 17, 2019
    #16
  17. James Pierce

    James Pierce Cyanogen Customer

    Joined:
    May 20, 2019
    Messages:
    66
    Ok...

    * Latest driver already installed (and using the adaptor recommended by SBIG).
    * Windows power management USB selective suspend was already disabled.
    * A couple of the USB hubs where however not set to disabled, so I've changed them all now.

    If it was a USB issue I would have thought that resetting the port would solve the problem would it not ?
     
    James Pierce, Jun 17, 2019
    #17
  18. Colin Haig

    Colin Haig Staff Member

    Joined:
    Oct 27, 2014
    Messages:
    7,962
    Location:
    Earth
    Not necessarily. You never know what state the device driver and port get into if the port was open when it is powered down by Windows. The port was open (the handle to the device from the software), but if windows powers it off, there will be no communications and the chip won't respond. The software has no clue other than the port went away or became unresponsive.
    Let's eliminate this as a problem.
     
    Colin Haig, Jun 17, 2019
    #18
  19. James Pierce

    James Pierce Cyanogen Customer

    Joined:
    May 20, 2019
    Messages:
    66
    Yup understand. Have disabled the power off option for all USB devices in Device Manager... will monitor and see how it goes from here.
     
    James Pierce, Jun 17, 2019
    #19
  20. James Pierce

    James Pierce Cyanogen Customer

    Joined:
    May 20, 2019
    Messages:
    66
    Another hour of log from before it crashed again this am.

    What do these codes that appear right through the log mean? They always appear in the log between the exposure and reading the CCD. They don't appear every exposure.

    #Note this is just a sample...

    06/18/2019 3:09:28 AM - Unknown status: 52

    06/18/2019 3:11:29 AM - Unknown status: 52
    06/18/2019 3:11:29 AM - Unknown status: 103

    06/18/2019 3:15:32 AM - Unknown status: 103

    06/18/2019 3:17:33 AM - Unknown status: 11

    06/18/2019 3:31:40 AM - Unknown status: 103
    06/18/2019 3:31:40 AM - Unknown status: 84
     

    Attached Files:

    James Pierce, Jun 17, 2019
    #20
Page 1 of 2

Share This Page