Resolved All Sky 340M crashing ~ daily.

Discussion in 'Legacy Models - Community Support' started by James Pierce, May 20, 2019.

  1. James Pierce

    James Pierce Cyanogen Customer

    Joined:
    May 20, 2019
    Messages:
    66
    Ok time for an update... I finally got up to my observatory and retrieved the camera. It's been sitting and running on my bench for over a week without fault. Same power supply, same USB adapter. The only differences are the computer (home machine vs EAGLE at the dome) and the serial cable (10m @ the dome, and running underground, so I didn't pull it out). I'm at a bit of a loss about what to try next - thoughts?
     
  2. Colin Haig

    Colin Haig Staff Member

    Joined:
    Oct 27, 2014
    Messages:
    7,379
    Location:
    Earth
    Really there's only a few things left to try.
    - same length of cable at home
    - home computer to the observatory
    - try setting a lower baud rate on the camera when it is at the observatory. Images will download slower, but it may work more reliably.
    - is that 10m of cable 18AWG and shielded or what?
    - a possibility is that the USB to RS232 adapter you are using doesnt have high enough drive voltage to send commands to the camera over 10m of cable. eg real serial ports supply +12/-12V on their data pins; the USB to serial adapters are usually +5/-5V or lower. You could put a voltmeter on the TX pin and GND on the serial adapter and see what it is.
    - a less likely possibility is that the USB to RS232 adapter isn't getting a full +5V from the USB of the Eagle. Try adding a POWERED USB Hub (eg has an AC wall plug adapter, or a StarTech industrial hub running off +12V).
    - reload the Eagle fresh, OS and all.
    BTW Which Eagle is it?
    Beyond that, no idea.
    Good luck,
    Colin
     
  3. James Pierce

    James Pierce Cyanogen Customer

    Joined:
    May 20, 2019
    Messages:
    66
    - same length of cable at home

    Yup - I'll order another 10m serial cable to test this I think.

    - home computer to the observatory

    Yup - Another big job / trips. But yes, I can certainly try this and rule out something with the eagle.

    - try setting a lower baud rate on the camera when it is at the observatory. Images will download slower, but it may work more reliably.

    Have already done this, no improvement.

    - is that 10m of cable 18AWG and shielded or what?

    Heavier duty than that, it's double insulated, but not shielded. It runs through a big conduit with cables for the bolt wood etc also.

    - a possibility is that the USB to RS232 adapter you are using doesnt have high enough drive voltage to send commands to the camera over 10m of cable. eg real serial ports supply +12/-12V on their data pins; the USB to serial adapters are usually +5/-5V or lower. You could put a voltmeter on the TX pin and GND on the serial adapter and see what it is.

    Using the adaptor suggested by SBIG but agree it could be an issue.

    - a less likely possibility is that the USB to RS232 adapter isn't getting a full +5V from the USB of the Eagle. Try adding a POWERED USB Hub (eg has an AC wall plug adapter, or a StarTech industrial hub running off +12V).

    - reload the Eagle fresh, OS and all.
    BTW Which Eagle is it?

    Current version 3 - In retrospect not something I would buy again however, because doing a full clean reinstall isn't possible. Also I'm not about to mess up everything else that's working perfectly because this camera is flakey...

    Beyond that, no idea.
    Good luck,
    Colin

    The thing that still doesn't make sense is that it takes a power cycle to get it going again. It's as if the camera itself is crashing somehow ... if it was just a flakey serial connection that doesn't make sense ... I still wonder if it's got a hardware fault, dry joint etc... and moving the camera jiggles it just right etc.
     
  4. Bradley Walter

    Bradley Walter Cyanogen Customer

    Joined:
    Feb 9, 2015
    Messages:
    31
    James and SBIG,
    Was there a resolution to this problem. We are having a similar problem with our all-sky 340. I am just curious why doesn't anyone make an all-sky IP cam that you can just hook into a network
     
  5. James Pierce

    James Pierce Cyanogen Customer

    Joined:
    May 20, 2019
    Messages:
    66
    I added a huge ground rod for the camera which improved things slightly but I've never got it stable at my dome. I am literally right now rebuilding my setup onto a new Dome computer. I can't replicate the fault at home - so perhaps there is something about the dome computer's USB setup? It's really frustrating, this is not a cheap camera, and it is 10+ year old technology with zero updates for years... it should be bullet proof. There is so much about astro imaging which is stupidly clunky or just really old software and hardware... the curse of a small market.
     
  6. Colin Haig

    Colin Haig Staff Member

    Joined:
    Oct 27, 2014
    Messages:
    7,379
    Location:
    Earth
    James, sorry to hear this. I'm suspecting the same thing - USB/Serial issues.

    Check which kind of serial port / USB to Serial adapter chipset you have. e.g. FTDIchip, as they are very reliable.
    The Prolific ones are not as reliable, but if you are stuck with one, go get the latest driver for ot.

    For the FTDIchip, the current and best driver version is 2.12.28.
    Windows sometimes reverts this to an older driver version, and this causes headaches.
    ftdichip.png
    Check the advanced settings for the driver - turn OFF [] Enable Selective Suspend, and OFF [] Serial Enumeration and ON [x] Disable Modem Control at Startup
    ftdiadvanced.png

    Next, check Windows Power Management settings - USB Selective Suspend should be disabled on all USB devices, USB hubs, USB Root hubs, and PCIe devices.
    For example:
    selectivesuspend.png
    and:
    serialpower.png

    usbnoturnoff.png
    etc.

    Ground loops can be an issue. In my personal observatory, I have a ground rod that is the single-point for all grounding, for the mount, computer, batteries, power control, everything. Check that the shield wire (outer shell) of the serial cabling is properly grounded. Some USB/RS232 adapters don't have proper grounding; Some also don't have more than +/-5V volts, instead of the oldschool reliable RS232.
    Opto-isolator devices also can be helpful, but at the cost of speed limitations.

    The AllSky340 has been bullet-proof, an absolutely rock-solid design, with some shared technology with our SG-4 autoguiders and ST-i cameras.
    In September of this year, ON Semiconductor, the folks that make the CCD chip used in these devices informed us they aren't going to be making these chips after March 2020.
    We'll probably revisit whether it makes sense to do a modernized AllSky camera in the future.
     
  7. Colin Haig

    Colin Haig Staff Member

    Joined:
    Oct 27, 2014
    Messages:
    7,379
    Location:
    Earth
    Hi Bradley, see my response to James' latest, where I outline a bunch of things to check.
    Add any details about your specific situation after you check over my recommendations, if it doesnt resolve it.

    I've been asking some of our customers for their requirements. I'd like to know what are your needs/wants ?
    e.g. minimum sky coverage, resolution, mono/colour, distance from camera to computer, etc.

    We've been thinking about this for some time. There's a bunch of factors - how much resolution is really needed at what frame rate; the optics; weatherproofing; temperature and humidity extremes; lightning/EMI/RFI/ESD protection; distances involved. Wired Ethernet can do a distance of a bit over 300 feet, which just isn't good enough for many professional observatory installations.
    Wireless technology is better, but suceptible to interference (RFI) and may need a licence in some countries.

    Long-term supportability - electronic components have very short lifecycles now, and a lot of these cameras are used for a decade or more, and designing a camera that can be repaired a decade from now is a challenge.

    Then there's the economics, as this tends to be low-quantity stuff, it is hard to get price breaks on components. If it was a security-camera solution, and we were selling tens or hunders of thousands of units, its easier to get the cost down.

    The homebrew - put a webcam on a cheap PC or just use an IP security cam isn't going to be long-term reliable.

    So, am all ears to understand what you're looking for. We may be able to develop something based on some new camera technology we're working on now.
     
  8. James Pierce

    James Pierce Cyanogen Customer

    Joined:
    May 20, 2019
    Messages:
    66
    Yup - I've tried all those things... and so much more. I've ultimately replaced my dome computer as a last resort. The camera has been totally reliable on the bench at home, and infuriating in the field! At this stage I am hopefully that the Eagle has a crappy USB chipset, or some other quirk which has been the cause of so much frustration. An expensive exercise.

    Re a new camera - Very pleased to hear you are considering a modern iteration. From my perspective...

    The primary use is to asses sky conditions remotely to check and second guess the weather station.
    A full sky view (or very close too) is essential.
    Mono is fine, thought a good colour camera would be fine also.
    More resolution would be nice, but not at the cost of sensitivity.
    One image every 5 mins is fine.
    A cooled chip, which allowed for lower noise, and practical usage for things like chasing meteorite showers etc would be nice... I can appreciate the potential impact on longevity though for an always on device.

    I can really appreciate the complexities of wired vs wireless and a standalone or tethered unit.
    I'd have thought ethernet was a pretty universal standard. If users need longer distances then there are simple ways to extend (fibre, wireless bridges etc)

    The longevity issue is true for nearly all good astro gear, reinstalling a new dome computer has been a painful reminder of how many quirks and hacks exist to get around old hardware, old protocols on new operating systems...
     
  9. William B

    William B Cyanogen Customer

    Joined:
    Jan 8, 2015
    Messages:
    548
    Location:
    Christchurch, Dorset UK
    Hi James & Bradley.

    James, did you ever try a galvanic isolator on the RS232 port (optical isolator) to eliminate the ground loop possibility, as discussed earlier in the thread (and in the All Sky manual) ?

    With a lifetime experience of installing and maintaining RS232 comm links in medical systems the commonest RS232 failures found during commissioning and during equipment lifetime maintenance are ground loops and incorrect cabling.

    Is the cable the correct specification for RS232 comms and has it been correctly terminated and installed?

    For reliable RS232 comms above 9600 baud, up to 115,200 baud, with cable runs over a couple of meters the cable must be low capacitance, low impedance and the Rx -Tx cores should be twisted-pairs with an overall external shield. For baud rates above 115,200 the Rx -Tx cores should be twisted-pairs with individual shields and an overall cable shield.

    In a correctly installed RS232 cable, the outer shield is connected to chassis ground only and not 0v (chassis ground and 0v are only allowed to be connected at a single point, usually in the power supply, and if the host computer and connected equipment are on different power supply networks the shield is connected to chassis ground at one end only and isolated at the other, if on the same power supply network the shield is connected to chassis ground at both ends.
    Where the host end and the client end are on different power supply networks a galvanic (optical) isolator on the RS232 cable is essential.
    The Rx and Tx twisted pairs are Rx + 0v and Tx +0v with the 0v’s of both Rx and Tx connected to signal 0v at both ends.
    The commonly held belief that only three cores are required, Rx, Tx and 0v over a parallel-cored cable is often quoted but is vulnerable to interference both from external sources and internally via cross-talk between the Rx and Tx signals and the problems increase with the length of the cable run.

    The commonest cabling errors are using unshielded and untwisted-pair (parallel-core) cabling, or using twisted-pair but where the Rx and Tx have been twisted together instead of Rx + 0v and Tx + 0v.

    The correct specification Rs232 cable (shielded w’ twisted pair cores) can be expensive. In the medical environment I am familiar with a four-core, twin twisted pair, single shielded, low-smoke toxicity RS232 cable is around €2-€3 per meter however for an observatory a suitable low-cost alternative for baud rates up to 115,200 is regular CAT5 STP cable (STP = Shielded Twisted Pair).
    If using CAT5 STP cable the unused cores must be connected to signal 0v at both ends and not left floating.

    In the systems I worked on, low frequency, sporadic, RS232 comms errors with a frequency extending over several hours or days tended to be ground-loop induced. As the load on the building power supply changes with usage and weather so the potential difference between mains neutral and ground varies, leading to variable ground-loop currents flowing along the RS232 cable. In some cases cable-duct fires were the result, and is the reason we always specified galvanic isolators for RS232 equipment interconnections outside of our own local domain. High frequency errors, occurring over minutes, tended to be the wrong specification of cable, or incorrectly terminated.

    If your system works reliably on the bench but not when installed then an impermissible ground loop would be the first place to look but if you are using an unshielded and un-twisted core cable that is running alongside other cables, particularly cables carrying mains AC and/or in metal ducting, then the cable specification can also be the reason.

    Start with the correct type of cable, properly installed and terminated, screened twisted-pair and unused cores tied to signal 0v, screen tied to chassis ground at one end only, if the problem remains install a galvanic isolator.

    A test of the RS232 Rx and Tx signals at host and client ends of the cable with an oscilloscope will show pretty much instantly if signal smearing, cross talk or mains frequency AC superimposition are present, if you have access to an oscilloscope or know anyone that can check for you it will save a lot of head scratching.

    HTH

    William.
     
  10. James Pierce

    James Pierce Cyanogen Customer

    Joined:
    May 20, 2019
    Messages:
    66
    James, did you ever try a galvanic isolator on the RS232 port (optical isolator) to eliminate the ground loop possibility, as discussed earlier in the thread (and in the All Sky manual) ?

    Yes ... I have used a number of different USB convertors, currently a StarTech Opto Isolator. Doesn't improve my issue.

    With a lifetime experience of installing and maintaining RS232 comm links in medical systems the commonest RS232 failures found during commissioning and during equipment lifetime maintenance are ground loops and incorrect cabling.

    Is the cable the correct specification for RS232 comms and has it been correctly terminated and installed?


    I believe so, I have tried a number of different commercial cables. No difference between them.

    For reliable RS232 comms above 9600 baud, up to 115,200 baud, with cable runs over a couple of meters the cable must be low capacitance, low impedance and the Rx -Tx cores should be twisted-pairs with an overall external shield. For baud rates above 115,200 the Rx -Tx cores should be twisted-pairs with individual shields and an overall cable shield.

    In a correctly installed RS232 cable, the outer shield is connected to chassis ground only and not 0v (chassis ground and 0v are only allowed to be connected at a single point, usually in the power supply, and if the host computer and connected equipment are on different power supply networks the shield is connected to chassis ground at one end only and isolated at the other, if on the same power supply network the shield is connected to chassis ground at both ends.

    Where the host end and the client end are on different power supply networks a galvanic (optical) isolator on the RS232 cable is essential.
    The Rx and Tx twisted pairs are Rx + 0v and Tx +0v with the 0v’s of both Rx and Tx connected to signal 0v at both ends.
    The commonly held belief that only three cores are required, Rx, Tx and 0v over a parallel-cored cable is often quoted but is vulnerable to interference both from external sources and internally via cross-talk between the Rx and Tx signals and the problems increase with the length of the cable run.

    The commonest cabling errors are using unshielded and untwisted-pair (parallel-core) cabling, or using twisted-pair but where the Rx and Tx have been twisted together instead of Rx + 0v and Tx + 0v.

    The correct specification Rs232 cable (shielded w’ twisted pair cores) can be expensive. In the medical environment I am familiar with a four-core, twin twisted pair, single shielded, low-smoke toxicity RS232 cable is around €2-€3 per meter however for an observatory a suitable low-cost alternative for baud rates up to 115,200 is regular CAT5 STP cable (STP = Shielded Twisted Pair).
    If using CAT5 STP cable the unused cores must be connected to signal 0v at both ends and not left floating.

    In the systems I worked on, low frequency, sporadic, RS232 comms errors with a frequency extending over several hours or days tended to be ground-loop induced. As the load on the building power supply changes with usage and weather so the potential difference between mains neutral and ground varies, leading to variable ground-loop currents flowing along the RS232 cable. In some cases cable-duct fires were the result, and is the reason we always specified galvanic isolators for RS232 equipment interconnections outside of our own local domain. High frequency errors, occurring over minutes, tended to be the wrong specification of cable, or incorrectly terminated.

    If your system works reliably on the bench but not when installed then an impermissible ground loop would be the first place to look but if you are using an unshielded and un-twisted core cable that is running alongside other cables, particularly cables carrying mains AC and/or in metal ducting, then the cable specification can also be the reason.

    The camera is at the other end of ~10m of serial cable which runs along side a number of other cables... Ethernet for an SQM and the serial cable for my Boltwood. Plus 12v for the camera itself. If I still have issues after the computer switch out them I'm prepared to pull the serial cable and swap it over, but it's under ground in a conduit and it will be a huge job.

    Start with the correct type of cable, properly installed and terminated, screened twisted-pair and unused cores tied to signal 0v, screen tied to chassis ground at one end only, if the problem remains install a galvanic isolator.

    A test of the RS232 Rx and Tx signals at host and client ends of the cable with an oscilloscope will show pretty much instantly if signal smearing, cross talk or mains frequency AC superimposition are present, if you have access to an oscilloscope or know anyone that can check for you it will save a lot of head scratching.


    Now that isn't something I had considered to test this... I can probably get my hands on something from an Amateur Radio buddy.

    HTH

    William.
     
  11. William B

    William B Cyanogen Customer

    Joined:
    Jan 8, 2015
    Messages:
    548
    Location:
    Christchurch, Dorset UK
    As you have used an opto-isolator you can rule out ground loops, at least on the signal cable.

    A couple of other things to try, fit ferrite cable clamps over the RS232 and camera power cables at the host computer and camera ends to reduce the chance of parasitic interference.
    (For reliability, ferrite clamps should be on both ends of all low voltage power and signalling cabling in an observatory).

    If excess cable is laying around it should not be tidied in continuous coils but tidied in a figure-of-eight pattern.

    If the oscilloscope test shows significant signal voltage drop on the RS232 cable and if your host computer has a spare expansion slot then a dedicated RS232 expansion card is a much better option as you will have the full 12v Rx - Tx signal available against just 5v with a USB-RS232 adaptor.

    William.
     
    Last edited: Dec 8, 2019
  12. William B

    William B Cyanogen Customer

    Joined:
    Jan 8, 2015
    Messages:
    548
    Location:
    Christchurch, Dorset UK
    One final thought...

    Make sure the power jack at the camera end really is 5.5/2.0 and not 5.5/2.1 or even 5.5/2.5, and inspect the plug to make sure the solder tags and jack pins are tightly riveted, had a few spurious faults over the years due to loose riveted solder tags on jack plugs or the wrong sized plugs supplied in wrongly labelled packs.
     
  13. James Pierce

    James Pierce Cyanogen Customer

    Joined:
    May 20, 2019
    Messages:
    66
    A week with the new dome computer up and running and zero faults. Fingers crossed this might finally be solved. I will indeed install some ferrules - I already have a box of them. I'll update this thread again in another week or so hopefully with the same positive prognosis.
     
  14. James Pierce

    James Pierce Cyanogen Customer

    Joined:
    May 20, 2019
    Messages:
    66
    And now another week and a half on the camera continues to work without a single fault. What a saga this has been.

    In summary regardless of which serial cable, power cable or USB serial convertor the same issue occurred with the EAGLE computer. It crashed the 340 camera such that a power cycle was the only way to recover.

    As a final (and rather expensive) step replacing the computer with a FIT PC IPC2 would appear to have finally resolved the problem. Thanks all for your patience, even when I was frustrated or frustrating!
     
  15. Colin Haig

    Colin Haig Staff Member

    Joined:
    Oct 27, 2014
    Messages:
    7,379
    Location:
    Earth
    Hi James, am glad to hear you are back in business.
    Merry Christmas!
     

Share This Page