I really hate to be back here with issues, but we continue to have problems with this 455 camera system. We are having two issues, one we have known about for some time and have been trying to solve on our end, and the other we just discovered, or rather diagnosed, would be more accurate. The first is that it wants to drop off line. We recently spent money to buy a new computer with higher-grade memory. Reloaded all the software and drivers from scratch, implementing all recommended settings, and replaced the fiber optic with high-grade commercial materials to operate in wide temperature ranges, and the camera still periodically disconnects. But worse, we have been getting weird results where exposures will result in a fraction of the anticipated ADU counts. Doug ran a set of test exposures in which each filter was repeated in a sequence of ten exposures, causing the filters to change after each exposure, and walla! The filter wheel is not consistently going to the proper filter slot. So, for example, the blue image will be a beautiful pinpoint image, then the next one will be donuts, and then you might get a couple more good ones, and the next will be the dark filter - no stars, then a good one, etc. It occurs to us - indeed, we hope - that there is a connection between the camera dropping off and this filter issue. When it works, it is great, but it is useless as it is. Please advise Bruce McMath
PS, new information. I shared the above with Danny Flippo. He indicates that he recently added new filters to the wheel, and when he did, some magnets fell out. He attempted to reinstall them, but it would seem this is likely the issue with the wheel. Please advise. Apparently they need to be glued and the orientation needs to be correct. Can we address this outselves, with your guidance, or must we ship it in again?
To clarify a bit, I installed industrial grade SFP+ modules with a -40 to 85C temperature range. We're using an OM-3 fiber optic cable between the camera and the PC.
Colin, so we got the filter issue resolved, but the camera remains unreliable. We keep getting failures that throw error messages like this: "07:56:26 **Pointing update error from MaxIm.CCDCamera: **Exposure failed after being successfully started"
Hi Bruce, What's that error message from? eg what software created that error message? Can you send the logs from whatever that software is, and the corresponding MaxIm logs?
Colin, sorry to be slow getting back on this, but we are trying to solve this ourselves before bothering you, but it keeps happening. Here is an excerpt from the ACP log and the Maxim Log is attached for the latest. 03:52:19$14 DL Imaging[1] : No AO sensor detected 03:53:17>4 Completed 56.000s 3186x2126 exposure, binned 3x3, ROI (0, 0) 03:53:27>4 Started 56.000s 3186x2126 exposure, binned 3x3, ROI (0, 0) 03:53:27$14 DL Imaging[1] : No AO sensor detected 03:54:24>4 Completed 56.000s 3186x2126 exposure, binned 3x3, ROI (0, 0) 03:54:34>4 Started 56.000s 3186x2126 exposure, binned 3x3, ROI (0, 0) 03:54:34$14 DL Imaging[1] : No AO sensor detected 03:55:15$5 Guider algorithm 'Single Star' ended 03:55:15$5 Guider Calibration Completed 03:55:15$4 Camera Exposure Aborted 03:56:10$5 Guider algorithm 'Single Star' ended 03:56:10$5 Guider Calibration Completed 03:56:57$14 DL Imaging[1] : Retry 1/3 :: Promise timed out. 03:56:57$14 DL Imaging[1] : Retry 2/3 :: Promise timed out. 03:57:09$3 Disconnected from filter wheel 1 "DL Imaging+FW" 03:57:09$3 Disconnected from camera 2 "DL Imaging" 03:57:09$3 Disconnected from camera 1 "DL Imaging" 03:57:35>3 Connecting cameras - thread ID 13492 03:57:37$3 Connected to camera 1 "DL Imaging" (CCDPlugDLImaging.dll v7.3.1.0) 03:57:38$3 Connected to camera 2 "DL Imaging" (CCDPlugDLImaging.dll v7.3.1.0) 03:57:38$3 Connected to filter wheel 1 "DL Imaging+FW" (CCDPlugDLImaging.dll v7.3.1.0) 03:57:39>4 Filter wheel moving to position 7 03:58:26>4 Filter wheel moving to position 1 04:15:21$5 Guider algorithm 'Single Star' ended 04:15:21$5 Guider Calibration Completed 04:26:54$4 Camera Exposure Aborted 04:26:54$14 DL Imaging[1] : Attempting connection to camera with S/N: AC455M-25012404 04:26:54$14 DL Imaging[1] : Connected to camera with S/N: AC455M-25012404 04:26:54$14 DL Imaging[1] : Aluma Camera Flag State: 0 04:26:54$14 DL Imaging[1] : Attempting connection to camera with S/N: SCE1300M-22032208 04:26:54$14 DL Imaging[1] : Connected to camera with S/N: SCE1300M-22032208 04:26:54$14 DL Imaging[1] : Aluma Camera Flag State: 0 04:26:54$14 DL Imaging[1] : Retry 1/3 :: 04:27:27*6 View, Information Window (closing) 04:30:23$3 Disconnected from filter wheel 1 "DL Imaging+FW" 04:30:23$3 Disconnected from camera 2 "DL Imaging" 04:30:23$3 Disconnected from camera 1 "DL Imaging" 04:30:26>3 Connecting cameras - thread ID 13492 04:30:28>3 Connecting cameras - thread ID 13492 04:30:29$3 Connected to camera 1 "DL Imaging" (CCDPlugDLImaging.dll v7.3.1.0) 04:30:29$3 Connected to camera 2 "DL Imaging" (CCDPlugDLImaging.dll v7.3.1.0) 04:30:29$3 Connected to filter wheel 1 "DL Imaging+FW" (CCDPlugDLImaging.dll v7.3.1.0) 04:30:30$14 DL Imaging[1] : Attempting connection to camera with S/N: AC455M-25012404 04:30:30$14 DL Imaging[1] : Connected to camera with S/N: AC455M-25012404 04:30:30$14 DL Imaging[1] : Aluma Camera Flag State: 0 04:30:30$14 DL Imaging[1] : Attempting connection to camera with S/N: SCE1300M-22032208 04:30:30$14 DL Imaging[1] : Connected to camera with S/N: SCE1300M-22032208 04:30:30$14 DL Imaging[1] : Aluma Camera Flag State: 0 04:30:30$14 DL Imaging[1] : Retry 1/3 :: 04:30:32>3 Camera 1 Cooler Off, Setpoint: -5.00° 04:30:32>3 Camera 2 Cooler Off, Setpoint: -20.00° 04:30:37$3 Disconnected from filter wheel 1 "DL Imaging+FW" 04:30:37$3 Disconnected from camera 2 "DL Imaging" 04:30:37$3 Disconnected from camera 1 "DL Imaging" 04:30:46>1 MaxIm DL Version 7.3.1.0 exited 04:50:22>1 MaxIm DL Version 7.3.1.0 started on RRT-2 04:50:37>3 Connecting cameras - thread ID 5560 04:50:40$3 Connected to camera 1 "DL Imaging" (CCDPlugDLImaging.dll v7.3.1.0) 04:50:40$3 Connected to camera 2 "DL Imaging" (CCDPlugDLImaging.dll v7.3.1.0) 04:50:40$3 Connected to filter wheel 1 "DL Imaging+FW" (CCDPlugDLImaging.dll v7.3.1.0) 04:50:41$14 DL Imaging[1] : Attempting connection to camera with S/N: AC455M-25012404 04:50:41$14 DL Imaging[1] : Connected to camera with S/N: AC455M-25012404 04:50:41$14 DL Imaging[1] : Aluma Camera Flag State: 0 04:50:41$14 DL Imaging[1] : Attempting connection to camera with S/N: SCE1300M-22032208 04:50:41$14 DL Imaging[1] : Connected to camera with S/N: SCE1300M-22032208 04:50:41$14 DL Imaging[1] : Aluma Camera Flag State: 0 04:50:41$14 DL Imaging[1] : Retry 1/3 :: 04:51:11>3 Camera 1 Cooler On, Setpoint: -5.00° 04:51:13>3 Camera 1 Cooler On, Setpoint: -5.00° 04:51:16$5 Guider algorithm 'Single Star' ended 04:51:16$5 Guider Calibration Completed 04:51:44$5 Guider algorithm 'Single Star' ended 04:51:44$5 Guider Calibration Completed 05:01:18>3 Camera 1 Cooler On, Setpoint: -5.00° 05:01:20>3 Camera 1 Cooler On, Setpoint: -5.00° 05:01:20$5 Guider algorithm 'Single Star' ended 05:01:20$5 Guider Calibration Completed 05:01:36$14 DL Imaging[1] : Retry 1/3 :: Promise timed out. 05:01:36$14 DL Imaging[1] : Retry 2/3 :: Promise timed out. 05:01:50$3 Disconnected from filter wheel 1 "DL Imaging+FW" 05:01:50$3 Disconnected from camera 2 "DL Imaging" 05:01:51$3 Disconnected from camera 1 "DL Imaging" 05:02:13>3 Connecting cameras - thread ID 5560 05:02:16$3 Connected to camera 1 "DL Imaging" (CCDPlugDLImaging.dll v7.3.1.0) 05:02:17$3 Connected to camera 2 "DL Imaging" (CCDPlugDLImaging.dll v7.3.1.0) 05:02:17$3 Connected to filter wheel 1 "DL Imaging+FW" (CCDPlugDLImaging.dll v7.3.1.0) 05:02:17$14 DL Imaging[1] : Attempting connection to camera with S/N: AC455M-25012404 05:02:17$14 DL Imaging[1] : Connected to camera with S/N: AC455M-25012404 05:02:17$14 DL Imaging[1] : Aluma Camera Flag State: 0 05:02:17$14 DL Imaging[1] : Attempting connection to camera with S/N: SCE1300M-22032208 05:02:17$14 DL Imaging[1] : Connected to camera with S/N: SCE1300M-22032208 05:02:17$14 DL Imaging[1] : Aluma Camera Flag State: 0 05:02:17$14 DL Imaging[1] : Retry 1/3 :: 05:02:17>4 Filter wheel moving to position 1 05:03:06>4 Started 10.000s 3585x3190 exposure, binned 2x2, ROI (598, 0) 05:03:06$14 DL Imaging[1] : No AO sensor detected 05:03:17>4 Completed 10.000s 3585x3190 exposure, binned 2x2, ROI (598, 0) 05:03:42>4 Started 10.000s 3585x3190 exposure, binned 2x2, ROI (598, 0) 05:03:42$14 DL Imaging[1] : No AO sensor detected 05:03:53>4 Completed 10.000s 3585x3190 exposure, binned 2x2, ROI (598,
Please don't copy/paste the whole log into the message - [Upload a File] is just fine. Please attach the ACP log files - the MaxIm log and a screen shot is attached. Unfortunately, the log isn't telling me which of the two cameras is disconnecting, but it seems like the guider is very active at the time, and one of the two cameras is timing out. Suggestions: 1. Update MaxIm to 7.3.2. Help... Check for Updates. 2. Download the latest firmware versions for the AC455 and the SC-4 guider. You can find them here: https://diffractionlimited.com/aluma-fpga-firmware/ 3. Then use MaxIm 7.3.2 to check what firmware version is in each camera. Camera Control - Camera Setup - Advanced - to check the firmware version in the cameras. Check the version of both the AC455 and the SC-4. Update the AC455 camera to the latest version (FPGA rev 31 as of today). Use the [Firmware] button on that advanced screen in MaxIm. Power cycle off for 60s after completing the firmware update. Exit MaxIm. Now repeat the update process for the SC-4. (version 8 came out in January), and update that as well. Make sure to power off the SC-4 after updating, and exit MaxIm. The firmware update process is similar to this: 4. Can someone provide a diagram of what is plugged into where, and what the lengths of cables are, and any hubs/extenders? 5. We could do a remote session next week.
6. Have you guys run continous images on either camera independently? e.g. Do several hundred throw-away images, one camera at a time, to see whether it is dropping out, and which camera is dropping out? Try this with only one camera connected in MaxIm. eg set the other one to "Simulator". 7. Have you looked in the Windows Event Log for any clues as to what is happening? eg hardware issues, windows updates running, application or system errors? Will let you know if I think of anything else.
Loading the latest firmware, the camera would not download an image at all. We are trying to get data for some students, so we reversed the firmware. Can't say at the moment how long the USB cables are, but the fiber optic cable runs directly from the AC455 camera to the PC and it is 10 meters long. It is using name brand industrial temperature range transceivers in both the camera and the PC. We recently upgraded the fiber optics and the computer in an effort to address this. I don't think we are concerned with the guide camera, only the AC455 appears to be losing its connection. We would appreciate a remote session.
When you say "the camera would not download an image at all", what was happening? Did the camera connect? Did the exposure start? Did it start reading and/or downloading? Were there any messages? When can we do a remote session? I'll need you (or whoever facilitates the session) to download and run this remote assistance tool: https://static.realvnc.help/generic/3.2.0/On-Demand-Assist.exe Send me a Private Message with a couple of times when someone can be available, and a telephone number.
Maxim DL 7 pro, version 7.3.2 installed and working. Alumina AC455 - 25012404, FPGA 27 NOTE: FPGA 28 and 29 fail to download images. SCE1300M-22032208, FPGA 8 We are not using this camera at the moment but would like to do so in the future. ACP observatory control guiding is deactivated. We have tried turning the camera off. It makes no difference. The AO is disabled in the set up. We have the standard optical train arrangement from DL. We are using 10 meter OME 3 fiber optics to connect to the camera. Note that the computer is relatively new and we built a custom support for the optical cable that serves as a strain relief. While there could be a mechanical problem we think that is unlikely since the camera doesn't download with versions 28 or 29, but at least downloads with version 27. Doug I updated the FPGA firmware on the camera to the latest which was version 31. After that, I connected Maxim to the camera and tried to take a test image. Maxim displayed a “downloading” message but the download would never finish. So I started reverting the FPGA firmware one version at a time and re-testing. I had to go all the way back to version 27 to get an image to download in Maxim. All of this was tested after installing the latest Maxim update. Danny Attached are logs for the period when the download would not happen.
Notes: Doug W and I did a remote session today. The camera is currently connected to the Intel x710 network card #2. We made the following changes (Danny should probably take note): 1. Force connection to the camera to stay at 10Gbps Full Duplex. Device Manager - Properties - Advanced Speed and Duplex - change from Auto Negotiation to 10 Gbps Full Duplex 2. Network interface metric to prevent Windows from trying to route all network traffic over the fastest link that only has a camera on it. Control Panel Network and Sharing Center For the 10GbE adapter that talks to the camera Properties Internet Protocol Version 4 (TCP/IPv4) Properties Advanced Uncheck (disable) [ ] Automatic Metric then set the value of Interface Metric: 1000 We then shot over 1100 images continuously, with no connection failures and no stuck in Download. Since it looks like it will be clear tonight, the users will be running the system. If there are any disconnect or other issues overnight, Doug will follow up with me (Colin) tomorrow. We decided to only make these necessary changes, and see how it goes. We did not do any optimization. We did not update the camera firmware from 27 to 31, to keep it simple and stable. My colleagues here want to do a bit of analysis work on the firmware differences.
