Application.CalMasterFlat question

Discussion in 'Scripting and Programming' started by CraigNZ, Dec 8, 2019.

  1. CraigNZ

    CraigNZ Cyanogen Customer

    Joined:
    Dec 29, 2014
    Messages:
    86
    Location:
    Ngutunui, New Zealand
    So far I have used:
    Application.CalMasterBias to create a master bias image and saved to disk.
    Application.CalMasterDark, same.

    Now I want to create a master flat using CalMasterFlat. Like the other two, I have used CalAddFlat to add the flat images, but how do I specify the Master Dark (created earlier) to dark correct the flat images?

    Do I simply do this:

    Application.CalClear
    Application.CalAddDark(master dark)
    Application.CalAddFlat(for each flat frame)
    Application.CalSet
    Application.CalMasterFlat

    Craig
     
  2. CraigNZ

    CraigNZ Cyanogen Customer

    Joined:
    Dec 29, 2014
    Messages:
    86
    Location:
    Ngutunui, New Zealand
    I didn't get any response so decided to go ahead and try it.

    When I get to CalAddDark to add the master Dark frame I get this error message:

    "Failed to add a file to the group, because it was corrupt".

    I manually opened the Master Dark in MaximDL and it loaded okay so I added this statement before the CalAddDark:

    CalOpenFile

    It opened with no problem in MaximDL, but here is what the FITS header looked like:

    SIMPLE = T
    BITPIX = -32 /8 unsigned int, 16 & 32 int, -32 & -64 real
    NAXIS = 2 /number of axes
    NAXIS1 = 3072 /fastest changing axis
    NAXIS2 = 2047 /next to fastest changing axis
    BSCALE = 1.0000000000000000 /physical = BZERO + BSCALE*array_value
    BZERO = 0.00000000000000000 /physical = BZERO + BSCALE*array_value
    EXPTIME = 6.0000000000000000 /Exposure time in seconds
    EXPOSURE = 6.0000000000000000 /Exposure time in seconds
    HISTORY Cal Master Dark, 40 inputs
    PEDESTAL = -100 /Correction to add for zero-based ADU
    CSTRETCH = 'Medium ' / Initial display stretch mode
    CBLACK = 1119 /Initial display black level in ADUs
    CWHITE = 1123 /Initial display white level in ADUs
    SWOWNER = 'Craig Young' / Licensed owner of software
    INPUTFMT = 'FITS ' / Format of file from which image was read

    So it is indeed a Master Dark, created by MaximDL from 40 inputs (see HISTORY) , but there is a problem in reloading it for use when doing a master flat.
    I am guessing there is some FITS keywords missing.

    Has anyone tried doing this? Has this been tested before?

    Craig
     
  3. CraigNZ

    CraigNZ Cyanogen Customer

    Joined:
    Dec 29, 2014
    Messages:
    86
    Location:
    Ngutunui, New Zealand
    Notice the HISTORY keyword does not have a value, there is no "=" sign. Could that be why I get a corrupt message?
     
  4. CraigNZ

    CraigNZ Cyanogen Customer

    Joined:
    Dec 29, 2014
    Messages:
    86
    Location:
    Ngutunui, New Zealand
    I removed the HISTORY keyword from the FITS header and tried again but it still failed, so it is not that.

    Maybe the CalAddDark method cannot be used to load a Master Dark frame. If not, then what method is needed so I can use the CalMasterFlat method?

    What is needed is more detailed documentation on how to create master calibration frames and how to calibrate light frames when using Scripting. The lack of responses seems to indicate this has never been done before. But I would think if the scripting methods were added then it must have been tested. In which case something has happened to the code since the CreateCalibrationGroups method was added or I am not using the methods correctly, hence lack of documentation.

    Craig
     
  5. Doug

    Doug Staff Member

    Joined:
    Sep 25, 2014
    Messages:
    6,961
  6. CraigNZ

    CraigNZ Cyanogen Customer

    Joined:
    Dec 29, 2014
    Messages:
    86
    Location:
    Ngutunui, New Zealand
    The Master Dark is 24MB in size and when I try to upload I get an error that it is too big. I do have a private FTP server I can upload to if that is better. If so, send me a private message and I will send you the FTP login details.

    Craig
     
  7. Doug

    Doug Staff Member

    Joined:
    Sep 25, 2014
    Messages:
    6,961
    You can upload to ftp.diffractionlimited.com, incoming folder.
     
  8. CraigNZ

    CraigNZ Cyanogen Customer

    Joined:
    Dec 29, 2014
    Messages:
    86
    Location:
    Ngutunui, New Zealand
    Hi Doug,
    I can connect to the FTP server but cannot retrieve a directory listing. I am using Filezilla and Host: ftp.diffractionlimited.com. No username or password.

    Status: Resolving address of ftp.diffractionlimited.com
    Status: Connecting to 204.225.249.110:21...
    Status: Connection established, waiting for welcome message...
    Status: Initializing TLS...
    Status: Verifying certificate...
    Status: TLS connection established.
    Status: Logged in
    Status: Retrieving directory listing...
    Command: PWD
    Response: 257 "/" is the current directory
    Command: TYPE I
    Response: 200 Type set to I
    Command: PASV
    Response: 227 Entering Passive Mode (204,225,249,110,137,83).
    Command: MLSD
    Error: Connection timed out after 20 seconds of inactivity
    Error: Failed to retrieve directory listing
     
  9. CraigNZ

    CraigNZ Cyanogen Customer

    Joined:
    Dec 29, 2014
    Messages:
    86
    Location:
    Ngutunui, New Zealand
    Tried again, this time using ftp.diffractionlimited.com/incoming and got this:

    Status: Resolving address of ftp.diffractionlimited.com
    Status: Connecting to 204.225.249.110:21...
    Status: Connection established, waiting for welcome message...
    Status: Initializing TLS...
    Status: Verifying certificate...
    Status: TLS connection established.
    Status: Logged in
    Status: Retrieving directory listing of "/incoming"...
    Command: CWD /incoming
    Response: 250 CWD command successful
    Command: PWD
    Response: 257 "/incoming" is the current directory
    Command: TYPE I
    Response: 200 Type set to I
    Command: PASV
    Response: 227 Entering Passive Mode (204,225,249,110,139,81).
    Command: MLSD
    Response: 550 /incoming: Permission denied
    Error: Failed to retrieve directory listing

    Looks like a permission issue.
     
  10. Colin Haig

    Colin Haig Staff Member

    Joined:
    Oct 27, 2014
    Messages:
    3,587
    You can PUT the file there, but you can't do a directory listing. It's a blind drop.
     
  11. CraigNZ

    CraigNZ Cyanogen Customer

    Joined:
    Dec 29, 2014
    Messages:
    86
    Location:
    Ngutunui, New Zealand
    Did you receive the file? Without any directory listing I cannot tell if you got it.

    Also, how do I upload to the incoming folder? If I open the link with a browser and then try to drag and drop on the web page I get an error.

    If I run an FTP client like Filezilla it won't transfer apparently because it cannot get a directory listing.

    Craig
     
    Last edited: Dec 11, 2019
  12. CraigNZ

    CraigNZ Cyanogen Customer

    Joined:
    Dec 29, 2014
    Messages:
    86
    Location:
    Ngutunui, New Zealand
    I found this on your web site:

    Third party FTP Clients such as "Filezilla" can be used to upload software to our FTP server as well.

    https://filezilla-project.org/download.php

    Once the software is installed, use the following credentials to connect:

    Host: ftp.diffractionlimited.com
    User Name: anonymous@diffractionlimited.com
    Password: (NO PASSWORD)

    Copy or drag your files into the window panel on the right to upload your files.

    Once the files have been successfully uploaded, you will *not* see them in the Filezilla program but we will receive them.

    Let us know when the transfer is complete.


    But I get an invalid login.
     
  13. Bob Denny

    Bob Denny Cyanogen Customer

    Joined:
    Oct 12, 2014
    Messages:
    801
    Location:
    DC-3 Dreams, SP, Mesa, Arizona +1 480 396 9700
    Put it into a Dropbox folder and send them the link./
     
  14. Doug

    Doug Staff Member

    Joined:
    Sep 25, 2014
    Messages:
    6,961
    There's a file 2018-12-03 Master Dark 006 sec.fit in the folder, but it is zero length. Perhaps you interrupted the upload?

    You can't see the files you put in there, for security reasons.
     
  15. CraigNZ

    CraigNZ Cyanogen Customer

    Joined:
    Dec 29, 2014
    Messages:
    86
    Location:
    Ngutunui, New Zealand
    What happens is when I try to upload the file using Filezilla (ftp client) it starts the download (which is why you get the 0 length file) and then gets an error message "permission denied" and stops.

    I have gone ahead and uploaded the file to my private FTP server and sent you a private email with the login details. Let me know you got the file and hopefully why I get the corrupted error message from MaximDL.

    Craig
     
  16. Doug

    Doug Staff Member

    Joined:
    Sep 25, 2014
    Messages:
    6,961
    Curious. Your header is missing the following necessary keyword:

    IMAGETYP = 'DARK ' / Type of image

    Also the CALSTAT keyword is important so the software knows if it's been bias-subtracted or not.

    Not sure why there would be a difference between GUI generated master files and script generated ones. Can you confirm they are different?

    Aside from that, I'm wondering about your sequence of operations. Why are you clearing the calibration data? Just leave the previously generated darks and/or bias frames in place, and add the flats.

    Also master frames are not required by MaxIm DL. When you load calibration frames it automatically generates internal master frames "under the hood". They don't need to be explicitly generated and saved out. They are internally generated as needed when you calibrate images.
     
  17. CraigNZ

    CraigNZ Cyanogen Customer

    Joined:
    Dec 29, 2014
    Messages:
    86
    Location:
    Ngutunui, New Zealand
    I am creating a Master Library consisting of BIAS, DARKS and FLATS for use in photometry reduction processing. The BIAS master is normally not used (ie, no scaling) because I create master darks for each exposure time (e.g., 60 seconds). Master darks are created two or three times a year because they do not change all that much over time. Master flats are created twice a month. When creating a master flat I take a collection of flat images in each filter and dark subtract them using a master dark (for each exposure time). The master dark is selected as being the closest (by date) to the flat. So I load the darks closest to the date of the flats, subtract and then combine to get a master flat. For calibration of science images I then do a similar process where I find the nearest master dark and flat for each filter and exposure time in the library and then apply to the light image. This way I don't have to recreate masters each time I do an observation run. If I have a light frame that used an exposure time not in the library then I simply take a bunch of dark images at the exposure time and calibrate, save to library.

    When doing a calibration run I clear the calibration data to make sure there is nothing left over from the previous calibration (e.g., dark or flat) that might cause confusion when creating a new master to be added to the calibration library. Another reason I do this is because if I ever have to go back in time to recreate or modify a calibration then I have my historical library to go back to.

    I use these commands when creating the Master Dark:

    MaximDLApplicationObject.CalClear;

    MaximDLApplicationObject.CalAddDark(wideString(s));

    MaximDLApplicationObject.CalAutoFlat(0);
    MaximDLApplicationObject.CalScaleType(0);
    MaximDLApplicationObject.CalSet;
    CalibrationDocumentObject := MaximDLApplicationObject.CalMasterDark;

    CalibrationDocumentObject.SaveFile ( wideString(fName), 3, FALSE, 3, FALSE);

    MaximDLApplicationObject.CloseAll;

    It appears to be working so if the FITS header is not valid then it may be due to the CalMasterDark not building the FITS header correctly or the SaveFile not working properly.

    Craig
     
  18. Doug

    Doug Staff Member

    Joined:
    Sep 25, 2014
    Messages:
    6,961
    Okay here's a simple workaround - add the missing keywords. I did this in Visual Basic because that's what I have handy.

    Set MaximDLApplicationObject = CreateObject("maxim.application")
    MaximDLApplicationObject.CalClear

    MaximDLApplicationObject.CalAddDark "Calib-001D.fit"
    MaximDLApplicationObject.CalAddDark "Calib-002D.fit"
    MaximDLApplicationObject.CalAddDark "Calib-003D.fit"
    MaximDLApplicationObject.CalSet

    Dim Doc As Object
    Set Doc = MaximDLApplicationObject.CalMasterDark
    Doc.SetFITSKey "IMAGETYP", "DARK"
    Doc.SetFITSKey "CALSTAT", "M"
    Doc.SaveFile "master.fit", 3, False, 3, False
     
  19. CraigNZ

    CraigNZ Cyanogen Customer

    Joined:
    Dec 29, 2014
    Messages:
    86
    Location:
    Ngutunui, New Zealand
    That worked .. thanks Doug. I will watch for an update to MaximDL which provides a permanent fix and then I can remove these statements from the code.

    Craig
     
  20. CraigNZ

    CraigNZ Cyanogen Customer

    Joined:
    Dec 29, 2014
    Messages:
    86
    Location:
    Ngutunui, New Zealand
    Doug,
    Additionally, the master flat frame is also missing the ImageType and Calstat keywords. I added these two lines to the code:

    Doc.SetFITSKey "IMAGETYP", "FLAT"
    Doc.SetFITSKey "CALSTAT", "DM"

    After fixing the FITS headers for both the master Dark and master Flat I was then able to successfully calibrate the science frames. I also used MaximDL to create the masters and then subtracted the script master from the MaximDL master, the result was an image with all 0's meaning the two masters were identical. So other than the two missing keywords, the scripting calibration functions work perfectly.

    Craig
     

Share This Page