Skip to content

How findr Works

Jump to bottom
danielchen1 edited this page Dec 8, 2015 · 38 revisions

How findr Works

sample.cfg

The sample.cfg file defines all the variables used in run_findr.py and findr_lib (variables are in italics): Jared Males's bitbucket url: https://bitbucket.org/jaredmales/acic/src

  • max_processes: max number of workers that can be ran simultaneously (defined as 2)
  • fileshifts: used for centering the images; defined as file_shifts.txt provided by Jared
  • darkmaster_path: path to darkmaster.cpp located in Jared Males's bitbucket
  • darksub_path: path to darksub.cpp located in Jared's bitbucket
  • fitscent_path: path to fitscent.cpp located in Jared's bitbucket
  • outputfname: the name of the .json and .tsv files created by run_findr.py (defined as metadata)
  • masterdark_filename: defined as masterdark.fits (the average of all the dark images in darks.list) generated from running darkmaster.cpp from Jared's bitbucket
  • darknorms_filename: defined as norm.dat (file containing median pixel intensity across the dark images) generated from running darkmaster.cpp
  • darklist_filename: defined as darks.list (list of all the dark images passed into darkmaster.cpp) generated from running darkmaster.cpp
  • smooth_window: the number of images to normalize in each iteration (defined as 20)
  • science_norms: defined as all_sci_median_norms.dat (contains the normalization of the science images)

run_findr.py

Necessary library and python script imports:

  • sys
  • json
  • os
  • multiprocessing
  • ConfigParser
  • astropy.io.fits
  • findr_lib
  1. If the files does not exists.
    • Gets the path and cfg file name from passed args
    • Reads config_vals into findr_lib from file name
  2. If no .json and .tsv file
    • Gets files in dir if they are .fits
    • Gets fieldnames from first fits file
    • Setups multiprocessing pool
    • Asynchronously gathers metadata
    • Generates tsv of metadata
    • Makes dictionary from list of all metadata
    • Creates json from list of metadata if config is set to allow
  3. If .json and .tsv files exists
    • Reads in data from .json files
  4. Sorts metadata into dictionary of lists based on VIMTYPE
  5. Removes science files from metadata dictionary if AOLOOPST is OPEN
  6. Runs darkmaster
  7. Runs subtractAndCenter
  8. TODO Klip-reduce
  9. Returns a dictionary of lists of good filenames sorted by type

findr_lib.py

Necessary library imports:

  • os: path, system
  • csv
  • json
  • numpy
  • datetime
  • time
  • astropy.io,fits
  1. Gets all the metadata from the fits file hdulist
  2. Removes comment field
  3. Creates a file called metadata.tsv for the output
  4. Sets up the writer, header fields, and delimiter
  5. Writes the header to the file
  6. Builds json file of metadata not sorted by VIMTYPE
  7. Sorts total_dic into dictionary by VIMTYPE
  8. Searches dictionary built by my other script
  9. Stores names of good files
  10. Returns those names
  11. Writes out a config file from a list
  12. Writes out a config file from a dictionary
  13. Writes dark images to config file
  14. Fills out required parameters
  15. Fills out bottom/top normalization coordinates if present
  16. Fills out height/width of centered normalization region (overrides normalization coordinates) if present
  17. Adds median options, if present
  18. Builds & calls darkmaster command
  19. Prepends text to filename
  20. Spawns a darksub command
  21. Spawns a fitscent command
  22. Calculates the centered moving average of the norm values for the image data set's top and bottom amplifiers
  23. Initial smoothing
  24. If the size is even, then smooth the smoother values
  25. Adjusts the science smoothed norm values based on the norm values
  26. Converts all timestamps to seconds since epoch
  27. Dark_time is now in seconds since epoch
  28. Sci_time is now in seconds since epoch
  29. Calculate deltas
  30. Shifts smoother norms
  31. Special case for the last dark and the last few sciences
  32. Builds list of science images to process
  33. Loads shift values from file to memory
  34. Builds norms and stores in memory
  35. Defines necessary variables
  36. Builds up commands for each science images
  37. Gets norm values
  38. Gets shift values
  39. Skips remaining task
  40. Builds subtraction task
  41. Builds centering task
  42. Executes subtraction tasks (parallel)
  43. Executes subtraction tasks (serial)
  44. Executes centering tasks (parallel)
  45. Executes centering tasks (serial)
  46. Returns list of final filenames

Clone this wiki locally