Skip to content

Commit

Permalink
Newcandidate 360rc1 (#1557)
Browse files Browse the repository at this point in the history
* Fix a bug in buildFileListOrig causing crash when updatewcs=True (#1549)

* Exception error now reported to include the name of the file being (#1552)

processed (e.g., ERROR: Cannot run astrodrizzle on
runastrodriz -m -i u482c701r_d0m.fits).

* test against pre-built binaries of numpy (#1553)

* Protect against using compute_sregion for no valid images (#1547)

* Ensure the software does not execute compute_sregion() if no valid images
are found for processing and AstroDrizzle is exiting.

* Added a note to the buildEmptyDRZ to explain further the need for this
empty (only header keywords, no data components) file.  Fixed a typo, and
changed the filename variable used when computing the sregion.

* Added info to CHANGELOG.

* Unit test for point kernel (#1542)

* Added fixture for test_drizzle and point kernel test

I've added a fixture to used for test_zero_input_weight and now a new test called test_point_kernel. I use the same pytest.mark.parameterize kernel parameters (for now).

* Working tests using the Get_Grid Class and appropriate dny

* Small text changes

* Added csv truth files.

* Added three additional tests for edge cases.

* Added now with the changes made to the test.

* Small comment/text change.

* New test_setup.py file with Get_Grid

Moved truth files to "truth_files" directory.  test_drizzle.py is now test_kernel and I've added a new test_cdriz that uses the same test_setup.py.

* Added test_cdriz tests.

* improved test_zero_input_weight, no setup defaults

* Added file creation on failure for non_sym. test

* Added other tests + return_pngs cdriz tests

* added warren's cdriz_setup, lanczos3 still fails.

* Tests working

* print statements and altered rtol

* temprary new pytest added

* added requirements.dev

* steve's pytest removed

* temporary assert command for test_point_kernel.

* test_point_kernel w/ np.testing.assert_array_equal

* test_kernel assert without mask

* testing test_point_kernel with zero_background

* better name for variable.

* updated truth files with zero background

* removing unused variable

* path to saved files added+new truth when missing.

* removed new_truth argument

* large_sqaure to large_square

* spelling corrections

* removed duplicated line.

* duplicated line removed..

* removed png path

* Implemented cdriz_setup.get_output_fullpath

* Added docstring to get_output_fullpath

* WFPC2: Issue a specific error code and create an empty manifest file for bad data (#1550)

* If analysis of a WFPC2 exposure determines the data cannot be processed
for some reason, the converted WFPC2 FLT file is currently deleted.  The
process will now exist with the return code of NO_VIABLE_DATA (65), and
an empty manifest file will be created. Also, fixed the option string,
added a little documentation, and added a BaseException to handle the
sys.exit().

* Updated the changelog.

* Fix changelog (#1556)

* Corrected the documentation for release candidates (RCs).  The
documenation for the separate RCs will be combined once the final
is ready.

* Fixed the Release Candidates(RCs) with dates into their own catagories.
When the final release is ready, the separate RCs will be combined.

---------

Co-authored-by: Mihai Cara <[email protected]>
Co-authored-by: Zach Burnett <[email protected]>
Co-authored-by: Steve Goldman <[email protected]>
  • Loading branch information
4 people authored May 10, 2023
1 parent bde8d90 commit 1ed7caf
Show file tree
Hide file tree
Showing 16 changed files with 971 additions and 78 deletions.
45 changes: 29 additions & 16 deletions CHANGELOG.rst
Original file line number Diff line number Diff line change
Expand Up @@ -17,40 +17,53 @@ number of the code change for that issue. These PRs can be viewed at:

https://github.com/spacetelescope/drizzlepac/pulls

3.6.0 (28-Apr-2023)
===================
3.6.0rc1 (10-May-2023)
======================

- For WFPC2 datasets which turn out to have no viable data to process and
a manifest file has been requested, force an empty manifest file to be
generated and issue the exit code NO_VIABLE_DATA (65). [#1550]

- Protect against writing the S_REGION keyword in intentionally empty DRZ/DRC
files in ``processinput.process`` to avoid messy crash. [#1547]

- Fix a bug in ``processinput.buildFileListOrig`` due to which astrodrizzle
might crash when ``updatewcs`` is set to ``True``. [#1549]

- Turn off use of ``verify_guiding()`` for WFPC2 images only as its use
incorrectly recognizes diffraction spikes from saturated stars as evidence
3.6.0rc0 (28-Apr-2023)
======================

- Turn off use of ``verify_guiding()`` for WFPC2 images only as its use
incorrectly recognizes diffraction spikes from saturated stars as evidence
of loss of lock and flags those exposures as 'bad'. [#1511]

- Ensure processing of all IMAGETYP=EXT WFPC2 targets. [#1505]

- Properly identify neighbor Projection Cells which overlap input
exposures. [#1503]

- Updates identify and remove any WFPC2 calibration exposures that
cannot be processed during standard pipeline alignment and drizzling.
The list of recognized calibration target names was updated to
accommodate WFPC2 and to identify exposures to be skipped and deleted
- Updates identify and remove any WFPC2 calibration exposures that
cannot be processed during standard pipeline alignment and drizzling.
The list of recognized calibration target names was updated to
accommodate WFPC2 and to identify exposures to be skipped and deleted
after converting the D0M images into FLT images. [#1514]

- Compute a default kernel for use with astrometric_utils.extract_sources()
function when the kernel parameter is None. The default kernel is based on
- Compute a default kernel for use with astrometric_utils.extract_sources()
function when the kernel parameter is None. The default kernel is based on
the fwhm parameter of the same function. [#1519]

- Address many ReadTheDocs issues. [#1521 - #1529]

- Write the EXPNAME keyword to the ACS SVM and MVM headers to avoid errors
- Write the EXPNAME keyword to the ACS SVM and MVM headers to avoid errors
and enforce consistency with WFC3. [#1530]
- Properly populate the S_REGION keyword with a closed polygon for the

- Properly populate the S_REGION keyword with a closed polygon for the
pipeline FLT/FLC images. [#1533]

- Compute the S_REGION values for pipeline drizzled products. [#1535]

- Ensure the DATE keyword is written to the primary header of all output
drizzled products. The DATE represents the date the file was written.
drizzled products. The DATE represents the date the file was written.
[#1537]

- Update to ensure the SVM FLT/FLC files all contain the S_REGION keyword
Expand All @@ -62,7 +75,7 @@ number of the code change for that issue. These PRs can be viewed at:
- Turn on use of ``verify_guiding()`` to ignore exposures where guide star
lock was lost and the stars are trailed. [#1443]

- Ensure when no sources are found and the variable thresh is zero, the
- Ensure when no sources are found and the variable thresh is zero, the
``verify_crthesh()`` properly indicates the catalog failed the CR threshold.
[#1450]

Expand Down
12 changes: 9 additions & 3 deletions drizzlepac/processInput.py
Original file line number Diff line number Diff line change
Expand Up @@ -704,7 +704,9 @@ def buildFileListOrig(input, output=None, ivmlist=None,
newfilelist, ivmlist = check_files.checkFiles(updated_input, ivmlist)

if updatewcs:
uw.updatewcs(','.join(set(newfilelist) - set(filelist)))
new_flist = set(newfilelist) - set(filelist)
if new_flist:
uw.updatewcs(','.join(new_flist))

if len(ivmlist) > 0:
ivmlist, filelist = list(zip(*ivmlist))
Expand Down Expand Up @@ -950,6 +952,11 @@ def buildEmptyDRZ(input, output):
pipeline can handle the Multidrizzle zero expossure time exception
where all data has been excluded from processing.
Notes
-----
The existance of the DRZ file is also necessary to satisfy the interface
expectations of the archive. Do NOT delete this file.
Parameters
----------
input : str
Expand Down Expand Up @@ -981,7 +988,7 @@ def buildEmptyDRZ(input, output):
if '_drz' not in output:
output = fileutil.buildNewRootname(output, extn='_drz.fits')

print('Building emtpy DRZ file with output name: %s' % output)
print('Building empty DRZ file with output name: %s' % output)

# Open the first image (of the excludedFileList?) to use as a template to build
# the DRZ file.
Expand Down Expand Up @@ -1022,7 +1029,6 @@ def buildEmptyDRZ(input, output):
fitsobj[0].header.add_history("** all input images were excluded from "
"processing.**")


# Change the filename in the primary header to reflect the name of the output
# filename.
fitsobj[0].header['FILENAME'] = str(output) # +"_drz.fits"
Expand Down
31 changes: 24 additions & 7 deletions drizzlepac/runastrodriz.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
:License: :doc:`LICENSE`
USAGE: runastrodriz.py [-fhdaibng] inputFilename [newpath]
USAGE: runastrodriz.py [-bdahfginv] inputFilename [newpath]
Alternative USAGE:
python
Expand All @@ -30,6 +30,9 @@
The '-g' option allows the user to TURN OFF alignment of the images to an external
astrometric catalog, such as GAIA, as accessible through the MAST interface.
See the main() at the end of this module for all available options and a
terse description.
Additional control over whether or not to attempt to align to an external
astrometric catalog, such as GAIA, is provided through the use of the
environment variables:
Expand Down Expand Up @@ -294,7 +297,16 @@ def process(inFile, force=False, newpath=None, num_cores=None, inmemory=True,
# Remove FLT file created here, since the calibration file can NOT be aligned or drizzled
os.remove(inFilename)
print("ERROR: Inappropriate input file. Deleting converted WFPC2 FLT file.")
return

# write out manifest file, if requested
mani_filename = inFile[:inFile.find('_d0m')] + '_manifest.txt'
if make_manifest:
if os.path.exists(mani_filename):
os.remove(mani_filename)
with open(mani_filename, 'w') as fout:
_ = [fout.write(f"{fname}\n") for fname in manifest_list]
print(f"Created manifest file: {mani_filename}")
sys.exit(analyze.Ret_code.NO_VIABLE_DATA.value)

infile_det = fits.getval(inFilename, 'detector')
cal_ext = None
Expand Down Expand Up @@ -698,8 +710,10 @@ def process(inFile, force=False, newpath=None, num_cores=None, inmemory=True,
drz_output = drz_product
# keep track of output files
manifest_list.append(drz_output)
# update s_region header keyword
processing_utils.compute_sregion(drz_product)
# update s_region header keyword only if the drz_output
# contains actual data, not just header information.
if asn_dicts is not None:
processing_utils.compute_sregion(drz_output)

else:
# Create default trailer file messages when astrodrizzle is not
Expand Down Expand Up @@ -2131,6 +2145,7 @@ def main():
force = True
if opt == "-i":
inmemory = True
# The "-m" option is specific for WFPC2 data
if opt == "-m":
make_manifest = True
if opt == "-v":
Expand Down Expand Up @@ -2161,11 +2176,13 @@ def main():

except Exception as errorobj:
print(str(errorobj))
print("ERROR: Cannot run astrodrizzle on %s." % sys.argv[1])
print("ERROR: Cannot run astrodrizzle on %s." % " ".join(sys.argv))
raise Exception(str(errorobj))

sys.exit()

# This except handles sys.exit() which raises the SystemExit exception which inherits from BaseException.
except BaseException:
exc_type, exc_value, exc_tb = sys.exc_info()
print(f"Return Value: {exc_value}")

if __name__ == "__main__":
main()
1 change: 1 addition & 0 deletions requirements-dev.txt
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@ git+https://github.com/spacetelescope/stsci.tools.git#egg=stsci.tools
astropy>=0.0.dev0
git+https://github.com/spacetelescope/stwcs.git#egg=stwcs
git+https://github.com/astropy/astroquery.git#egg=astroquery
numpy>=0.0.dev0
157 changes: 157 additions & 0 deletions tests/drizzle/cdriz_setup.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,157 @@
import os
import numpy as np
from astropy import wcs
from drizzlepac import cdriz


def get_wcs(_grid, pscale=0.04):
w = wcs.WCS()
w.wcs.ctype = ["RA---TAN", "DEC--TAN"]
w.wcs.crpix = [_grid[0] / 2.0, _grid[1] / 2.0] # define as center pixel in array
w.wcs.crval = [10.000, 10.000] # RA/Decl. in decimal degrees
w.wcs.cdelt = [pscale / 3600.0, pscale / 3600.0]
w.wcs.set()
return w


class Get_Grid:
"""Sets up inputs to call the python wrapper of the c code: cdriz.tdriz.
The size of the input and ouput grids can be specified in the arguments for
the init(). For an input grid of 4 x 4 and output grid of 5 x 5 you would
use the following call of the class.
Get_Grid(inx=4,iny=4, outx=5, outy=5)
mapping = cdriz.DefaultWCSMapping(
input_wcs, output_wcs,
input_wcs.pixel_shape[0], input_wcs.pixel_shape[1],
stepsize
)
"""

def __init__(self, inx, iny, outx, outy):
np.random.seed(0) # keep same random across each instance
self.in_grid = (iny, inx) # use numpy indexing order (y,x)
self.out_grid = (outy, outx) # use numpy indexing order (y,x)
self.insci = np.random.randn(self.in_grid[0], self.in_grid[1]).astype("float32")
self.inwht = np.ones(self.in_grid, dtype=np.float32)
self.dny = self.insci.shape[1] # input y_grid
self.outsci = np.zeros(self.out_grid, dtype=np.float32)
self.outwht = np.zeros(self.out_grid, dtype=np.float32)
self.outctx = np.zeros(self.out_grid, dtype=np.int32)
self.w1 = get_wcs(self.in_grid)
self.w2 = get_wcs(self.out_grid)
self.mapping = cdriz.DefaultWCSMapping(
self.w1, self.w2, self.in_grid[0], self.in_grid[1], 1
)

def zero_background(self):
super().__init__()
self.insci = np.zeros(self.in_grid, dtype=np.float32)


def cdriz_call(_set_kernel_pars, kernel):
"""
parameters explained in c code (arrdrizmodule.c)
_vers, nmiss, nskip = cdriz.tdriz(insci, inwht, outsci, outwht,
outctx, uniqid, ystart, 1, 1, _dny,
pix_ratio, 1.0, 1.0, 'center', pixfrac,
kernel, in_units, expscale, wt_scl,
fillval, nmiss, nskip, 1, mapping)
"""
cdriz.tdriz(
_set_kernel_pars.insci,
_set_kernel_pars.inwht,
_set_kernel_pars.outsci,
_set_kernel_pars.outwht,
_set_kernel_pars.outctx,
1, # uniqid
0, # ystart
1, # xmin pixel to start reading; fits so starts at 1
1, # ymin pixel to start reading; fits so starts at 1
_set_kernel_pars.dny, # _dny
1.0, # pix_ratio
1.0, # xscale; plate scale variations
1.0, # yscale; plate scale variations
"corner", # center of pixel is what 0 corresponds to
1.0, # pixfrac
kernel, # kernel
"cps", # units
1.0, # expscale
1.0, # wt_scale
"INDEF", # fillval
0, # nmiss
0, # nskip
1, # vflag; historical value only (not used)
_set_kernel_pars.mapping,
)


def get_output_fullpath(relative_path, output_filename):
"""Returns the final output path given a name and relative path.
Parameters
----------
relative_path : str
relative path from this file to desired path.
output_filename : str
desired name of output file (png or csv).
Returns
-------
output_fullpath: str
final output path
"""
local_path = os.path.dirname(__file__)
output_name = os.path.join(relative_path, output_filename)
output_fullpath = os.path.join(local_path, output_name)
print(f"Output Path: {output_fullpath}")
return output_fullpath


def save_array(_data, _name):
np.savetxt(
_name,
X=_data,
fmt="%1.8f",
delimiter=",",
)


def generate_png(_set_kernel_pars, _name):
"""Added function for displaying the test results as an image map.
This should show the expected point kernel response.
Parameters
----------
_set_kernel_pars : Class instance
an instance of the kernel_pars class.
_name : str
The name of the output png file.
"""
# for generating truth files
import matplotlib.pyplot as plt

fig = plt.figure(figsize=(4, 2))
ax1 = fig.add_subplot(111, projection=_set_kernel_pars.w1)
ax1.imshow(_set_kernel_pars.outsci, origin="lower", cmap="Greys")
ax1.set_ylabel(" ")
ax1.set_xlabel(" ")
fig.savefig(_name)


def error_message(_data, _name):
"""Saves new truth csv file on failure of test.
Parameters
----------
_data : np.array
data to save to new truth file
_name : str
new name of truth file, should be slightly
different than current truth file
"""
save_array(_data, _name)
Loading

0 comments on commit 1ed7caf

Please sign in to comment.