JWST Build 7.8.1

Changes in jwst v1.3.0

associations

• Ensure no Lv3_WFSC associations created on group candidates [#6131]

datamodels

• Add new PATTTYPE values for MIRI Coronagraphic flats:
  4QPM_LFLAT, 4QPM_PFLAT, LYOT_LFLAT, LYOT_PFLAT. [#6232]

• Update DarkModel to use uint32 for DQ array. [#6228]

• Add NOUTPUTS keyword to the DarkModel schema. [#6213]

lib

• Add overriding of the matrix calculations to set_telescope_pointing.py [#5843]

• Add guide star-based pointing algorithm to set_telescope_pointing.py [#5843]

resample

Fix the extreme memory consumption seen in resampling of variance arrays. [#6251]

tweakreg

• Add an upper tweak threshold of 10 arcsec to tweakreg [#6252]

wfs_combine

-- Add option to flip the dither locations so that images with different
filters will have the same pixel locations [#6101]

-- Fixed the refine option to correctly use the cross correlation to align
the images if the WCS is off [#6101]

JWST Build 7.8

Release Notes

The JWST calibration pipeline (v1.2.3) has been made available for installation on PyPi and github. A high-level list of updates in this build is given below.

Association generator rules updates

• Properly segregate NIRCam + NIRISS parallel Wide Field Slitless Spectroscopy (WFSS) exposures by instrument
• Process MIRI MRS dedicated background observations as stand-alone science
• Only associate direct images obtained using the same instrument channel with WFSS grism images
• Generate "spec3" associations for MIRI MRS observations that don't use a dither pattern
• Don't create ASN's for NIRCam Engineering Template exposures that use EXP_TYPE='NRC_GRISM' and PUPIL='CLEAR'
• Only use exposures from other primary nod positions as background members for NIRSpec fixed-slit observations that employ sub-pixel dithering

Updates to the Level-1b set_telescope_pointing script to:

• Moved the JWST_[XYZ] and JWST_D[XYZ] keywords from the primary header to the SCI extension header

Updates to Level-2a processing for all exposure modes include:

• Updated the "refpix" step to properly handle subarray exposures that use 4 amplifiers for readout
• Updated the "jump" step to detect jumps in integrations containing only 3 or 4 groups
• Refactored the Ordinary Least Squares (OLS) algorithms in the "ramp_fit" step

Updates to Level-2b processing for all exposure modes include:

• Fix the computation of S_REGION for exposures that straddle the RA=0 boundary
• Added new step parameter to the "background" step that allows saving the combined background image

Updates to Level-2b spectroscopic (calwebb_spec2) processing include:

• Updated the "srctype" step to populate SRCTYPE based on the "is_extended" flag for WFSS exposures
• "cube_build" step updates to improve processing efficiency
• Implemented variance and error propagation in the "extract_1d" step

Updates to the Level-3 imaging (calwebb_image3) processing include:

• Updated the "resample" step to propagate variance and error arrays
• Modified the "source_catalog" step to compute photometric errors
• Modified the "source_catalog" step to store "is_extended" instead of "is_star" for each source
• Modified the "source_catalog" step to compute concentration indexes as flux ratios instead of magnitude differences

Updates to the Level-3 spectroscopic (calwebb_spec3) processing include:

• Updated the "resample_spec" step to propagate variance and error arrays
• "cube_build" step updates to improve processing efficiency
• Implemented variance and error propagation in the "extract_1d" step

Updates to the Level-3 TSO (calwebb_tso3) pipeline include:

• Modified the "outlier_detection" step to use simple sigma clipping up the stack of integrations within an exposure

Updates to the Level-3 AMI (calwebb_ami3) pipeline include:

• Fix sign error in AMI pupil phase results

The complete change log follows:

datamodels

• Add back and use "CALCULATED" for ENGQLPTG. [#6135]

• Convert incoming Path objects to strings in datamodels.open [#6130]

JWST Build 7.8 rc3

ami_analyze

• Fix to AMI pupil phases sign error [#6128]

datamodels

• Update moving target schema to match b7.8 keyword schema. [#6129]

JWST Build 7.8 rc2

associations

• Asn_Lv2WFSS: Add instrument constraint. [#6114]

• Asn_Lv2NRSLAMPSpectral: Allow msaspec only if msametfl is available. [#6085]

combine_1d

• Added SRCTYPE to COMBINE1D output extension headers, propagated from
  EXTRACT1D inputs [#6079]

cube_build

• Fix some typos in the the arguments documentation. [#6077]

datamodels

• Updated enum lists for ENGQLPTG and PATTTYPE keywords [#6081]

• Removed obsolete keyword NDITHPTS and updated attributes for NRIMDTPT [#6083]

• Added units to CombinedSpecModel table output [#6082]

• Added keywords OSS_VER, DETMODE, CMD_TSEL, NOD_TYPE, and GS_V3_PA to
  the core schema [#6086]

• Remove ModelContainer schema and refactor use of association table
  metadata within. [#6094]

general

• Make CRDS context reporting pytest plugin disabled by default. [#6070]

• Removed all usage of sys.path, in associations and jwst.stpipe [#6098]

lib

• Updated set_telescope_pointing to populate ENGQLPTG keyword with new
  allowed values [#6088]

outlier_detection

• Avoid using 'linear' interpolation method as default for blot due to
  a bug in the implimentation of the bilinear interpolator in the drizzle
  package. Now the default value will be 'poly5'. [#6116]

ramp_fitting

• Re-enable multiprocessing in RampFitStep by moving code back from
  stcal package. [#6119]

scripts

• Add migrate_data command with support for migrating spec_table in
  x1d files produced with <= 1.1.0 of this package. [#6055]

tweakreg

• Remove attached tweakreg catalog from datamodel before exiting step [#6102]

JWST Build 7.8 rc1

ami_analyze

• Create copy of input datamodel to avoid overwriting input. [#5828]

assign_wcs

• Convert the ra values to array in util.wrap_ra, but if input is a list return
  a list [#6031]

• Moved the routine wrap_ra from cube_build to assign_wcs.util. The s_region is
  now correct for data that cross ra boundary. [#6026]

• Changed evaluation of grism bounding box center from averaged extrema of
  transformed bounding box to transformed centroid of source_cat object [#5809]

• Added pixel shift to MSA slits due to 0-indexing in NIRSpec slit validation
  code, fixing difference between bounding box locations during the separate
  halves of assign_wcs runs [#5927]

• Added logic to prevent the sending of an empty list of slits to the
  validate_open_slits function, so a proper error message is provided to
  the user [#5939]

• Added computed spectral_region to model.meta.wcsinfo. [#5969]

associations

• Add rule Asn_MIRMRSBackground to treat background as science. [#6046]

• Updated level2b WFSS rules to only consider exposures from the same
  instrument channel when matching direct images with grism images in
  NIRCam WFSS observations. [#5786]

• Removed PATTTYPE='None' constraint from Lv3MIRMRS association rule to
  generate spec3 associations for undithered MRS observations. [#5804]

• Updated level2b WFSS rules to only consider exposures using the same
  PUPIL value (cross filter) when matching direct images with grism images
  in NIRISS WFSS observations. [#5896]

• Updated level2b and level3 TSO rules to exclude exposures with
  EXP_TYPE=NRC_TSGRISM and PUPIL=CLEAR, which can result from NIRCam
  engineering template observations. [#5946]

• Updated level2b NIRSpec FS rules to exclude exposures sharing a primary
  dither location from the list of background exposures [#5994]

background

• Remove unused SubtractImagesStep [#5919]

• Added new step parameter to optionally save the combined, average
  background image: save_combined_background. [#5954]

calwebb_spec2

• Updated documentation to indicate that master_background is applied to
  NIRSpec MOS exposures in the calwebb_spec2 pipeline [#5913]

calwebb_spec3

• Updated documentation to indicate that master_background is applied to
  NIRSpec MOS exposures in the calwebb_spec2 pipeline [#5913]

csv_tools

• The csv_tools subpackage was removed [#6006]

cube_build

• Fixed typo in CubeBuildStep spec for grating [#5839]

• Update code to read in spectral and spatial size of exposure on the sky [#5991]

• For calspec2 pipeline skip determining the dq plane in cube_build [#5991]

• Remove certain WCS keywords that are irrelevant after cube_build. [#6032]

datamodels

• Added is_star to slitmeta [#5788]

• Update keyword comments for NIRSpec grating wheel (GWA) keywords [#5844]

• Moved functions in dqflags and dynamic_mask to stcal [#5898]

• API change - stcal.dqflags.interpret_bit_flags and stcal.dynamicdq.dynamic_mask
  now require the mnemonic_map as input. [#5898, #5914]

• Implemented new data models SpecKernelModel, SpecProfileModel,
  SpecTraceModel, and WaveMapModel for use by new NIRISS SOSS
  reference files in optimized 1D extraction [#5925]

• Added FULLP to SUBARRAY enum list in core, subarray,
  and keyword_psubarray schemas [#5947]

• Moved JWST_[XYZ] and JWST_[DXDYDZ] keywords from primary to SCI extension
  header and updated their comment fields to indicate they'll now be in the
  barycentric frame. Also added the new OBSGEO[XYZ] keywords to the SCI
  extension header, which are in the geocentric frame. [#6050]

• Added a new datamodel, SegmentationMapModel that has an uint32 data array
  for storing the segmentation map output from source_catalog. [#6051]

documentation

• Update documentation, deprecating primary use of CFG files [#5901]

• Update pipeline introduction document to include segmentation map (segm)
  in list of data products [#5956]

• Update assign_mtwcs step docs and reference the assign_mtwcs step in the
  calwebb_image3 and calwebb_spec3 pipeline docs [#6024]

extract_1d

• Implemented error and variance propagation for all modes but those
  utilizing IFU cubes [#6014]

extract_2d

• For WFSS removed setting srctype to UNKNOWN; added setting is_star in slitmeta [#5788]

• In NRC_TSGRISM mode replaced FITS WCS keywords with JWST specific ones. [#6005]

• Added specsys to slits. [#6005]

general

• Update file naming conventions documentation to clarify when optional components
  will be used. [#5796]

• Update DQFLAGS table in RTD docs with new definitions for persistence and
  ad_floor in bits five and six [#5815]

• Update data products, calwebb_image3, and source_catalog docs to include
  information about the segmentation map product [#5949]

• Replace documentation references to ambiguous class names with full
  paths. [#6017]

jump

• Update the step to detect jumps in three and four group integations [#5915].

• Change the default S/N ratio for not flagging neighbors to be a higher value to
  better reflect the correct IPC.

lib

• Update update_mt_kwds function in set_telescope_pointing.py to populate
  the TARG_RA/TARG_DEC [#5808]

• moved basic_utils.multiple_replace to stcal. [#5898]

• Implemented window clipping algorithm for WFSS contamination corrections. [#5978]

• Updated set_velocity_aberration and utc_to_tdb to access the JWST
  position and velocity keywords from the SCI extension header, rather than the
  primary header. [#6050]

master_background

• Updated documentation to more fully describe the various ways in which the
  step is applied [#5913]

outlier_detection

• Outlier detection on non-dithered images is implemented with a simple sigma
  clipping, dithered outlier detection cleaned up and HST specific steps removed
  and additional tests added. [#5822]

ramp_fitting

• Refactoring OLS code for ramp fitting to improve readability and maintenance.
  Also, reference to nreads is being removed and replaced with ngroups
  to remove and confusion on functionality. [#5872]

• Refactoring ramp fit code separating OLS and GLS code into their own file. [#5951]

• Refactoring ramp fit code in preparation for moving code to STCAL. [#6010]

• Moved ramp fit code to STCAL. [#6023]

• Now that ramp fitting has been moved to STCAL, for the JWST unit tests to
  pass need to use STCAL 0.2.1 or greater. The bug fix for JP-1920 were made
  in STCAL, which affected JWST unit tests for ramp fitting. [#6038]

refpix

• Added code to handle NIR subarrays that use 4 readout amplifiers. Uses and
  applies reference pixel signal from available amplifiers and side reference
  pixel regions, including odd-even column separation if requested [#5926]

• Fixed a bug introduced in #5926 that affected refpix calibration of 1-amp NIR
  subarrays [#5937]

• Added regression test and unit test for NIR 4-amp subarray correction [#5967]

resample

• Fix resample_spec output size from input images crossing RA=0 [#5929]

• Propagate variance arrays into SlitModel used as input for ResampleSpecStep [#5941]

• Remove certain WCS keywords that are irrelevant after resampling. [#5971]

• Propagate error and variance arrays in ResampleStep for imaging data. [#6036]

• Propagate error and variance arrays in ResampleSpecStep for 2D spectral data [#6041]

• Record pixel_scale_ratio and pixfrac from ResampleStep in header
  keywords PXSCLRT and PIXFRAC, respectively, or meta.resample.pixel_scale_ratio
  and meta.resample.pixfrac. [#6044]

source_catalog

• Updated the concentration indices to be calculated as flux ratios
  instead of magnitude differences. The CI column names have also been
  renamed to list the larger EE first, e.g. CI_50_30. [#5810]

• Aperture-corrected total fluxes and magnitudes are now computed for
  all sources. [#5996]

• Photometric errors are now computed using the new resampled total
  error array. [#5997]

• The nn_dist column was replaced by a nn_label column
  indicating the label number of the nearest neighbor. [#5998]

• The is_star column was replaced by a is_extended column with
  inverted boolean values. [#6018]

• Circular aperture sizes now scale in the case of non-native pixel
  scales in the resampled image. [#6045]

• Segmentation map output dtype is now uint32 [#6051]

srctype

• Added section for WFSS mode data to set srctype based on is_star value [#5788]

transforms

• Added is_star to GrismObject [#5788]

tweakreg

• Fix a bug due to models_grouped now returning odict_values instead
  of lists. [#6022]

• Updated documentation to include the new "rshift" option for fit geometry [#5899]

wfss_contam

• Implemented basic step structure to apply WFSS contamination corrections, along with
  the necessary grism library modules [#5508]

JWST v1.1.0 (Build 7.7.1)

assign_mtwcs

• Fixed a bug which caused the step to fail with MultiSlitModel input. [#JP-1907]

assign_wcs

• Added velocity aberration-corrected frame 'v2v3vacorr' to the WCS
  pipeline which takes into account DVA effects. [#5602]

• Renamed MIRI frame 'V2_V3_spatial' to 'v2v3_spatial' and
  'V2_V3_vacorr_spatial' to 'v2v3vacorr_spatial'. Added axes names
  to the 'v2v3' frame for nircam, niriss, miri, and fgs.
  Renamed axes for nirspec from V2 and V3 to
  v2 and v3. [#5765]

• Changed units of the 'v2v3' frame for nircam from u.deg to
  u.arcsec [#5765]

• Added spectral frames to the output WCS frame of TSO and WFSS observations. [#5771]

associations

• Warn about duplicate product names and do not write duplicate associations [#5721]

• Added new Lvl2 rule, Asn_Lv2NRSLAMPImage, to run Image2 pipeline for NRSLAMP
  exposures with OPMODE=image [#5740]

• Ignore duplicate product names while handling Level 2 associations [#5780]

• Constraint added to Asn_Lv3Coron to remove background exposures [#5781]

combine_1d

• Pull source_id from input x1d headers (from source_catalog) to populate
  c1d output headers [#5759]

cube_build

• Added support for cross-dichroic configurations [#5722]

• Added infrastructure to support NIRSpec opaque + grating options to build lamp mode data [#5757]

• When building MIRI internal_cal type cubes removed the requirement that cdelt1=cdelt2 [#5757]

datamodels

• Updated keyword_readpatt, core, preadpatt schemas for new MIRI detector
  readout patterns 'FASTR1', 'FASTR100' and 'SLOWR1' [#5670]

• Added extr_x and extr_y to multispec datamodel. These values are center
  of extraction region for IFU data [#5685]

• Added segmentation map output file name to core schema keywords, under
  keyword 'SEGMFILE' [#5730]

• Added '1LOS' to PATTTYPE enum list in core.schema datamodel [#5728]

• Added 'IMAGE' to OPMODE enum list [#5745]

• Added source_id to combinedspec and multicombinedspec schemas to populate
  combine1d output headers [#5759]

extract_1d

• Adding writing SRCTYPE, EXTR_X, and EXTR_Y to extracted spec for IFU data [#5685]

• Only update the output x1d data using the PRIMARY input data. Prevents SCI data in x1d data [#5694]

• Fixed bug in background region fitting for image columns/rows that have zero weight
  for all pixels [#5696]

• Determine the background using sigma clipping of entire extended region for
  extended source IFU data [#5743]

group_scale

• Fix premature model closing in group_scale_step [#5692]

lib

• Make EngDB_Value public for JSDP use [#5669]

• Update code in set_velocity_aberration.py functions based on Colin Cox
  suggestions: simplify DVA scale computation and improve apparent RA and
  DEC aberrated position computation. Also, attributes ra_offset and
  dec_offset of datamodel.meta.velocity_aberration have been renamed to
  va_ra_ref and va_dec_ref and their corresponding FITS keywords
  have been renamed from DVA_RA and DVA_DEC to
  VA_RA and VA_DEC. [#5666]

• Make get_wcs_values_from_siaf public for JSDP use [#5669]

outlier_detection

• Remove hard-coded MRS outlier detection values now that a parameter reference
  file exists. [#5753]

photom

• Fixed handling of NIRSpec IFU extended source data, so that the flux
  calibration gets converted to surface brightness [#5761]

pipeline

• Remove references to Numpy globals np.int, np.float, np.bool and
  np.str in the package. [#5769]

ramp_fitting

• Fixed bug in handling NGROUPS=2 exposures for pixels that saturate in group 2.
  Proper slope, err, and other quantities are now computed from the good data
  in group 1. [#5700]

• Update documentation to define optimal weighting algorithm [#5682]

resample

• Make inverse variance weight_type="ivm" the default weighting scheme for
  multiple exposures resampled into a single output. [#5738]

source_catalog

• Added the segmentation map as an output data file, with
  suffix "segm". [#5730]

srctype

• Changed default SRCTYPE for non-primary NIRSpec slits in a FIXEDSLIT
  exposure to 'EXTENDED' rather than 'POINT' [#5671]

• Changed logic for handling NIRSpec MOS exposures to blank out the "global"
  value of SRCTYPE, to ensure that only the individual slit-specific values
  of SRCTYPE get used downstream. [#5754]

stpipe

• Make jwst.stpipe independent of the rest of the jwst package and move
  core code to spacetelescope/stpipe. [#5695, #5720, #5752]

JWST Build 7.7.1 rc1

Final release notes are in the jwst v1.1.0 (Build 7.7.1) section.

JWST Build 7.7

Summary of changes in the 0.18.x release

Updates to Level-2a processing for all exposure modes

• Fixed bug in the minimum number of groups per integration for the jump detection step by changing it from 3 to 5
• Reworked the jump step to reduce memory usage
• Updated the ramp_fit step to add 'DO_NOT_USE' DQ flag to pixels that have all groups flagged as saturated
• Turned the MIRI "reset" step back on in the level-2a pipeline
• Updated the saturation step to use the A-to-D limit of 65535 as the saturation threshold for pixels flagged with "NO_SAT_CHECK" in the saturation reference file
• Updated the saturation step to flag groups having values below the A-to-D floor (zero)

Updates to Level-2b processing for all exposure modes

• Enhance the flat_field step to flag MIRI imager NON_SCIENCE pixels as DO_NOT_USE
• Enhancements to assign_wcs to add FITS SIP approximation keywords to imaging mode WCS

Updates to Level-2b spectroscopic (calwebb_spec2) processing

• Enable the resample_spec step for NIRSpec line lamp exposures
• Enhancements and bug fixes to the extract_1d step for extracting and applying aperture corrections to IFU mode data
• Bug fixes to the wavecorr step to produce correct wavelength shifts
• Enable processing of NRS_LAMP exposures in the assign_wcs and resample_spec steps
• Enhancements and bug fixes to cube_build for handling single cubes
• Bug fix for extract_1d when determining source RA/Dec in input resampled Slit data
• Enhanced extract_1d to use an extract1d reference file to guide processing for NIRCam TSGRISM exposures, allowing for column-by-column fitting and subtraction of background signal
• Bug fix for extract_1d when there's only one spectrum instance for a WFSS source
• Updated flat_field step branch logic to handle NRS_LAMP as a spectroscopic exposure
• Updated NIRSpec fixed-slit processing in the flat_field and photom steps to compute and save correction values for both point and uniform sources in the primary slit when it contains a point source, in order to support master background corrections

Updates to the Level-3 imaging (calwebb_image3) processing

• Bug fix in outlier_detection to account for background signal when blotting the median-combined image back to individual exposure's space
• Bug fix in the skymatch step to prevent failures when attempting to compute statistics on a single pixel
• Implemented memory checks in the outlier_detection and resample steps to prevent the creation of huge arrays and subsequent memory allocation errors
• Added "pixel_scale_ratio" parameter to the resample step to allow for finer output grid
• Added support for the new fitting geometry mode "rshift" in the tweakreg step

Updates to the Level-3 spectroscopic (calwebb_spec3) processing

• Update the NIRSpec MOS master background logic to only proceed with processing after verifying that there are both background and source slits available in the input dataset
• Enhancement to the combine_1d step to produce separate extensions for each spectral order

Updates to the Level-3 TSO (calwebb_tso3) pipeline

• Updated the tso_photometry step to place the photometric aperture at the location given by the XREF_SCI, YREF_SCI keyword values, instead of using CRPIX1, CRPIX2
• Fixed the flux units in the source_catalog output
• Updated the white_light step to allow for the specification of wavelength limits to restrict the portion of the spectrum used to compute the integrated flux

Updates to the Level-3 AMI (calwebb_ami3) pipeline

• Complete overhaul of the ami_analyze step to use the latest ImPlaneIA algorithms from the AMI WG
• Enhancements to the ami_analyze step to allow two user-settable parameter

Detailed notes

documentation

• Update documentation introduction to include installation and CRDS setup
  instructions. [#5659]

combine1d

• Fixed code error in combine1d, creating extensions per spectral order
  with the same input data [#5644]

ramp_fitting

• Fix a bug in estimating the max number of segments that will be needed
  to fit any pixel [#5653]

set_telescope_pointing

• Update the check in set_telescope_pointing that determines whether an
  exposure is TSO mode to always consider hardwired TSO EXP_TYPEs as TSO,
  regardless of TSOVISIT and NINTS settings. [#5657]

white_light

• Fixed error causing multi-segment data to reject int_times
  for MJDs [#5566]

JWST Build 7.7 rc3

associations

• JWSTDMS-410 Asn_Lv2NRSLAMPSpectral: Break out the negative cases [#5635]

• Update MIRI LRS-Fixedslit ALONG-SLIT-NOD backgrounds strategies [#5620]

cube_build

• Do not allow varibles defined in spec (part of the cube_build_step class) to
  be changed, to allow calspec2 to loop over a list of files and run the
  pipeline. [#5603]

datamodels

• Updated schemas for new keywords CROWDFLD, PRIDTYPE, PRIDTPTS, PATTNPTS, SMGRDPAT,
  changed name of SUBPXPNS to SUBPXPTS, and new allowed values for PATTTYPE. [#5618]

flat_field

• Added DO_NOT_USE to pixels flagged as NON_SCIENCE for non-NIRSpec data [#5601]

outlier_detection

• Account for the background subtracted data in the blot image for determining
  the noise image used in flagging outliers [#5601]

set_telescope_pointing

• Updated to populate XREF_SCI, YREF_SCI keywords for all TSO exposures,
  not just NRC_TSGRISM mode. [#5616]

JWST Build 7.7 rc2

combine1d

• Output FITS now contains separate combine1d extensions for each spectral
  order present in the data [#5204]

datamodels

• Fix missing CHANNEL entry in distortion reffile schema. [#5553]

extract_1d

• For IFU data (NIRSpec and MIRI) the extraction radius is now a varying size
  based on wavelength. The apcorr correction is a function of wavelength and
  radius size. Fixes a bug in units conversion for applying the apcorr correction.
  The units are now correctly converted from arcseconds to pixels. Added an
  new method to apply the apcorr correction for IFU data. [#5506]

pipeline

• Removed all unnecessary parameter settings from cfg files for all steps
  and pipelines, and removed references to step config files from most
  pipeline modules (only kept those that are necessary for intended
  functionality). [#5574]

skymatch

• Fixed a bug due to which sky matching may fail under certain circumstances
  such as using 'mode' statistics on a single pixel (after sigma-clipping). [#5567]

stpipe

• Removed unused LinearPipeline class. [#5590]

wavecorr

• Fixed bugs in wavecorr. [#5570]