Add support for CASA-based MS calibration tables

CHANGELOG.md

@@ -4,6 +4,19 @@ All notable changes to this project will be documented in this file.
 ## [Unreleased]
 
 ### Added
+- Support for writing "MODEL_DATA" and "CORRECTED_DATA" columns has been added to the
+`UVData.write_ms` method.
+- Support for "flexible-Jones" `UVCal` objects -- where different spectral windows can
+contain different Jones vectors/polarization information -- has been added, along with
+methods for converting `UVCal` objects to and from flex-Jones format (`remove_flex_jones`,
+`_make_flex_jones`, and `convert_to_flex_jones`). Support for flex-Jones has been added
+to `MSCal` and `CalH5` subclasses.
+- A new class called `MSCal` has been added as a subclass of `UVCal`, which adds support
+for reading and writing of CASA Measurement Set gains, bandpass, and delays tables. To
+support this, several new optional parameters have been added to the `UVCal` class,
+including `phase_center_catalog`, `phase_center_id_array`, `antenna_diameters`,
+`scan_id_array`, and `ref_antenna_array` (which affords the ability to record a
+different reference antenna per time-axis entry).
 - A new HDF5 file format for calibration solutions called `calh5`, which supports
 writing out all types of UVCal objects and supports partial reads (select on read).
 - A `UVCal.get_lst_array` method similar to the `get_time_array` method that either
@@ -37,6 +50,10 @@ times for those solutions.
 - Updated minimum optional dependency versions: astropy-healpix>=1.0.2
 
 ### Fixed
+- Fixed a bug in `UVBase` where `allowed_failures` was being ignored if a parameter had
+`required=True` set.
+- Fixed a bug where selection and addition/concat methods did not operate correctly on
+flex-spw `UVCal` objects where `Nspws>1`.
 - Fixed a bug writing UVData objects to UVFITS formats with extra phase center catalog
 entries resulted in an error on read.
 - Fixed a bug where calling `UVData._set_app_coords_helper` with extra phase center

README.md

@@ -52,7 +52,10 @@ It is probably not compatible with other interferometric HDF5 files defined by o
 * MWA correlator FITS files (read-only support, tested against Cotter outputs and FHD)
 
 ## UVCal file formats
-* calfits: a new format defined in pyuvdata, details in the [calfits_memo](docs/references/calfits_memo.pdf).
+* calh5: a new format defined in pyuvdata,  details to come in a forthcoming memo.
+* Measurement Set calibration files (read and write, gains/delay/bandpass supported, beta version).
+Tested against a limited set of SMA, LWA, and VLA calibration files generated in CASA.
+* calfits: a custom format defined in pyuvdata, details in the [calfits_memo](docs/references/calfits_memo.pdf).
 Note that this format was recently defined and may change in coming versions,
 based on user needs. Consider it to be a beta version, but we will strive to
 make future versions backwards compatible with the current format.
@@ -253,6 +256,7 @@ pyuvdata is maintained by the RASG Managers, which currently include:
 
  - Adam Beardsley (Winona State University)
  - Bryna Hazelton (University of Washington)
+ - Garrett "Karto" Keating (Smithsonian Astrophysical Observatory)
  - Daniel Jacobs (Arizona State University)
  - Matt Kolopanis (Arizona State University)
  - Paul La Plante (University of Nevada, Las Vegas)

ci/pyuvdata_min_deps_tests.yml

@@ -10,7 +10,7 @@ dependencies:
   - pyerfa>=2.0
   - pyyaml>=5.4.1
   - coverage
-  - pytest>=6.2.0
+  - pytest>=6.2.5
   - pytest-cases>=3.8.3
   - pytest-cov
   - pytest-xdist

docs/uvcal_tutorial.rst

@@ -195,6 +195,21 @@ b) CalFITS to CalH5
   >>> cal.write_calh5(os.path.join('.', 'tutorial_cal.calh5'), clobber=True)
 
 
+c) MSCal to CalH5
+*******************
+.. code-block:: python
+
+  >>> import os
+  >>> from pyuvdata import UVCal
+  >>> from pyuvdata.data import DATA_PATH
+  >>> filename = os.path.join(DATA_PATH, 'sma.ms.amp.gcal')
+  >>> # Here we use the ``from_file`` class method, can also use the ``read`` method.
+  >>> # Can optionally specify the ``file_type`` to either method
+  >>> cal = UVCal.from_file(filename, use_future_array_shapes=True)
+
+  >>> cal.write_calh5(os.path.join('.', 'tutorial_cal.ms'), clobber=True)
+
+
 UVCal: Initializing from a UVData object
 ----------------------------------------
 The :meth:`pyuvdata.UVCal.initialize_from_uvdata` method allows you to initialize a UVCal

docs/uvdata_tutorial.rst

@@ -922,15 +922,15 @@ Note: there is now support for reading in only part of a uvfits, uvh5 or miriad
   >>> import matplotlib.pyplot as plt
   >>> from pyuvdata import UVData
   >>> from pyuvdata.data import DATA_PATH
-  >>> filename = os.path.join(DATA_PATH, 'day2_TDEM0003_10s_norx_1scan.uvfits')
+  >>> filename = os.path.join(DATA_PATH, 'day2_TDEM0003_10s_norx_1src_1spw.uvfits')
   >>> uvd = UVData.from_file(filename, use_future_array_shapes=True)
 
   >>> print(uvd.data_array.shape)
-  (1414, 128, 4)
+  (1360, 64, 4)
   >>> print(uvd.Ntimes)
-  17
+  15
   >>> print(uvd.Nfreqs)
-  128
+  64
   >>> # get the data for a single baseline and polarization
   >>> waterfall_data = uvd.get_data((1, 2, uvd.polarization_array[0]))
   >>> # get the corresponding times for this waterfall
@@ -952,27 +952,6 @@ Note: there is now support for reading in only part of a uvfits, uvh5 or miriad
 .. image:: Images/amplitude_waterfall.png
     :width: 600
 
-.. code-block:: python
-
-  >>> # The plot above has a discontinuity in the frequency axis because this
-  >>> # data set has spectral windows. Let's just plot the 0th one
-  >>> window_sel = uvd.flex_spw_id_array == 0
-  >>> freqs_use = uvd.freq_array[window_sel]
-  >>> fig, ax = plt.subplots(1, 1)
-  >>> _ = plt.imshow(np.abs(waterfall_data[:,  window_sel]))
-  >>> _ = ax.set_yticks([0, waterfall_times.size - 1])
-  >>> _ = ax.set_yticklabels([waterfall_times[0], waterfall_times[1]])
-  >>> freq_tick_inds = np.concatenate((np.arange(0, freqs_use.size, 16), [freqs_use.size-1]))
-  >>> _ = ax.set_xticks(freq_tick_inds)
-  >>> _ = ax.set_xticklabels([f"{val:.3f}" for val in freqs_use[freq_tick_inds]*1e-9])
-  >>> _ = ax.set_xlabel("Frequency (GHz)")
-  >>> plt.show() # doctest: +SKIP
-  >>> plt.savefig("Images/amplitude_waterfall_spw0.png", bbox_inches='tight')
-  >>> plt.clf()
-
-.. image:: Images/amplitude_waterfall_spw0.png
-    :width: 600
-
 
 UVData: Location conversions
 ----------------------------