From 074bf1185cbc2b93de11d60ed1a7e01fcc60883b Mon Sep 17 00:00:00 2001 From: Bradley Sappington Date: Mon, 16 Dec 2024 13:14:37 -0500 Subject: [PATCH 1/5] Readme update --- README.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.rst b/README.rst index f23f5d6..0d17cfb 100644 --- a/README.rst +++ b/README.rst @@ -22,7 +22,7 @@ STPSF: Simulated Point Spread Functions for the James Webb and Nancy Grace Roman :target: http://ascl.net/1504.007 -**ADVISORY: STPSF IS CURRENTLY BEING MIGRATED FROM WEBBPSF - THIS REPOSITORY IS NOT READY FOR USE** +**STPSF HAS BEEN MIGRATED FROM WEBBPSF** STPSF produces simulated PSFs for the James Webb Space Telescope, NASA's flagship infrared space telescope. STPSF can simulate images for any of the From 45f2ceb1994d544bf1e23b9256bcaf50ebc3615c Mon Sep 17 00:00:00 2001 From: Bradley Sappington Date: Mon, 16 Dec 2024 17:28:47 -0500 Subject: [PATCH 2/5] update release notes --- docs/release.rst | 136 +++++++++++++++++++++++++++-------------------- 1 file changed, 79 insertions(+), 57 deletions(-) diff --git a/docs/release.rst b/docs/release.rst index 0bf9468..57400c0 100644 --- a/docs/release.rst +++ b/docs/release.rst @@ -4,71 +4,93 @@ Developer Notes: Releasing a new version of STPSF Prerequisites ============= - + * If you are making a release for `poppy` at the same time as a release in STPSF, do that first. + Update the dependency requirement to the new version of poppy, in ``stpsf/pyproject.toml``. * Is the `develop` build `passing on Github Actions? `_ with all desired release items included? + * Does the latest release install properly? + ``$ conda env remove --name ENV_NAME`` + ``$ conda create -n ENV_NAME python=3.xx`` + ``$ conda activate ENV_NAME`` + ``$ pip install -e .`` + ``$ pip uninstall tox`` + ``$ pip install tox tox-conda`` Releasing new data packages =========================== - #. Run ``dev_utils/master_data_release.sh`` (details below) to make a gzipped tarred archive of the STPSF data + #. Run ``dev_utils/master_data_release.sh`` to make a gzipped tarred archive of the STPSF data: + #. If you are on the Institute network + #. ``$ cd stpsf/dev_utils/`` + #. ``$ ./master_data_release.sh 0.X.Y`` + #. If you're working from a local data root + #. ``$ cd stpsf/dev_utils/`` + #. ``$ DATAROOT="/Users/you/stpsf-data-sources/" ./make-data-sdist.sh 0.X.Y`` + #. ``$ cp ./stpsf-data-0.X.Y.tar.gz /where/ever/you/want/`` #. If the new data package is **required** (meaning you can't run STPSF without it, or you can run but may get incorrect results), you must bump ``DATA_VERSION_MIN`` in ``__init__.py`` to ``(0, X, Y)`` #. Extract the resulting data archive and check that you can run the STPSF tests with ``STPSF_PATH`` pointing to it #. Copy the data archive into public web space. This now means on Box. The following steps need to be performed in this sequence in order to preserve the naming conventions. - #. Find stpsf-data-LATEST.tar.gz, and click on "more options" and "Update Version". Choose the newest version of stpsf-data-#.#.#.tar.gz - #. This will change the name of stpsf-data-LATEST.tar.gz to be what you just uploaded, rename the file back to "stpsf-data-LATEST.tar.gz" - #. Upload to Box a separate version of stpsf-data-#.#.#.tar.gz shared data folder for future storage. - #. Find minimal-stpsf-data-LATEST.tar.gz, and click on "more options" and "Update Version". Choose the newest version of minimal-stpsf-data-#.#.#.tar.gz - #. This will change the name of minimal-stpsf-data-LATEST.tar.gz to be what you just uploaded, rename the file back to "minimal-stpsf-data-LATEST.tar.gz" - #. Upload to Box a separate version of minimal-stpsf-data-#.#.#.tar.gz shared data folder for future storage. - #. Verify the shared link of stpsf-data-latest.tar.gz is the same that exists in ``docs/installation.rst`` ("copy shared link" then "link settings") - - #. A shared copy will be automatically configured in STScI Central Store with updated symlink ``/grp/jwst/ote/stpsf-data`` - #. Update the URLS in ``installation.rst`` under :ref:`data_install` - -Details for using `master_data_release.sh`: -------------------------------------------- - -Invoke ``dev_utils/master_data_release.sh`` one of the following ways to make a gzipped tarred archive of the STPSF data suitable for distribution. - -**If you are on the Institute network:** :: - - $ cd stpsf/dev_utils/ - $ ./master_data_release.sh 0.X.Y - -**If you're working from a local data root:** :: - - $ cd stpsf/dev_utils/ - $ DATAROOT="/Users/you/stpsf-data-sources/" ./make-data-sdist.sh 0.X.Y - $ cp ./stpsf-data-0.X.Y.tar.gz /where/ever/you/want/ - -Releasing new versions -====================== - -If you are making a release for `poppy` at the same time as a release in STPSF, do that first. -Update the dependency requirement to the new version of poppy, in ``stpsf/pyproject.toml``. - -When you are ready, proceed with the STPSF release as follows: - - #. Get the `develop` branch into the state that you want, including all PRs merged, updated release notes. This includes all tests passing both locally and on GitHub Actions. - #. Tag the commit with `v`, being sure to sign the tag with the `-s` option. - * ``git tag -s v -m "Release v"`` - - #. Push tag to github, on `develop` - #. On github, make a PR from `develop` to `stable` (this can be done ahead of time and left open, until all individual PRs are merged into `develop`.). - #. After verifying that PR is complete and tests pass, merge it. (Once merged, both the `stable` and `develop` branches should match). + #. Find ``stpsf-data-LATEST.tar.gz``, and click on "more options" and "Update Version". Choose the newest version of ``stpsf-data-#.#.#.tar.gz`` + #. This will change the name of ``stpsf-data-LATEST.tar.gz`` to be what you just uploaded, rename the file back to ``stpsf-data-LATEST.tar.gz`` + #. Upload to Box a separate version of ``stpsf-data-#.#.#.tar.gz`` shared data folder for future storage. + #. Find ``minimal-stpsf-data-LATEST.tar.gz``, and click on "more options" and "Update Version". Choose the newest version of ``minimal-stpsf-data-#.#.#.tar.gz`` + #. This will change the name of ``minimal-stpsf-data-LATEST.tar.gz`` to be what you just uploaded, rename the file back to ``minimal-stpsf-data-LATEST.tar.gz`` + #. Upload to Box a separate version of ``minimal-stpsf-data-#.#.#.tar.gz`` shared data folder for future storage. + #. Verify the shared link of ``stpsf-data-latest.tar.gz`` is the same that exists in ``docs/installation.rst`` ("copy shared link" then "link settings") + #. A shared copy will be automatically configured in STScI Central Store with updated symlink ``/grp/stpsf/stpsf-data`` + #. Verify code base is still up to date with box links and version names (they should be) + #. Verify ``installation.rst`` with link to box data (this shouldn't need to change the box link, but verify it hasn't changed) + #. update minimal in the ci setup (``stpsf/.github/workflows/download_data.yml``) (this also shouldn't need to change as the box link is for latest) + #. update ``stpsf/stpsf/__init__.py`` with version number (DATA_VERSION_MIN) + #. CITATIONS.cff with new version + #. Generate the release notes + #. You can do a draft release on github to autogenerate the relnotes + #. In github set new release to be pre-release and make a release candidate tag - 1.3.0.rc1 + #. Auto-generate the release notes + #. Copy output to ``docs/relnotes.rst`` + #. Verify everything on sphinx locally on your computer (in docs directory $make html) + #. make sure sphinx is installed properly with its needed dependencies + #. navigate to docs folder + #. ``$ pip install -r requirements.txt`` + #. ``$ pip uninstall graphviz`` + #. ``$ conda install graphviz`` + #. ``$ make clean`` (not needed for initial run, just to reset everything) + #. ``$ make html`` + #. Merge in any changes you've made including release notes + #. NOTE: If the builds fail, this may be because we dont have the data cache fetched yet if so: + #. Go to https://github.com/spacetelescope/stpsf/actions/workflows/download_data.yml + #. Open "Run Workflow" box, change it to point to your pre-release-xxx branch (or whatever your pre release branch is named), run it + #. It should then make a cache for you of the new version data + #. If you look at https://github.com/spacetelescope/stpsf/actions/caches, you will see what is available that the CI can pick from + #. THIS may still not solve the problem, as the cache is only in your branch, not the actual PR. so if the branch passes, the PR should + #. Theoretically be fine (despite its failure). You can merge, and then re-run the "dowload data" action for develop, and then re-run your failed jobs in develop. + #. Do not release unless develop is passing all tests + #. Update the test_readthedocs branch. Force development there. Test it on readthedocs (it should be hidden on the actual site). + #. Checkout the branch you want to overwrite (test_readthedocs) + #. ``$git checkout test_readthedocs`` + #. Reset the target branch to match the source branch (develop) + #. ``$git reset --hard develop`` + #. Push to the github repo (probably upstream, may be origin, just dont do your personal one) + #. ``$git push upstream test_readthedocs --force`` + #. Once readthe docs looks all good test your release on test pypi. + #. Create new env and install STPSF + #. ``$ pip install build twine`` + #. ``$ python -m build`` + #. ``$ twine check dist/*`` + #. ``$ twine upload --repository-url https://test.pypi.org/legacy/ dist/* --verbose`` (NOTE: API token is the password in your ~/.pypirc testpypi token) + #. test that you can download and install in fresh env (have pypi as backup for libraries that aren't on testpypi): + #. ``$ pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ stpsf==`` + #. Tag a version in develop and push it to git (do it through local terminal, not through website) + #. ``$ git tag -a -m “webbpsf v1.4.0”`` ( is just the version number --> 1.4.0) + #. ``$ git push upstream `` + #. Go to stable branch, and look at where it says how many commits behind it is from develop. Click that to generate a pull request (do not squash when you merge here) + #. When tests pass merge them to stable #. Release on Github: - - #. On Github, click on "[N] Releases". - #. Select "Draft a new release". + #. On Github, click on ``[N] Releases`` + #. Select ``Draft a new release``. #. Specify the version number, title, and brief description of the release. - #. Press "Publish Release". - - #. Release to PyPI. This should now happen automatically on GitHub Actions. This will be triggered by a GitHub Actions build of a tagged commit on the `stable` branch, so it will happen automatically on the prior step for the PR into `stable`. - - -Finishing the release -===================== - + #. Press ``Publish Release`` + #. Release to PyPI should now happen automatically on GitHub Actions. This will be triggered by a GitHub Actions build of a tagged commit on the `stable` branch. + #. Verify that files stored in ``/grp/stpsf/stpsf-data`` (symlink directory) have the correct permissions. + #. ``$ cd /grp/stpsf/`` + #. ``$ find . -type f -exec chmod 755 {} \;`` (current and all subdirectories should be rwxr-xr-x) #. Email an announcement to ``stpsf-users@maillist.stsci.edu`` - - From 9523b595925b705650d37b871e571dfb0b375ee3 Mon Sep 17 00:00:00 2001 From: Bradley Sappington Date: Mon, 16 Dec 2024 17:29:26 -0500 Subject: [PATCH 3/5] update file paths to new mount --- dev_utils/compute_psf_library.py | 2 +- dev_utils/grab_opd_info.py | 3 +-- dev_utils/make-data-sdist.sh | 3 +-- dev_utils/master_data_release.sh | 8 ++++---- docs/installation.rst | 2 +- stpsf/stpsf_core.py | 2 +- 6 files changed, 9 insertions(+), 11 deletions(-) diff --git a/dev_utils/compute_psf_library.py b/dev_utils/compute_psf_library.py index 47d45a1..6a8c631 100644 --- a/dev_utils/compute_psf_library.py +++ b/dev_utils/compute_psf_library.py @@ -15,7 +15,7 @@ assert exists(os.environ['PYSYN_CDBS']), "Can't load synthetic photometry files!" if not os.environ.get('STPSF_PATH'): - os.environ['STPSF_PATH'] = '/grp/jwst/ote/stpsf-data' # TODO SAPP change to stpsf when new mount is ready + os.environ['STPSF_PATH'] = '/grp/stpsf/stpsf-data' import stpsf # noqa N_PROCESSES = 16 diff --git a/dev_utils/grab_opd_info.py b/dev_utils/grab_opd_info.py index f0c9555..d56df84 100644 --- a/dev_utils/grab_opd_info.py +++ b/dev_utils/grab_opd_info.py @@ -7,8 +7,7 @@ from scipy import ndimage import stpsf -# TODO SAPP - this should transition to stpsf location after new mount -matfile = '/grp/jwst/ote/stpsf-data-source/OTE_source/JWST_thermal_response_data05_31_2017.mat' +matfile = '/grp/stpsf/stpsf-data-source/OTE_source/JWST_thermal_response_data05_31_2017.mat' def load_matfile(matfile): diff --git a/dev_utils/make-data-sdist.sh b/dev_utils/make-data-sdist.sh index 957ae7e..e709e17 100755 --- a/dev_utils/make-data-sdist.sh +++ b/dev_utils/make-data-sdist.sh @@ -7,9 +7,8 @@ if ! [[ $1 ]]; then echo " ./make-data-sdist.sh 0.3.3" exit 1 fi -# TODO SAPP move this to proper location stpsf after new mount if ! [[ $DATAROOT ]]; then - DATAROOT="/grp/jwst/ote/stpsf-data-source/" + DATAROOT="/grp/stpsf/stpsf-data-source/" fi echo "Using data from $DATAROOT" diff --git a/dev_utils/master_data_release.sh b/dev_utils/master_data_release.sh index 6cff401..71fd7e6 100755 --- a/dev_utils/master_data_release.sh +++ b/dev_utils/master_data_release.sh @@ -1,7 +1,7 @@ #!/bin/bash # Top-level script to make a distributable version of the data files -# See /grp/stpsf/stpsf-data-source/README_DEVEL.txt # TODO SAPP VERIFY THIS IS CREATED +# See /grp/stpsf/stpsf-data-source/README_DEVEL.txt if ! [[ $1 ]]; then echo "Provide a version string, e.g.:" @@ -15,10 +15,10 @@ TMPDIR="/tmp/stpsf-data" ./make-data-sdist.sh $VER echo -echo "Copying latest data to /grp/jwst/ote for internal stsci use..." -main_directory="/grp/jwst/ote" # TODO SAPP - update when stpsf new data mount +echo "Copying latest data to /grp/stpsf for internal stsci use..." +main_directory="/grp/stpsf" new_directory="$main_directory/stpsf-data-$VER" -symlink_directory="/grp/jwst/ote/stpsf-data" +symlink_directory="/grp/stpsf/stpsf-data" legacy_webbpsf_symlink_directory="/grp/jwst/ote/webbpsf-data" cp "$PWD/stpsf-data-$VER.tar.gz" "$main_directory" diff --git a/docs/installation.rst b/docs/installation.rst index 700af08..94a968f 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -87,7 +87,7 @@ You should now be able to successfully ``import stpsf`` in a Python session. .. Note:: **For STScI Users Only:** Users at STScI may access the required data files from the Central Storage network. Set the following environment variables in your ``bash`` shell. (You will probably want to add this to your ``.bashrc``.) :: - export STPSF_PATH="/grp/jwst/ote/stpsf-data" + export STPSF_PATH="/grp/stpsf/stpsf-data" export PYSYN_CDBS="/grp/hst/cdbs" Software Requirements diff --git a/stpsf/stpsf_core.py b/stpsf/stpsf_core.py index ad6a1e7..8617295 100644 --- a/stpsf/stpsf_core.py +++ b/stpsf/stpsf_core.py @@ -1638,7 +1638,7 @@ def load_wss_opd(self, filename, output_path=None, backout_si_wfe=True, verbose= output_path : str Downloaded OPD are saved in this location. - This option is convinient for STScI users using /grp/jwst/ote/stpsf-data/. + This option is convinient for STScI users using /grp/stpsf/stpsf-data/. Default is $STPSF_PATH/MAST_JWST_WSS_OPDs backout_si_wfe : bool From 943d693a19283ea25043fb2b21e2d07df4fe168c Mon Sep 17 00:00:00 2001 From: Bradley Sappington Date: Mon, 16 Dec 2024 17:29:46 -0500 Subject: [PATCH 4/5] use SHA in workflow --- .github/workflows/ci_workflows.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci_workflows.yml b/.github/workflows/ci_workflows.yml index 6c5b003..2dffac5 100644 --- a/.github/workflows/ci_workflows.yml +++ b/.github/workflows/ci_workflows.yml @@ -4,7 +4,7 @@ on: [push, pull_request] jobs: retrieve_cache: - uses: spacetelescope/stpsf/.github/workflows/retrieve_cache.yml@develop # 2090d50be6b75f2c241c07ae0538bff16e5be25e # v1.4.0 # TODO SAPP change to initial release + uses: spacetelescope/stpsf/.github/workflows/retrieve_cache.yml@098e3d3515d906bbfa12680d2282ec8d6726527b # v1.5.2 with: minimal: true tests: From e9f8a0a8b175222722a1c1b544350dcf6308ea5c Mon Sep 17 00:00:00 2001 From: Bradley Sappington Date: Tue, 17 Dec 2024 08:54:39 -0500 Subject: [PATCH 5/5] Document WebbPSF as pre 2.0.0 --- docs/index.rst | 1 + docs/relnotes.rst | 6 ++++++ 2 files changed, 7 insertions(+) diff --git a/docs/index.rst b/docs/index.rst index 7815eec..f8a77e2 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -1,6 +1,7 @@ Documentation for STPSF =============================== +**(Formerly WebbPSF, for versions < 2.0)** STPSF is a Python package that computes simulated point spread functions (PSFs) for NASA's James Webb Space Telescope (JWST) and Nancy Grace Roman Space Telescope. STPSF transforms models of telescope and instrument optical state into PSFs, taking into account detector pixel scales, rotations, filter profiles, and point source spectra. It is rather a tool for transforming optical path difference (OPD) maps, created with observatory systems engineering models, into the resulting PSFs as observed with JWST's or Roman's instruments. .. figure:: ./fig_instrument_comparison.png diff --git a/docs/relnotes.rst b/docs/relnotes.rst index bce2b70..926b6fd 100644 --- a/docs/relnotes.rst +++ b/docs/relnotes.rst @@ -23,6 +23,12 @@ See https://github.com/spacetelescope/stpsf/issues for currently open issues and Version History and Change Log ------------------------------- +Version 2.0.0 +============= +*2024 December* +Ready for Production Release +All versions before 1.5.1 were part of the WebbPSF repository + Version 1.5.2 ============= *2024 December*