Skip to content

Commit

Permalink
Merge branch 'feat-logger/1-remove-stdout-suppression' into feat-logg…
Browse files Browse the repository at this point in the history
…er/2-replace-print-logger-analysis_db

# Conflicts:
#	src/icesat2waves/analysis_db/A02c_IOWAGA_thredds_prior.py
#	src/icesat2waves/analysis_db/B02_make_spectra_gFT.py
#	src/icesat2waves/analysis_db/B03_plot_spectra_ov.py
#	src/icesat2waves/analysis_db/B06_correct_separate_var.py
  • Loading branch information
hollandjg committed Mar 11, 2024
2 parents 416940c + 5613097 commit 1258f9d
Show file tree
Hide file tree
Showing 78 changed files with 2,888 additions and 33,095 deletions.
26 changes: 26 additions & 0 deletions .devcontainer/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Use the official Python 3.11 image from the Docker Hub
FROM python:3.11

# Update the system and install the packages
RUN apt-get update && apt-get install -y \
git-lfs \
openmpi-bin \
openmpi-doc \
libopenmpi-dev \
gdal-bin \
libgdal-dev


# Set the working directory to /app
WORKDIR /app

# Copy the current directory contents into the container at /app
COPY . /app

# Create the output directory
RUN mkdir -p /app/output

# Install any needed packages specified in pyproject.toml
RUN pip install --editable ".[dev]"

CMD ["/bin/bash"]
26 changes: 26 additions & 0 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
// For format details, see https://aka.ms/devcontainer.json. For config options, see the
// README at: https://github.com/devcontainers/templates/tree/main/src/docker-existing-dockerfile
{
"name": "Existing Dockerfile",
"build": {
// Sets the run context to one level up instead of the .devcontainer folder.
"context": "..",
// Update the 'dockerFile' property if you aren't using the standard 'Dockerfile' filename.
"dockerfile": "Dockerfile"
}

// Features to add to the dev container. More info: https://containers.dev/features.
// "features": {},

// Use 'forwardPorts' to make a list of ports inside the container available locally.
// "forwardPorts": [],

// Uncomment the next line to run commands after the container is created.
// "postCreateCommand": "cat /etc/os-release",

// Configure tool-specific properties.
// "customizations": {},

// Uncomment to connect as an existing user other than the container default. More info: https://aka.ms/dev-containers-non-root.
// "remoteUser": "devcontainer"
}
10 changes: 10 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Set update schedule for GitHub Actions

version: 2
updates:

- package-ecosystem: "github-actions"
directory: "/"
schedule:
# Check for updates to GitHub Actions every month
interval: "monthly"
9 changes: 4 additions & 5 deletions .github/workflows/publish_pypi.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: Publish Icesat2 to PyPI
name: Publish icesat2waves to PyPI

on:
release:
Expand All @@ -13,10 +13,10 @@ jobs:
id-token: write
steps:
- name: Checkout code
uses: actions/checkout@v2
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v2
uses: actions/setup-python@v5
with:
python-version: 3.11

Expand All @@ -32,5 +32,4 @@ jobs:
- name: Publish to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
with:
packages_dir: dist
repository-url: https://test.pypi.org/legacy/
packages_dir: dist
4 changes: 2 additions & 2 deletions .github/workflows/python_ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@ jobs:
- name: install mpi macos
if: matrix.os == 'macos-latest'
run: brew install open-mpi
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: setup python
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.version }} # install the python version needed
cache: "pip"
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/test_icesat2_tracks_pipeline.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,9 @@ jobs:
steps:
- name: install mpi
run: sudo apt update && sudo apt-get install openmpi-bin openmpi-doc libopenmpi-dev
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: setup python
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.version }} # install the python version needed
cache: "pip"
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/test_test_suite.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,11 @@ jobs:
steps:
- name: install mpi
run: sudo apt update && sudo apt-get install openmpi-bin openmpi-doc libopenmpi-dev
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
lfs: true
- name: setup python
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.version }} # install the python version needed
cache: "pip"
Expand Down
48 changes: 0 additions & 48 deletions Dockerfile

This file was deleted.

162 changes: 8 additions & 154 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,15 @@
# ICESAT2 Track Analysis

- [ICESAT2 Track Analysis](#icesat2-track-analysis)
- [Installation for Developers](#installation-for-developers)
- [Command line interface](#command-line-interface)
- [Sample workflow](#sample-workflow)

## Installation for Developers

Prerequisites:
- A POSIX-compatible system (Linux or macOS)
- Python 3.9 (run `python --version` to check that your version of python is correct)
- Python 3.10 (run `python --version` to check that your version of python is correct)
- MPI (e.g. from `brew install open-mpi` on macOS)
- HDF5 (e.g. from `brew install hdf5` on macOS)

Expand Down Expand Up @@ -50,162 +55,11 @@ Installation:
> pip install --upgrade --editable ".[dev]"
> ```

- Check the module `icesat2_tracks` is available by loading the module:
- Check the module `icesat2waves` is available by loading the module:
```shell
python -c "import icesat2_tracks; print(icesat2_tracks.__version__)"
python -c "import icesat2waves; print(icesat2waves.__version__)"
```

## Installing on Oscar (Deprecated)

If any of these commands fail, check the conda configuration (listed below) before retrying for possible fixes.

Load a conda module.

```shell
module load miniconda/23.1.0
```

Follow any prompts from the module load command, like running the following line:
```shell
source /gpfs/runtime/opt/miniconda/23.1.0/etc/profile.d/conda.sh
```

Create a new environment using:
```shell
conda create --name "2021-icesat2-tracks"
```

Activate the environment using:
```shell
conda activate "2021-icesat2-tracks"
```

Install or update the packages in the environment with those listed in the `environment.yml` file using:
```shell
conda env update --file environment.yml
```

(You can create and install dependencies in the environment in a single command using:
```shell
conda env create --name "2021-icesat2-tracks" --file environment.yml
```
... but this has more steps and is thus more likely to fail. Since the installation step takes a long period of time, it is recommended to use the separate commands instead.)

## Conda Configuration (Deprecated)

Conda draws its configuration from multiple places, and will behave differently when the configuration is different, even when using the same `environment.yml` file.

#### `.condarc`

The `.condarc` file in your home directory sets your conda configuration. If the file doesn't exist, you can create it with:
```shell
touch ~/.condarc
```
#### `pkgs_dirs`
`pkgs_dirs` is the location where conda downloads package files from registries like `anaconda.org`.
If you use the defaults, when trying to install packages you may get a warning like:
```
WARNING conda.lock:touch(51): Failed to create lock, do not run conda in parallel processes [errno 13]
...
ERROR Could not open file /gpfs/runtime/opt/miniconda/4.12.0/pkgs/cache/b63425f9.json
```
In this case, you might be trying to download packages to the global directory `/gpfs/runtime/opt/miniconda/4.12.0/pkgs` where you have no write-permissions, rather than your home directory where you have write-permissions.
View the conda configuration:
```shell
conda config --show
```
Check that the `pkgs_dirs` setting points to a location in your home directory:
```yaml
pkgs_dirs:
- /users/username/anaconda/pkg
```
If it doesn't, update this using:
```shell
conda config --add pkgs_dirs ~/anaconda/pkg
```

(Use `--remove` instead of `--add` to remove an entry.)

#### `envs_dirs`

`envs_dirs` is the location where there is a separate directory per environment containing the installed packages.

View the conda configuration:
```shell
conda config --show
```

Check that the `envs_dirs` setting to a location in your home directory:
```yaml
envs_dirs:
- /users/username/anaconda/envs
```

... and update this using:
```shell
conda config --add envs_dirs ~/anaconda/envs
```

(Use `--remove` instead of `--add` to remove an entry.)

Always re-check the configuration after running the `conda config` command.

#### Environment Variables

Note that modules (like `miniconda/23.1.0`) set environment variables like `CONDA_ENVS_PATH` which override the conda config.

You might view the conda config and see the following entries:
```yaml
envs_dirs:
- /users/username/anaconda
- /users/username/anaconda/envs
```

If you try to run
```shell
conda config --remove envs_dirs ~/anaconda
```
... you'll get a warning:
```
'envs_dirs': '/users/username/anaconda' is not in the 'envs_dirs' key of the config file
```
... and find that the value is still there when you rerun `conda config --show`:
```yaml
envs_dirs:
- /users/username/anaconda # <-- still here!
- /users/username/anaconda/envs
```
The value might have been silently set by the `module load` command using an environment variable.
Check for environment variables by running:
```shell
$ printenv | grep ^CONDA_
CONDA_SHLVL=0
CONDA_EXE=/gpfs/runtime/opt/miniconda/23.1.0/bin/conda
CONDA_ENVS_PATH=~/anaconda # <- this is the offending variable
CONDA_PYTHON_EXE=/gpfs/runtime/opt/miniconda/23.1.0/bin/python
```
To unset a value like `CONDA_ENVS_PATH` use:
```shell
unset CONDA_ENVS_PATH
```
... then check that rerun `conda config --show` no longer shows the has modified the conda config to match the values you wanted:
```yaml
envs_dirs:
- /users/username/anaconda/envs
```
## Command line interface

The `icesat2waves` package comes with a command-line interface (CLI) that facilitates interaction with the package directly from your terminal. This can be particularly useful for scripting and automation. You can access the help documentation for the CLI by running the following command:
Expand Down
Loading

0 comments on commit 1258f9d

Please sign in to comment.