Replace ORCA with non-global state (ActivitySim#654)

* recode mazs in maz-to-maz table * basic implementation of parquet pipeline * fix overrun caused by dtype optimization * pipeline is a real class * whales * refactoring * refactoring [ci-skip] * refactoring [skip ci] * refactoring * refactoring * refactor * refactoring * refactoring * refactoring * refactoring * refactoring * refactoring * refactor * refactoring * single process mtc ok * clean logging * refactor MP * refactor-mp * core.tests * fix mtc * style * test cli * bump python to 3.10 * 3.10 as str * test summarize * clean up checkpointing * clean checkpoints * various tests * fix vts * optimize imports * optimize imports * fix is_open and is_readonly * refactoring * estimation only when settings exist * fix circular import * test_mtc_progressive * mtc progressive test * persist sharrow cache * asim create always makes a subdir * cleaning out inject * testing xborder * run_if_exists * cleanup * refactor * refactoring * fix persons_merged * refactor * refactor * refactor * refactor * external examples * refactor * refactor * eval_str on annotations * GA external regional models * actions/cache@v3 * clean * state prep * whale to state * fixes * fixes * fix annotations as str * zarr fix * xborder fix * bugfix * fixes * extending * docstring * refactor * refactor * optional altair * viz clean * hist bounds * dataset accessor * steps use default args * duplicate_step_execution * trace validation * bugfixing * bugfixing * bugfixing * more error logging when external example fails * require appdirs * missing state * fix tests * cleanup * fix multizone * allow tar.gz trace validation * fix marin * test fixing * fix random seed * mtc_extended * mtc_ext * mtc_ext * mtc_ext * mtc_ext * don't require pytest for run * stray bracket * testing * fix xborder * ext pipeline * testing * better test * test parquet store * fix loading pickled tables in mp * doc * exercise pickling in store test * est updates * test est fix * cdap for est * fix t0 on error * allow aborted steps * repair sandag test * repaid sandag * reformat tables as requested * change in cdap est targets due to change in randomness * fix env * datastore * clean up * fix test * list checkpoint names * more * dependency changes * repair merge errors and inconsistencies * blacken * ignores * change index dtype to signed int for consistency * partial fix of external tests * ruff * rm datastore * state.run.all now handles mp * fix psrc example * trip scheduling logic_version * make skim_dataset known before getting it * allow missing skims * prevent inf recursion when checkpoint is None * update external link * fix outside auto own test * fix joint freq comp * upgrade testing * repair zone coding in test * ignore parquet * validate assign on settings * allow parquet inputs * allow disable sharrow in WFH * zarr write time * nan on missing * start docs * memory optimizations * check_for_variability cannot be None * accessibility improvements * pinderella * pandas in setup.cfg deps * doc nullstore * external:prototype_psrc_in_development * some docs * doc updates [makedocs] appdirs -> platformdirs sphinx accessor autodoc * branch docs on 3.10 * manual doc build [makedocs] * document download_external_example * repair cdap for joint * external examples doc [makedocs] * so much doc [makedocs] * checkpoints * checkpoint docs [makedocs] * estimation example * ext test * require sharrow 2.6 * add zstandard to envs * fix error in unpacking * update psrc ext link * allow maz_ceiling to be any kind of int * initialize_filesystem doc * doc updates [makedocs] * more docs * documentation [makedocs] * docs on decorators [makedocs] * cache env [makedocs] * various bug fixes * chunk_method is literal * interactive run docs [makedocs] * fix link * doc steps * class no init template * api on correct pages [makedocs] * fix bug in log file names closes ActivitySim#668 * fix error in MP resume_after closes ActivitySim#669 * doc duplicate_step_execution * fix link * docs * cdap est coefficients * test logging * annotate rng * update semcog model * docs for logging * fix semcog example test * only one add_size_tables func * better error msg * remove extra lines * multiple output dirs for tests * fix for chunking * blacken * remove vestigial log_df * undo junk * docstring * write log to output dir * more docs on logging * bigger chunksize, too small times out * clean up when missing TAZ input table * historian * allow state to be None * change recode_pipeline_columns default to false * update prototype_psrc_in_development * fix cdap for sharrow * import skim_dataset * recode pipeline for mtc * Multiprocessing Performance * Arithmetic on Logical Values * clarify * repair problems with tracing * suppress excessive performance warnings addresses ActivitySim#685 * cleanup * drop core.contrast from this PR * fix link * drop "warn" option for duplicate_step_execution * add check that land_use_taz matches skims * move land_use_taz check * land_use_taz check fix
camsys · Jul 25, 2023 · 180dcca · 180dcca
1 parent d9ead70
commit 180dcca
Show file tree

Hide file tree

Showing 301 changed files with 16,864 additions and 9,157 deletions.
diff --git a/.github/workflows/branch-docs.yml b/.github/workflows/branch-docs.yml
@@ -1,13 +1,15 @@
 name: ActivitySim Branch Docs
 # This workflow is provided as a service for forks to build branch-specific documentation.
 
-on: push
+on:
+  - push
+  - workflow_dispatch
 
 jobs:
   docbuild:
     if: "contains(github.event.head_commit.message, '[makedocs]') && (github.repository_owner != 'ActivitySim') && (github.ref_name != 'develop')"
     # develop branch docs are built at the end of the core test workflow, regardless of repository owner or commit message flags
-    name: ubuntu-latest py3.9
+    name: ubuntu-latest py3.10
     runs-on: ubuntu-latest
     defaults:
       run:
@@ -16,36 +18,49 @@ jobs:
       - uses: actions/checkout@v3
         with:
           fetch-depth: 0  # get all tags, lets setuptools_scm do its thing
-      - name: Set up Python 3.9
-        uses: actions/setup-python@v2
-        with:
-          python-version: 3.9
-      - name: Install dependencies
+
+      - name: Setup Mambaforge
         uses: conda-incubator/setup-miniconda@v2
         with:
           miniforge-variant: Mambaforge
           miniforge-version: latest
           use-mamba: true
-          environment-file: conda-environments/docbuild.yml
-          python-version: 3.9
+          python-version: "3.10"
           activate-environment: docbuild
           auto-activate-base: false
           auto-update-conda: false
+
+      - name: Set cache date for year and month
+        run: echo "DATE=$(date +'%Y%m')" >> $GITHUB_ENV
+
+      - uses: actions/cache@v3
+        with:
+          path: /usr/share/miniconda3/envs/docbuild
+          key: linux-64-conda-${{ hashFiles('conda-environments/docbuild.yml') }}-${{ env.DATE }}-${{ env.CACHE_NUMBER }}
+        id: cache
+
+      - name: Update environment
+        run: mamba env update -n docbuild -f conda-environments/docbuild.yml
+        if: steps.cache.outputs.cache-hit != 'true'
+
       - name: Install activitysim
         run: |
           python -m pip install .
+
       - name: Conda checkup
         run: |
           conda info -a
           conda list
           echo REPOSITORY ${{ github.repository }}
           echo REF ${{ github.ref }}
           echo REF_NAME ${{ github.ref_name }}
+
       - name: Build the docs
         run: |
           cd docs
           make clean
           make html
+
       - name: Push to GitHub Pages
         uses: peaceiris/[email protected]
         with:

diff --git a/.github/workflows/core_tests.yml b/.github/workflows/core_tests.yml
@@ -10,14 +10,14 @@ on:
       - '*'
 
 env:
-  CACHE_NUMBER: 1  # increase to reset cache manually
+  CACHE_NUMBER: 0  # increase to reset cache manually
 
 jobs:
   foundation:
 
     strategy:
       matrix:
-        python-version: [3.9]
+        python-version: ["3.10"]
     defaults:
       run:
         shell: bash -l {0}
@@ -38,7 +38,7 @@ jobs:
       - name: Set cache date for year and month
         run: echo "DATE=$(date +'%Y%m')" >> $GITHUB_ENV
 
-      - uses: actions/cache@v2
+      - uses: actions/cache@v3
         with:
           path: /usr/share/miniconda3/envs/asim-test
           key: linux-64-conda-${{ hashFiles('conda-environments/github-actions-tests.yml') }}-${{ env.DATE }}-${{ env.CACHE_NUMBER }}
@@ -81,6 +81,10 @@ jobs:
         run: |
           python -m pytest --pyargs activitysim.cli
 
+      - name: Test activitysim.examples.test
+        run: |
+          python -m pytest --pyargs activitysim.examples.test
+
 
   cross-platform:
     # also test foundation cross platforms, but do not require a successful
@@ -92,12 +96,12 @@ jobs:
           - os: macos-latest
             label: macOS
             prefix: /Users/runner/miniconda3/envs/asim-test
-            python-version: 3.9
+            python-version: "3.10"
 
           - os: windows-latest
             label: win-64
             prefix: C:\Miniconda3\envs\asim-test
-            python-version: 3.9
+            python-version: "3.10"
 
     defaults:
       run:
@@ -120,7 +124,7 @@ jobs:
       - name: Set cache date for year and month
         run: echo "DATE=$(date +'%Y%m')" >> $GITHUB_ENV
 
-      - uses: actions/cache@v2
+      - uses: actions/cache@v3
         with:
           path: ${{ matrix.prefix }}
           key: ${{ matrix.label }}-conda-${{ hashFiles('conda-environments/github-actions-tests.yml') }}-${{ env.DATE }}-${{ env.CACHE_NUMBER }}
@@ -164,11 +168,11 @@ jobs:
           python -m pytest --pyargs activitysim.cli
 
 
-  regional_models:
+  builtin_regional_models:
     needs: foundation
     env:
       mamba-env-prefix: /usr/share/miniconda3/envs/asim-test
-      python-version: 3.9
+      python-version: "3.10"
       label: linux-64
     strategy:
       matrix:
@@ -204,7 +208,7 @@ jobs:
       - name: Set cache date for year and month
         run: echo "DATE=$(date +'%Y%m')" >> $GITHUB_ENV
 
-      - uses: actions/cache@v2
+      - uses: actions/cache@v3
         with:
           path: ${{ env.mamba-env-prefix }}
           key: ${{ env.label }}-conda-${{ hashFiles('conda-environments/github-actions-tests.yml') }}-${{ env.DATE }}-${{ env.CACHE_NUMBER }}
@@ -235,7 +239,7 @@ jobs:
       #  - name: Get a random number
       #    run: echo "RANDOM_SUFFIX=${RANDOM}${RANDOM}" >> $GITHUB_ENV
       #
-      #  - uses: actions/cache@v2
+      #  - uses: actions/cache@v3
       #    # store the regional model's cache directory in github actions cache
       #    # this will (almost) never hit on primary key due to the random number
       #    # but will pull the most recent cache from restore-keys... and then
@@ -253,11 +257,71 @@ jobs:
         run: |
           python -m pytest activitysim/examples/${{ matrix.region }}/test --durations=0
 
+  external_regional_models:
+    needs: foundation
+    env:
+      mamba-env-prefix: /usr/share/miniconda3/envs/asim-test
+      python-version: "3.10"
+      label: linux-64
+    strategy:
+      matrix:
+        region:
+          - prototype_mtc
+          - prototype_psrc_in_development
+      fail-fast: false
+    defaults:
+      run:
+        shell: bash -l {0}
+    name: ${{ matrix.region }}
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Setup Mambaforge
+        uses: conda-incubator/setup-miniconda@v2
+        with:
+          miniforge-variant: Mambaforge
+          miniforge-version: latest
+          activate-environment: asim-test
+          use-mamba: true
+          python-version: ${{ env.python-version }}
+
+      - name: Set cache date for year and month
+        run: echo "DATE=$(date +'%Y%m')" >> $GITHUB_ENV
+
+      - uses: actions/cache@v3
+        with:
+          path: |
+            ${{ env.mamba-env-prefix }}
+            ~/.cache/ActivitySim
+          key: ${{ env.label }}-conda-${{ hashFiles('conda-environments/github-actions-tests.yml') }}-${{ env.DATE }}-${{ env.CACHE_NUMBER }}
+        id: cache
+
+      - name: Update environment
+        run: mamba env update -n asim-test -f conda-environments/github-actions-tests.yml
+        if: steps.cache.outputs.cache-hit != 'true'
+
+      - name: Install activitysim
+        # installing without dependencies is faster, we trust that all needed dependencies
+        # are in the conda environment defined above.  Also, this avoids pip getting
+        # confused and reinstalling tables (pytables).
+        run: |
+          python -m pip install -e . --no-deps
+
+      - name: Conda checkup
+        run: |
+          mamba info -a
+          mamba list
+
+      - name: Test ${{ matrix.region }}
+        run: |
+          python -m activitysim test ${{ matrix.region }}
+
   random_seed_generation:
     needs: foundation
     env:
       mamba-env-prefix: /usr/share/miniconda3/envs/asim-test
-      python-version: 3.9
+      python-version: "3.10"
       label: linux-64
     defaults:
       run:
@@ -280,7 +344,7 @@ jobs:
       - name: Set cache date for year and month
         run: echo "DATE=$(date +'%Y%m')" >> $GITHUB_ENV
 
-      - uses: actions/cache@v2
+      - uses: actions/cache@v3
         with:
           path: ${{ env.mamba-env-prefix }}
           key: ${{ env.label }}-conda-${{ hashFiles('conda-environments/github-actions-tests.yml') }}-${{ env.DATE }}-${{ env.CACHE_NUMBER }}
@@ -310,7 +374,7 @@ jobs:
     needs: foundation
     env:
       mamba-env-prefix: /usr/share/miniconda3/envs/asim-test
-      python-version: 3.9
+      python-version: "3.10"
       label: linux-64
     defaults:
       run:
@@ -332,7 +396,7 @@ jobs:
       - name: Set cache date for year and month
         run: echo "DATE=$(date +'%Y%m')" >> $GITHUB_ENV
 
-      - uses: actions/cache@v2
+      - uses: actions/cache@v3
         with:
           path: ${{ env.mamba-env-prefix }}
           key: ${{ env.label }}-conda-${{ hashFiles('conda-environments/github-actions-tests.yml') }}-${{ env.DATE }}-${{ env.CACHE_NUMBER }}
@@ -343,7 +407,7 @@ jobs:
         if: steps.cache.outputs.cache-hit != 'true'
 
       - name: Install Larch
-        run: mamba install "larch>=5.5.3"
+        run: mamba install "larch>=5.7.1"
 
       - name: Install activitysim
         # installing without dependencies is faster, we trust that all needed dependencies
@@ -373,18 +437,18 @@ jobs:
       - uses: actions/checkout@v3
         with:
           fetch-depth: 0  # get all tags, lets setuptools_scm do its thing
-      - name: Set up Python 3.9
+      - name: Set up Python 3.10
         uses: actions/setup-python@v2
         with:
-          python-version: 3.9
+          python-version: "3.10"
       - name: Install dependencies
         uses: conda-incubator/setup-miniconda@v2
         with:
           miniforge-variant: Mambaforge
           miniforge-version: latest
           use-mamba: true
           environment-file: conda-environments/docbuild.yml
-          python-version: 3.9
+          python-version: "3.10"
           activate-environment: docbuild
           auto-activate-base: false
           auto-update-conda: false

diff --git a/.gitignore b/.gitignore
@@ -77,3 +77,7 @@ _test_est
 **/output/
 **/_generated_version.py
 docs/**/_generated
+activitysim/examples/prototype_mtc_extended/test/*.ipynb
+activitysim/examples/prototype_mtc/*.ipynb
+Untitled.ipynb
+docs/dev-guide/_generated2
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -1,14 +1,14 @@
 repos:
 
 - repo: https://github.com/pre-commit/pre-commit-hooks
-  rev: v4.3.0
+  rev: v4.4.0
   hooks:
   - id: end-of-file-fixer
     exclude: .*\.ipynb
   - id: trailing-whitespace
 
 - repo: https://github.com/pycqa/isort
-  rev: 5.10.1
+  rev: 5.12.0
   hooks:
   - id: isort
     args: ["--profile", "black", "--filter-files"]
@@ -18,7 +18,7 @@ repos:
   hooks:
     - id: black
 
-- repo: https://github.com/PyCQA/flake8
-  rev: 5.0.4
-  hooks:
-    - id: flake8
+#- repo: https://github.com/PyCQA/flake8
+#  rev: 5.0.4
+#  hooks:
+#    - id: flake8
diff --git a/AAA-BreakingChanges.md b/AAA-BreakingChanges.md
@@ -0,0 +1,3 @@
+
+
+- The 'run_list' key in settings.yaml is no longer supported.
diff --git a/activitysim/abm/__init__.py b/activitysim/abm/__init__.py
@@ -1,3 +1,4 @@
 # ActivitySim
 # See full license in LICENSE.txt.
-from . import misc, models, tables
+from activitysim.abm import misc, models, tables
+from activitysim.core.steps import output  # also contains workflow.step # noqa: F401