Merge branch 'release/2.30.0'

.github/workflows/ibllib_ci.yml

@@ -10,28 +10,9 @@ on:
     branches: [ master, develop ]
 
 jobs:
-  detect-outstanding-prs:  # Don't run builds for push events if associated with PR
-    runs-on: ubuntu-latest
-    env:
-      GH_TOKEN: ${{ github.token }}
-    outputs:
-      abort: ${{ steps.debounce.outputs.abort }}
-    steps:
-      - name: Debounce
-        if: github.event_name == 'push'
-        id: debounce
-        run: |
-          pr_branches=$(gh pr list --json headRefName --repo $GITHUB_REPOSITORY)
-          if [[ $(echo "$pr_branches" | jq -r --arg GITHUB_REF '.[].headRefName | select(. == $GITHUB_REF)') ]]; then
-            echo "This push is associated with a pull request. Skipping the job."
-            echo "abort=true" >> "$GITHUB_OUTPUT"
-          fi
-
   build:
     name: build (${{ matrix.python-version }}, ${{ matrix.os }})
     runs-on: ${{ matrix.os }}
-    needs: debounce
-    if: needs.debounce.outputs.abort != 'true'
     strategy:
       fail-fast: false  # Whether to stop execution of other instances
       max-parallel: 2

brainbox/behavior/dlc.py

@@ -1,6 +1,4 @@
-"""
-Set of functions to deal with dlc data
-"""
+"""Set of functions to deal with dlc data."""
 import logging
 import pandas as pd
 import warnings
@@ -48,22 +46,23 @@ def insert_idx(array, values):
 
 def likelihood_threshold(dlc, threshold=0.9):
     """
-    Set dlc points with likelihood less than threshold to nan
+    Set dlc points with likelihood less than threshold to nan.
+
+    FIXME Add unit test.
     :param dlc: dlc pqt object
     :param threshold: likelihood threshold
     :return:
     """
     features = np.unique(['_'.join(x.split('_')[:-1]) for x in dlc.keys()])
     for feat in features:
         nan_fill = dlc[f'{feat}_likelihood'] < threshold
-        dlc[f'{feat}_x'][nan_fill] = np.nan
-        dlc[f'{feat}_y'][nan_fill] = np.nan
-
+        dlc.loc[nan_fill, (f'{feat}_x', f'{feat}_y')] = np.nan
     return dlc
 
 
 def get_speed(dlc, dlc_t, camera, feature='paw_r'):
     """
+    FIXME Document and add unit test!
 
     :param dlc: dlc pqt table
     :param dlc_t: dlc time points

examples/data_release/data_release_behavior.ipynb

@@ -16,6 +16,10 @@
   },
   {
    "cell_type": "markdown",
+   "id": "dd157e91",
+   "metadata": {
+    "collapsed": false
+   },
    "source": [
     "## Overview of the Data\n",
     "We have released behavioral data throughout learning from our standardized training pipeline, implemented across 9 labs in 7 institutions. Users can download behavioral data from mice throughout their training, and analyse the transition from novice to expert behavior unfold. The behavioral data is associated with 198 mice up until 2020-03-23, as used in [The International Brain Laboratory et al. 2020](https://elifesciences.org/articles/63711). This dataset contains notably information on the sensory stimuli presented to the mouse, as well as mouse decisions and response times.\n",
@@ -35,10 +39,7 @@
     "Note:\n",
     "\n",
     "* The tag associated to this release is `2021_Q1_IBL_et_al_Behaviour`"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
+   ]
   }
  ],
  "metadata": {
@@ -58,9 +59,9 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.7"
+   "version": "3.11.6"
   }
  },
  "nbformat": 4,
  "nbformat_minor": 5
-}
+}

examples/data_release/data_release_brainwidemap.ipynb

@@ -16,6 +16,10 @@
   },
   {
    "cell_type": "markdown",
+   "id": "e9be5894",
+   "metadata": {
+    "collapsed": false
+   },
    "source": [
     "## Overview of the Data\n",
     "We have released data from 459 Neuropixel recording sessions, which encompass 699 probe insertions, obtained in 139 subjects performing the IBL task across 12 different laboratories. As output of spike-sorting, there are 376730 units; of which 45085 are considered to be of good quality. In total, 138 brain regions were recorded in sufficient numbers for inclusion in IBL’s analyses [(IBL et al. 2023)](https://www.biorxiv.org/content/10.1101/2023.07.04.547681v2).\n",
@@ -33,12 +37,18 @@
     "\n",
     "* The tag associated to this release is `Brainwidemap`\n",
     "\n",
-    "## Receive updates on the data\n",
-    "To receive a notification that we released new datasets, please fill up [this form](https://forms.gle/9ex2vL1JwV4QXnf98)\n"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
+    "\n",
+    "## Updates on the data\n",
+    "Note: The section [Overview of the Data](#overview-of-the-data) contains the latest numbers released.\n",
+    "\n",
+    "### Receive updates on the data\n",
+    "To receive a notification that we released new datasets, please fill up [this form](https://forms.gle/9ex2vL1JwV4QXnf98)\n",
+    "\n",
+    "### 15 February 2024\n",
+    "We have added data from an additional 105 recording sessions, which encompass 152 probe insertions, obtained in 24 subjects performing the IBL task. As output of spike-sorting, there are 81229 new units; of which 12319 are considered to be of good quality.\n",
+    "\n",
+    "We have also replaced and removed some video data. Application of additional quality control processes revealed that the video timestamps for some of the previously released data were incorrect. We corrected the video timestamps where possible (285 videos: 137 left, 148 right) and removed video data for which the correction was not possible (139 videos: 135 body, 3 left, 1 right). We also added 31 videos (16 left, 15 right). **We strongly recommend that all data analyses using video data be rerun with the data currently in the database (please be sure to clear your cache directory first).**"
+   ]
   }
  ],
  "metadata": {
@@ -58,7 +68,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.7"
+   "version": "3.11.6"
   }
  },
  "nbformat": 4,

examples/data_release/data_release_repro_ephys.ipynb

@@ -57,9 +57,9 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.7"
+   "version": "3.11.6"
   }
  },
  "nbformat": 4,
  "nbformat_minor": 5
-}
+}

examples/data_release/data_release_spikesorting_benchmarks.ipynb

@@ -89,9 +89,9 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.7"
+   "version": "3.11.6"
   }
  },
  "nbformat": 4,
  "nbformat_minor": 5
-}
+}