Merge pull request OSOceanAcoustics#277 from brandynlucca/docs_echopr…

…o_vs_echopop Documentation for differences between `EchoPro` and `Echopop`
brandynlucca · Oct 17, 2024 · d5677cd · d5677cd
2 parents a92a9ee + 42fdb2d
commit d5677cd
Show file tree

Hide file tree

Showing 16 changed files with 272 additions and 4 deletions.
diff --git a/docs/_config.yml b/docs/_config.yml
@@ -12,6 +12,7 @@ execute:
 
 parse:
   myst_enable_extensions:
+  - amsmath
   - colon_fence
   - dollarmath
   - linkify
@@ -46,7 +47,7 @@ sphinx:
   extra_extensions: [
     'sphinx_automodapi.automodapi',
     'numpydoc',
-    # 'sphinx.ext.autodoc',
+    'sphinx.ext.autodoc',
     'sphinxcontrib.bibtex',
     'sphinx.ext.intersphinx',
     'sphinx.ext.mathjax',

diff --git a/docs/_toc.yml b/docs/_toc.yml
@@ -9,6 +9,7 @@ parts:
   - file: install
   - file: input_files
   - file: core_data_structure
+  - file: echopro_vs_echopop
   - file: example_notebooks
     title: Example workflows and analyses
     sections:
@@ -19,8 +20,9 @@ parts:
     - file: theory/acoustics
     - file: theory/bio_estimates
     - file: theory/stratification
-    # - file: theory/kriging
-    - file: theory/apportion
+    # - file: theory/kriging    
+    - file: theory/apportion_biomass
+    - file: theory/apportion_abundance
     - file: theory/kriging_eq
     - file: theory/semivariogram_eq
     # - file: theory/other

diff --git a/docs/echopro_vs_echopop.md b/docs/echopro_vs_echopop.md
@@ -0,0 +1,36 @@
+(echopro-vs-echopop)=
+# Differences between the `EchoPro` and `Echopop`
+
+There are documented differences between the `EchoPro` MATLAB program and `Echopop` workflows. Changes include new features (or changes to methods in `EchoPro`) not previously implemented in `Echopop` (<span style="font-size:5mm;">✨</span>), bugfixes (<span style="font-size:5mm;">🐛</span>), language-specific differences between MATLAB and Python (<span style="font-size:5mm;">👽</span>), features `EchoPro` that have not yet been implemented in `Echopop` (<span style="font-size:5mm;">🧩</span>), and features in `EchoPro` that are not currently in development for `Echopop` (<span style="font-size:5mm;">📕</span>).
+
+## Data ingestion
+- <span style="font-size:6mm;">[✨]</span> The creation of external `*.xlsx` transect-region-haul key files from Echoview `*.xlsx` exports are no longer required
+- <span style="font-size:6mm;">[🧩]</span> Manually mapping transect numbers, regions, and haul numbers together from Echoview `*.xlsx` exports is currently not implemented
+
+## Transect analysis
+- <span style="font-size:6mm;">[🐛]</span> Age bins can now be manually defined in the configuration YAML file and do not automatically subsume out-of-bounds values (e.g. $\alpha >= 21$ years all being included within $\alpha = 20$)
+- <span style="font-size:6mm;">[🐛]</span> Sex-specific TS-length regressions no longer arbitrarily redefine 'unsexed' fish as being 'female'
+- <span style="font-size:6mm;">[👽]</span> Interpolated weights across $\ell$ can produce different results in `EchoPro` and `Echopop` due to differences in numerical precision and how missing data are handled
+- <span style="font-size:6mm;">[🧩]</span> Incorporating net selectivity into biological distributions (across $\ell$ and $\alpha$) is currently not implemented
+- <span style="font-size:6mm;">[✨/🧩]</span> Filtering out specific haul numbers, ages, sexes, and other plausible contrasts have not yet been implemented
+- <span style="font-size:6mm;">[📕]</span> Support for incorporating observer data within `Echopop` is currently not supported
+
+## Stratified analysis
+- <span style="font-size:6mm;">[✨]</span> Confidence intervals at user-defined significance levels are now provided for CV. This includes bootstrap/resampling bias calculations and several methods for computing the confidence intervals: empirical, percentile, standard, $t$-standard, $t$-jackknife, bias-corrected (BC), and bias-corrected and accelerated (BCa)
+- <span style="font-size:6mm;">[✨]</span> Estimators and corresponding confidence intervals are now provided for (resampled) mean abundance/biomass
+- <span style="font-size:6mm;">[🧩]</span> The Jolly and Hampton (1990) resampling approach for computing overall variability in $\textit{NASC}$ estimates has not been implemented in `Echopop`. It also currently limited to being computed only over INPFC strata
+
+## General spatial methods
+- <span style="font-size:6mm;">[✨]</span> A new alternative method for cropping the kriging mesh has been implemented based on the convex hull shape of the survey transects  (i.e. `crop_method="convex_hull"`)
+- <span style="font-size:6mm;">[✨/🐛]</span> The implementation of the `EchoPro` kriged mesh cropping method relied on interpolating the transect line extents of three discrete regions to account for the island of Haida Gwaii. In `EchoPro`, these are manually defined. `Echopop` defaults to this implementation (i.e. `crop_method="transect_ends"`) by discretizing the transect lines into each of these three regions based on their respective headings in cardinal directions 
+- <span style="font-size:6mm;">[🐛]</span> Modifications were made to prevent erroneous gaps in the survey coverage shape that subsequently included unexpected mesh nodes
+
+
+## Variogram analysis
+- <span style="font-size:6mm;">[✨]</span> The semivariogram fitting GUI (i.e. `Survey.variogram_gui()`) has been enabled to allow for user changes to optimization algorithm parameters (e.g. `gradient_tolerance`)
+- <span style="font-size:6mm;">[👽]</span> Differences between the non-linear least squares optimization algorithm in MATLAB and Python (via `lmfit`) differs in a number of ways that can produce different parameter estimates. These differences can range from precision error to much larger; however, the relative importance placed on parameter order remains the same in `Echopop` as it does in `EchoPro`
+
+## Kriging analysis
+- <span style="font-size:6mm;">[✨]</span> Kriged unaged biomass estimates are reapportioned along length and age based on the distributions computed from aged fish. When certain length-bins are missing, `EchoPro` imputes values using the closest length bin corresponding to either male or female fish. `Echopop` does this imputation using the closest sex-specific length bins instead
+- <span style="font-size:6mm;">[🧩]</span> `Echopop` currently does not support kriged abundance or $\textit{NASC}$ back-calculation from biomass estimates
+- <span style="font-size:6mm;">[🧩]</span> Kriging has only been fully tested, validated, and implemented for biomass
diff --git a/docs/images/abundance_to_nasc_conversion.jpg b/docs/images/abundance_to_nasc_conversion.jpg
diff --git a/docs/images/aged_abundance_apportionment.jpg b/docs/images/aged_abundance_apportionment.jpg
diff --git a/docs/images/aged_aged_sex_length_age_number_proportion.jpg b/docs/images/aged_aged_sex_length_age_number_proportion.jpg
diff --git a/docs/images/aged_all_sex_number_proportion.jpg b/docs/images/aged_all_sex_number_proportion.jpg
diff --git a/docs/images/aged_and_unaged_apportioned_sexed_abundance.jpg b/docs/images/aged_and_unaged_apportioned_sexed_abundance.jpg
diff --git a/docs/images/biomass_to_abundance_conversion.jpg b/docs/images/biomass_to_abundance_conversion.jpg
diff --git a/docs/images/initial_biomass_summation.jpg b/docs/images/initial_biomass_summation.jpg
diff --git a/docs/images/total_apportioned_abundance.jpg b/docs/images/total_apportioned_abundance.jpg
diff --git a/docs/images/unaged_abundance_apportionment.jpg b/docs/images/unaged_abundance_apportionment.jpg
diff --git a/docs/images/unaged_all_sex_number_proportion.jpg b/docs/images/unaged_all_sex_number_proportion.jpg
diff --git a/docs/images/unaged_unaged_sex_length_number_proportion.jpg b/docs/images/unaged_unaged_sex_length_number_proportion.jpg
diff --git a/docs/theory/apportion_abundance.md b/docs/theory/apportion_abundance.md
@@ -0,0 +1,229 @@
+(apportion-abundance)=
+# Back-calculating abundances from kriged biomass estimates
+
+```{attention} 
+`Echopop` currently does support kriged abundance back-calculation from biomass estimates. Refer to the <b>[kriged biomass apportionment](apportion_biomass.md)</b> for more information on how `Echopop` incorporates kriging into population estimates.
+```
+
+```{note}
+It is worth noting that all calculations are done for each stratum, $i$. Refer to the <b>[stratification](stratification.md)</b> documentation for more information.
+```
+
+Biomass estimates for each $s$ ($\textrm{M}$ and $\textrm{F}$) along transect interval $k$ are summed across $\ell$ and $\alpha$ via:
+
+$$ 
+B_{\textrm{M}}^{k} = \sum_{\textrm{M}, \ell, \alpha} B_{\textrm{M}, \ell, \alpha}^{k, \textrm{aged}} +  \sum_{\textrm{M}, \ell, \alpha} B_{\textrm{M}, \ell, \alpha}^{k, \textrm{unaged}}
+\label{eq:biomass_M} \tag{1}
+$$
+
+$$
+B_{\textrm{F}}^{k} = \sum_{\textrm{F}, \ell, \alpha} B_{\textrm{F}, \ell, \alpha}^{k, \textrm{aged}} + \sum_{\textrm{F}, \ell, \alpha} B_{\textrm{F}, \ell, \alpha}^{k, \textrm{unaged}}
+\label{eq:biomass_F} \tag{2}
+$$
+
+Similarly, biomass estimates for all fish ($B^{k}$), which is inclusive of both sexed and unsexed fish, are also summed (see: <b>[kriged biomass summation for more details](apportion_biomass.md#total-biomass-apportioned-with-length-and-age)</b>).
+
+
+These kriged biomass estimates are then converted to sexed ($\hat{N}_{s}^{k}$) and total ($\hat{N}^{k}$) abundance by using averaged length-weight regression output ($\overline{W}(\ell)$). Consequently, $\overline{W}(\ell)$ can be defined either by using the average length-weight relationship produced or parameterizing $\overline{W}(\ell)$ with the mean length ($\bar{\ell}$). It is important to note, however, that both $\hat{N}_{s}^{k}$ and $\hat{N}^{k}$ are calculated using $\overline{W}(\ell)$ fit from <b>all</b> individuals (i.e. male, female, and unsexed).
+
+$$
+\hat{N}^{k} = \frac{B^{k}}{\overline{W}(\ell)}
+\label{eq:abundance} \tag{3}
+$$
+
+```{note} 
+With $\hat{N}_{\textrm{All}}^{k}$ calculated, $\hat{\textit{NASC}^{k}}$ can then be back-calculated by using the averaged $i^{\text{th}}$ differential backscattering cross-section ($\bar{\sigma}_{\textrm{bs}}$):
+$
+\hat{\textit{NASC}^{k}} = \hat{N}^{k} \bar{\sigma}_{\textrm{bs}}
+$
+```
+
+## Apportioning the back-calculated abundance estimates
+
+### Summing fish counts
+
+The back-calculated $\hat{N}^{k}$ $\eqref{eq:abundance}$ is subsequently apportioned similarly to the [<b>weight proportions</b>](apportion_biomass.md#unaged-biomass-apportioned-with-sex-length-and-age) across sex, length, and age. 
+
+#### Unaged fish
+
+This process is first done across $\ell$ for unaged fish, and both $\ell$ *and* $\alpha$ for aged fish. First, the total number counts for unaged fish ($n_{s,\ell}^{\textrm{unaged}}$):
+
+$$
+\begin{equation}
+\begin{aligned}
+    n_{\textrm{M},\ell}^{\textrm{unaged}} &= \sum_{j \in J_{\textrm{M},\ell}^{\textrm{unaged}}}n_j \nonumber \\
+    n_{\textrm{F},\ell}^{\textrm{unaged}} &= \sum_{j \in J_{\textrm{F},\ell}^{\textrm{unaged}}}n_j \nonumber
+\end{aligned}
+\label{eq:total_unaged_sex_length} \tag{4}
+\end{equation}
+$$
+
+These are then summed across all fish of $s$:
+
+$$
+\begin{equation}
+\begin{aligned}
+    n_{\textrm{M}}^{\textrm{unaged}} &= \sum_{\textrm{M},\ell}n_{\textrm{M},\ell}^{\textrm{unaged}} \nonumber \\
+    n_{\textrm{F}}^{\textrm{unaged}} &= \sum_{\textrm{F}, \ell}n_{\textrm{F},\ell}^{\textrm{unaged}} \nonumber
+\end{aligned}
+\label{eq:total_unaged_sex} \tag{5}
+\end{equation}
+$$
+
+#### Aged fish
+
+The total counts for aged fish across both $\ell$ and $\alpha$ are similarly calculated via:
+
+$$
+\begin{equation}
+\begin{aligned}
+    n_{\textrm{M},\ell,\alpha}^{\textrm{aged}} &= \sum_{j \in J_{\textrm{M},\ell,\alpha}^{\textrm{aged}}}n_j \nonumber \\
+    n_{\textrm{F},\ell,\alpha}^{\textrm{aged}} &= \sum_{j \in J_{\textrm{F},\ell,\alpha}^{\textrm{aged}}}n_j \nonumber
+\end{aligned}
+\label{eq:total_aged_sex_length_age} \tag{6}
+\end{equation}
+$$
+
+These are then summed across all fish of $s$:
+
+$$
+\begin{equation}
+\begin{aligned}
+    n_{\textrm{M}}^{\textrm{aged}} &= \sum_{\textrm{M},\ell,\alpha}n_{\textrm{M},\ell,\alpha}^{\textrm{aged}} \nonumber \\
+    n_{\textrm{F}}^{\textrm{aged}} &= \sum_{\textrm{F},\ell,\alpha}n_{\textrm{F},\ell,\alpha}^{\textrm{aged}} \nonumber \\    
+\end{aligned}
+\label{eq:total_aged_sex} \tag{7}
+\end{equation}
+$$
+
+### Number proportions
+
+The sex-specific abundances for unaged $\eqref{eq:total_unaged_sex}$ and aged $\eqref{eq:total_aged_sex}$ fish are then summed together to calculate the total unaged ($n^{\textrm{unaged}}$), aged ($n^{\textrm{aged}}$), and all ($n$) fish:
+
+$$
+\begin{equation}
+\begin{aligned}
+    n^{\textrm{unaged}} &= n_{\textrm{M}}^{\textrm{unaged}} + n_{\textrm{F}}^{\textrm{unaged}} \nonumber \\
+    n^{\textrm{aged}} &= n_{\textrm{M}}^{\textrm{aged}} + n_{\textrm{F}}^{\textrm{aged}} \nonumber \\
+    n &= n^{\textrm{unaged}} + n^{\textrm{aged}} \nonumber
+\end{aligned}
+\label{eq:total_counts} \tag{8}
+\end{equation}
+$$
+
+#### Unaged fish
+
+The number counts of unaged fish across $\ell$ for each $s$ $\eqref{eq:total_unaged_sex_length}$ relative to the sex-specific totals $\eqref{eq:total_unaged_sex}$, $r_{n,s,\ell}^{\textrm{unaged/unaged}}$, are:
+
+$$
+\begin{equation}
+\begin{aligned}
+    r_{n,~\textrm{M},\ell}^{\textrm{unaged/unaged}} &= \frac{n_{\textrm{M},\ell}^{\textrm{unaged}}}{n_{\textrm{M}}^{\textrm{unaged}}} \nonumber \\
+    r_{n,~\textrm{F},\ell}^{\textrm{unaged/unaged}} &= \frac{n_{\textrm{F},\ell}^{\textrm{unaged}}}{n_{\textrm{F}}^{\textrm{unaged}}} \nonumber    
+\end{aligned}
+\label{eq:number_proportions_unaged_sex_length} \tag{9}
+\end{equation}
+$$
+
+In a similar manner, the unaged fish number counts relative to the sum of unaged and aged number counts $\eqref{eq:total_counts}$, $r_{n,s,\ell}^{\textrm{unaged/all}}$, are then calculated via:
+
+$$
+\begin{equation}
+\begin{aligned}
+    r_{n,~\textrm{M},\ell}^{\textrm{unaged/all}} = \frac{n_{\textrm{M},\ell}^{\textrm{unaged}}}{n} \nonumber \\
+    r_{n,~\textrm{F},\ell}^{\textrm{unaged/all}} = \frac{n_{\textrm{F},\ell}^{\textrm{unaged}}}{n} \nonumber
+\end{aligned}
+\label{eq:number_proportions_unaged_sex} \tag{10}
+\end{equation}
+$$
+
+The number proportions referencing unaged $\eqref{eq:number_proportions_unaged_sex_length}$ and all $\eqref{eq:number_proportions_unaged_sex}$ fish are then combined to calculate the overall sex-specific number proportions:
+
+$$
+\begin{equation}
+\begin{aligned}
+    r_{n,~\textrm{M}}^{\textrm{unaged}} &= r_{n,~\textrm{M},\ell}^{\textrm{unaged/unaged}} r_{n,~\textrm{M},\ell}^{\textrm{unaged/all}} \nonumber \\
+    r_{n,~\textrm{F}}^{\textrm{unaged}} &= r_{n,~\textrm{F},\ell}^{\textrm{unaged/unaged}} r_{n,~\textrm{F},\ell}^{\textrm{unaged/all}} \nonumber
+\end{aligned}
+\label{eq:number_proportions_unaged} \tag{11}
+\end{equation}
+$$
+
+#### Aged fish
+
+Similar to unaged fish, the number counts of aged fish across $\ell$ and $\alpha$ for each $s$ $\eqref{eq:total_aged_sex_length_age}$ relative to the sex-specific totals $\eqref{eq:total_aged_sex}$, $r_{n,s,\ell,\alpha}^{\textrm{aged/aged}}$, are:
+
+$$
+\begin{equation}
+\begin{aligned}
+    r_{n,~\textrm{M},\ell,\alpha}^{\textrm{aged/aged}} &= \frac{n_{\textrm{M},\ell,\alpha}^{\textrm{aged}}}{n_{\textrm{M}}^{\textrm{aged}}} \nonumber \\
+    r_{n,~\textrm{F},\ell,\alpha}^{\textrm{aged/aged}} &= \frac{n_{\textrm{F},\ell,\alpha}^{\textrm{aged}}}{n_{\textrm{F}}^{\textrm{aged}}} \nonumber
+\end{aligned}
+\label{eq:number_proportions_aged_sex_length_age} \tag{12}
+\end{equation}
+$$
+
+In a similar manner, the unaged fish number counts relative to the sum of unaged and aged number counts $\eqref{eq:total_counts}$, $r_{n,s,\ell,\alpha}^{\textrm{aged/all}}$, are then calculated via:
+
+$$
+\begin{equation}
+\begin{aligned}
+    r_{n,~\textrm{M},\ell,\alpha}^{\textrm{aged/all}} = \frac{n_{\textrm{M},\ell,\alpha}^{\textrm{aged}}}{n} \nonumber \\
+    r_{n,~\textrm{F},\ell,\alpha}^{\textrm{aged/all}} = \frac{n_{\textrm{F},\ell,\alpha}^{\textrm{aged}}}{n} \nonumber
+\end{aligned}
+\label{eq:number_proportions_aged_sex} \tag{13}
+\end{equation}
+$$
+
+The number proportions referencing unaged $\eqref{eq:number_proportions_aged_sex_length_age}$ and all $\eqref{eq:number_proportions_aged_sex}$ fish are then combined to calculate the overall sex-specific number proportions:
+
+$$
+\begin{equation}
+\begin{aligned}
+    r_{n,~\textrm{M}}^{\textrm{aged}} &= r_{n,~\textrm{M},\ell,\alpha}^{\textrm{aged/aged}} r_{n,~\textrm{M},\ell,\alpha}^{\textrm{aged/all}} \nonumber \\
+    r_{n,~\textrm{F}}^{\textrm{aged}} &= r_{n,~\textrm{F},\ell,\alpha}^{\textrm{aged/aged}} r_{n,~\textrm{F},\ell,\alpha}^{\textrm{aged/all}} \nonumber
+\end{aligned}
+\label{eq:number_proportions_aged} \tag{14}
+\end{equation}
+$$
+
+### Apportioning abundances
+
+#### Unaged fish
+
+Total unaged fish abundance estimates for $k$ $\eqref{eq:abundance}$ are then apportioned for each $s$ across $\ell$, $\hat{N}_{s,\ell}^{k, \textrm{unaged}}$ using the computed number proportions $\eqref{eq:number_proportions_unaged_sex_length}$. The sexed estimates are then summed to compute the total unaged fish abundance estimates, $\hat{N}_{\ell}^{k, \textrm{unaged}}$:
+
+$$
+\begin{equation}
+\begin{aligned}
+    \hat{N}_{\textrm{M},\ell}^{k, \textrm{unaged}} &= \hat{N}^{k} r_{n,~\textrm{M}}^{\textrm{unaged}} \nonumber \\
+    \hat{N}_{\textrm{F},\ell}^{k, \textrm{unaged}} &= \hat{N}^{k} r_{n,~\textrm{F}}^{\textrm{unaged}} \nonumber \\
+    \hat{N}_{\ell}^{k, \textrm{unaged}} &= \hat{N}_{\textrm{M},\ell}^{k, \textrm{unaged}} + \hat{N}_{\textrm{F},\ell}^{k, \textrm{unaged}} \nonumber
+\end{aligned}
+\label{eq:abundance_unaged} \tag{15}
+\end{equation}
+$$
+
+#### Aged fish
+
+Total unaged fish abundance estimates for $k$ $\eqref{eq:abundance}$ are then apportioned for each $s$ across $\ell$ and $\alpha$, $\hat{N}_{s,\ell,\alpha}^{k, \textrm{aged}}$ using the computed number proportions $\eqref{eq:number_proportions_aged_sex_length_age}$. The sexed estimates are then summed to compute the total unaged fish abundance estimates, $\hat{N}_{\ell,\alpha}^{k, \textrm{aged}}$:
+
+$$
+\begin{equation}
+\begin{aligned}
+    \hat{N}_{\textrm{M},\ell,\alpha}^{k, \textrm{aged}} &= \hat{N}^{k} r_{n,~\textrm{M}}^{\textrm{aged}} \nonumber \\
+    \hat{N}_{\textrm{F},\ell,\alpha}^{k, \textrm{aged}} &= \hat{N}^{k} r_{n,~\textrm{F}}^{\textrm{aged}} \nonumber \\
+    \hat{N}_{\ell,\alpha}^{k, \textrm{aged}} &= \hat{N}_{\textrm{M},\ell,\alpha}^{k, \textrm{aged}} + \hat{N}_{\textrm{F},\ell,\alpha}^{k, \textrm{aged}} \nonumber
+\end{aligned}
+\label{eq:abundance_aged} \tag{16}
+\end{equation}
+$$
+
+#### Combining unaged and aged estimates
+
+Lastly, unaged $\eqref{eq:abundance_unaged}$ and aged $\eqref{eq:abundance_aged}$ abundance estimates can be consolidated to apportion the total abundances across $\ell$ irrespective of $\alpha$:
+
+$$
+\hat{N}_{\ell}^{k,i} = \hat{N}_{\ell}^{k, \textrm{unaged}} + \sum_{\alpha} \hat{N}_{\ell,\alpha}^{k, \textrm{aged}}
+\label{eq:abundance_length} \tag{17}
+$$
diff --git a/docs/theory/apportion.md → docs/theory/apportion_biomass.md b/docs/theory/apportion.md → docs/theory/apportion_biomass.md
@@ -1,4 +1,4 @@
-(apportion)=
+(apportion-biomass)=
 # Apportioning kriged biomass density
 
 The challenges associated with apportioning the kriged biomass is to properly combine the various pieces of information to distribute the biomass into different sex, length, and age groups. This is because fish samples obtained from a haul (trawl) are processed at two different stations that report different biometric data: