-
Notifications
You must be signed in to change notification settings - Fork 8
/
Copy pathindex.Rmd
280 lines (203 loc) · 21.2 KB
/
index.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
---
knit: "bookdown::render_book"
title: "Open Soil Spectral Library"
author: "Woodwell Climate Research Center, University of Florida, OpenGeoHub and contributors"
description: "Open Soil Spectroscopy Library technical documentation. Use these documentation to access OSSL data, fit models, and use the reference models."
url: 'https\://soilspectroscopy.github.io/ossl-manual'
bibliography: ./tex/refs.bib
csl: ./tex/apa.csl
documentclass: svmono
classoption: graybox,natbib,nospthms
geometry: "paperwidth=18.90cm, paperheight=24.58cm, top=2.1cm, bottom=2.1cm, inner=2cm, outer=2cm"
monofont: "Source Code Pro"
monofontoptions: "Scale=0.7"
biblio-style: spbasic
fig_caption: yes
link-citations: yes
gihub-repo: soilspectroscopy/ossl-manual
twitter-handle: soilspec
cover-image: cover.png
site: bookdown::bookdown_site
---
# About {.unnumbered}
[{.cover width="250"}](https://soilspectroscopy.org/)
[](https://doi.org/10.5281/zenodo.5759693)
> "Man’s most human characteristic is not his ability to learn, which he shares
> with many other species, but his ability to teach and store what others have
> developed and taught him."
> Margaret Mead, Culture and Commitment: The New Relationships Between the Generations in the 1970s
## Soil Spectroscopy for Global Good {.unnumbered}
[**SoilSpec4GG**](https://soilspectroscopy.org/) is a USDA-funded [Food and Agriculture Cyberinformatics
Tools Coordinated Innovation Network NIFA Award #2020-67021-32467](https://nifa.usda.gov/press-release/nifa-invests-over-7-million-big-data-artificial-intelligence-and-other) project. It brings together soil
scientists, spectroscopists, informaticians, data scientists and
software engineers to overcome some of the current bottlenecks
preventing wider and more efficient use of soil spectroscopy. A series
of working groups will be formed to address topics including calibration
transfer, model choice, outreach & demonstration, and use of
spectroscopy to inform global carbon cycle modeling. For more info refer
to: <https://soilspectroscopy.org/>.
<img src="img/soilspec4gg-logo_fc.png" width="60%" />
R tutorials and software developed to implement OSSL is available via: <https://github.com/soilspectroscopy>.
Soil spectroscopy for global good project works with other global initiatives including the **[FAO Global Soil Partnership](https://www.fao.org/global-soil-partnership/glosolan/en/)** and the **[IEEE P4005 Standards and Protocols for Soil Spectroscopy](https://sagroups.ieee.org/4005/)** Working Group.
## What is soil spectroscopy? {.unnumbered}
Soil spectroscopy is the measurement of light absorption when light in the [visible, near infrared or mid infrared (Vis–NIR–MIR) regions of the electromagnetic spectrum](https://en.wikipedia.org/wiki/Electromagnetic_spectrum) is applied to a soil surface. The proportion of the incident radiation reflected by soil is sensed through Vis–NIR–MIR reflectance spectroscopy. These characteristic spectra (see Fig. below) can then be used to estimate numerous soil attributes including: particle size distribution, mineral and organic compounds, and water.
```{r soil-pnts, echo=FALSE, fig.cap="Schematic explanation of the soil spectroscopy. For more info see: https://soilspectroscopy.org/.", out.width="100%"}
knitr::include_graphics("img/Spectroscopy-Schematic.jpeg")
```
```{r soil-visnir, echo=FALSE, fig.cap="Example of spectral signatures for large number of VisNIR scans (KSSL).", out.width="80%"}
knitr::include_graphics("img/VNIR_Raw_Spectra.png")
```
## Open Soil Spectral Library {.unnumbered}
[](https://doi.org/10.5281/zenodo.5759693)
The **Open Soil Spectral Library** (OSSL) is a suite of datasets, web-services, software, and tutorials. It includes (see also <https://github.com/soilspectroscopy>):
- A soil spectral database shared through several formats (mongoDB, cloud bucket, and API);
- API with database access and a prediction service available from [https://api.soilspectroscopy.org](https://api.soilspectroscopy.org/__docs__/#/);
- Front-end solutions: **[OSSL Engine](https://engine.soilspectroscopy.org/)** and **[OSSL Explorer](https://explorer.soilspectroscopy.org/)**;
- Registry of global and local prediction models (<https://github.com/soilspectroscopy/>);
- Tutorials included in this book.
**MIR spectral diversity**
<img src="https://github.com/soilspectroscopy/ossl-imports/blob/main/img/plot_pca_scores_mir_ossl.png?raw=true" heigth=100% width=100%>
**MIR locations**
<img src="https://github.com/soilspectroscopy/ossl-imports/blob/main/img/mir.pnts_sites.png?raw=true" heigth=100% width=100%>
**VisNIR locations**
<img src="https://github.com/soilspectroscopy/ossl-imports/blob/main/img/visnir.pnts_sites.png?raw=true" heigth=100% width=100%>
## Importing new datasets to OSSL {.unnumbered}
The OSSL database has been prepared following the OSSL schema which is available at [ossl-imports](https://github.com/soilspectroscopy/ossl-imports).
The github repository [ossl-imports](https://github.com/soilspectroscopy/ossl-imports) has all the importing codes for preparing and integrating new datasets into the OSSL. The folder [ossl-imports/dataset](https://github.com/soilspectroscopy/ossl-imports/tree/main/dataset) contains all the datasets separated by its name/code, with an internal `README.Rmd` file describing in detail the steps necessary for preparing and standardizing the files into the OSSL format.
The original files are placed on an internal server to avoid the storage of big files in the github repository, but some of them can be found on persistent online repositories if they have a public license. Other original datasets that do not have a public license or are shared to this project with some restrictions will no be shared publicly. As we keep the original files in a local repository to run the importing notebooks (each `ossl-imports/dataset/../README.Rmd`), this operation can only be run at our internal server.
New contributors are encouraged to host their files on a public repository and draft the importing step on github through pull requests. The final checks and consolidation, however, will be concluded on the internal server.
The `README.Rmd` files follow a basic structure. Each one has a description of the dataset at the top of the notebook which links to the dataset source and references. An extended description is provided in [this book](https://soilspectroscopy.github.io/ossl-manual/soil-spectroscopy-tools-and-users.html#registered-soil-spectral-libraries).
For preparing the import, the following subsections are defined:
- Basic description.
- Soil site information.
- Soil laboratory (wet chemistry) data
- Mid-infrared (MIR) spectroscopy data (optional).
- Visible-near-infrared (VisNIR) spectroscopy data (optional).
- Quality control.
- References.
Each subsection has its own reading and processing functions, and the outputs can be joined using shared id columns. The output files share the same name and pattern but can be retrieved with different folder names, which are binded together in a final stage. The outputs are named as `ossl_soilsite_<version>.qs`, `ossl_soillab_<version>.qs`, `ossl_mir_<version>.qs`, `ossl_visnir_<version>.qs`. The version number is adjusted for recurring updates.
## Soil properties of interest {.unnumbered}
The contrasting methods used for analytically determining (wet chemistry) a given soil property has been a subject of internal discussion in this project. Some global initiatives have been facing this same issue in their soil databases but there still no clear or full consensus on how to harmonize those different methods. This has been a topic of great discussion and research development at the [Global Soil Partnership’s Global Soil Laboratory Network (GLOSOLAN)](https://www.fao.org/global-soil-partnership/glosolan/en/).
In order to maximize transparency, for now, we have decided to produce two different levels for the OSSL database. `Level 0` takes into account the original methods employed in each dataset but tries to initially fit them to two reference lists: [KSSL Guidance – Laboratory Methods and Manuals](https://www.nrcs.usda.gov/resources/guides-and-instructions/kssl-guidance) and ISO standards. A copy of the KSSL procedures and coding scheme is archived in [ossl-imports](https://github.com/soilspectroscopy/ossl-imports/blob/main/out/kssl_procedures.csv).
If a reference method does not fall in any previous method, then we create a new variable sharing at least a common property and unit. A final harmonization takes place in the OSSL `Level 1`, where those common properties sharing different methods are converted to a target method using some publicly available transformation rule, or in the worst scenario, they are naively binded or kept separated to produce its specific model. All the implementations are documented in the [ossl-import/ossl_level0_to_level1_soillab_harmonization.csv](https://github.com/soilspectroscopy/ossl-imports/blob/main/out/ossl_level0_to_level1_soillab_harmonization.csv) repository.
In addition, [GLOSOLAN's Standard Operating Procedures (SOPs)](http://www.fao.org/global-soil-partnership/glosolan/soil-analysis) list four groups of soil variables of interest to international soil spectroscopy projects:
Soil chemical variables:
- pH,
- Carbon,
- Phosphorous,
- Potassium,
- Nitrogen,
- Exchangeable cations and CEC,
- Extractable microelements,
- Trace and major element analyses,
- Gypsum,
- Electrical conductivity and total soluble salt content,
- Soluble sulfate and chloride analysis,
- Special analysis for peats, mineral and organic soils, agriculture and forest,
Soil physical variables:
- Bulk density,
- Coarse fragments,
- Particle-size distribution,
- Water retention curve,
- Porosity,
- Hydraulic conductivity function,
- Aggregate stability,
- Moisture content,
Soil biological variables:
- Microbial biomass,
- Soil Respiration,
- Enzyme activity,
- Microbial identification,
Soil contaminants:
- Heavy metal elements: As, Hg, Cu, Cd, Pb and similar,
- Other soil pollutants,
## Contributing data {.unnumbered}
We encourage public and private entities to help this project and share SSL data. The following four modes of data sharing are especially encouraged:
1. Publish your data open acces by releasing it under a Creative Commons license ([CC-BY](https://creativecommons.org/licenses/by/4.0/), [CC-BY-SA](https://creativecommons.org/licenses/by-sa/4.0/))
or the Open Data Commons Open Database License ([ODbL](https://opendatacommons.org/licenses/odbl/)).
This data can then directly imported into the OSSL.
2. Donate a small part (e.g. 5%) of your data (release under [CC-BY](https://creativecommons.org/licenses/by/4.0/), [CC-BY-SA](https://creativecommons.org/licenses/by-sa/4.0/) and/or [ODbL](https://opendatacommons.org/licenses/odbl/)).
This data can then be directly imported into the OSSL.
3. Allow SoilSpectroscopy.org project direct access to your data so that we can perform data mining
and then release ONLY results under some Open Data license.
4. Use OSSL data to produce new derivative products, then share them through own
infrastructures OR contact us for providing hosting support.
We can sign professional **Data Sharing Agreements** with data producers
that specify in detail how will the data will be used. Our primary interest is in enabling research,
sharing and use of models (calibration and prediction) and collaboration of groups
across borders.
We take especial care that your data is secured, encrypted where necessary,
and kept safely, closely following our [privacy policy and terms of use](https://www.woodwellclimate.org/privacy-policy/).
## Contributing documentation {.unnumbered}
Please feel free to contribute to this technical documentation. Check the [GitHub repository](https://github.com/soilspectroscopy/ossl-manual) for more detailed instructions.
Information outdated or missing? Please [open an issue](https://github.com/soilspectroscopy/ossl-manual/issues) or best do a
correction in the text and then make a [pull request](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request).
## Contributors {.unnumbered}
If you've contributed to this manual, add your name, Twitter handle, ORCID or blog link below:
[Jonathan Sanderman](https://twitter.com/sandersoil),
[Tomislav Hengl](https://twitter.com/tom_hengl),
[Katherine Todd-Brown](https://www.essie.ufl.edu/people/name/kathe-todd-brown/),
[Leandro L. Parente](https://www.linkedin.com/in/leal-parente/),
[Wanderson de Sousa Mendes](https://www.linkedin.com/in/wanderson-de-sousa-mendes-75a16195/),
[Dellena Bloom](https://www.linkedin.com/in/dellena-bloom-541817149/),
[José Lucas Safanelli](https://www.linkedin.com/in/joselucassafanelli/),
[Henning Teickner](https://github.com/henningte).
## Acknowledgments {.unnumbered}
Open Soil Spectral Library was possible due to the contributions by public and private
organizations. Listed based on the date of import:
- **[USDA-NRCS Kellogg Soil Survey Laboratory (KSSL)](https://ncsslabdatamart.sc.egov.usda.gov/)** MIR spectral library [@wijewardane2018predicting;@sanderman2020mid] and the VisNIR library from RaCA project [@Wills2014;@Wijewardane2016]. We are especially grateful to **Rich Ferguson & Scarlett Murphy (NRCS USDA)** for their help with importing and using the KSSL Soil Spectral Library;
- **[ICRAF - World Agroforestry](https://www.worldagroforestry.org/sd/landhealth/soil-plant-spectral-diagnostics-laboratory/soil-spectra-library) & [ISRIC - World Soil Information](https://www.isric.org/explore/isric-soil-data-hub)** for their VisNIR soil spectral library [@garrity2004globally; @aitkenhead2018exploring] based on the Soil Information System (ISIS) of ISRIC;
- **[Africa Soil Information Service (AfSIS)](https://data.worldagroforestry.org/dataverse/icraf_soils) phases I and II** MIR spectral libraries [@Vagen_2020], a collaborative project funded by the Bill and Melinda Gates Foundation (BMGF). Partners included: CIAT-TSBF, ISRIC, CIESIN, The Earth Institute at Columbia University and World Agroforestry (ICRAF);
- **[European Soil Data Centre](https://esdac.jrc.ec.europa.eu/)** for sharing the LUCAS topsoil (VisNIR) Soil Spectral Library [@orgiazzi2018lucas]. We have degraded location accuracy of points so that exact locations are about 1-km off;
- Laura Summerauer from **ETH Zurich**, for sharing the [Central African Soil Spectral Library](https://github.com/laura-summerauer/ssl-central-africa) described in detail in @Summerauer2021;
- Marcus Schiedung from **University of Zurich** for sharing a [MIR soil spectral library](https://doi.org/10.5281/zenodo.6024831) from forest soils of North Canada [@Schiedung2022];
- Loretta Garrett from **Scion Research** for sharing a [MIR soil spectral library](https://figshare.com/articles/dataset/Soil_data_from_Pinus_radiata_forests_in_New_Zealand_FR380_trial_series/20506587/2) from forest soils in New Zealand [@Garrett2022];
- Branislav Jović from **University of Novi Sad** for sharing a MIR soil spectral library from Serbian soils [@Jovi2019];
We are also grateful to [Wanderson de Sousa Mendes](https://www.linkedin.com/in/wanderson-de-sousa-mendes-75a16195/) for
the help with initial screening of the data and the development of the initial R code for processing soil spectroscopy data.
For more advanced uses of the soil spectral libraries **we advise to contact the original data producers** especially to get help with using, extending and improving the original SSL data.
We are also grateful to **[USDA National Institute of Food and Agriculture #2020-67021-32467](https://cris.nifa.usda.gov/cgi-bin/starfinder/0?path=fastlink1.txt&id=anon&pass=&search=R=89483&format=WEBFMT6NT)** for providing funding for this project.
## Disclaimer {.unnumbered}
Whilst utmost care has been taken by the Soil Spectroscopy project and data authors while
collecting and compiling the data, the data is provided _"as is"_. [Woodwell Climate Research Center](https://www.woodwellclimate.org/), [University of Florida](https://faculty.eng.ufl.edu/ktoddbrown/), [OpenGeoHub foundation](https://opengeohub.org/about) and its
suppliers and licensors hereby disclaim all warranties of any kind, express or implied,
including, without limitation, the warranties of merchantability, fitness for a particular
purpose and non-infringement. Neither [Woodwell Climate Research Center](https://www.woodwellclimate.org/), [University of Florida](https://faculty.eng.ufl.edu/ktoddbrown/), [OpenGeoHub foundation](https://opengeohub.org/about) nor its suppliers and licensors,
makes any warranty that the Website will be error free or that access thereto will be
continuous or uninterrupted. You understand that you download from, or otherwise obtain
content or services through, the Website at your own discretion and risk.
In no event shall the data authors, the Soil Spectroscopy project, or relevant funding
agencies be liable for any actual, incidental or consequential damages arising from use of the data.
By using the Soil Spectroscopy project data, the user expressly acknowledges that the Data
may contain some nonconformities, defects, or errors. No warranty is given that the data will meet
the user's needs or expectations or that all nonconformities, defects, or errors can or will be
corrected. The user should always verify actual data; therefore the user bears all responsibility in
determining whether the data is fit for the user’s intended use.
This document is **under construction**. If you notice an error or outdated information,
please submit a correction/pull request or **[open an issue](https://github.com/soilspectroscopy/ossl-manual/issues)**.
This is a community project. No profits are being made from building and serving
Open Spectral Library. If you would like to become a sponsor of the project, please
contact us via: <https://soilspectroscopy.org/contact/>.
## License {.unnumbered}
This website/book and attached software is free to use, and is licensed under [the MIT License](https://en.wikipedia.org/wiki/MIT_License). The OSSL training data and models,
if not otherwise indicated, are available either under the [Creative Commons Attribution 4.0 International CC-BY](https://creativecommons.org/licenses/by/4.0/legalcode)
and/or [CC-BY-SA](https://creativecommons.org/licenses/by-sa/4.0/legalcode) license / [Open Data Commons Open Database License (ODbL) v1.0](https://opendatacommons.org/licenses/odbl/1-0/).
## Suggested literature {.unnumbered}
Some other connected publications and initiatives describing collation,
import and use of soil spectroscopy data:
- Angelopoulou, T., Balafoutis, A., Zalidis, G., & Bochtis, D. (2020). [From laboratory to proximal sensing spectroscopy for soil organic carbon estimation—a review](https://doi.org/10.3390/su12020443). Sustainability, 12(2), 443. <https://doi.org/10.3390/su12020443>
- Ayres, E. (2019). [Quantitative Guidelines for Establishing and Operating Soil Archives](https://doi.org/10.2136/sssaj2019.02.0050). Soil Science Society of America Journal, 83(4), 973-981. <https://doi.org/10.2136/sssaj2019.02.0050>
- Benedetti, F. and van Egmond, F. (2021). [Global Soil Spectroscopy Assessment](https://doi.org/10.4060/cb6265en). Spectral soil data – Needs and capacities. Rome, FAO. <https://doi.org/10.4060/cb6265en>
- Dudek, M., Kabała, C., Łabaz, B., Mituła, P., Bednik, M., & Medyńska-Juraszek, A. (2021). [Mid-Infrared Spectroscopy Supports Identification of the Origin of Organic Matter in Soils](https://doi.org/10.3390/land10020215). Land, 10(2), 215. <https://doi.org/10.3390/land10020215>
- [GLOSOLAN's Standard Operating Procedures (SOPs)](http://www.fao.org/global-soil-partnership/glosolan/soil-analysis/standard-operating-procedures/en/#c763834);
- Nocita, M., Stevens, A., van Wesemael, B., Aitkenhead, M., Bachmann, M., Barthès, B., ... & Wetterlind, J. (2015). [Soil spectroscopy: An alternative to wet chemistry for soil monitoring](https://doi.org/10.1016/bs.agron.2015.02.002). Advances in agronomy, 132, 139-159. <https://doi.org/10.1016/bs.agron.2015.02.002>
- Sanderman, J., Savage, K., Dangal, S. R., Duran, G., Rivard, C., Cavigelli, M. A., ... & Stewart, C. (2021). [Can Agricultural Management Induced Changes in Soil Organic Carbon Be Detected Using Mid-Infrared Spectroscopy?](https://doi.org/10.3390/rs13122265). Remote Sensing, 13(12), 2265. <https://doi.org/10.3390/rs13122265>
- Sanderman, J., Savage, K., & Dangal, S. R. (2020). [Mid-infrared
spectroscopy for prediction of soil health indicators in the United
States](https://doi.org/10.1002/saj2.20009). Soil Science Society of America Journal, 84(1), 251–261.
<https://doi.org/10.1002/saj2.20009>
- Wijewardane, N. K., Ge, Y., Wills, S., & Libohova, Z. (2018). [Predicting
physical and chemical properties of US soils with a mid-infrared
reflectance spectral library](https://doi.org/10.2136/sssaj2017.10.0361). Soil Science Society of America Journal,
82(3), 722–731. <https://doi.org/10.2136/sssaj2017.10.0361>
- Wadoux, A.M.J.-C., Malone, B., McBratney, A.B., Fajardo, M., Minasny, B., (2021). [Soil Spectral Inference with R: Analysing Digital Soil Spectra Using the R Programming Environment](https://books.google.nl/books?id=4kQgEAAAQBAJ). Progress in Soil Science, Springer Nature, ISBN: 9783030648961, 274 pp.