This is a community maintained open-source repository and is not affiliated with any of the projects below. There are links and resources to academic research, Mindat & API, R / Python packages, and more. 🪨
Authors: Xiaogang Ma, Jolyon Ralph, Jiyin Zhang, Xiang Que, Anirudh Prabhu, Shaunna M. Morrison, Robert M. Hazen, Lesley Wyborn, Kerstin Lehnert
- Accessible at https://rmets.onlinelibrary.wiley.com/doi/10.1002/gdj3.204
- Open Access Paper; an inspiring research initiative!
- The Mindat API and improvements to the data establishes many practical affordances to those working with mineral information !
Authors: Cheng Deng, Tianhang Zhang, Zhongmou He, Yi Xu, Qiyuan Chen, Yuanyuan Shi, Luoyi Fu, Weinan Zhang, Xinbing Wang, Chenghu Zhou, Zhouhan Lin, Junxian He, Zhouhan Lin, Junxian He
- Accessible at: https://arxiv.org/abs/2306.05064
- Code Source: https://github.com/davendw49/k2
- Paper Preprint available on arXiv (2306.05064); paper accepted The 17th ACM International Conference on Web Search and Data Mining
The world's most comprehensive mineral database and mineralogical reference website"
Source: https://www.mindat.org/a/how_to_get_my_mindat_api_key
- The first step in getting started with the Mindat API
- The article "How do I get my API Key?" explains how a user can obtain a key
- An API "key" is a long string of letters and numbers that uniquely identifies the user when they are interacting with the Mindat API.
- The article has several thousand views (which is a metric to the interest in the extended function of Mindat).
Source: https://api.mindat.org/schema/redoc/
- The API endpoints are downloadable as a .YAML file above and are in OpenAPI format
- Start here if you are a developer or in need of accessing the Mindat API endpoints directly.
-
Depending on context, you may not need to use the full search functions of the Mindat API! See the advanced search functions below for
If you are using Python or R for Data Science, there are completed packages ready for use with the Mindat API.
Right now the Mindat API is most accessible through R and Python. However, anyone who can use Excel can get started with Python or R.
First, download the Python programming language at Python.org. Next, visit this URL (https://colab.research.google.com/github/ChuBL/How-to-Use-Mindat-API/blob/main/How_to_Use_Mindat_API.ipynb ) With no programming experience whatsoever, you can follow the steps within this Jupyter Notebook to get started with the API.
The current footprint of the total Mindat database is measurable in gigabytes. It's lighter than a backpack full of those same geomaterials !
🚧 Items I would like to add to this guide:
- Downloading the entire Mindat database to a local copy.
- Verifying a local database has the most correct information
It is important to make a distinction in the usage of the Mindat API: querying and downloading data from the Mindat website is not the same as data analytics and data science with the mineralogy data. R and Python allow for both.
- Source: https://cran.r-project.org/web/packages/OpenMindat/index.html
- Author(s): Xiang Que [aut, cre], Xiaogang Ma [aut]
- "Allows access to the Mindat.org resources through the R programming language for data science activities."
- ⭐
- "This is a test version of the OpenMindat Python package, designed to facilitate querying and retrieving data on minerals and geomaterials from the Mindat API."
- An excellent resource with documentation.
- Author(s): Jiyin Zhang
- ⭐
🚧 Not all people interested in using the Mindat API are familar with the programming of digital computers. There are many great resources for folks to learn and anyone interested can get started within an hour. However, the usecase of Mindat to .CSV outputs is important and I think that the Mindat API can facilitate this. At present, Python and R are the only completed official implementations.
🚧
The API for Mindat.org removes the need for web scraping. While you can use web-scraping methods on the Mindat.org website example of old web scraper, this method is now outdated and it.
The Julia Programming Language is capable of data science activities alongside or in parallel with Python and R. Anything that can be used for data science activities with the Mindat API should be used for data science activities !
There exists an R package that allows calls to the Julia lang from R (https://cran.r-project.org/web/packages/JuliaCall/readme/README.html).
There exists a Julia package that allows calls to Python package that may be interoperable with the Python Mindat package. The Julia package may be found at (https://juliahub.com/ui/Packages/General/PyCall).
-
A language model (LLM) toolchain could, if connected to the internet, query the Mindat database for factual retrieval of geomaterial properties.
-
RAG functionality with Mindat's data would be excellent for systems that would otherwise confabulate answers pertaining to geomaterials, localities, etc.
-
⚠️ ©️ ❔ Fair use and commercial applications are outside the scope of this document. Please respect the Mindat team and researchers. -
Could there be a more effecient method to encode mineralogy data into a LLM dataset like K2?
Running Rust via web-assembly may be usable through WebAssembly to handle JSON from the Mindat server.
- Rust can enable command line functionality for queries, such as a bash script, that can be useful when for users looking for a single page of results.
- Rust may be helpful in parsing the JSON responses from the server
- Source: https://github.com/fros1y/mindatGIS
- Language: Python
- Author: Martin G / "fros1y"
- "Export Mindat Mineral Information to GeoPackages for GIS"
- Source: https://github.com/MisterSirCode/Mindat-Data-Collector
- Language: JavaScript
- Author: Schneider / "MisterSirCode"
- "This is a simple little NodeJS tool I threw together to show how to use the Mindat API to repeatedly grab pages of info from mindat to copy an entire database locally."
- "A simple python example for the (alpha) version of the API. Requires an API key "
- Source: https://github.com/jolyonralph/mindat_api_test
- Author: Jolyon Ralph
- Outdated with the full release of the Mindat API
- Python Programming Language
- Source: https://github.com/MrHamel/mindat_api_test
- Author(s): Ryan Hamel
- Python Programming Language
- Outdated with the full release of the Mindat API
- Source: https://github.com/MelorGloom/Crystal-Info-Scrapper
- PostgreSQL and Python
- Made redundant with the MindatAPI, which allows for this task to be done much more effeciently!
- https://www.earthdatascience.org/
- Free Earth Data Science Courses & Textbooks
- Learn scientific programming, reproducible open science workflows and data science today.
- Python and Earth Data Science
- Source: https://opengeology.org/
- An excellent collection of open source textbooks!
- Source(s): https://www.c82.net/mineralogy/
- "All 2,242 illustrations from James Sowerby’s compendium of knowledge about mineralogy in Great Britain and beyond, drawn 1802–1817 and arranged by color."
- Interactive website
Python is a programming language that is accessible and well-suited for data science. If you are new to computer programming, Python is an excellent choice.
- FreeCodeCamp.com has an interactive "Scientific Computing With Python" course.
- The 'Python IDLE' is a programming environment associated with the Python language.
Advancing into Analytics: From Excel to Python and R (O’Reilly) https://stringfestanalytics.com/book/
This repository is a student practicing documentation. The OpenMindat paper and Mindat API are really exciting and interesting. There are many excellent and amazing things that are still to come of this resource being made accessible. Please create a pull request if you can correct, clairfy, or expand on the resources above. :)