Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add a Profiler that output a dictionary #358

Merged
merged 5 commits into from
Oct 24, 2024
Merged

feat: add a Profiler that output a dictionary #358

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
96ce088
feat: add a Profiler that output a dictionary
12rambau Oct 23, 2024
696b762
refactor: mypy error
12rambau Oct 23, 2024
5fe2eef
refactor: make intermediaries methods private
12rambau Oct 24, 2024
9d2ee63
fix: remove the special character from the column names
12rambau Oct 24, 2024
e535f0c
fix: hide the profiler behind a geetools namespace
12rambau Oct 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/usage/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,4 +42,5 @@ Then you can open a PR with the new file and it will be reviewed and merged.
template
export
plot/index
asset
asset
profile
388 changes: 388 additions & 0 deletions docs/usage/profile.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,388 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Profile Earth Engine computation\n",
"\n",
"The Earth Engine API provides tools for profiling the performance of your computations but they are not always the easiest to use to get the number you are looking for. The `geetools` library supercharge the original profiler to make any computation evaluation the easiest possible."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[![github](https://img.shields.io/badge/-see%20sources-white?logo=github&labelColor=555)](https://github.com/gee-community/geetools/blob/main/docs/usage/profile.ipynb)\n",
"[![colab](https://img.shields.io/badge/-open%20in%20colab-blue?logo=googlecolab&labelColor=555)](https://colab.research.google.com/github/gee-community/geetools/blob/main/docs/usage/profile.ipynb)"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"tags": [
"remove-input"
]
},
"outputs": [],
"source": [
"import ee, pytest_gee, os\n",
"\n",
"if \"EARTHENGINE_SERVICE_ACCOUNT\" in os.environ:\n",
" pytest_gee.init_ee_from_service_account()\n",
"elif \"EARTHENGINE_PROJECT\" in os.environ:\n",
" pytest_gee.init_ee_from_token()\n",
"else:\n",
" raise ValueError(\"Cannot authenticate with Earth Engine.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Set up environment\n",
"\n",
"Install all the requireed libs if necessary. and perform the import satements upstream."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"# uncomment if installation of libs is necessary\n",
"# !pip install earthengine-api geetools"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"import ee\n",
"import geetools\n",
"import pandas as pd"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"# uncomment if authetication to GEE is needed\n",
"# ee.Authenticate()\n",
"# ee.Intialize(project=\"<your_project>\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Example data\n",
"\n",
"The following examples rely on a `ee.FeatureCollection` composed of three ecoregion features that define regions by which to reduce image data. The Image data are PRISM climate normals, where bands describe climate variables per month; e.g., July precipitation or January mean temperature.\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"ecoregions = (\n",
" ee.FeatureCollection(\"projects/google/charts_feature_example\")\n",
" .select([\"label\", \"value\",\"warm\"])\n",
")\n",
"\n",
"normClim = ee.ImageCollection('OREGONSTATE/PRISM/Norm91m').toBands()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## default profiler \n",
"\n",
"The default profiler from Earth Engine can be called as a context manager, it will print at the end of the cell the extensive description of your computation. "
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
" EECU·s PeakMem Count Description\n",
" 0.256 654k 87 Loading assets: projects/google/charts_feature_example\n",
" 0.244 59k 6 Algorithm Image.reduceRegions\n",
" 0.134 362k 831 (plumbing)\n",
" 0.025 606k 86 no description available\n",
" 0.010 200k 11 Loading assets: OREGONSTATE/PRISM/Norm91m/08@1662731245955723\n",
" 0.009 198k 11 Loading assets: OREGONSTATE/PRISM/Norm91m/01@1662731626359925\n",
" 0.009 202k 11 Loading assets: OREGONSTATE/PRISM/Norm91m/07@1662732032798195\n",
" 0.008 211k 11 Loading assets: OREGONSTATE/PRISM/Norm91m/05@1662731334196830\n",
" 0.008 199k 11 Loading assets: OREGONSTATE/PRISM/Norm91m/09@1662730604988590\n",
" 0.008 212k 11 Loading assets: OREGONSTATE/PRISM/Norm91m/04@1662730567169297\n",
" 0.008 198k 11 Loading assets: OREGONSTATE/PRISM/Norm91m/10@1662731228874571\n",
" 0.008 210k 11 Loading assets: OREGONSTATE/PRISM/Norm91m/03@1662731338127317\n",
" 0.008 205k 11 Loading assets: OREGONSTATE/PRISM/Norm91m/02@1662731486284455\n",
" 0.008 209k 11 Loading assets: OREGONSTATE/PRISM/Norm91m/06@1662731651724226\n",
" 0.008 198k 11 Loading assets: OREGONSTATE/PRISM/Norm91m/12@1662731114457874\n",
" 0.008 199k 11 Loading assets: OREGONSTATE/PRISM/Norm91m/11@1662731554688435\n",
" 0.007 352 72 Reprojecting pixels from GEOGCS[\"GCS_North_American_1983\",DATUM[\"North_American_Datum_1983\",SPHEROID[...] to GEOGCS[\"GCS_North_American_1983\",DATUM[\"North_American_Datum_1983\",SPHEROID[...]\n",
" 0.003 3.2k 13 Algorithm Collection.reduceColumns with reducer Reducer.toList\n",
" 0.002 9.9k 15 Algorithm ImageCollection.toBands\n",
" 0.001 5.8k 15 Algorithm Image.rename\n",
" 0.001 4.0k 15 Algorithm Image.select\n",
" 0.001 3.1k 14 Algorithm ReduceRegions.AggregationContainer\n",
" 0.000 61k 51 Loading assets: OREGONSTATE/PRISM/Norm91m\n",
" 0.000 448 3 Listing collection\n",
" 0.000 112k 3 Computing image mask from geometry\n",
" - 90k 26 Algorithm Collection.reduceColumns\n",
" - 46k 19 Algorithm List.map\n",
" - 30k 5 Loading assets: OREGONSTATE/PRISM/Norm91m/12\n",
" - 30k 5 Loading assets: OREGONSTATE/PRISM/Norm91m/11\n",
" - 30k 5 Loading assets: OREGONSTATE/PRISM/Norm91m/10\n",
" - 30k 5 Loading assets: OREGONSTATE/PRISM/Norm91m/09\n",
" - 30k 5 Loading assets: OREGONSTATE/PRISM/Norm91m/08\n",
" - 30k 5 Loading assets: OREGONSTATE/PRISM/Norm91m/07\n",
" - 30k 5 Loading assets: OREGONSTATE/PRISM/Norm91m/06\n",
" - 30k 5 Loading assets: OREGONSTATE/PRISM/Norm91m/05\n",
" - 30k 5 Loading assets: OREGONSTATE/PRISM/Norm91m/04\n",
" - 30k 5 Loading assets: OREGONSTATE/PRISM/Norm91m/03\n",
" - 30k 5 Loading assets: OREGONSTATE/PRISM/Norm91m/02\n",
" - 30k 5 Loading assets: OREGONSTATE/PRISM/Norm91m/01\n",
" - 24k 40 Algorithm AggregateFeatureCollection.array\n",
" - 11k 15 Algorithm Collection.loadTable\n",
" - 10k 15 Algorithm ImageCollection.load\n",
" - 8.6k 1 Algorithm (user-defined function)\n",
" - 8.5k 4 Algorithm Projection\n",
" - 7.1k 4 Algorithm ReduceRegions.ReduceRegionsEnumerator\n",
" - 5.4k 15 Algorithm Collection.map\n",
" - 3.6k 14 Algorithm Dictionary.fromLists\n",
" - 3.4k 14 Algorithm Feature.select\n",
" - 3.4k 10 Algorithm If\n",
" - 3.2k 10 Algorithm Number.eq\n",
" - 3.2k 4 Algorithm String.compareTo\n",
" - 3.2k 4 Algorithm Reducer.forEach\n",
" - 3.1k 37 Algorithm String\n",
" - 3.1k 10 Algorithm ObjectType\n",
" - 2.9k 20 Loading assets: OREGONSTATE/PRISM\n",
" - 2.8k 20 Loading assets: projects/google\n",
" - 1.9k 1 Algorithm Number.format\n",
" - 600 5 Algorithm Reducer.mean\n",
" - 432 7 Expression evaluation\n",
" - 288 72 Algorithm Image.load computing pixels\n"
]
}
],
"source": [
"with ee.profilePrinting():\n",
" normClim.geetools.byBands(\n",
" regions = ecoregions,\n",
" reducer = \"mean\",\n",
" scale = 500,\n",
" regionId = \"label\",\n",
" bands = [f\"{i:02d}_tmean\" for i in range(1,13)],\n",
" ).getInfo()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This result is extremely useful but cannot be further explored in the notebook."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## geetools profiler\n",
"\n",
"The `geetools` profiler is a context manager object that fill a dictionary member (`profile`) with the content of the string profile. This dictionary can be transformed into a table easily."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'EECU·s': [0.001, None],\n",
" 'PeakMem': [4720, 3100],\n",
" 'Count': [3, 3],\n",
" 'Description': ['(plumbing)', 'Algorithm Number.add']}"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# example with a simple function\n",
"with ee.Profiler() as p:\n",
" ee.Number(3.14).add(0.00159).getInfo()\n",
"p.profile"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"With a bigger method we can valorized the results as a pandas dataframe and extract key informations."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>EECU·s</th>\n",
" <th>PeakMem</th>\n",
" <th>Count</th>\n",
" <th>Description</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>0.368</td>\n",
" <td>59000</td>\n",
" <td>6</td>\n",
" <td>Algorithm Image.reduceRegions</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>0.227</td>\n",
" <td>655000</td>\n",
" <td>87</td>\n",
" <td>Loading assets: projects/google/charts_feature...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>0.147</td>\n",
" <td>356000</td>\n",
" <td>831</td>\n",
" <td>(plumbing)</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>0.026</td>\n",
" <td>592000</td>\n",
" <td>86</td>\n",
" <td>no description available</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>0.011</td>\n",
" <td>209000</td>\n",
" <td>11</td>\n",
" <td>Loading assets: OREGONSTATE/PRISM/Norm91m/06@1...</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" EECU·s PeakMem Count Description\n",
"0 0.368 59000 6 Algorithm Image.reduceRegions\n",
"1 0.227 655000 87 Loading assets: projects/google/charts_feature...\n",
"2 0.147 356000 831 (plumbing)\n",
"3 0.026 592000 86 no description available\n",
"4 0.011 209000 11 Loading assets: OREGONSTATE/PRISM/Norm91m/06@1..."
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"with ee.Profiler() as p:\n",
" normClim.geetools.byBands(\n",
" regions = ecoregions,\n",
" reducer = \"mean\",\n",
" scale = 500,\n",
" regionId = \"label\",\n",
" bands = [f\"{i:02d}_tmean\" for i in range(1,13)],\n",
" ).getInfo()\n",
"df = pd.DataFrame(p.profile)\n",
"df.head()"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.8999999999999999"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# total EECU cost of the computation\n",
"float(df[\"EECU·s\"].sum())"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "geetools",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.9"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Loading