From 459c79a9fc45fbc154a60464b25960714a245204 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Martin=20Sch=C3=A4tz?= <61615369+martinschatz-cz@users.noreply.github.com> Date: Thu, 1 Feb 2024 11:17:04 +0100 Subject: [PATCH] Update 02_Pandas_operations.ipynb solving #12 --- .../01_pandas_statistics/02_Pandas_operations.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/advanced_image_analysis_mb100t01/01_pandas_statistics/02_Pandas_operations.ipynb b/advanced_image_analysis_mb100t01/01_pandas_statistics/02_Pandas_operations.ipynb index d773bfb..a6c8c09 100644 --- a/advanced_image_analysis_mb100t01/01_pandas_statistics/02_Pandas_operations.ipynb +++ b/advanced_image_analysis_mb100t01/01_pandas_statistics/02_Pandas_operations.ipynb @@ -1 +1 @@ -{"cells":[{"cell_type":"markdown","source":["\"Open

"],"metadata":{"id":"_kbesz9rsBo5"}},{"cell_type":"markdown","metadata":{"id":"Vkf1B-vMwpVB"},"source":["# Basic Operations"]},{"cell_type":"markdown","metadata":{"id":"fpPtZgqIvuXz"},"source":["Inspiration and some of the parts came from: Python Data Science [GitHub repository](https://github.com/jakevdp/PythonDataScienceHandbook/tree/master), [MIT License](https://github.com/jakevdp/PythonDataScienceHandbook/blob/master/LICENSE-CODE) and [Introduction to Pandas](https://colab.research.google.com/notebooks/mlcc/intro_to_pandas.ipynb) by Google, [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)\n","\n","If running this from Google Colab, uncomment the cell below and run it. Otherwise, just skip it."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"5saSBc40voZF"},"outputs":[],"source":["#!pip install seaborn\n","#!pip install watermark"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"TL7zwFk5sA2o"},"outputs":[],"source":["import pandas as pd\n","import numpy as np\n","import seaborn as sns"]},{"cell_type":"markdown","metadata":{"id":"ZkUd2sa-yP5e"},"source":["## Learning Objectives:\n","\n"," * operations\n"," * selection\n"," * filtering\n"," * concat\n"," * NaNs\n"]},{"cell_type":"markdown","metadata":{"id":"oa5wfZT7VHJl"},"source":["For this notebook, we will continue using the cities and california housing dataframes."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"Hn50qDhssA2p","outputId":"b7f1c937-012c-430a-b4d8-1beca9dcd07f"},"outputs":[{"data":{"text/html":["

\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
City namePopulation
0San Francisco852469
1San Jose1015785
2Sacramento485199
\n","
"],"text/plain":[" City name Population\n","0 San Francisco 852469\n","1 San Jose 1015785\n","2 Sacramento 485199"]},"execution_count":3,"metadata":{},"output_type":"execute_result"}],"source":["city_names = pd.Series(['San Francisco', 'San Jose', 'Sacramento'])\n","population = pd.Series([852469, 1015785, 485199])\n","\n","cities_dataframe = pd.DataFrame({ 'City name': city_names, 'Population': population })\n","cities_dataframe"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":206},"executionInfo":{"elapsed":400,"status":"ok","timestamp":1692082096397,"user":{"displayName":"Martin Schätz","userId":"14609383414092679868"},"user_tz":-120},"id":"av6RYOraVG1V","outputId":"acbcb340-2789-428a-c4dc-9c92b57d52c6"},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
longitudelatitudehousing_median_agetotal_roomstotal_bedroomspopulationhouseholdsmedian_incomemedian_house_value
0-114.3134.1915.05612.01283.01015.0472.01.493666900.0
1-114.4734.4019.07650.01901.01129.0463.01.820080100.0
2-114.5633.6917.0720.0174.0333.0117.01.650985700.0
3-114.5733.6414.01501.0337.0515.0226.03.191773400.0
4-114.5733.5720.01454.0326.0624.0262.01.925065500.0
\n","
"],"text/plain":[" longitude latitude housing_median_age total_rooms total_bedrooms \\\n","0 -114.31 34.19 15.0 5612.0 1283.0 \n","1 -114.47 34.40 19.0 7650.0 1901.0 \n","2 -114.56 33.69 17.0 720.0 174.0 \n","3 -114.57 33.64 14.0 1501.0 337.0 \n","4 -114.57 33.57 20.0 1454.0 326.0 \n","\n"," population households median_income median_house_value \n","0 1015.0 472.0 1.4936 66900.0 \n","1 1129.0 463.0 1.8200 80100.0 \n","2 333.0 117.0 1.6509 85700.0 \n","3 515.0 226.0 3.1917 73400.0 \n","4 624.0 262.0 1.9250 65500.0 "]},"execution_count":4,"metadata":{},"output_type":"execute_result"}],"source":["california_housing_dataframe = pd.read_csv(\"https://download.mlcc.google.com/mledu-datasets/california_housing_train.csv\", sep=\",\")\n","california_housing_dataframe.head()"]},{"cell_type":"markdown","metadata":{"id":"OzPb3cEJsA2r"},"source":["## Manipulating Data\n","\n","### Applying functions\n","\n","You may apply Python's basic arithmetic operations to `Series`. For example:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"ZPvYGXKisA2r","outputId":"f13f9c0e-cafd-4407-da38-e6d738a7b6b2"},"outputs":[{"data":{"text/plain":["0 852.469\n","1 1015.785\n","2 485.199\n","dtype: float64"]},"execution_count":5,"metadata":{},"output_type":"execute_result"}],"source":["population / 1000"]},{"cell_type":"markdown","metadata":{"id":"Z9QVl3qksA2s"},"source":["*pandas* `Series` can be used as arguments to most NumPy functions:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"FXAIj5tUsA2s","outputId":"72ac6c7b-c545-4a6b-ef05-5ded8445d398"},"outputs":[{"data":{"text/plain":["0 13.655892\n","1 13.831172\n","2 13.092314\n","dtype: float64"]},"execution_count":6,"metadata":{},"output_type":"execute_result"}],"source":["np.log(population)"]},{"cell_type":"markdown","metadata":{"id":"hRofbzevsA2s"},"source":["For more complex single-column transformations, you can use `Series.apply`. Like the Python [map function](https://docs.python.org/2/library/functions.html#map),\n","`Series.apply` accepts as an argument a [lambda function](https://docs.python.org/2/tutorial/controlflow.html#lambda-expressions), which is applied to each value.\n","\n","The example below creates a new `Series` that indicates whether `population` is over one million:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"2eN3aG_xsA2s","outputId":"36b769eb-77e3-441e-d8d8-5fe0a91b8d83"},"outputs":[{"data":{"text/plain":["0 False\n","1 True\n","2 False\n","dtype: bool"]},"execution_count":7,"metadata":{},"output_type":"execute_result"}],"source":["big_city = population.apply(lambda val: val > 1000000)\n","big_city"]},{"cell_type":"markdown","metadata":{"id":"XfzCh1qT1Pn3"},"source":["### Filtering"]},{"cell_type":"markdown","metadata":{"id":"sh7L0H2HsA2t"},"source":["One can use this result as a binary mask to make a sub-dataframe."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"XzNvFVotsA2t","outputId":"754ca69f-10c5-454d-cbd5-75fbf0825ec5"},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
City namePopulation
1San Jose1015785
\n","
"],"text/plain":[" City name Population\n","1 San Jose 1015785"]},"execution_count":8,"metadata":{},"output_type":"execute_result"}],"source":["cities_dataframe[big_city]"]},{"cell_type":"markdown","metadata":{"id":"6KDilny_sA2t"},"source":["Here is another way of generating a binary mask without explicitly using a `lamba` function."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"VpA39aYVsA2t","outputId":"0948b7dd-fc97-444a-fbb7-d9858a0e26fc"},"outputs":[{"data":{"text/plain":["0 False\n","1 True\n","2 False\n","Name: Population, dtype: bool"]},"execution_count":9,"metadata":{},"output_type":"execute_result"}],"source":["big_city = cities_dataframe['Population'] > 1000000\n","big_city"]},{"cell_type":"markdown","metadata":{"id":"5S6iUV5WsA2t"},"source":["### Adding new columns\n","\n","Modifying `DataFrames` is also straightforward. For example, the following code adds two `Series` to an existing `DataFrame`. One of them is the result of a computation of 2 existing columns."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"WOGb96mosA2t","outputId":"1721868d-eb6b-457b-881e-59662cd14f7f"},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
City namePopulationArea square milesPopulation density
0San Francisco85246946.8718187.945381
1San Jose1015785176.535754.177760
2Sacramento48519997.924955.055147
\n","
"],"text/plain":[" City name Population Area square miles Population density\n","0 San Francisco 852469 46.87 18187.945381\n","1 San Jose 1015785 176.53 5754.177760\n","2 Sacramento 485199 97.92 4955.055147"]},"execution_count":10,"metadata":{},"output_type":"execute_result"}],"source":["cities_dataframe['Area square miles'] = pd.Series([46.87, 176.53, 97.92])\n","cities_dataframe['Population density'] = cities_dataframe['Population'] / cities_dataframe['Area square miles']\n","cities_dataframe"]},{"cell_type":"markdown","metadata":{"id":"ZvqbT9tP030O"},"source":["### Concatenating DataFrames\n","\n","Let's imagine we collect another similar data sample, like the one below."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"7tYdJEJOsA2u","outputId":"845945fb-71c9-4be2-8674-f371d9c6132d"},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
City namePopulationArea square milesPopulation density
0Sao Paulo12400232587.3421112.527667
1Sao Luis1108975319.363472.491859
2Salvador2886698268.0010771.261194
\n","
"],"text/plain":[" City name Population Area square miles Population density\n","0 Sao Paulo 12400232 587.34 21112.527667\n","1 Sao Luis 1108975 319.36 3472.491859\n","2 Salvador 2886698 268.00 10771.261194"]},"execution_count":11,"metadata":{},"output_type":"execute_result"}],"source":["city_names = pd.Series(['Sao Paulo', 'Sao Luis', 'Salvador'])\n","population = pd.Series([12400232, 1108975, 2886698])\n","\n","another_cities_dataframe = pd.DataFrame({ 'City name': city_names, 'Population': population })\n","\n","another_cities_dataframe['Area square miles'] = pd.Series([587.34, 319.36, 268])\n","another_cities_dataframe['Population density'] = another_cities_dataframe['Population'] / another_cities_dataframe['Area square miles']\n","another_cities_dataframe"]},{"cell_type":"markdown","metadata":{"id":"mFZVreVhsA2u"},"source":["Before concatenating, it is probably a good idea to insert an identifier column so that we keep track where data came from.\n","\n","We can easily do that by creating a new column in each dataframe **before** concatenating."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"iIVZNMQTsA2u"},"outputs":[],"source":["cities_dataframe['Country'] = 'USA'\n","another_cities_dataframe['Country'] = 'Brazil'"]},{"cell_type":"markdown","metadata":{"id":"OO5Iw5TYsA2u"},"source":["We can now concatenate similar dataframes with the `pandas.concat` functions."]},{"cell_type":"code","execution_count":null,"metadata":{"executionInfo":{"elapsed":17,"status":"ok","timestamp":1692082096398,"user":{"displayName":"Martin Schätz","userId":"14609383414092679868"},"user_tz":-120},"id":"wlPXyLYw06UV","outputId":"76547221-538c-4d71-e86c-9871e7318ee2"},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
City namePopulationArea square milesPopulation densityCountry
0San Francisco85246946.8718187.945381USA
1San Jose1015785176.535754.177760USA
2Sacramento48519997.924955.055147USA
0Sao Paulo12400232587.3421112.527667Brazil
1Sao Luis1108975319.363472.491859Brazil
2Salvador2886698268.0010771.261194Brazil
\n","
"],"text/plain":[" City name Population Area square miles Population density Country\n","0 San Francisco 852469 46.87 18187.945381 USA\n","1 San Jose 1015785 176.53 5754.177760 USA\n","2 Sacramento 485199 97.92 4955.055147 USA\n","0 Sao Paulo 12400232 587.34 21112.527667 Brazil\n","1 Sao Luis 1108975 319.36 3472.491859 Brazil\n","2 Salvador 2886698 268.00 10771.261194 Brazil"]},"execution_count":13,"metadata":{},"output_type":"execute_result"}],"source":["result = pd.concat([cities_dataframe, another_cities_dataframe])\n","result"]},{"cell_type":"markdown","metadata":{"id":"WICecJHvsA2u"},"source":["We now have a longer dataframe with some repeated indices. To have unique indices, we can use `.reset_index`."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"BvRjm2vUsA2u","outputId":"0fe65ba1-8b8c-4ca4-d3b3-7a0674385dfa"},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
City namePopulationArea square milesPopulation densityCountry
0San Francisco85246946.8718187.945381USA
1San Jose1015785176.535754.177760USA
2Sacramento48519997.924955.055147USA
3Sao Paulo12400232587.3421112.527667Brazil
4Sao Luis1108975319.363472.491859Brazil
5Salvador2886698268.0010771.261194Brazil
\n","
"],"text/plain":[" City name Population Area square miles Population density Country\n","0 San Francisco 852469 46.87 18187.945381 USA\n","1 San Jose 1015785 176.53 5754.177760 USA\n","2 Sacramento 485199 97.92 4955.055147 USA\n","3 Sao Paulo 12400232 587.34 21112.527667 Brazil\n","4 Sao Luis 1108975 319.36 3472.491859 Brazil\n","5 Salvador 2886698 268.00 10771.261194 Brazil"]},"execution_count":14,"metadata":{},"output_type":"execute_result"}],"source":["result = result.reset_index(drop=True)\n","result"]},{"cell_type":"markdown","metadata":{"id":"uy2tR0XR3vbE"},"source":["### NaNs\n","\n","`DataFrame` objects can be created by passing a `dict` mapping `string` column names to their respective `Series`. If the `Series` don't match in length, missing values are filled with special [NA/NaN](http://pandas.pydata.org/pandas-docs/stable/missing_data.html) values. We cannot assume what these values are, because that would distort th results. So we need to deal with these NaNs values."]},{"cell_type":"markdown","metadata":{"id":"bGYpAX7t4bky"},"source":["We can test the missing values using `isnull()` function."]},{"cell_type":"markdown","metadata":{"id":"P0BwiPFhIbfV"},"source":["We can work with one of the `seaborn` training datasets *Penguins*"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"9Spm3StwImkL"},"outputs":[],"source":["penguins = sns.load_dataset(\"penguins\")"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":424},"executionInfo":{"elapsed":17,"status":"ok","timestamp":1692082096400,"user":{"displayName":"Martin Schätz","userId":"14609383414092679868"},"user_tz":-120},"id":"GYJkXKke42gp","outputId":"ab0c346a-952e-4fa1-aca9-e0d96c964520"},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
speciesislandbill_length_mmbill_depth_mmflipper_length_mmbody_mass_gsex
0FalseFalseFalseFalseFalseFalseFalse
1FalseFalseFalseFalseFalseFalseFalse
2FalseFalseFalseFalseFalseFalseFalse
3FalseFalseTrueTrueTrueTrueTrue
4FalseFalseFalseFalseFalseFalseFalse
........................
339FalseFalseTrueTrueTrueTrueTrue
340FalseFalseFalseFalseFalseFalseFalse
341FalseFalseFalseFalseFalseFalseFalse
342FalseFalseFalseFalseFalseFalseFalse
343FalseFalseFalseFalseFalseFalseFalse
\n","

344 rows × 7 columns

\n","
"],"text/plain":[" species island bill_length_mm bill_depth_mm flipper_length_mm \\\n","0 False False False False False \n","1 False False False False False \n","2 False False False False False \n","3 False False True True True \n","4 False False False False False \n",".. ... ... ... ... ... \n","339 False False True True True \n","340 False False False False False \n","341 False False False False False \n","342 False False False False False \n","343 False False False False False \n","\n"," body_mass_g sex \n","0 False False \n","1 False False \n","2 False False \n","3 True True \n","4 False False \n",".. ... ... \n","339 True True \n","340 False False \n","341 False False \n","342 False False \n","343 False False \n","\n","[344 rows x 7 columns]"]},"execution_count":16,"metadata":{},"output_type":"execute_result"}],"source":["penguins.isnull()"]},{"cell_type":"markdown","metadata":{"id":"m9jBF0A55OBn"},"source":["But it is more practical to test if there are any NaNs, than looking for it. We can use `.isnull().values.any()` approach."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":18,"status":"ok","timestamp":1692082096401,"user":{"displayName":"Martin Schätz","userId":"14609383414092679868"},"user_tz":-120},"id":"PucyKyW87geJ","outputId":"ba37a65a-dc0c-47a6-8bba-ed09be8ee1c7"},"outputs":[{"data":{"text/plain":["True"]},"execution_count":17,"metadata":{},"output_type":"execute_result"}],"source":["penguins.isnull().values.any()"]},{"cell_type":"markdown","metadata":{"id":"C5dD79015mYf"},"source":["Or we can explore each column using `.isnull().sum()`."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":17,"status":"ok","timestamp":1692082096401,"user":{"displayName":"Martin Schätz","userId":"14609383414092679868"},"user_tz":-120},"id":"GWIZ4l_z5tYQ","outputId":"bf5f4e7b-9e1c-406d-fade-2e543db8f0ff"},"outputs":[{"data":{"text/plain":["species 0\n","island 0\n","bill_length_mm 2\n","bill_depth_mm 2\n","flipper_length_mm 2\n","body_mass_g 2\n","sex 11\n","dtype: int64"]},"execution_count":18,"metadata":{},"output_type":"execute_result"}],"source":["penguins.isnull().sum()"]},{"cell_type":"markdown","metadata":{"id":"PcKxJ_vNLW3X"},"source":["We will want to drop all rows with unknown entries with `.dropna()` function."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":17,"status":"ok","timestamp":1692082096402,"user":{"displayName":"Martin Schätz","userId":"14609383414092679868"},"user_tz":-120},"id":"Vfqe-aMDMBpi","outputId":"6cd4d58f-78d6-47ff-bc4c-17423e4bd683"},"outputs":[{"data":{"text/plain":["species 0\n","island 0\n","bill_length_mm 0\n","bill_depth_mm 0\n","flipper_length_mm 0\n","body_mass_g 0\n","sex 0\n","dtype: int64"]},"execution_count":19,"metadata":{},"output_type":"execute_result"}],"source":["penguins_cleaned = penguins.dropna()\n","penguins_cleaned.isnull().sum()"]},{"cell_type":"markdown","metadata":{"id":"VksvzteBsA2z"},"source":["## Exercise\n","\n","The table below contains shape and intensity measurements from a biological sample. Make a subset with the columns `Area` and `Mean`. Remove all rows that contain NaNs from it and count the remaining rows.\n","\n","Afterwards, take the initial table again and make a new subset with the columns `Major` and `Minor`. Remove NaNs and count the remaining rows again.\n","\n","What do you conclude?"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"ftJB1JvXsA20","outputId":"e888baf3-5fdb-45a9-d860-f7f570e709e5"},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
AreaMeanStdDevMinMaxXYXMYMMajorMinorAngle%AreaType
118.0730.389103.354592.0948.0435.0004.722434.9624.6975.9873.828168.425100A
2126.0718.33390.367556.01046.0388.0878.683388.1838.68716.5599.688175.471100A
3NaNNaNNaN608.0964.0NaNNaNNaN7.6657.359NaN101.121100A
468.0686.98561.169571.0880.0126.1478.809126.1928.81115.1365.720168.133100A
5NaNNaN69.438566.0792.0348.5007.500NaN7.508NaN3.088NaN100A
.............................................
387152.0801.599111.328582.01263.0348.487497.632348.451497.67517.77310.88911.829100A
38817.0742.70669.624620.0884.0420.500496.382420.513NaNNaN3.66349.457100A
38960.0758.03377.309601.0947.0259.000499.300258.990499.2899.4768.06290.000100A
39012.0714.83367.294551.0785.0240.167498.167240.179498.1484.6063.317168.690100A
39123.0695.04367.356611.0846.049.891503.02249.882502.9796.4544.53773.243100A
\n","

391 rows × 14 columns

\n","
"],"text/plain":[" Area Mean StdDev Min Max X Y XM \\\n"," \n","1 18.0 730.389 103.354 592.0 948.0 435.000 4.722 434.962 \n","2 126.0 718.333 90.367 556.0 1046.0 388.087 8.683 388.183 \n","3 NaN NaN NaN 608.0 964.0 NaN NaN NaN \n","4 68.0 686.985 61.169 571.0 880.0 126.147 8.809 126.192 \n","5 NaN NaN 69.438 566.0 792.0 348.500 7.500 NaN \n",".. ... ... ... ... ... ... ... ... \n","387 152.0 801.599 111.328 582.0 1263.0 348.487 497.632 348.451 \n","388 17.0 742.706 69.624 620.0 884.0 420.500 496.382 420.513 \n","389 60.0 758.033 77.309 601.0 947.0 259.000 499.300 258.990 \n","390 12.0 714.833 67.294 551.0 785.0 240.167 498.167 240.179 \n","391 23.0 695.043 67.356 611.0 846.0 49.891 503.022 49.882 \n","\n"," YM Major Minor Angle %Area Type \n"," \n","1 4.697 5.987 3.828 168.425 100 A \n","2 8.687 16.559 9.688 175.471 100 A \n","3 7.665 7.359 NaN 101.121 100 A \n","4 8.811 15.136 5.720 168.133 100 A \n","5 7.508 NaN 3.088 NaN 100 A \n",".. ... ... ... ... ... ... \n","387 497.675 17.773 10.889 11.829 100 A \n","388 NaN NaN 3.663 49.457 100 A \n","389 499.289 9.476 8.062 90.000 100 A \n","390 498.148 4.606 3.317 168.690 100 A \n","391 502.979 6.454 4.537 73.243 100 A \n","\n","[391 rows x 14 columns]"]},"execution_count":21,"metadata":{},"output_type":"execute_result"}],"source":["dataframe = pd.read_csv('../../data/Results.csv', index_col=0, delimiter=';')\n","dataframe"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"7IPdbT2NsA20"},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{"id":"fH1zusN7GKCx"},"source":["**Watermark**"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":3,"status":"ok","timestamp":1692082096620,"user":{"displayName":"Martin Schätz","userId":"14609383414092679868"},"user_tz":-120},"id":"iH1jL0baGMJW","outputId":"8639a749-8bab-4fd8-d6be-29230edccdea"},"outputs":[{"name":"stdout","output_type":"stream","text":["Last updated: 2023-08-24T14:25:40.097708+02:00\n","\n","Python implementation: CPython\n","Python version : 3.9.17\n","IPython version : 8.14.0\n","\n","Compiler : MSC v.1929 64 bit (AMD64)\n","OS : Windows\n","Release : 10\n","Machine : AMD64\n","Processor : Intel64 Family 6 Model 165 Stepping 2, GenuineIntel\n","CPU cores : 16\n","Architecture: 64bit\n","\n","watermark : 2.4.3\n","numpy : 1.23.5\n","pandas : 2.0.3\n","seaborn : 0.12.2\n","pivottablejs: 0.9.0\n","\n"]}],"source":["from watermark import watermark\n","watermark(iversions=True, globals_=globals())\n","print(watermark())\n","print(watermark(packages=\"watermark,numpy,pandas,seaborn,pivottablejs\"))"]}],"metadata":{"colab":{"provenance":[],"toc_visible":true},"kernelspec":{"display_name":"Python 3 (ipykernel)","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.9.17"}},"nbformat":4,"nbformat_minor":0} \ No newline at end of file +{"cells":[{"cell_type":"markdown","source":["\"Open

"],"metadata":{"id":"_kbesz9rsBo5"}},{"cell_type":"markdown","metadata":{"id":"Vkf1B-vMwpVB"},"source":["# Basic Operations"]},{"cell_type":"markdown","metadata":{"id":"fpPtZgqIvuXz"},"source":["Inspiration and some of the parts came from: Python Data Science [GitHub repository](https://github.com/jakevdp/PythonDataScienceHandbook/tree/master), [MIT License](https://github.com/jakevdp/PythonDataScienceHandbook/blob/master/LICENSE-CODE) and [Introduction to Pandas](https://colab.research.google.com/notebooks/mlcc/intro_to_pandas.ipynb) by Google, [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)\n","\n","If running this from Google Colab, uncomment the cell below and run it. Otherwise, just skip it."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"5saSBc40voZF"},"outputs":[],"source":["#!pip install seaborn\n","#!pip install watermark"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"TL7zwFk5sA2o"},"outputs":[],"source":["import pandas as pd\n","import numpy as np\n","import seaborn as sns"]},{"cell_type":"markdown","metadata":{"id":"ZkUd2sa-yP5e"},"source":["## Learning Objectives:\n","\n"," * operations\n"," * selection\n"," * filtering\n"," * concat\n"," * NaNs\n"]},{"cell_type":"markdown","metadata":{"id":"oa5wfZT7VHJl"},"source":["For this notebook, we will continue using the cities and california housing dataframes."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"Hn50qDhssA2p","outputId":"b7f1c937-012c-430a-b4d8-1beca9dcd07f"},"outputs":[{"data":{"text/html":["

\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
City namePopulation
0San Francisco852469
1San Jose1015785
2Sacramento485199
\n","
"],"text/plain":[" City name Population\n","0 San Francisco 852469\n","1 San Jose 1015785\n","2 Sacramento 485199"]},"execution_count":3,"metadata":{},"output_type":"execute_result"}],"source":["city_names = pd.Series(['San Francisco', 'San Jose', 'Sacramento'])\n","population = pd.Series([852469, 1015785, 485199])\n","\n","cities_dataframe = pd.DataFrame({ 'City name': city_names, 'Population': population })\n","cities_dataframe"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":206},"executionInfo":{"elapsed":400,"status":"ok","timestamp":1692082096397,"user":{"displayName":"Martin Schätz","userId":"14609383414092679868"},"user_tz":-120},"id":"av6RYOraVG1V","outputId":"acbcb340-2789-428a-c4dc-9c92b57d52c6"},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
longitudelatitudehousing_median_agetotal_roomstotal_bedroomspopulationhouseholdsmedian_incomemedian_house_value
0-114.3134.1915.05612.01283.01015.0472.01.493666900.0
1-114.4734.4019.07650.01901.01129.0463.01.820080100.0
2-114.5633.6917.0720.0174.0333.0117.01.650985700.0
3-114.5733.6414.01501.0337.0515.0226.03.191773400.0
4-114.5733.5720.01454.0326.0624.0262.01.925065500.0
\n","
"],"text/plain":[" longitude latitude housing_median_age total_rooms total_bedrooms \\\n","0 -114.31 34.19 15.0 5612.0 1283.0 \n","1 -114.47 34.40 19.0 7650.0 1901.0 \n","2 -114.56 33.69 17.0 720.0 174.0 \n","3 -114.57 33.64 14.0 1501.0 337.0 \n","4 -114.57 33.57 20.0 1454.0 326.0 \n","\n"," population households median_income median_house_value \n","0 1015.0 472.0 1.4936 66900.0 \n","1 1129.0 463.0 1.8200 80100.0 \n","2 333.0 117.0 1.6509 85700.0 \n","3 515.0 226.0 3.1917 73400.0 \n","4 624.0 262.0 1.9250 65500.0 "]},"execution_count":4,"metadata":{},"output_type":"execute_result"}],"source":["california_housing_dataframe = pd.read_csv(\"https://download.mlcc.google.com/mledu-datasets/california_housing_train.csv\", sep=\",\")\n","california_housing_dataframe.head()"]},{"cell_type":"markdown","metadata":{"id":"OzPb3cEJsA2r"},"source":["## Manipulating Data\n","\n","### Applying functions\n","\n","You may apply Python's basic arithmetic operations to `Series`. For example:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"ZPvYGXKisA2r","outputId":"f13f9c0e-cafd-4407-da38-e6d738a7b6b2"},"outputs":[{"data":{"text/plain":["0 852.469\n","1 1015.785\n","2 485.199\n","dtype: float64"]},"execution_count":5,"metadata":{},"output_type":"execute_result"}],"source":["population / 1000"]},{"cell_type":"markdown","metadata":{"id":"Z9QVl3qksA2s"},"source":["*pandas* `Series` can be used as arguments to most NumPy functions:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"FXAIj5tUsA2s","outputId":"72ac6c7b-c545-4a6b-ef05-5ded8445d398"},"outputs":[{"data":{"text/plain":["0 13.655892\n","1 13.831172\n","2 13.092314\n","dtype: float64"]},"execution_count":6,"metadata":{},"output_type":"execute_result"}],"source":["np.log(population)"]},{"cell_type":"markdown","metadata":{"id":"hRofbzevsA2s"},"source":["For more complex single-column transformations, you can use `Series.apply`. Like the Python [map function](https://docs.python.org/2/library/functions.html#map),\n","`Series.apply` accepts as an argument a [lambda function](https://docs.python.org/2/tutorial/controlflow.html#lambda-expressions), which is applied to each value.\n","\n","The example below creates a new `Series` that indicates whether `population` is over one million:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"2eN3aG_xsA2s","outputId":"36b769eb-77e3-441e-d8d8-5fe0a91b8d83"},"outputs":[{"data":{"text/plain":["0 False\n","1 True\n","2 False\n","dtype: bool"]},"execution_count":7,"metadata":{},"output_type":"execute_result"}],"source":["big_city = population.apply(lambda val: val > 1000000)\n","big_city"]},{"cell_type":"markdown","metadata":{"id":"XfzCh1qT1Pn3"},"source":["### Filtering"]},{"cell_type":"markdown","metadata":{"id":"sh7L0H2HsA2t"},"source":["One can use this result as a binary mask to make a sub-dataframe."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"XzNvFVotsA2t","outputId":"754ca69f-10c5-454d-cbd5-75fbf0825ec5"},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
City namePopulation
1San Jose1015785
\n","
"],"text/plain":[" City name Population\n","1 San Jose 1015785"]},"execution_count":8,"metadata":{},"output_type":"execute_result"}],"source":["cities_dataframe[big_city]"]},{"cell_type":"markdown","metadata":{"id":"6KDilny_sA2t"},"source":["Here is another way of generating a binary mask without explicitly using a `lamba` function."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"VpA39aYVsA2t","outputId":"0948b7dd-fc97-444a-fbb7-d9858a0e26fc"},"outputs":[{"data":{"text/plain":["0 False\n","1 True\n","2 False\n","Name: Population, dtype: bool"]},"execution_count":9,"metadata":{},"output_type":"execute_result"}],"source":["big_city = cities_dataframe['Population'] > 1000000\n","big_city"]},{"cell_type":"markdown","metadata":{"id":"5S6iUV5WsA2t"},"source":["### Adding new columns\n","\n","Modifying `DataFrames` is also straightforward. For example, the following code adds two `Series` to an existing `DataFrame`. One of them is the result of a computation of 2 existing columns."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"WOGb96mosA2t","outputId":"1721868d-eb6b-457b-881e-59662cd14f7f"},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
City namePopulationArea square milesPopulation density
0San Francisco85246946.8718187.945381
1San Jose1015785176.535754.177760
2Sacramento48519997.924955.055147
\n","
"],"text/plain":[" City name Population Area square miles Population density\n","0 San Francisco 852469 46.87 18187.945381\n","1 San Jose 1015785 176.53 5754.177760\n","2 Sacramento 485199 97.92 4955.055147"]},"execution_count":10,"metadata":{},"output_type":"execute_result"}],"source":["cities_dataframe['Area square miles'] = pd.Series([46.87, 176.53, 97.92])\n","cities_dataframe['Population density'] = cities_dataframe['Population'] / cities_dataframe['Area square miles']\n","cities_dataframe"]},{"cell_type":"markdown","metadata":{"id":"ZvqbT9tP030O"},"source":["### Concatenating DataFrames\n","\n","Let's imagine we collect another similar data sample, like the one below."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"7tYdJEJOsA2u","outputId":"845945fb-71c9-4be2-8674-f371d9c6132d"},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
City namePopulationArea square milesPopulation density
0Sao Paulo12400232587.3421112.527667
1Sao Luis1108975319.363472.491859
2Salvador2886698268.0010771.261194
\n","
"],"text/plain":[" City name Population Area square miles Population density\n","0 Sao Paulo 12400232 587.34 21112.527667\n","1 Sao Luis 1108975 319.36 3472.491859\n","2 Salvador 2886698 268.00 10771.261194"]},"execution_count":11,"metadata":{},"output_type":"execute_result"}],"source":["city_names = pd.Series(['Sao Paulo', 'Sao Luis', 'Salvador'])\n","population = pd.Series([12400232, 1108975, 2886698])\n","\n","another_cities_dataframe = pd.DataFrame({ 'City name': city_names, 'Population': population })\n","\n","another_cities_dataframe['Area square miles'] = pd.Series([587.34, 319.36, 268])\n","another_cities_dataframe['Population density'] = another_cities_dataframe['Population'] / another_cities_dataframe['Area square miles']\n","another_cities_dataframe"]},{"cell_type":"markdown","metadata":{"id":"mFZVreVhsA2u"},"source":["Before concatenating, it is probably a good idea to insert an identifier column so that we keep track where data came from.\n","\n","We can easily do that by creating a new column in each dataframe **before** concatenating."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"iIVZNMQTsA2u"},"outputs":[],"source":["cities_dataframe['Country'] = 'USA'\n","another_cities_dataframe['Country'] = 'Brazil'"]},{"cell_type":"markdown","metadata":{"id":"OO5Iw5TYsA2u"},"source":["We can now concatenate similar dataframes with the `pandas.concat` functions."]},{"cell_type":"code","execution_count":null,"metadata":{"executionInfo":{"elapsed":17,"status":"ok","timestamp":1692082096398,"user":{"displayName":"Martin Schätz","userId":"14609383414092679868"},"user_tz":-120},"id":"wlPXyLYw06UV","outputId":"76547221-538c-4d71-e86c-9871e7318ee2"},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
City namePopulationArea square milesPopulation densityCountry
0San Francisco85246946.8718187.945381USA
1San Jose1015785176.535754.177760USA
2Sacramento48519997.924955.055147USA
0Sao Paulo12400232587.3421112.527667Brazil
1Sao Luis1108975319.363472.491859Brazil
2Salvador2886698268.0010771.261194Brazil
\n","
"],"text/plain":[" City name Population Area square miles Population density Country\n","0 San Francisco 852469 46.87 18187.945381 USA\n","1 San Jose 1015785 176.53 5754.177760 USA\n","2 Sacramento 485199 97.92 4955.055147 USA\n","0 Sao Paulo 12400232 587.34 21112.527667 Brazil\n","1 Sao Luis 1108975 319.36 3472.491859 Brazil\n","2 Salvador 2886698 268.00 10771.261194 Brazil"]},"execution_count":13,"metadata":{},"output_type":"execute_result"}],"source":["result = pd.concat([cities_dataframe, another_cities_dataframe])\n","result"]},{"cell_type":"markdown","metadata":{"id":"WICecJHvsA2u"},"source":["We now have a longer dataframe with some repeated indices. To have unique indices, we can use `.reset_index`."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"BvRjm2vUsA2u","outputId":"0fe65ba1-8b8c-4ca4-d3b3-7a0674385dfa"},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
City namePopulationArea square milesPopulation densityCountry
0San Francisco85246946.8718187.945381USA
1San Jose1015785176.535754.177760USA
2Sacramento48519997.924955.055147USA
3Sao Paulo12400232587.3421112.527667Brazil
4Sao Luis1108975319.363472.491859Brazil
5Salvador2886698268.0010771.261194Brazil
\n","
"],"text/plain":[" City name Population Area square miles Population density Country\n","0 San Francisco 852469 46.87 18187.945381 USA\n","1 San Jose 1015785 176.53 5754.177760 USA\n","2 Sacramento 485199 97.92 4955.055147 USA\n","3 Sao Paulo 12400232 587.34 21112.527667 Brazil\n","4 Sao Luis 1108975 319.36 3472.491859 Brazil\n","5 Salvador 2886698 268.00 10771.261194 Brazil"]},"execution_count":14,"metadata":{},"output_type":"execute_result"}],"source":["result = result.reset_index(drop=True)\n","result"]},{"cell_type":"markdown","metadata":{"id":"uy2tR0XR3vbE"},"source":["### NaNs\n","\n","`DataFrame` objects can be created by passing a `dict` mapping `string` column names to their respective `Series`. If the `Series` don't match in length, missing values are filled with special [NA/NaN](http://pandas.pydata.org/pandas-docs/stable/missing_data.html) values. We cannot assume what these values are, because that would distort th results. So we need to deal with these NaNs values."]},{"cell_type":"markdown","metadata":{"id":"bGYpAX7t4bky"},"source":["We can test the missing values using `isnull()` function."]},{"cell_type":"markdown","metadata":{"id":"P0BwiPFhIbfV"},"source":["We can work with one of the `seaborn` training datasets *Penguins*"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"9Spm3StwImkL"},"outputs":[],"source":["penguins = sns.load_dataset(\"penguins\")"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":424},"executionInfo":{"elapsed":17,"status":"ok","timestamp":1692082096400,"user":{"displayName":"Martin Schätz","userId":"14609383414092679868"},"user_tz":-120},"id":"GYJkXKke42gp","outputId":"ab0c346a-952e-4fa1-aca9-e0d96c964520"},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
speciesislandbill_length_mmbill_depth_mmflipper_length_mmbody_mass_gsex
0FalseFalseFalseFalseFalseFalseFalse
1FalseFalseFalseFalseFalseFalseFalse
2FalseFalseFalseFalseFalseFalseFalse
3FalseFalseTrueTrueTrueTrueTrue
4FalseFalseFalseFalseFalseFalseFalse
........................
339FalseFalseTrueTrueTrueTrueTrue
340FalseFalseFalseFalseFalseFalseFalse
341FalseFalseFalseFalseFalseFalseFalse
342FalseFalseFalseFalseFalseFalseFalse
343FalseFalseFalseFalseFalseFalseFalse
\n","

344 rows × 7 columns

\n","
"],"text/plain":[" species island bill_length_mm bill_depth_mm flipper_length_mm \\\n","0 False False False False False \n","1 False False False False False \n","2 False False False False False \n","3 False False True True True \n","4 False False False False False \n",".. ... ... ... ... ... \n","339 False False True True True \n","340 False False False False False \n","341 False False False False False \n","342 False False False False False \n","343 False False False False False \n","\n"," body_mass_g sex \n","0 False False \n","1 False False \n","2 False False \n","3 True True \n","4 False False \n",".. ... ... \n","339 True True \n","340 False False \n","341 False False \n","342 False False \n","343 False False \n","\n","[344 rows x 7 columns]"]},"execution_count":16,"metadata":{},"output_type":"execute_result"}],"source":["penguins.isnull()"]},{"cell_type":"markdown","metadata":{"id":"m9jBF0A55OBn"},"source":["But it is more practical to test if there are any NaNs, than looking for it. We can use `.isnull().values.any()` approach."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":18,"status":"ok","timestamp":1692082096401,"user":{"displayName":"Martin Schätz","userId":"14609383414092679868"},"user_tz":-120},"id":"PucyKyW87geJ","outputId":"ba37a65a-dc0c-47a6-8bba-ed09be8ee1c7"},"outputs":[{"data":{"text/plain":["True"]},"execution_count":17,"metadata":{},"output_type":"execute_result"}],"source":["penguins.isnull().values.any()"]},{"cell_type":"markdown","metadata":{"id":"C5dD79015mYf"},"source":["Or we can explore each column using `.isnull().sum()`."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":17,"status":"ok","timestamp":1692082096401,"user":{"displayName":"Martin Schätz","userId":"14609383414092679868"},"user_tz":-120},"id":"GWIZ4l_z5tYQ","outputId":"bf5f4e7b-9e1c-406d-fade-2e543db8f0ff"},"outputs":[{"data":{"text/plain":["species 0\n","island 0\n","bill_length_mm 2\n","bill_depth_mm 2\n","flipper_length_mm 2\n","body_mass_g 2\n","sex 11\n","dtype: int64"]},"execution_count":18,"metadata":{},"output_type":"execute_result"}],"source":["penguins.isnull().sum()"]},{"cell_type":"markdown","metadata":{"id":"PcKxJ_vNLW3X"},"source":["We will want to drop all rows with unknown entries with `.dropna()` function."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":17,"status":"ok","timestamp":1692082096402,"user":{"displayName":"Martin Schätz","userId":"14609383414092679868"},"user_tz":-120},"id":"Vfqe-aMDMBpi","outputId":"6cd4d58f-78d6-47ff-bc4c-17423e4bd683"},"outputs":[{"data":{"text/plain":["species 0\n","island 0\n","bill_length_mm 0\n","bill_depth_mm 0\n","flipper_length_mm 0\n","body_mass_g 0\n","sex 0\n","dtype: int64"]},"execution_count":19,"metadata":{},"output_type":"execute_result"}],"source":["penguins_cleaned = penguins.dropna()\n","penguins_cleaned.isnull().sum()"]},{"cell_type":"markdown","metadata":{"id":"VksvzteBsA2z"},"source":["## Exercise\n","\n","The table below contains shape and intensity measurements from a biological sample. Make a subset with the columns `Area` and `Mean`. Remove all rows that contain NaNs from it and count the remaining rows.\n","\n","Afterwards, take the initial table again and make a new subset with the columns `Major` and `Minor`. Remove NaNs and count the remaining rows again.\n","\n","What do you conclude?"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"ftJB1JvXsA20","outputId":"e888baf3-5fdb-45a9-d860-f7f570e709e5"},"outputs":[{"data":{"text/html":["
\n","\n","\n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n"," \n","
AreaMeanStdDevMinMaxXYXMYMMajorMinorAngle%AreaType
118.0730.389103.354592.0948.0435.0004.722434.9624.6975.9873.828168.425100A
2126.0718.33390.367556.01046.0388.0878.683388.1838.68716.5599.688175.471100A
3NaNNaNNaN608.0964.0NaNNaNNaN7.6657.359NaN101.121100A
468.0686.98561.169571.0880.0126.1478.809126.1928.81115.1365.720168.133100A
5NaNNaN69.438566.0792.0348.5007.500NaN7.508NaN3.088NaN100A
.............................................
387152.0801.599111.328582.01263.0348.487497.632348.451497.67517.77310.88911.829100A
38817.0742.70669.624620.0884.0420.500496.382420.513NaNNaN3.66349.457100A
38960.0758.03377.309601.0947.0259.000499.300258.990499.2899.4768.06290.000100A
39012.0714.83367.294551.0785.0240.167498.167240.179498.1484.6063.317168.690100A
39123.0695.04367.356611.0846.049.891503.02249.882502.9796.4544.53773.243100A
\n","

391 rows × 14 columns

\n","
"],"text/plain":[" Area Mean StdDev Min Max X Y XM \\\n"," \n","1 18.0 730.389 103.354 592.0 948.0 435.000 4.722 434.962 \n","2 126.0 718.333 90.367 556.0 1046.0 388.087 8.683 388.183 \n","3 NaN NaN NaN 608.0 964.0 NaN NaN NaN \n","4 68.0 686.985 61.169 571.0 880.0 126.147 8.809 126.192 \n","5 NaN NaN 69.438 566.0 792.0 348.500 7.500 NaN \n",".. ... ... ... ... ... ... ... ... \n","387 152.0 801.599 111.328 582.0 1263.0 348.487 497.632 348.451 \n","388 17.0 742.706 69.624 620.0 884.0 420.500 496.382 420.513 \n","389 60.0 758.033 77.309 601.0 947.0 259.000 499.300 258.990 \n","390 12.0 714.833 67.294 551.0 785.0 240.167 498.167 240.179 \n","391 23.0 695.043 67.356 611.0 846.0 49.891 503.022 49.882 \n","\n"," YM Major Minor Angle %Area Type \n"," \n","1 4.697 5.987 3.828 168.425 100 A \n","2 8.687 16.559 9.688 175.471 100 A \n","3 7.665 7.359 NaN 101.121 100 A \n","4 8.811 15.136 5.720 168.133 100 A \n","5 7.508 NaN 3.088 NaN 100 A \n",".. ... ... ... ... ... ... \n","387 497.675 17.773 10.889 11.829 100 A \n","388 NaN NaN 3.663 49.457 100 A \n","389 499.289 9.476 8.062 90.000 100 A \n","390 498.148 4.606 3.317 168.690 100 A \n","391 502.979 6.454 4.537 73.243 100 A \n","\n","[391 rows x 14 columns]"]},"execution_count":21,"metadata":{},"output_type":"execute_result"}],"source":["dataframe = pd.read_csv('https://github.com/vmcf-konfmi/MB100T01/raw/main/data/Results.csv', index_col=0, delimiter=';')\n","dataframe"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"7IPdbT2NsA20"},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{"id":"fH1zusN7GKCx"},"source":["**Watermark**"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":3,"status":"ok","timestamp":1692082096620,"user":{"displayName":"Martin Schätz","userId":"14609383414092679868"},"user_tz":-120},"id":"iH1jL0baGMJW","outputId":"8639a749-8bab-4fd8-d6be-29230edccdea"},"outputs":[{"name":"stdout","output_type":"stream","text":["Last updated: 2023-08-24T14:25:40.097708+02:00\n","\n","Python implementation: CPython\n","Python version : 3.9.17\n","IPython version : 8.14.0\n","\n","Compiler : MSC v.1929 64 bit (AMD64)\n","OS : Windows\n","Release : 10\n","Machine : AMD64\n","Processor : Intel64 Family 6 Model 165 Stepping 2, GenuineIntel\n","CPU cores : 16\n","Architecture: 64bit\n","\n","watermark : 2.4.3\n","numpy : 1.23.5\n","pandas : 2.0.3\n","seaborn : 0.12.2\n","pivottablejs: 0.9.0\n","\n"]}],"source":["from watermark import watermark\n","watermark(iversions=True, globals_=globals())\n","print(watermark())\n","print(watermark(packages=\"watermark,numpy,pandas,seaborn,pivottablejs\"))"]}],"metadata":{"colab":{"provenance":[],"toc_visible":true},"kernelspec":{"display_name":"Python 3 (ipykernel)","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.9.17"}},"nbformat":4,"nbformat_minor":0}