Skip to content

Commit

Permalink
Merge pull request #1237 from tilburgsciencehub/organize/python-section
Browse files Browse the repository at this point in the history
  • Loading branch information
alexandervossen authored Aug 9, 2024
2 parents f1db6e7 + 019b196 commit 17dbd76
Show file tree
Hide file tree
Showing 6 changed files with 212 additions and 102 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: "Python Coding Style Guidelines"
description: "Follow nice language-specific coding styles to make your code easier to understand and easier to the eye."
keywords: "style, code, python, best practices, guidelines"
weight: 2
weight: 4
date: 2020-11-11T22:01:14+05:30
draft: false
aliases:
Expand All @@ -12,9 +12,15 @@ aliases:

# Overview

We follow Python's style guide [PEP 8](https://www.python.org/dev/peps/pep-0008/#programming-recommendations), emphasizing readability and consistency. In this topic, we highlight some of the best practices with code examples.
We follow Python's style guide [PEP 8](https://www.python.org/dev/peps/pep-0008/#programming-recommendations), emphasizing readability and consistency. In this topic, we highlight some of the best practices with code examples. Note that sometimes your own judgment is required, and what suits best for your project does not always comply with the guidelines!


{{% tip %}}
- If you haven't installed Python yet, refer to our [Set up Guide](/install/python)
- [Get Started With Python](/learn/python) through simple examples of the fundamental basics, such as variables and data types, functions, etcetera.

{{% /tip %}}

Note that sometimes your own judgment is required, and what suits best for your project does not always comply with the guidelines!

## Straightforward code

Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: "Configuring Python for Web Scraping"
title: "Configure Python for Web Scraping"
description: "Learn to web scrape using ChromeDriver, Selenium, and Webdriver manager for python. Web scraping using an automated browser by installing ChromeDrive"
keywords: "web scraping, scraping, automation, browser, chromedriver, install chromedriver, selenium"
weight: 5
Expand All @@ -10,24 +10,25 @@ aliases:
- /get/chromedriver
---

# Web Scraping Using an Automated Browser
## Overview

Sometimes when we scrape the web, we need to automate our computer to open a web browser to gather information from each page. This is especially true when the site we want to scrape has content that is loaded dynamically with javascript.
Web scraping often requires automating a web browser to gather information, especially when dealing with sites that dynamically load content using JavaScript.

We will install one package to help us here: [ChromeDriver](https://chromedriver.chromium.org). Below we show two different ways of installing it.
This guide will show you how to set up your environment to use the package [ChromeDriver](https://chromedriver.chromium.org) with Selenium and WebDriver Manager for Python. We will show you two different ways of installing these tools below.

## Install ChromeDriver

In order to install ChromeDriver, make sure you have already installed:
Before installing ChromeDriver, make sure you have the following already installed:

* [Google Chrome](https://www.google.com/chrome/browser/desktop/index.html).

* **Selenium**: by typing in the command `pip install selenium`.
- Alternatively, open Anaconda Prompt (Windows) or the Terminal (Mac), type the command `conda install selenium`, and agree to whatever the package manager wants to install or update (usually by pressing `y` to confirm your choice).
* *Selenium*: Install it using the command `pip install selenium`.
- Alternatively, you can install it via Anaconda Prompt (Windows) or the Terminal (Mac) with the command `conda install selenium`, and agreeing to whatever the package manager wants to install or update (usually by pressing `y` to confirm your choice).

* **Webdriver Manager for Python**: by typing in the command `pip install webdriver_manager`
* *Webdriver Manager for Python*: Install it using the command `pip install webdriver_manager`.

Once these packages are installed, you can set up ChromeDriver with the following Python code:

Once you have obtained these packages, you can now install ChromeDriver as follows:
{{% codeblock %}}
```Python
# Make selenium and chromedriver work for Untappd.com
Expand All @@ -46,7 +47,8 @@ driver.get(url)
{{% /codeblock %}}

## Manually Installing ChromeDriver
If for any reason the prior did not work or if you simply prefer installing ChromeDriver manually, follow the operating-system-specific steps below.

If the automatic installation does not work, or if you simply prefer to install ChromeDriver manually, follow the instructions for your operating system below.

### Windows Users

Expand All @@ -56,13 +58,13 @@ Watch our YouTube video, in which we walk you through the setup on Windows.

* Install Google Chrome from [here](https://www.google.com/chrome/browser/desktop/index.html).
* Download the windows version of Chromedriver from [here](https://chromedriver.storage.googleapis.com/index.html?path=2.41/).
* Extract the contents from the zip file, and extract them into a new directory under `C:\chromedriver`. If you do not have admin rights, you can put the file also in another folder, for example `C:\Program Files\chromedriver`, or `C:\Users\[your-username]\chromedriver`. It does not matter where exactly the file will be put, as long as you remember where it is (it's not a good idea though to leave it in your downloads folder).
* Extract the contents of the zip file into a new directory under `C:\chromedriver`. If you do not have admin rights, you can use a different folder, like `C:\Program Files\chromedriver`, or `C:\Users\[your-username]\chromedriver`. The exact location of the file does not matter, as long as you remember where you placed it. (However, it's not advisable to leave it in your Downloads folder.)
* Make sure that the chromedriver.exe file is directly under the PATH you specified, i.e. under `C:\chromedriver` (or an alternative path). If your zip unpacker created a new folder with a different name inside your specified folder, move the .exe file to that path.
* Add the directory `C:\chromedriver` (or whichever directory you chose above) to your PATH as described before (for instructions, see below)
* If this went successfully, open a terminal/command prompt, and enter `chromedriver --version`, you should get output that looks like `ChromeDriver [version number]`

{{% warning %}}
**Making `chromedriver` available via the PATH settings on Windows.**
*Adding ChromeDriver to PATH on Windows.*

We need to update our PATH settings; these settings are a set of directories that Windows uses to "look up" software to startup.

Expand Down Expand Up @@ -90,29 +92,35 @@ We need to update our PATH settings; these settings are a set of directories tha

#### Let's install Homebrew first!

Make sure your `Homebrew` package is up-to-date. To do so, open a terminal and enter
* Make sure your `Homebrew` package is installed and up-to-date. To do so, open a terminal and enter:

```
{{% codeblock %}}
```bash
brew update
```
{{% /codeblock %}}

If that returns an error, `Homebrew` is not installed.
If `Homebrew` is not installed, an error will return.

- To install Homebrew, open a terminal and paste the following command:
* To install Homebrew, open a terminal and paste the following command:

```
{{% codeblock %}}
```bash
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
```
{{% /codeblock %}}

- To verify that Homebrew installed correctly, enter the following into your terminal
* Verify the installation by entering the following command into your terminal:

```
{{% codeblock %}}
```bash
brew doctor
```
{{% /codeblock %}}

...and you should see the following output
You should see the following output:

```
```bash
Your system is ready to brew
```

Expand All @@ -122,36 +130,50 @@ Sometimes, `brew doctor` returns some warnings. While it's advisable to fix them

* We assume you have Google Chrome installed. If not, do this first, please.

* Install `chromedriver` via Homebrew:
* Install `chromedriver` via Homebrew with:

```
{{% codeblock %}}
```bash
brew install chromedriver --cask
```
{{% /codeblock %}}

* Verify your install, by entering the following in your terminal. The expected output is `ChromeDriver XX`
* Verify the installation by running the following in the terminal:

```
```bash
chromedriver --version
```

The expected output is the version number (`ChromeDriver XX`).

### Linux Users

* Open a terminal session
* Install Google Chrome for Debian/Ubuntu by pasting the following and then pressing `Return`
```
* Install Google Chrome for Debian/Ubuntu by pasting the following and then pressing `Return`:

{{% codeblock %}}
```bash
sudo apt-get install libxss1 libappindicator1 libindicator7
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb

sudo dpkg -i google-chrome*.deb
sudo apt-get install -f
```
{{% /codeblock %}}

* Install `xvfb` so chrome can run 'headless' by pasting the following and then pressing `Return`
```
* Install `xvfb` for "headless" browsing with:

{{% codeblock %}}
```bash
sudo apt-get install xvfb
```
{{% /codeblock %}}

* Install Chromedriver by pasting the following and then pressing `Return`:
```

* Install Chromedriver with:

{{% codeblock %}}
```bash
sudo apt-get install unzip

wget -N https://chromedriver.storage.googleapis.com/2.41/chromedriver_linux64.zip
Expand All @@ -162,7 +184,14 @@ sudo mv -f chromedriver /usr/local/share/chromedriver
sudo ln -s /usr/local/share/chromedriver /usr/local/bin/chromedriver
sudo ln -s /usr/local/share/chromedriver /usr/bin/chromedriver
```
* Your install worked, you should get `ChromeDriver XX` returned if the installation was successful
```
{{% /codeblock %}}

* Verify the installation by running the following in the terminal:

{{% codeblock %}}
```bash
chromedriver --version
```
{{% /codeblock %}}

You should see the version number if the installation was successful.
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: "Learn Python"
title: "Introduction to Coding With Python"
description: "Learn how to program in Python."
keywords: "python, data science, data, programming, manipulation, analysis, tutorial"
draft: false
Expand All @@ -12,7 +12,18 @@ aliases:

## Overview

We introduce you to Python, with the fundamental basics which prepare you to start using and experimenting with this cool programming language yourself! Once you understand the basics, the opportunities of Python are almost infinite.
We introduce you to [Python]((https://www.python.org/)), starting with the fundamental basics to help you begin experimenting with this cool programming language yourself! Once you understand these essentials, the opportunities of Python are almost infinite. This guide covers:

- Variables
- Data types
- Operators
- Functions
- Loops

{{% tip %}}

If you haven't installed Python yet, refer to our [Set up Guide](/install/python)
{{% /tip %}}

## Variables

Expand Down Expand Up @@ -625,10 +636,15 @@ Output:
5
```

## See more
{{% summary %}}

With the basics of Python under your belt, you're ready to start experimenting and expanding your skills. Check out these resources to further your learning:

- [Python Coding Style Guidelines](python/style-guides): Discover the importance of coding style with these essential guidelines.
- Learn Python with these great step-by-step tutorials from [Real Python](https://realpython.com).
- [QuantEcon.org](https://quantecon.org/) offers fantastic resources for you to
get started using Python
get started using Python.

In particular, we would like to refer you to their [open-source lectures](https://quantecon.org/lectures/), covering: [Quantitative Economics with Python](https://python.quantecon.org/), including a fantastic, open-source book companion, and [QuantEcon Data Science](https://datascience.quantecon.org/).

In particular, we would like to refer you to their [open-source lectures](https://quantecon.org/lectures/), covering: [Quantitative Economics with Python](https://python.quantecon.org/), including a fantastic, open-source book companion, and [QuantEcon Data Science](https://datascience.quantecon.org/).
{{% /summary %}}
Original file line number Diff line number Diff line change
Expand Up @@ -4,87 +4,84 @@ description: "Learn how to install new Python packages with package management t
keywords: "python, pip, packages, selenium"
#date: 2020-11-11T22:02:51+05:30
draft: false
weight: 4
weight: 2
aliases:
- /get/python-packages
- /get/pip
- /install/python-packages
---

## What are Python packages
## Overview

Python packages are collections of modules and functions that provide additional functionality beyond what is available in the Python standard library. They allow developers to easily extend Python's capabilities by including pre-written code for specific tasks, such as data analysis, web development, machine learning, and more.
[Python packages](/https://docs.python.org/3/tutorial/modules.html#packages) are collections of modules and functions that provide additional functionalities beyond the standard library. They enable developers to extend Python's capabilities easily with pre-written code for specific tasks, such as data analysis, web development, machine learning, and more. This guide will teach you how to install and update Python packages.

## Installing Python packages

Anaconda's Python Distribution comes with many of the packages we need to do scientific computing. If you're interested in all the packages included, click [here](https://docs.continuum.io/anaconda/packages/pkg-docs) and go to the Python 3.x tab.
The [Anaconda Python Distribution](/install/python) includes many of the packages we need for scientific computing. If you're interested in all the included packages, click [here](https://docs.continuum.io/anaconda/packages/pkg-docs) and select the `Python 3.x` tab.

However, you may come across packages that are not installed by default. In this case we recommend you use the `pip` package management tool to install them.
### Installation steps
However, you may come across packages that are not installed by default. In such cases, we recommend using the `pip` package management tool to install them.

{{% warning %}}
If your python 3 was found via `python3 --version` on the previous page, then type `pip3` instead of `pip` for all of the following python plugins.

If you verified your Python installation in the [setup process](/get/python) using `python3 --version`, use `pip3` instead of `pip` for all the following commands.

{{% /warning %}}

First let us update pip by typing the following into the terminal
### The installation steps

First, update `pip` by typing the following command in the terminal:

{{% codeblock %}}
```bash
pip install --upgrade pip
```
{{% /codeblock %}}

If you get an error, try the following command instead:

If you get an error, try typing instead:
{{% codeblock %}}
```bash
python -m pip install --upgrade pip
```
{{% /codeblock %}}


You also need the package `Selenium` as part of the web scraping tool kit we will build up. First let us install a depency for it via
To build your web scraping toolkit, you will need the `selenium` package. Start by installing a dependency:

{{% codeblock %}}
```bash
pip install msgpack
```
{{% /codeblock %}}

Next, install `selenium` by entering:

We then install `selenium` by entering the following into a terminal:
{{% codeblock %}}
```bash
pip install selenium
```
{{% /codeblock %}}


`pip` will then go through and install the package we asked for, and any other dependencies.
If this succeeded, the last line it printed out should be:
`pip` will install the requested package and any necessary dependencies. If this succeeded, the last line displayed will be:

```bash
Successfully installed selenium-4.x.0
```


## Updating packages

In order to update already installed packages on your system use the following command in your terminal:
To update an already installed package on your system, use the following command in your terminal:

{{% codeblock %}}
```bash
pip install --upgrade selenium
```
{{% /codeblock %}}

### Common problems

{{% tip %}}
**No administrator rights?**

If you do not have administrator rights to the computer you are using,
please install packages only for your account.
You can do so by typing `pip install --user packagename`, e.g., `pip install --user selenium`.
*No administrator rights?*

If you do not have administrator rights on the computer you are using, install packages only for your account. You can do this by typing `pip install --user packagename`, e.g., `pip install --user selenium`.

{{% /tip %}}
Loading

0 comments on commit 17dbd76

Please sign in to comment.