Add documentation

AlexMili · Dec 11, 2024 · d8970a6 · d8970a6
1 parent 1fccdce
commit d8970a6
Show file tree

Hide file tree

Showing 7 changed files with 285 additions and 9 deletions.
diff --git a/.github/workflows/deploy_doc.yml b/.github/workflows/deploy_doc.yml
@@ -0,0 +1,31 @@
+name: Deploy documentation
+
+on:
+  push:
+    branches:
+      - main
+
+permissions:
+  contents: write
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Configure Git Credentials
+        run: |
+          git config user.name github-actions[bot]
+          git config user.email github-actions[bot]@users.noreply.github.com
+      - uses: actions/setup-python@v5
+        with:
+          python-version: 3.x
+      - run: echo "cache_id=$(date --utc '+%V')" >> $GITHUB_ENV 
+      - uses: actions/cache@v4
+        with:
+          key: mkdocs-material-${{ env.cache_id }}
+          path: .cache
+          restore-keys: |
+            mkdocs-material-
+      - run: pip install -r requirements-docs.txt 
+      - run: mkdocs gh-deploy --force
diff --git a/docs/index.md b/docs/index.md
@@ -0,0 +1,123 @@
+# Extract Favicon
+
+---
+
+**Documentation**: <a href="https://alexmili.github.io/extract_favicon" target="_blank">https://alexmili.github.io/extract_favicon</a>
+
+**Source Code**: <a href="https://github.com/alexmili/extract_favicon" target="_blank">https://github.com/alexmili/extract_favicon</a>
+
+---
+
+**Extract Favicon** is designed to easily retrieve favicons from any website. Built atop robust `reachable` and `BeautifulSoup`, it aims to deliver accurate and efficient favicon extraction for web scraping and data analysis workflows.
+
+Key features include:
+
+* **Automatic Extraction**: Detects multiple favicon references like `<link>`, `<meta>` and inline base64-encoded icons.
+* **Smart Fallbacks**: When explicit icons aren’t defined, it checks standard fallback routes (like `favicon.ico`) to provide consistent results even on sites without standard declarations.
+* **Size Guessing**: Dynamically determines favicon dimensions, even for images lacking explicit size information, by partially downloading and parsing their headers.
+* **Base64 Support**: Easily handles inline data URLs, decoding base64-encoded images and validating them on-the-fly.
+* **Availability Checks**: Validates each favicon’s URL, following redirects and marking icons as reachable or not.
+* **Async Support**: Offers asynchronous methods (via `asyncio`) to efficiently handle multiple favicon extractions concurrently, enhancing overall performance when dealing with numerous URLs.
+
+## Installation
+
+Create and activate a virtual environment and then install `extract_favicon`:
+
+```console
+$ pip install extract_favicon
+```
+
+## Usage
+
+
+### Extracting Favicons from HTML
+
+The `from_html` function allows you to parse a given HTML string and extract all favicons referenced within it. It looks for common `<link>` and `<meta>` tags that reference icons (e.g., `icon`, `shortcut icon`, `apple-touch-icon`, etc.). If `include_fallbacks` is set to `True`, it will also check standard fallback paths like `favicon.ico` when no icons are explicitly defined.
+
+**Example:**
+```python
+html_content = """
+<!DOCTYPE html>
+<html>
+<head>
+  <link rel="icon" href="https://example.com/favicon.ico" />
+  <link rel="apple-touch-icon" href="/apple-touch-icon.png">
+</head>
+<body>
+  <p>Sample page</p>
+</body>
+</html>
+"""
+
+favicons = from_html(html_content, root_url="https://example.com", include_fallbacks=True)
+for favicon in favicons:
+    print(favicon.url, favicon.width, favicon.height)
+```
+
+### Extracting Favicons from a URL
+
+If you only have a URL and want to directly extract favicons, `from_url` fetches the page, parses it, and returns a set of `Favicon` objects. It uses `Reachable` internally to check if the URL is accessible. If `include_fallbacks` is True, fallback icons (like `/favicon.ico`) are also considered.
+
+```python
+favicons = from_url("https://example.com", include_fallbacks=True)
+for favicon in favicons:
+    print(favicon.url, favicon.format, favicon.width, favicon.height)
+```
+
+### Downloading Favicons
+
+Depending on the mode, you can choose to download:
+
+* "all": Download all favicons.
+* "biggest": Download only the largest favicon (by area).
+* "smallest": Download only the smallest favicon.
+
+If `include_unknown` is False, favicons without known dimensions are skipped. The `sort` option sorts the returned favicons by size, and `sleep_time` controls how long to wait between requests to avoid rate limits.
+
+The result is a list of `RealFavicon` objects, which contain additional information like the loaded image or raw SVG data.
+
+```python
+favicons = from_url("https://example.com")
+real_favicons = download(favicons, mode="all", sort="DESC")
+
+for real_favicon in real_favicons:
+    print(real_favicon.url.url, real_favicon.valid, real_favicon.width, real_favicon.height)
+```
+
+### Checking Favicon Availability
+
+Sends a HEAD request for each favicon URL to determine if it’s reachable. If the favicon has been redirected, it updates the URL accordingly. It also sets the reachable attribute on each Favicon. The `sleep_time` parameter lets you pause between checks to reduce the load on the target server.
+
+```python
+favicons = from_url("https://example.com")
+checked_favicons = check_availability(favicons)
+
+for favicon in checked_favicons:
+    print(favicon.url, favicon.reachable)
+```
+
+### Guessing Favicon Sizes
+
+If some extracted favicons don’t have their dimensions specified, `guess_missing_sizes` can attempt to determine their width and height. For base64-encoded favicons (data URLs), setting `load_base64_img` to `True` allows the function to decode and load the image in memory to get its size. For external images, it partially downloads the image to guess its dimensions without retrieving the entire file.
+
+```python
+favicons = from_url("https://example.com")
+# Some favicons may not have width/height info
+favicons_with_sizes = guess_missing_sizes(favicons, load_base64_img=True)
+
+for favicon in favicons_with_sizes:
+    print(favicon.url, favicon.width, favicon.height)
+```
+
+## Dependencies
+
+When you install `extract_favicon` it comes with the following dependencies:
+
+* <a href="https://www.crummy.com/software/BeautifulSoup" target="_blank"><code>BeautifulSoup</code></a> - to parse HTML content.
+* <a href="https://github.com/python-pillow/Pillow" target="_blank"><code>Pillow</code></a> - to load images to get real size once downloaded and to guess image size based on its streamed headers.
+* <a href="https://github.com/alexmili/reachable" target="_blank"><code>Reachable</code></a> - to check availability of favicons' URLs, download content and handle redirects, HTTP errors and some simple anti-bot protections.
+* <a href="https://github.com/tiran/defusedxml" target="_blank"><code>DefusedXML</code></a> - to parse and check validity of SVG files.
+
+## License
+
+This project is licensed under the terms of the MIT license.
diff --git a/docs/reference.md b/docs/reference.md
@@ -0,0 +1 @@
+::: extract_favicon.main
diff --git a/mkdocs.yml b/mkdocs.yml
@@ -0,0 +1,115 @@
+site_name: Extract Favicon
+site_description: Extract favicon of any website.
+repo_name: alexmili/extract_favicon
+repo_url: https://github.com/alexmili/extract_favicon
+copyright: Copyright &copy; 2024 Alexandre Milisavljevic
+theme:
+    name: material
+    palette:
+        - media: "(prefers-color-scheme)"
+          toggle:
+            icon: material/lightbulb-auto
+            name: Switch to light mode
+        - media: '(prefers-color-scheme: light)'
+          scheme: default
+          accent: amber
+          toggle:
+            icon: material/lightbulb
+            name: Switch to dark mode
+        - media: '(prefers-color-scheme: dark)'
+          scheme: slate
+          accent: amber
+          toggle:
+            icon: material/lightbulb-outline
+            name: Switch to system preference
+    features:
+        - content.code.annotate
+        - content.code.copy
+        # - content.code.select
+        - content.footnote.tooltips
+        - content.tabs.link
+        - content.tooltips
+        - navigation.footer
+        - navigation.indexes
+        - navigation.instant
+        - navigation.instant.prefetch
+        # - navigation.instant.preview
+        - navigation.instant.progress
+        - navigation.path
+        - navigation.tabs
+        - navigation.tabs.sticky
+        - navigation.top
+        - navigation.tracking
+        - search.highlight
+        - search.share
+        - search.suggest
+        - toc.follow
+
+    # Icon to use for repo_url on top right
+    icon:
+        repo: fontawesome/brands/github-alt
+
+extra:
+    social:
+        - icon: fontawesome/brands/github-alt
+          link: https://github.com/alexmili/extract_favicon
+        - icon: fontawesome/brands/linkedin
+          link: https://www.linkedin.com/in/amili
+plugins:
+    search:
+    mkdocstrings:
+        handlers:
+            python:
+                options:
+                    show_source: false
+                    show_root_heading: false
+                    show_if_no_docstring: false
+                    show_root_toc_entry: false
+                    filters: ["!^_"]
+                    members_order: source
+nav:
+- Extract Favicon: index.md
+- reference.md
+
+markdown_extensions:
+    # Python Markdown
+    abbr:
+    attr_list:
+    footnotes:
+    md_in_html:
+    tables:
+    toc:
+        permalink: true
+
+    # Python Markdown Extensions
+    pymdownx.betterem:
+    pymdownx.caret:
+    pymdownx.highlight:
+        line_spans: __span
+    pymdownx.inlinehilite:
+    pymdownx.keys:
+    pymdownx.mark:
+    pymdownx.superfences:
+        custom_fences:
+            - name: mermaid
+              class: mermaid
+              format: !!python/name:pymdownx.superfences.fence_code_format
+    pymdownx.tilde:
+
+    # pymdownx blocks
+    pymdownx.blocks.admonition:
+        types:
+            - note
+            - attention
+            - caution
+            - danger
+            - error
+            - tip
+            - hint
+            - warning
+            # Custom types
+            - info
+            - check
+    pymdownx.blocks.details:
+    pymdownx.tabbed:
+        alternate_style: true
diff --git a/pyproject.toml b/pyproject.toml
@@ -16,17 +16,17 @@ classifiers = [
 ]
 keywords = []
 dependencies = [
-    "bs4",
+    "beautifulsoup4",
     "pillow",
     "reachable",
     "defusedxml",
 ]
 
 [project.urls]
-Homepage = "https://github.com/AlexMili/Extract_Favicon"
+Homepage = "https://alexmili.github.io/extract_favicon"
 Issues = "https://github.com/AlexMili/Extract_Favicon/issues"
 Repository = "https://github.com/AlexMili/Extract_Favicon"
-Documentation = "https://github.com/AlexMili/Extract_Favicon"
+Documentation = "https://alexmili.github.io/extract_favicon"
 
 
 [tool.hatch.build.targets.wheel]

diff --git a/requirements-docs.txt b/requirements-docs.txt
@@ -0,0 +1,7 @@
+mkdocs
+# For the theme
+mkdocs-material
+mkdocs-material-extensions
+# For auto generating reference API pages
+mkdocstrings[python]
+griffe-typingdoc
diff --git a/src/extract_favicon/main.py b/src/extract_favicon/main.py
@@ -154,8 +154,7 @@ def from_html(
     Args:
         html: HTML to parse.
         root_url: Root URL where the favicon is located.
-        include_default_favicon: Include /favicon.ico in the list when no other
-            favicons have been found.
+        include_fallbacks: Whether to include fallback favicons like `/favicon.ico`.
 
     Returns:
         A set of favicons.
@@ -254,7 +253,7 @@ def _get_root_url(url: str) -> str:
 
     Returns:
         The root URL, including the scheme and netloc, but without any
-        additional paths, queries, or fragments.
+            additional paths, queries, or fragments.
     """
     parsed_url = urlparse(url)
     url_replaced = parsed_url._replace(query="", path="")
@@ -551,7 +550,7 @@ def guess_size(favicon: Favicon, chunk_size: int = 512) -> Tuple[int, int]:
 
 def guess_missing_sizes(
     favicons: Union[list[Favicon], set[Favicon]],
-    chunk_size=512,
+    chunk_size: int = 512,
     sleep_time: int = 1,
     load_base64_img: bool = False,
 ) -> list[Favicon]:
@@ -616,7 +615,7 @@ def check_availability(
     favicons: Union[list[Favicon], set[Favicon]],
     sleep_time: int = 1,
     client: Optional[Client] = None,
-):
+) -> list[Favicon]:
     """
     Checks the availability and final URLs of a collection of favicons.
 
@@ -638,7 +637,7 @@ def check_availability(
 
     Returns:
         A list of `Favicon` objects with updated `reachable` statuses and potentially
-        updated URLs if redirects were encountered.
+            updated URLs if redirects were encountered.
     """
     favs = list(favicons)