Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minimal scraper working from end-to-end #1

Merged
merged 1 commit into from
Oct 3, 2024
Merged

Conversation

benoit74
Copy link
Contributor

@benoit74 benoit74 commented Sep 27, 2024

Fix #2

This is the first PR putting mostly everything in place for this scraper:

  • Python code
    • minimal structure, with very small client allowing to fetch icon + description from a Libretexts library home page
  • Vue.JS code
    • minimal structure, with support for old browsers, based on Youtube ZIM UI
  • Github workflows
    • based on python-bootstrap with addition for Vue.JS
  • Documentation

End-to-end test:

docker build -t local-libretexts2zim .
docker run --rm -it -v "$PWD/output":/output local-libretexts2zim libretexts2zim --library-slug geo --library-name Geosciences --file-name-format "tests_en_libretexts-geo"

This will create a tests_en_libretexts-geo.zim file in output subfolder, ready to use in kiwix-serve or any other reader.

@benoit74 benoit74 self-assigned this Sep 27, 2024
Copy link

codecov bot commented Sep 27, 2024

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered ☂️

@benoit74 benoit74 force-pushed the minimal_structure branch 3 times, most recently from d11f364 to 4f4c5ba Compare September 27, 2024 08:54
@benoit74 benoit74 marked this pull request as ready for review September 27, 2024 08:56
@benoit74 benoit74 requested a review from rgaudin September 27, 2024 08:56
@benoit74 benoit74 mentioned this pull request Sep 27, 2024
Copy link
Member

@rgaudin rgaudin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finished! Most of my comments are on the python side where I have more experience. I did look at the frontends but it's both limited and good looking.

.vscode/settings.json Outdated Show resolved Hide resolved
CONTRIBUTING.md Show resolved Hide resolved
scraper/src/libretexts2zim/entrypoint.py Show resolved Hide resolved
scraper/src/libretexts2zim/entrypoint.py Outdated Show resolved Hide resolved
scraper/src/libretexts2zim/generator.py Outdated Show resolved Hide resolved
scraper/src/libretexts2zim/generator.py Outdated Show resolved Hide resolved
scraper/src/libretexts2zim/generator.py Outdated Show resolved Hide resolved
zimui/src/components/HeaderBar.vue Outdated Show resolved Hide resolved
zimui/src/stores/counter.ts Outdated Show resolved Hide resolved
@benoit74 benoit74 force-pushed the minimal_structure branch 2 times, most recently from b569a7d to 46e6072 Compare October 3, 2024 12:06
@benoit74 benoit74 requested a review from rgaudin October 3, 2024 12:49
Copy link
Member

@rgaudin rgaudin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All good

@benoit74 benoit74 merged commit 9fa9b7a into main Oct 3, 2024
8 checks passed
@benoit74 benoit74 deleted the minimal_structure branch October 3, 2024 14:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Create minimal scraper
2 participants