Technical design documentation
This is the main class of this application. Core.py handles logging, DB-Connection and overall scraping for the 25 given Vendors. It does so by using the following Methods …
core = Core( vendor_list=vendor_list, logger=logger )
Initializes Vendors-to-run. Not for every run we want to scrape every 25 Vendor Websites for information.
Core.get_product_catalog()
The Core-Module initializes the Vendor-specific Scraping Class and extracts all necessary information from given Vendor-specific website. Scraping takes place with Python and the Selenium package. The result includes Firmwares Download-Link, Versionnumber, Product-Line, Product-Name, potentially scraped MD5-checksum as a List[dict].
This firmware's Metadata List of dictionaries is then temporarily saved in the DB as seperate table.
db.compare_products()
- Comparison between Vendor's firmware metadata List[dict] with DB's
- Based on MINUS Operator (LEFT JOIN)
- Keys are firmware Versionnumber, scraped-MD5_checksum, …
- returns firmware metadata that is unknown to the DB
- thus this unknown firmware gets downloaded
Core.download_firmware()
Triggers download of new firmwares into downloads/ Since some firmware files are several hundred megabytes, we recommend to have sufficient storage.
CLI
You can contribute to this project by adding more Vendors.
create a Vendors.py in src/Vendor
touch init.py in that folder (create empty init.py file in same folder)
add your Vendor in src/init.py accordingly
add your Vendor into the src/config.json file