Releases: Lookyloo/lookyloo
Releases · Lookyloo/lookyloo
February release, if it was a bissextile year
New features
- Beta support for monitoring: the system will trigger a recurring capture and allow to compare them over time.
- Beta support for comparing captures. For now, it focuses on the redirects from the URL captured to the landing page, and the URLs of the resources loaded on that page. An upcoming release of this feature will compare the rendered content of the landing page too.
- Takedown endpoint (API only): gather contact information (whois on domain, IP, ASN, and security.txt file if available) for all the URLs up to the landing page. Can be used to notify owners in case of a malicious URL.
- Flag known Cloudflare IPs on the hostnode popups
- Trigger AAAA DNS requests (was only A before)
Bugfixes
- When the capture is a file that was uploaded by the user, some modules cannot be triggered. Avoiding exceptions.
- In some edge cases, a lock file for generating the tree could be left there even if the initiating script was dead. We now clean them up automatically.
Changes
- Use Lacus v1.3.0 / LacusCore v1.3.0 / PlaywrightCapture v1.18.0
- Allow to disable defanging URLs in emails
- Many improvements in the rendering of the menus on the tree page
- [Lookyloo] Bump javascript and python dependencies
- Har2Tree Maintenance, update dependencies
HoHoHoliday season release
Breaking change
Poetry v1.3.0 or more recent is now required, please upgrade to the latest version.
New features:
- See Lacus release notes for v1.2.0 if you want more details on the changes regarding the captures.
- Temporary local storage of captures if Lacus web-service isn't reachable temporarily
- Submit and view a HAR file captured somewhere else, or a full Lookyloo capture from an other instance
- Show status of captures when they're submitted in bulk
- List of all downloadable contents for a capture in a modal (from the tree view ->
Download capture elements):
Bugfixes
- Catch and stop script when generating a tree takes too long
- [har2tree] Avoid exception when a node doesn't have a pageref
- [har2tree] Better use of BeautifulSoup
- [har2tree] Better handling of nodes that aren't loading a URL
- [har2tree] Improve mimetime detection in HTTP responses
Changes
- [har2tree] Improve decoding of POSTed data
- Use more recent Flask and flask-restx
- Bump deps (Python ans JS)
- Major logging improvements in Lookyloo, LacusCore, PlaywrightCapture and har2tree
October release - Spooky capture at a distance
Breaking change
This release requires poetry v1.2.0 or more recent. Run the following command to upgrade it: poetry self update
New Features
- Move to Lacus/LacusCore, many changes to make lookyloo's code compatible with it
With lacus, the captures are more reliable and using Lacus as a web service allows to monitor them better. If you want to use the webservice, you'll need to:
- Install Lacus
- make sure it is running by loading http://127.0.0.1:7100 on the machine you have it running on (7100 is the default port, you can of course change it)
- Edit the config file
config/generic.json(keyremote_lacus):- set
enabletotrue - set
urlto the url your loolyloo instance can use to connect to lacus:http://<ip>:<port>
- set
- Restart lookyloo & try it
Changes
- Make hashlookup visible to everyone
- Improve logging
- Maintenance: use poetry 1.2, bump deps (Python and JS), bump Github actions
- Improve caching with Lacus
- Retry failing captures when it might be a temporary issue (typically domain resolution)
Fixes
- Avoid triggering a capture (and failing) when the URL and documents are missing
- Issue with urlscan when the capture had no referer
- Better handling of exceptions in VT module
- Better handling of devices offered by Playwright and their user-agents
Late august release aka "panda goes on holiday"
Breaking change
- Lookyloo requires Redis 7.0 or more decent. The upgrade process is as follows:
- Go to the Redis directory (should be in the same directory as where you cloned Lookyloo)
- Run the following commands
git fetch git checkout 7.0 make distclean make -j4 make test - You now have the new version of redis in place, you can update lookyloo as usual.
New features
-
Use pre-configured devices from Playwright (mobile only for now)
-
Download files when the URLs points to a downloadable content
- Submit downloadable content to Pandora (if available)
- Automatically select the most appropriate browser engine based on the user-agent
Fixes
- Make sure all the gunicorn instances displays all the recent captures
- Other bugfixes and GUI improvements
Changes
- Improve capture page with radio button to select which user-agent to submit
- Bump dependencies
August 2022 - Summer release
New features
- Trigger a capture on a web enabled document provided as a file instead of a URL. Useful for HTML files attached to emails, or HTML body in email.
- Compress (gzip) the HAR file in archived captures - saves a lot of disk space.
- Support for RiskIQ Passive DNS (requires API key)
- Display SSL/TLS information available in the HAR dump from Playwright
- Optional DoNotTrack HTTP header in capture
- Display size of rendered page on hostnode popup.
- [WiP] Download files when the URL captured to a downloadable file (PDF, Office doc, ...) (Important note the downloaded file is not exposed to the user yet)
- [WiP] List all hashes available in the capture, sort them by frequency. Makes it easier to find phishing sides using the same resources.
Fixes
- Major speed improvements when displaying the hostnode popups (only show the recent cached captures by default)
- Improvements in the caching mechanism
- Cleanup data showed by monitoring script
- Avoid crashes when RiskIQ isn't reachable
Changes
- Update dependencies (js, python)
- Improve logging in archiver
- Improve config file
June 2022 - #FIRSTCON22 release
Maintenance and bug-fixes release
All releases don't need to contain new features, sometimes, it is just some cleanup, and it is okay.
- Properly handle exceptions in some edge cases (fixes in har2tree)
- Properly display an error message if the capture fails
- Use the same default User-Agent in when a capture is submitted via the API as via the web interface.
- Cleanup some legacy code
- Bump all dependencies (JS/CSS and Python)
Still, there is a new-ish thing
We revamped the package generator, and it should be more usable. If it is not, let us know!
May 2022 - Replace Splash with Playwright
New Features
Playwright
The captures are now made via Playwright instead of Splash. It is a major improvement as Playwright uses actual up-to-date browsers, in headless mode (instead of qt-webkit from ~2016). You can read more about the research that lead to this change in the discussion.
The main other advantages of using playwright are the following:
- Easier to install: it doesn't requires Docker in order to use Splash
- Much better control of what happen in the browser while capturing: Playwright makes it extremely simple to instrument everything in the browsers. The capturing module already tries to solve reCaptcha if it detects it on the page.
The capture is made by a standalone python module that you can use in your own tools if you wish to.
De-duplication
If the exact same capture is triggered multiple times within 5 min, it is skipped and the requestor is redirected to the capture done before.
Fixes
- Avoid discarding a capture on network error: when a redirect is broken down the line, we keep the chain up to that point
- Issue when the MISP was submitted as un-published
- [Docker] Properly handle archiving
- [Docker] Init SRI hashes
Changes
- Improve subsequent capture template on long URLs
- Improve view of the capture page on small-ish screens
- General maintenance and code cleanup
- Improvement in the tree generation on edge cases
- Bump JS/CSS libraries
- Update bundled-in User-Agent file
- Use pydeep2, comes with a bundled-in libfuzzy, easier to install.
March 2022 - Maitenance and improvements
New Feature
- Trigger multiple captures at once from web interface
Fixes
- Improve MISP event publishing (make it asynchronous)
- Improve legend with titles on hoover
- Fix caches in modules
- Improve stats page
- Normalize buttons color
- Improve rendering of capture page
Changes
- Updates all web and python dependencies
- Use bootstrap 5
December 2021 - Hashlookup, Web extension support
New features
- Hashlookup integration - Documentation
- Pass arbitrary HTTP headers to captures - Documentation
- Pass arbitrary User-Agents to captures - Documentation
- Get hashes of all the resources using any algorithm supported by Python (API)
- Add configuration setting to make captures private by default - See
default_publicin the Documentation - Add CORS settings to allow JavaScript submissions (required for the browser extension)
- Defang URLs in email notifications
Fixes
- Avoid exception when the timestamp of a capture has no millisecond
- Avoid exceptions in archiver when indexes are broken
Changes
- Improve logging
- Improve capture page
- Normalize tooltips across the app
- Save redis databases to disk less often
- Programmatically shutdown redis databases (synchronous)
- Bump dependencies
September 2021 - Add phishtank support, improve monitoring, logging and cache
New features
- Integration with Phishtank via Phishtank Lookup - Documentation.
- Simple monitoring script to keep an eye on the health of the instance, run it in a tmux/screen with watch.
- Link in the tree menu to re-trigger a capture on the same URL.
Fixes
- Improve logging entries, the date was incomplete.
- Add UUID file in export.
- Inform users when a capture failed critically and we have nothing to show.
- Catch timeout when pushing to MISP (avoid exception)
Changes
- Major improvements in caching, better handling of exceptions and keep a limited amount of pickles in memory.
- Simplify code in the async capture script.
- Add urlscan.io permalink in MISP export
- Add phishtank permalink in MISP export
- Move modules to dedicated files