August 2021 - Make the code nicer, urlscan.io integration.

New Features:

• Integration with urlscan.io - Documentation
• Trigger a capture from the URL - #248
• Archiving: the captures more than 6 month old (configurable) are moved to an archive directory so they're not listed on the index anymore, but the captures can still be accessed by UUID (doesn't break permanent URLs)
• Index file by directory for each captures (archived or not). Greatly reduces the I/O when initializing the known captures in redis.

Fixes:

• Missing 3rd party web dependencies in docker (thanks to @FafnerKeyZee)

Changes - This release is implementing a lot of back end changes :

• The captures are now stored a by year and month (instead of in a single directory) to avoid having too many entries in the same directory (ext4 dislikes it). All the new captures are following this new architecture, but you need to run tools/change_captures_dir.py to move the existing ones to the new format (only useful if you feel restarting the app takes too much time)
• Move all the capture-related code from Lookyloo to AsyncCapture
• Move all the services management code to abstractmanager
• Use redis pooling to manage connections to the database in Lookyloo and Indexing
• New process to trigger occasional actions, currently: generate the daily user-agent file if Lookyloo is using the UAs of its own users.
• Reinitialize the list of captures UUIDs when starting the app instead of the in website itself
• Improvements in processes handling (TL;DR: don't stop redis until all the async captures processes are down)
• Move some methods from Lookyloo to the helpers
• Simplify code in Lookyloo to make it more readable, remove dead code.
• Bump dependencies, add hiredis to speed up redis interactions
• Return proper HTTP error codes (mostly 4XX), when appropriate

July 2021 - Proxy and API improvements

The two main changes in this release are:

• Add support for passing a proxy to a capture, thanks to @Felalex57 - Documentation
• Major improvement in the API using flask-restx - Documentation on the demo Lookyloo instance.
• Add lookup against a MISP instance - Documentation
• Add sample config for log rotate thanks to @FafnerKeyZee - Documentation

The other changes are mainly bugfixes and small changes:

• Avoid receiving notifications from bots
• Upgrade the bundled-in list of user agents
• Improve generation of the pickles and avoid doing it twice
• Add reference to parent in the case a capture is initiated from an other one
• Improve MISP export

May 2021 - Speed improvements, maintenance

This release contains lots of changes in the backend (described below), and a few improvements on the web interface:

• Hide the captures with error from the index (see hide_captures_with_error)
• Return resources as text instead of in a zip file
• Crop and blur screenshot if it is too big to be displayed as-is
• Redesign of the menus
• Fix rendering of image resources in the investigation popup

The backend changes are mainly improving the overall performance of Lookyloo, with a few new features:

• All the captures (web and API) are using the asynchronous capture script, and the priority of each capture is weighted depending on the origin and the user(see priority, the number of async capture process is configurable (see async_capture_processes)
• The index is cached in memory by the webserver, making the index view lot faster after first load
• Improve auto-trigger of 3rd party modules (configurable per module)
• Add optional integration of whois queries with uWhoisd
• Disable FLoC globally
• Many bug fixes in har2tree and the creation of the tree
• Fix and improve MISP export, support subsequent captures as extended events
• Update all dependencies

Subsequent captures, UI updates and bugfixes

Once again, many of the changes in this release wouldn't have been possible without the support of Internews and the BASICS Project. And we would also like to thank Credit Agricole and @FafnerKeyZee for the continuous bug reports!

The main new feature of this release is the possibility to capture URLs present in a capture you already made, all that while keeping the context your browser was in (cookies, user-agent, referer) in the subsequent captures. It is especially useful when the page you're landing on expects the user to click on a link in order to load the content, the website checks the referer and/or cookies, and bounce you if you're not presenting the right session.
This feature will be further extended in the upcoming releases to allow other types of requests (POST), and let the user choose the link(s) to captures from the screenshot of the page itself.

This release also adds a new background indexer so the captures queued with the API are automatically cached even if they are never opened in the browser.

And there are the usual bunch of bugfixes, improvements and dependencies upgrades. And we also require python 3.8+.

You can also search for hostnames, URLs, hashes, and cookies names from the /search entry point. This entry point is not listed yet in the documentation, but it will be added soon.

MISP integration & UI

Once again, many of the changes in this release wouldn't have been possible without the support of Internews and the BASICS Project.

On the UI front, we now have a better support of huge screenshots and many more tooltips are shown when the mouse goes over icons and different parts of the tree. It should make the tree easier to read for users discovering the platform.

The main new feature of this release is the integration with MISP. It is now possible to export a capture directly to a pre-configured MISP instance:

The documentation in order to get it to work is also available.

There were also quite a few changes for the administrators of a Lookyloo instance, especially the authentication.

And for more details, you should have a look at the git changelogs.

Christmas release

Thanks to Internews and the BASICS Project we were able to greatly improve Lookyloo over the last few months.

These release notes won't be exhaustive and if you want to see all the changes, you should have a look at the git changelog.

Here is a short overview of the main changes in the last ~6 months:

• Major rewrite of the user interface based on the user tests realized thank to the BASICS Project and other user feedbacks
• Major improvements in the investigation popup
• Documentation website, also thanks to the BASICS Project
• Indexing of cookies and hashes of resources
• Add support for marking specific resources as known (libraries, icons, ... related to a specific domain or not) or malicious (phishing, malwares)
• DNS resolution (IP and CNAMEs)
• Query third party services via a modules system (SaneJS, VirusTotal, Phishing Initiative)
• Configuration via config files
• Update script
• Statistics of the whole instance
• Export in MISP format
• Systemd templates
• Docker image
• Update dependencies and bug fixes all over the place.