Skip to content

Releases: alephdata/aleph

4.0.2

23 Nov 13:17
@stchris stchris
4.0.2
This tag was signed with the committer’s verified signature.
stchris Christian Stefanescu
SSH Key Fingerprint: d6saNz+X1lhBCYQP+G45qW3LjsNnzVnlP8RILMEOzb0 Learn about vigilant mode.
6610201
This commit was signed with the committer’s verified signature.
stchris Christian Stefanescu
SSH Key Fingerprint: d6saNz+X1lhBCYQP+G45qW3LjsNnzVnlP8RILMEOzb0 Learn about vigilant mode.
Compare
Choose a tag to compare
4.0.2 Latest
Latest

We're announcing the release of Aleph 4.0.2 (and ingest-file 4.0.2) and highly recommend users of the 4.x branches to update to this release.

What's changed

  • Update to servicelayer 1.23.2 which fixes a significant performance regression noticeable especially when there are > 10000 tasks per dataset being processed

Full Changelog: 4.0.1...4.0.2

Assets 2
Loading

4.0.1

07 Nov 10:23
@stchris stchris
4.0.1
This tag was signed with the committer’s verified signature.
stchris Christian Stefanescu
SSH Key Fingerprint: d6saNz+X1lhBCYQP+G45qW3LjsNnzVnlP8RILMEOzb0 Learn about vigilant mode.
e790959
This commit was signed with the committer’s verified signature.
stchris Christian Stefanescu
SSH Key Fingerprint: d6saNz+X1lhBCYQP+G45qW3LjsNnzVnlP8RILMEOzb0 Learn about vigilant mode.
Compare
Choose a tag to compare
4.0.1

We're announcing the release of Aleph 4.0.1 (and ingest-file 4.0.1) and highly recommend users of the 4.x branches to update to this release.

What's changed

Bugfix

  • Update to using servicelayer 1.23.1 which fixes an issue with improper clean-up when a task exhausts it's maximum number of retries

Full Changelog: 4.0.0...4.0.1

Assets 2
Loading

4.0.0

15 Oct 10:24
@stchris stchris
4.0.0
This tag was signed with the committer’s verified signature.
stchris Christian Stefanescu
SSH Key Fingerprint: d6saNz+X1lhBCYQP+G45qW3LjsNnzVnlP8RILMEOzb0 Learn about vigilant mode.
d452633
This commit was signed with the committer’s verified signature.
stchris Christian Stefanescu
SSH Key Fingerprint: d6saNz+X1lhBCYQP+G45qW3LjsNnzVnlP8RILMEOzb0 Learn about vigilant mode.
Compare
Choose a tag to compare
4.0.0

Hello Aleph community! We’re excited to announce Aleph 4.0.0, a release focused on powerful new features, performance improvements, and expanded options for investigation sharing and user metrics. In addition, this release includes a few other small enhancements, bug fixes and dependency upgrades.

🚀 Bigger Changes 🚀

  • RabbitMQ based task queueing backend
    • Configurable AlephWorker Stages
    • Priority Buckets for Processing
    • System Status Page Enhancements
  • Updated Prometheus Metrics
  • Documentation Restructure and Enhancements
  • Improved Error Handling in Elasticsearch Upgrades

As always, we’d love to hear your feedback to keep improving. Feel free to reach out and share your thoughts!

What's Changed

Features

RabbitMQ

4.0.0 introduces a change to the way background tasks are scheduled. Previously Aleph used a Redis-based task queue, which was well designed but showed its limitations with large payloads and a risk of data loss. RabbitMQ queues are persisted to disk, but the flexibility in the way messages are queued, routed and fetched allows for certain optimizations which Aleph benefits from because of the widely varying degree of task loads.

Migration notes from Redis to RabbitMQ

Due to the significant changes in terms of task status persistence, switching between Aleph versions with RabbitMQ and Redis-based task queues requires some manual steps in order to ensure data consistency.

Perform the following steps every time you are either upgrading to a version with the RabbitMQ task queue or rolling back to the Redis-based task queue:

  1. Let all pending jobs run to completion (check the status page).
  2. Put Aleph into maintenance mode.
  3. Stop all workers (worker, ingest-file processes).
  4. (optional) Save the current state of redis in case you want to roll back using the BGSAVE command.
  5. Clear Redis (by issuing FLUSHDB from redis-cli from the redis container). If you get the error message "Unknown command FLUSHDB" then this command is disabled and you can resort to this shell invocation: echo 'KEYS *' | redis-cli | grep -v '^aleph:' | sed 's/^/DEL /' | redis-cli.
  6. (optional, if previous versions had conflicting RabbitMQ queue settings) Delete existing queues using rabbitmqctl delete queue {ingest,pruneentity,updateentity,exportxref,analyze,flushmapping,reingest,exportsearch,index,xref,reindex,loadmapping}. NOTE: queues are named after the stages found in ALEPH_WORKER_STAGES.
  7. Perform the upgrade or rollback to the desired version of Aleph.
  8. Ensure that all expected processes have started correctly.

Related changes:

Prometheus metrics

We have extended the Prometheus metrics exposed by Aleph to provide more information about active users and the data in your Aleph instance. For example, you can now query for the number of active users within the past 30 days or the number of investigations related to a particular language. For details about the available metrics please refer to the metrics reference in the technical documentation.

Sharing investigations

Due to the sensitive nature of dataset access we have made some changes to the way datasets are shared, no longer allowing email addresses to autocomplete. This means one needs to know the exact email address of another user if they want to share an investigation.

  • Feature: Allow sharing of investigations by @tillprochaska in #3865
  • Remove sharing options from create investigation screen by @stchris in #3862
  • Multiple small UX enhancements related to investigation sharing/user suggestion component by @tillprochaska in #3868

Other new features

Bug fixes and other changes

Documentation updates

Dependency updates

Full Changelog: 3.17.0...4.0.0

Assets 2
Loading

3.17.0

27 May 16:53
@tillprochaska tillprochaska
3.17.0
a63d208
This commit was signed with the committer’s verified signature.
tillprochaska Till Prochaska
GPG key ID: 04ED744349702E1F
Learn about vigilant mode.
Compare
Choose a tag to compare
3.17.0

Note

Please note that we skipped version 3.16.0. That means the previous version before this version is 3.15.6.

Important

This version upgrades a dependency, react-pdf, in order to mitigate a security vulnerability in a transitive dependency. The Aleph UI uses react-pdf to render PDF previews. We recommend that you update Aleph instances that you operate to the latest patched releases.

What's Changed

New Contributors

Full Changelog: 3.15.6...3.17.0

Contributors

stchris, DavidLemayian, and 7 other contributors
Assets 2
Loading

3.15.6

22 Apr 12:44
@tillprochaska tillprochaska
3.15.6
This tag was signed with the committer’s verified signature.
stchris Christian Stefanescu
SSH Key Fingerprint: d6saNz+X1lhBCYQP+G45qW3LjsNnzVnlP8RILMEOzb0 Learn about vigilant mode.
8a34798
This commit was signed with the committer’s verified signature.
stchris Christian Stefanescu
SSH Key Fingerprint: d6saNz+X1lhBCYQP+G45qW3LjsNnzVnlP8RILMEOzb0 Learn about vigilant mode.
Compare
Choose a tag to compare
3.15.6

During a routine security audit of Aleph we’ve become aware of ⚠️ security vulnerabilities ⚠️ in Aleph and ingest-file, the component that handles files uploaded to Aleph. We recommend that you update Aleph instances you operate to the latest patched releases:

Please find detailed information about the patched vulnerabilities below:

Downloaded source files are opened automatically

Summary

As part of the investigations feature, users can upload files to Aleph. The detail view in Aleph offers a sanitized preview of a file, but Aleph also allows users to download (unsanitized) source files. When downloading a source file, Aleph displays a confirmation prompt warning that source files may contain malware or notify the originator of the file.

After downloading a source file, files are opened automatically in the same browser window if the file’s MIME type is supported by the browser. This contradicts the warning that is displayed before downloading the file and potentially enables phishing attacks. For instance, an HTML file resembling the Aleph login interface could be uploaded for this purpose.

Affected versions

Aleph versions up to and including 3.15.5.

The vulnerability is exploitable if you have configured your Aleph instance to use Google Cloud Storage or AWS S3 (or a service compatible with S3) as a storage backend for files uploaded to Aleph via the “ARCHIVE_TYPE” configuration option. The default storage backend that stores files on the local file system is not affected.

Solution

Aleph versions 3.15.6 and newer contain a patch for this vulnerability. Patched versions set the “Content-Disposition” header to instruct browsers to download files as an attachment instead of opening them after the download has completed.

HTML injection in notification emails

Summary

Aleph sends a daily notification digest via email to users. Notification digests are enabled by default and can be disabled by users.

When a user creates an investigation and then shares it with another user who has daily notification digests enabled, the name of the user who created the investigation and the name of the investigation aren’t properly sanitized or encoded.

This means that links and other HTML markup included in the user’s name or in the investigation name will be rendered as is in the notification email which can enable (targeted) phishing campaigns.

Affected versions

Aleph versions up to and including 3.15.5.

The vulnerability is exploitable if you have set up email sending for your Aleph instance via the “ALEPH_MAIL_*” configuration options.

Solution

Aleph versions 3.15.6 and newer contain a patch for this vulnerability. Patched versions properly encode user-controlled data in notification emails.

Unauthorized access to mapping metadata

Summary

Aleph allows users to create entity mappings for uploaded spreadsheets. Using this feature, rows in a spreadsheet can be converted to FollowTheMoney entities in an investigation.

The access controls in the API endpoints for the mappings feature contain a bug that allows users without read or write access to the collection to view, update, trigger, and delete mappings as well as to delete or modify entities generated using a mapping.

The bug allows unauthorized access to the following mapping metadata:

  • Mapping definition (this includes column names in the source spreadsheet)
  • ID of the investigation a mapping belongs to
  • User ID of the user who created the mapping
  • Creation and update timestamps
  • Mapping status (“pending”/”successful”/”error” and the error message in case the status is “error”)
  • Entity ID of the source table

The bug does not allow users to view the entities generated from the mappings or the contents of the source spreadsheet.

Affected versions

Aleph versions up to and including 3.15.5.

Solution

Aleph versions 3.15.6 and newer contain a patch for this vulnerability. Patched versions properly verify user permissions when sending requests to the API endpoints for the mappings feature.

Unauthorized overrides of investigation and dataset metadata

Summary

Aleph allows users to manage metadata for investigations and datasets, including a label and a description as well as URLs to the publisher and source of the data. The metadata is displayed in the Aleph UI when viewing investigations and datasets.

Aleph allows users to specify a “foreign_id” when creating new investigations or datasets. The “foreign_id” can be used to reference the investigation or dataset when using the Aleph API or the alephclient CLI.

Due to a bug, when creating a new investigation or dataset with a “foreign_id” that is already used by another investigation or dataset, Aleph updates the metadata of the existing investigation/dataset instead of failing.

This bug allows users without the necessary permissions to update investigation and dataset metadata.

However, the bug does not allow unauthorized users to view investigation and dataset metadata or data added or uploaded to the investigation or dataset.

Affected versions

Aleph versions up to and including 3.15.5.

Solution

Aleph versions 3.15.6 and newer contain a patch for this vulnerability. Patched versions properly verify user permissions when creating or updating investigations or datasets.

Unauthorized access to uploaded files

Summary

Aleph allows uploading files to investigations and datasets. When a file is uploaded Aleph computes a checksum of the file contents and stores the checksum in the database. The uploaded file can later be retrieved using checksum as a reference. File checksums are represented as strings of hexadecimal characters, for example “ae9ce53fa78166704f5990601ec412d73fb1698a”.

Due to a bug in ingest-file users are able to upload specifically crafted files in order to create file records in the database with arbitrary checksums. This allows users to download files they do not have access to if they know the checksum of the file contents.

Affected versions

ingest-file versions up to and including 3.20.2. ingest-file is the component responsible for handling files you upload to Aleph.

Solution

ingest-file versions 3.20.3 and newer contain a patch for this vulnerability. The patch removes the ability to upload JSONL files that contain entities in the FollowTheMoney format to Aleph. If you have previously used this feature to create FollowTheMoney entities in Aleph in bulk, we recommend that you use the bulk endpoint of the Aleph API instead.

Assets 2
Loading

3.15.5

23 Jan 10:13
@stchris stchris
3.15.5
This tag was signed with the committer’s verified signature.
catileptic
GPG key ID: D9187F27E731E730
Learn about vigilant mode.
562723e
This commit was signed with the committer’s verified signature.
catileptic
GPG key ID: D9187F27E731E730
Learn about vigilant mode.
Compare
Choose a tag to compare
3.15.5

What's Changed

  • Bump followthemoney to 3.5.8
  • Bump ingest-file to 3.20.0 (also using followthemoney 3.5.8)

Full Changelog: 3.15.4...3.15.5

Assets 2
Loading

3.15.4

02 Nov 13:30
@stchris stchris
3.15.4
This tag was signed with the committer’s verified signature.
stchris Christian Stefanescu
SSH Key Fingerprint: d6saNz+X1lhBCYQP+G45qW3LjsNnzVnlP8RILMEOzb0 Learn about vigilant mode.
2a1e81d
This commit was signed with the committer’s verified signature.
stchris Christian Stefanescu
SSH Key Fingerprint: d6saNz+X1lhBCYQP+G45qW3LjsNnzVnlP8RILMEOzb0 Learn about vigilant mode.
Compare
Choose a tag to compare
3.15.4

What's Changed

⚠️ Because of this change the minimum Kubernetes version for the Aleph helm chart is now 1.23 ⚠️

  • Github Actions: use auth action before setup-gcloud in #3415

Full Changelog: 3.15.3...3.15.4

Contributors

richardjennings-occrp
Assets 2
Loading

3.15.3

26 Oct 12:42
@stchris stchris
3.15.3
This tag was signed with the committer’s verified signature.
stchris Christian Stefanescu
SSH Key Fingerprint: d6saNz+X1lhBCYQP+G45qW3LjsNnzVnlP8RILMEOzb0 Learn about vigilant mode.
a95f61b
This commit was signed with the committer’s verified signature.
stchris Christian Stefanescu
SSH Key Fingerprint: d6saNz+X1lhBCYQP+G45qW3LjsNnzVnlP8RILMEOzb0 Learn about vigilant mode.
Compare
Choose a tag to compare
3.15.3

What's Changed

  • Improved Dockerfile to reduce image size by @tdurieux in #2801
  • Updated SECURITY.md by @tillprochaska in #3218
  • New user guide by @tillprochaska in #3223
  • Make it possible to manually trigger a docs deployment by @tillprochaska in #3226
  • Add redirects for old user guide links by @tillprochaska in #3229
  • Updated dev environment by @monneyboi in #3205
    ⚠️ NOTE ⚠️: this will upgrade the postgres container in the development environment from version 10 to 15 and since the data files are binary incompatible you need to either manually update them (for instance using this) or by cleaning out your local postgres volumes (docker volume rm aleph_postgres-data aleph_postgres-data-e2e followed by make upgrade)
  • Remove query_string_query function, remove fields from highlight query by @monneyboi in #3280
  • Add make format-check as pull request "check" by @monneyboi in #3282

Dependency upgrades

New Contributors

Full Changelog: 3.15.1...3.15.3

Contributors

tillprochaska, tdurieux, and 2 other contributors
Assets 2
Loading

3.15.1

08 Sep 09:09
@stchris stchris
3.15.1
This tag was signed with the committer’s verified signature.
stchris Christian Stefanescu
SSH Key Fingerprint: d6saNz+X1lhBCYQP+G45qW3LjsNnzVnlP8RILMEOzb0 Learn about vigilant mode.
cab5fb7
This commit was signed with the committer’s verified signature.
stchris Christian Stefanescu
SSH Key Fingerprint: d6saNz+X1lhBCYQP+G45qW3LjsNnzVnlP8RILMEOzb0 Learn about vigilant mode.
Compare
Choose a tag to compare
3.15.1

What's Changed

Dependency upgrades

New Contributors

Full Changelog: 3.15.0...3.15.1

Contributors

tillprochaska, zekehuntergreen, and dependabot
Assets 2
Loading

3.15.0

17 Jul 09:22
@stchris stchris
3.15.0
This tag was signed with the committer’s verified signature.
stchris Christian Stefanescu
SSH Key Fingerprint: d6saNz+X1lhBCYQP+G45qW3LjsNnzVnlP8RILMEOzb0 Learn about vigilant mode.
bab61e8
This commit was signed with the committer’s verified signature.
stchris Christian Stefanescu
SSH Key Fingerprint: d6saNz+X1lhBCYQP+G45qW3LjsNnzVnlP8RILMEOzb0 Learn about vigilant mode.
Compare
Choose a tag to compare
3.15.0

What's Changed

Dependency upgrades

New Contributors

Full Changelog: 3.14.1-rc15...3.15.0-rc2

Contributors

micahflee, stchris, and 3 other contributors
Assets 2
Loading