Skip to content

Releases: RunLLM/aqueduct

v0.3.6

07 Jun 19:29
840f917
Compare
Choose a tag to compare

0.3.6

Released on June 6, 2023.

Key Features

  • Enables users to selectively or completely disable Aqueduct's artifact snapshotting functionality. When defining a workflow, users can now decide to disable snapshotting for an individual artifact or for all artifacts in a workflow; if selected, these artifacts will only be generated during computation time and then garbage collected. No data will be persisted beyond any data that is explicitly saved. See our documentation for more details.

Enhancements

  • Enables on-demand Kubernetes clusters on Google Cloud to use GPU nodes in an autoscaling capacity.

Bugfixes

  • Fixes bug where resources could not be deleted even when unused.
  • Fixes bug where workflows that succeeded with a warning check were shown in a regular success state instead of a warning state.
  • Fixes page load jitter when a large number of workflows are present.

All Changes

v0.3.5...v0.3.6

All commits

v0.3.5

01 Jun 04:56
86c10db
Compare
Choose a tag to compare

Released on May 31, 2023.

Key Features

  • Adds support for on-demand Kubernetes clusters on Google Cloud; you can now select the Google Cloud option when asking Aqueduct to create a Kubernetes cluster, and Aqueduct will automatically use the Google Kubernetes Engine. See our documentation here.
  • Adds support for Google Artifact Registry as a conatiner registry in Aqueduct. You can now use custom Docker images that are pushed to GAR. See our documentation here.

Bugfixes

  • Fixes bug where editing certain resource configurations from the UI would cause the resource edit dialog to error out.
  • Fixes bug where updating a workflow that has been previously registered but not run would cause the workflow to be duplicated.
  • Fixes bug where certain resources could be registered without specifying a name for the resource.

All Changes

v0.3.4...v0.3.5

All commits

v0.3.4

25 May 05:17
cc9a6fb
Compare
Choose a tag to compare

Released on May 24, 2023.

Enhancements

  • Improves consistency of icon sizing on resources page.
  • Merges flow_id and name into a single argument when retrieving workflows
    from the SDK.
  • Adds ability to parametrize save operators when using AWS S3.

Bugfixes

  • Fixes bug where race condition could occur when syncing Airflow workflows to
    Aqueduct.
  • Fixes bug where listing Snowflake data objects would return an error.

All Changes

v0.3.3...v0.3.4

All commits

v0.3.3

17 May 23:50
e9a34c0
Compare
Choose a tag to compare

Released on May 17, 2023.

Enhancements

  • Adds ability to use a parameter in save operators to dynamically name saves.
  • Updates the artifact storage card on the resources page to be consistent with
    other resource cards.
  • Shows how many workflows are using notification resources.
  • Improves form validation with react-hook-forms.
  • Replaces the icon that signifies a workflow that has been registered but not
    executed.
  • On the resouces page, displays the Conda integration as modifying the Aqueduct
    server execution layer rather than as a separate execution layer.
  • Improves BigQuery error checking to ensure that a dataset exists before
    accessing it.

Bugfixes

  • Fixes bug where documentation links on the home page were broken.
  • Fixes bug where the SDK's generated docstrings were broken.

All Changes

v0.3.2...v0.3.3

All commits
  • ENG_2960: Fix broken documentation link for Aqueduct demo resource. by @agiron123 in #1306
  • Update the artifact storage card presentation to be consistent with the other data integrations by @kenxu95 in #1297
  • Add appropriate workflow count to our connected notification cards and details page by @kenxu95 in #1302
  • ENG-2767 react hook form submission by @agiron123 in #1232
  • README typo fix by @cw75 in #1313
  • Update the workflow "No Run" logo by @kenxu95 in #1312
  • Fixes broken storybook build by @agiron123 in #1317
  • Eng 2884 test node routes by @eunice-chan in #1299
  • Fix generated docstrings when viewing resource methods in Jupyter by @kenxu95 in #1315
  • Fix bug where duplicate fields showing in resource details dropdown by @kenxu95 in #1321
  • [2/n] Adds gh actions to publish test pypi packages by @likawind in #1262
  • [UI] Merge the Conda resource into the Aqueduct Server by @kenxu95 in #1311
  • Add ability to parameterize the save operator by @kenxu95 in #1320
  • Adds error checking for whether big query dataset exists by @saurav-c in #1319
  • Fixes cloudpickle serialization for Python 3.7 by @saurav-c in #1324
  • Fix Snowflake save missing schema bug by @kenxu95 in #1331
  • ENG-2979 S3 Dialog Validation Fixes by @agiron123 in #1328
  • ENG-2979: Fixes bigquery and gcs dialogs that were crashing after file uploads. by @agiron123 in #1326
  • Move any overflow rows on the details header fields into an additional column by @kenxu95 in #1322
  • Eng 2885 create and test metric and check routes by @eunice-chan in #1323
  • Changes schedule for spark periodic tests by @hsubbaraj-spiral in #1336
  • Rc v033 by @jpurusho65 in #1334

v0.3.2

10 May 23:40
6341917
Compare
Choose a tag to compare

Released on May 10, 2023.

Enhancements

  • Improves error handling when the user attempts to take an action without
    connecting to the Aqueduct server.
  • When connecting an S3 resource to Aqueduct, if the specified subdirectory
    does not exist, it is now automatically created.

Bugfixes

  • Replaces outdated integration terminology on the UI.
  • Fixes bug where previously created resource names were not editable.

All Changes

v0.3.1...v0.3.2

All commits

v0.3.1

04 May 17:59
Compare
Choose a tag to compare

Key Features

  • Introduces the aqueduct.llm_op API and the aqueduct-llm package. Aqueduct
    now has support for invoking LLMs with a single API call and comes with
    pre-built Docker images optimized for executing LLMs on Kubernetes. The
    llm_op API supports both ad hoc execution, as pictured below, as well as
    batch execution over a list of inputs or a Pandas Series. See our
    documentation for more details.
    from aqueduct import Client, llm_op
    
    client = Client() # initialize Aqueduct client so we can check if the engine name below is valid
    vicuna = llm_op('vicuna_7b', engine='my_k8s_engine')
    vicuna('What is the best LLM?')
  • Reorganizes integrations around the concept of resources. Resources are any
    external tool, system, or API that Aqueduct can connect to; existing data
    and compute integrations are automatically converted into resources. A
    container registry resource is added in this release, and future releases
    will introduce new resource types. The recommended SDK API for accessing
    resources is now client.resource, with client.integration slated to
    deprecated in a future release.
  • Allows users to specify a custom Docker image when running an Aqueduct
    operator on Kubernetes. The Docker image is required to have the Aqueduct
    executor scaffolding installed; for more details, please see our
    documentation here.

Enhancements

  • Improves logging and error handling when an operator fails because it's able
    to successfully generate a result, typically in the setup phase.
  • Enables connecting a Databricks cluster to Aqueduct via the Python SDK.

Bugfixes

  • Fixes bug where installing pre-requisites for using Aqueduct-managed
    Kubernetes clusters would fail on an M1 Mac with certain configurations.

All Changes

v0.2.12...v0.3.1

All commits
  • Fix spark dkr build by @likawind in #1250
  • Eng 2142 add ability to run the conda integration test by @likawind in #1216
  • Fix bug where on-demand k8s integration registration fails on Mac by @cw75 in #1242
  • [1/n] Allow pypi pkg version injection by splitting version into a separate file by @likawind in #1259
  • Updates README.md by @vsreekanti in #1263
  • Improve logging when operator failed before generating artifact by @likawind in #1252
  • Adds DatabricksConfig to connect_config by @hsubbaraj-spiral in #1256
  • Adds support for running integration tests against Airflow by @saurav-c in #1204
  • Adds status checks for integration test Redshift cluster by @saurav-c in #1255
  • Allow users to specify custom Docker image when running operators on K8s by @cw75 in #1264
  • Increase server setup timeouts in integration tests by @likawind in #1254
  • Implement use argument for LLM for decorators by @cw75 in #1234
  • Add Docker images that includes the LLM library and dependencies by @cw75 in #1235
  • Do not schedule execution of save operator if other computer operator… by @jpurusho65 in #1253
  • Small improvement to ECR error surfacing by @cw75 in #1272
  • Improves error message returned to user when Airflow creds are incorrect by @saurav-c in #1275
  • Add MANIFEST file to LLM python package by @cw75 in #1278
  • This PR contains everything that's part of resource reframing milestone 1 by @cw75 in #1277
  • Rename integrations to resources on the SDK and example notebooks by @kenxu95 in #1276
  • Add more white space to the bottom of the integrations page by @cw75 in #1279
  • ENG-2895 Updates link for GCS storage documentation in GCS dialog. by @agiron123 in #1281
  • Remove additional log line by @jpurusho65 in #1282

v0.2.12

26 Apr 04:36
3321a42
Compare
Choose a tag to compare

Released on April 25, 2023.

Enhancements

  • Improves the describe method on integration objects in the SDK; if there
    was an error connecting or validating the integration, the error will be
    shown.
  • Eagerly validates integration connections, so users will be notified
    immediately if they are using a mis-configured integration when, for
    example, attempting to access data.
  • Replaces placeholder values in integration connection forms with more
    realistic values.
  • Adds support for the most recent version of numpy.

All Changes

v0.2.11...v0.2.12

All commits * Fix isort in our code and automated linter by @kenxu95 in https://github.com//pull/1227 * Bump numpy version to 1.24.2 by @kenxu95 in https://github.com//pull/1228 * Eng 2635 Add automated periodic testing coverage for MariaDB by @eunice-chan in https://github.com//pull/1225 * Track the execution state of every integration connection on the backend by @kenxu95 in https://github.com//pull/1220 * [SDK] Fail when using any integrations that aren't connected yet by @kenxu95 in https://github.com//pull/1223 * Improve placeholder values for integrations by @hsubbaraj-spiral in https://github.com//pull/1233 * Refactor Spark Python Executor by @hsubbaraj-spiral in https://github.com//pull/1231 * Fix documentation bug for missing local file by @Fanjia-Yan in https://github.com//pull/1230 * Eng 2735 Add order by and limit parameters for V2 workflow results by @eunice-chan in https://github.com//pull/1240 * Rc v0212 by @likawind in https://github.com//pull/1249

v0.2.11

19 Apr 01:21
b9bcf68
Compare
Choose a tag to compare

Released on April 18, 2023.

Enhancements

  • Extends the Aqueduct-Databricks integration to optionally support long-standing instance pools. If specified, Aqueduct will automatically use nodes in the instance pool, and otherwise, Aqueduct will create a cluster with new instances.
  • Improves form validation during integration connection to ensure that valid and invalid form fields are correctly detected and visualized.

Bugfixes

  • Fixes bug where MySQL dependency installation could fail on an M-series Mac.
  • Fixes bug where integration connections dialogs could re-render repeatedly, causing slight performance degradation.
  • Fixes bug where loading a Tensorflow object from a file could fail due to library version mismatches.
  • Fixes concurrency bug when multiple processes attempt to simultaneously create the same Kubernetes secret.
  • Fixes bug where the release notification banner introduced in the last release was not picking up new Aqueduct versions from PyPI.
  • Fixes bug where CUDA versions specified via the SDK may not have been faithfully installed at execution time.

All Changes

v0.2.10...v0.2.11

All commits

v0.2.10

12 Apr 01:01
6be8f10
Compare
Choose a tag to compare

Released on April 11, 2023.

Enhancements

  • Enables subdirectory support when configuring the AWS S3 integration. When
    a subdirectory is specified, everything outside of that directory will be
    ignored by Aqueduct.
  • Extends support for creating artifact from the local file system to all
    types
    supported by Aqueduct.
  • If executing code on the Aqueduct server without Conda, the server will now
    proactively ensure that the Python version from your environment matches
    the server's Python environment. If there's a mismatch, an error will be
    thrown.
  • Adds a dismissable banner to the Aqueduct UI notifying users of new releases.

Bugfixes

  • Fixes bug where errors occurring during the execution of a metric were not
    being properly surfaced.
  • Fixes bug where checks with a severity level of warning were being shown as
    errors in check history.
  • Fixes bug where creating a schedule with helper functions on the SDK would
    fail for monthly schedules.

All Changes

v0.2.9...v0.2.10

All commits

v0.2.9

05 Apr 22:21
8e93e58
Compare
Choose a tag to compare

Released on April 5, 2023.

Key Features

  • [Beta] Aqueduct now has support for loading dataframe and image data from
    the local filesystem as parameters. See client.create_param() for details.

Enhancements

  • Improves the artifact storage management process. The integrations page now
    shows which system is being used for artifact storage, and if a migration
    between artifact stores fails, the error will be surfaced on the
    integration details page until the user triggers a new artifact migration.
  • Adds metadata to read and write operators' sidesheets. Both sidesheets now
    show which system is being used for the IO operation, and the save operator
    now shows to what location and (if relevant) in what format the data is
    being saved.

Bugfixes

  • Fixes typos in MongoDB connection dialog.
  • Fixes bug where workflow status at the top of the workflow details page would
    not update in sync with other parts of the page.
  • Fixes a bug where stopping the Aqueduct server immediately after starting it
    could potentially trigger and then kill a workflow run that would then
    stay in a pending state permanently.

All Changes

v0.2.8...v0.2.9

All commits