Release 3.1.0
Release notes - Feature Store - 3.1.0
Epic
FSTORE-472 Metadata Management
FSTORE-474 Feature Flags
FSTORE-475 Storage Connectors
Task
FSTORE-33 Improve HSFS parameter description
FSTORE-35 Permission matrix
FSTORE-42 Not very helpful doc "you can do it via the UI" (for external FG)
FSTORE-43 Documentation sample data has syntax issue (missing comma)
FSTORE-52 Fix typo in GE tutorial
FSTORE-309 Add support for like operator in query filter
FSTORE-311 Support datetime64[ns, TZ] typed columns
FSTORE-315 Upgrading GE version to 0.14.12 to support Jinja2 version 3.0.0
FSTORE-316 Add Timestamp/Timezone Guide
FSTORE-322 Enable update of feature view metadata in python client
FSTORE-325 Make driver field optional in Redshift connector configuration
FSTORE-328 throw intelligible error message if user set primary and partition keys doesn't exist
FSTORE-330 Adapt python client to the change introduced for the single expectation api
FSTORE-332 implement method for get_or_create_feature_view
FSTORE-334 explicit provenance - feature groups, feature views, training datasets
FSTORE-336 Improve error message in get_batch_data
FSTORE-337 Single expectation API & Expectation Suite Smart Update
FSTORE-339 Add Great Expectations Validation Activity
FSTORE-340 Add validation for great expectations kwars for feature names
FSTORE-341 Validation History for an expectation
FSTORE-342 Add java tests to github pipeline
FSTORE-345 Update documentation to reflect supported methods in hsfs engines
FSTORE-346 Fail to create feature view with query having `as_of`
FSTORE-347 Make documentation build strict
FSTORE-348 Refactor Tutorials documentation page
FSTORE-350 Clean up Key Files for Storage Connectors on Delete
FSTORE-352 Clean up hudi delta streamer job config files in resources
FSTORE-354 Write Home Credit Default Risk hopsworks tutorial
FSTORE-358 Add icon to external links in documentation navigation
FSTORE-359 Add return types to transformation functions in 3.1
FSTORE-360 Add "Client install guide" to docs
FSTORE-364 Fail to run DeltaStreamer job
FSTORE-369 Remove feature group details endpoint
FSTORE-371 Cannot modify a FG on the UI
FSTORE-379 Add documentation on how to share a feature store store and how to access it.
FSTORE-380 Expand query documentation
FSTORE-381 Label does not consider prefixed feature name
FSTORE-385 Handle same instance of transformation function for multiple features properly
FSTORE-387 Add feature_store_id and feature_group_id in expectation suite DTO
FSTORE-397 Fail to get batch query in java client
FSTORE-398 Add validation history method to python client
FSTORE-406 Document time related parameter in feature view
FSTORE-407 API Doc should include full package name for the RestAPIError
FSTORE-408 Fix feature group integration tests
FSTORE-411 Sharing featurestore fails
FSTORE-416 Problem with get_training_data
FSTORE-422 PIT join does not return event time column of right feature group
FSTORE-425 throw meaningful error if user tries to set time_travel_format=None in a Python kernel
FSTORE-429 Add data validation tutorial to loadtest
FSTORE-430 Add return type and code example in feature view python client
FSTORE-431 Fix typo in user guide for data validation
FSTORE-433 Accept list in event_time
FSTORE-441 Enable shortening of expectation_config field in ValidationResult
FSTORE-442 Increase kwargs varchar field in expectation table
FSTORE-446 user guide update for 3.1 data validation
FSTORE-447 Improve documentation for Tags
FSTORE-448 Feature view documentation problems
FSTORE-451 Bump onlinefs version to 1.4-SNAPSHOT on master
FSTORE-452 Show entire stack trace when exception in engine occurs
FSTORE-460 The code part of the Data Validation is confusing
FSTORE-467 Make it easier to start the backfill job from python
FSTORE-470 Add tests for running workflows with hudi commits
FSTORE-477 Change Online FS ALTER table algorithm
FSTORE-478 Add workflow tests for pyspark fraud batch and fraud online
FSTORE-481 Unify python and pyspark test code in load tests
FSTORE-482 Workflow test for onlinefs append feature
FSTORE-484 Storage connector feature flag
FSTORE-485 Update Hive Grafana dashbaord to reflect new GC configuration
FSTORE-488 Ensure type consistency between write/read/online/offline python/spark
FSTORE-489 Bump spark driver memory to 2GB and executor memory to 4GB
FSTORE-494 add documentation how to add new filters and/or join new FGs to feature view query
FSTORE-496 Test improvements
FSTORE-497 Change integration test after validation time in future bug fix
FSTORE-498 Throw error when filter feature not exist
FSTORE-499 add python code snippets to API docs
FSTORE-505 Remove old feature store tour files
FSTORE-509 Update expectation suite metadata in backend on setter call.
FSTORE-510 Implement reliable ingestionResult for Validation Report
FSTORE-513 Make .parquet file default format for training datasets
FSTORE-515 Prevent deleting Expectation Suite activity on deletion of the suite
FSTORE-519 Redshift storage connector documentation is out of sync
FSTORE-523 Improve user messages for forbidden actions.
FSTORE-526 Don't cast all types when reading from feature view
FSTORE-529 Documentation for Permission matrix
FSTORE-530 Add override logic to save_expectation_suite
FSTORE-541 Investigate missing data in onlinefs load test
FSTORE-542 Add server version check to python client to make correct expectation suite call
FSTORE-543 Add get_batch_data and get_feature_vector functions to workflows
FSTORE-545 Add 3.1 new data validation methods to loadtest
FSTORE-548 Remove deprecated np. types from tests and type conversions
FSTORE-554 cleanup training dataset statistics
FSTORE-555 Add explicit provenance documentation
FSTORE-556 Add examples about include_primary_key and include_event_time
FSTORE-562 Add load test to integrate benchmark for vector lookup
FSTORE-567 Fix data validation loadtest to avoid uploading reports with same timestamp.
FSTORE-579 Tutorials 3.1 Update
FSTORE-581 Review feature view documentation
FSTORE-589 Document behaviour of get_feature_vector(s) for missing PK values
FSTORE-600 Fix training_dataset creation in Electricity project
FSTORE-606 Upgrade Fraud Batch/Online Tutorials
FSTORE-613 Make Kafka Topic partition default configurable for APP upgrade
FSTORE-621 Make unit tests run on hsfs branch-3.1
FSTORE-658 Incomplete doc string of create_external_feature_group
FSTORE-662 Deployment error. 3.1.
FSTORE-669 Enable GCS and BigQuery storage connector feature flags by default
FSTORE-700 Improve Bitcoin Tutorial
FSTORE-726 when computing pandas statistics check if mean is numeric type
Bug
FSTORE-4 FGs with 3 rows - when creating a TD, get an error creating statistics
FSTORE-5 Snowflake connector documentation assumes using the old classic snowflake UI
FSTORE-7 Deleting a Storage Connector does not work
FSTORE-8 Snowflake connector: trying to read with no query throw an error
FSTORE-10 Snowflake storage connector Warehouse is NOT optional
FSTORE-11 Feature group create documentation points to dev (3.1.0-SNAPSHOT)
FSTORE-13 Fraud Online Tutorial Step 7: Create Training Dataset. FeatureStore exception
FSTORE-15 Weight & Biases Colab failed
FSTORE-16 Filtering null values on a decimal feature failed
FSTORE-20 UI changes iris ML tutorial
FSTORE-21 Cannot create streaming feature group without primary key
FSTORE-23 Hopsworks Tutorials - Colabs
FSTORE-24 Fraud retail Colab failed
FSTORE-25 Query difference between external/internal Hive clients
FSTORE-27 Failed to get training data on a feature view if multiple splits with non standard names
FSTORE-29 Problem with Online Feature Group creation
FSTORE-31 Fraud Online Tutorial error
FSTORE-32 Data Validation with GE Tutorial
FSTORE-34 Weights & Biases Tutorial
FSTORE-36 Documentation not updated for train_validation_test_split method
FSTORE-37 api-key documentation broken image and screenshots from old UI
FSTORE-40 Creating online feature without online type
FSTORE-45 Missing entries when uploading large number of rows
FSTORE-53 apply_transformation_function provided Training Dataset instead of Transformation Functions
FSTORE-56 Fail to delete a FV in clients if some FG are deleted.
FSTORE-312 Multiple minor documentation issues in Redshift Feature Ingestion Tutorial
FSTORE-314 Fix td -> fv migration
FSTORE-318 Provenance NullPointerException for a feature group
FSTORE-319 replace license for ImportJobConf to Hopsworks instead of LC
FSTORE-320 Feature group documentation in feature group list points to nowhere
FSTORE-323 Resolve OnlineFS ClusterJ Session usage issue
FSTORE-326 Bigquery connector removed after updating it through the UI
FSTORE-327 Incorrect filling of default kwargs in great expectations table
FSTORE-329 CommitDTO type field is not returned anymore after Payara5 merge
FSTORE-331 BigQuery connector doesn't work if the application is executed on multiple nodes
FSTORE-333 Featurestore insert in Hudi table does not support certificate rotation
FSTORE-351 epipe - provenance index - handle resource folders correctly
FSTORE-366 Shorten result field is throwing error
FSTORE-370 NPE when time_travel_format=None
FSTORE-372 User Guide typos
FSTORE-376 Error in getting batch query when label is the only column selected from a feature group.
FSTORE-377 Corrupted FGs in Hopsworks after deleting a FG
FSTORE-378 Fixing pre-registered great expectations to fix the UI
FSTORE-383 Feature View method documentation
FSTORE-388 Weird values in Online FS, after using .append_feature
FSTORE-389 Activity logs reports activities from other users
FSTORE-390 Exception thrown when query feature group with appended features with default value
FSTORE-391 Ingesting dataframe with nullable=False column causes Exception in Hudi/Hive/Spark
FSTORE-392 Change GCS authentication properties
FSTORE-393 Fix bug introduced by single expectation API
FSTORE-395 great_expectation_engine in python client called with outdated arg list
FSTORE-396 Registered Great Expectations Kwargs set in json should be list
FSTORE-400 Integration Tests: trainingdataset_spec
FSTORE-401 Integration Tests: featuregroup_spec
FSTORE-402 Integration Tests: feature_store_activity_spec
FSTORE-403 Integration Tests: featurestore_code_spec
FSTORE-410 create feature view join.prefix is NoneType
FSTORE-412 json ignore on `isTimeTravel()` in Query.java
FSTORE-413 query prefix throws TypeError
FSTORE-414 feature_view.create_train_test_split always returns empty df
FSTORE-417 Problem with create_train_validation_test_split
FSTORE-418 Write options have no effect with .insert method in Python engine
FSTORE-420 Feature Group insertion job from Python does not apply spark resources correctly
FSTORE-423 expectation meta field not parsed on init
FSTORE-426 Errors/out of date info in the Data Validation documentation
FSTORE-427 Reading training data from HopsFS stopped working with pyarrow 10.0.0
FSTORE-432 Fix event_time in hopsworks-tutorial
FSTORE-435 Fetching report which have results with deleted expectation throws null pointer exception.
FSTORE-436 HSFS Java doesn't serialize JSONs correctly
FSTORE-437 Key error 'type' in inode.from_response_json
FSTORE-438 Methods get_train_validation_test_split and train_validation_test_split not returning same column type
FSTORE-443 Do not allow sending feature group feature offline type to be null
FSTORE-444 Data validation minor bug fixes for python client
FSTORE-445 Minor fix to tutorials
FSTORE-453 Fix convert_event_time_to_timestamp for datetimes with millisecond precision (hudi timestamps)
FSTORE-454 Show warning in output when data validation fails, right now it silently moves on
FSTORE-455 Default great expectation configuration not working
FSTORE-456 Validation time in the future
FSTORE-457 Data Validation code example is broken
FSTORE-459 Wrong FS name in API example for sharing FG
FSTORE-461 Append feature default value not working for online feature store
FSTORE-463 Import data to feature group job failure
FSTORE-465 Can't assign filter to feature view query
FSTORE-468 Cannot set timestamp type as partition_key in feature group
FSTORE-476 Schema not updated for hsfs java client after append
FSTORE-480 Create new Dynamic Object for each row in transaction
FSTORE-501 Can't insert streaming dataframe in streaming feature group using `insert_stream` method
FSTORE-507 figure out features and their transformation functions when attaching transformation functions with prefixes
FSTORE-508 NPE when creating feature view with labels
FSTORE-512 Typo in expectation_type leads to failure to execute validate job
FSTORE-518 Problem with Feature Groups Join
FSTORE-520 Feature view - explicit provenance if parent feature group is deleted
FSTORE-522 HSFS get redshift storage connector fails
FSTORE-527 Fix deletion issue with expectation suite due to activity table constrain
FSTORE-532 Redshift connector NPE when using Instance Role for authentication
FSTORE-534 Redshift Connector IAM Role authentication jackson issue with expiration field
FSTORE-538 Redshift connector read throws validation error for query argument inspite of table name specified on connector
FSTORE-551 Fix loadtest incorrect assert
FSTORE-552 Fix validate for pyspark when providing pandas dataframe
FSTORE-560 In onlinefs configuration, Hopsworks domain name is not configured by chef
FSTORE-639 Pin SQLalchemy version or upgrade to new 2.0.0 version
FSTORE-640 Fix QuickStart and Credit_Scores tutorials
FSTORE-653 get_feature_vector is broken
FSTORE-657 in dev branch tutorials imports are missing
FSTORE-660 Docs: Syntax issue for creating external feature group
FSTORE-661 Creating external feature group in Jupyter Notebook didn't work
FSTORE-663 Model training tutorial:Error on predictions after model deployment
FSTORE-682 Fix fg save_expectation_suite
FSTORE-689 Shallow dataframe copy hides TZ bug
FSTORE-691 Train test splits contain NaN column values
FSTORE-693 Insert with validation suite throws date format exception
FSTORE-730 Fix Multiple backfill job started in dataval loadtest
Release notes - Hopsworks - 3.1.0
Bug
HWORKS-1 Documentation link at /account/api should point to https://docs.hopsworks.ai/3.0/user\_guides/projects/api\_key/create\_api\_key/ and not https://docs.hopsworks.ai/3.0/user\_guides/integrations/databricks/api\_key/
HWORKS-5 When adding member to project it is always set to data scientist
HWORKS-11 Github documentation expect you to know the platform
HWORKS-26 Clone repo documentation could be more hand holding
HWORKS-36 Deactivate the "Save" button on "Edit profile" page if there are no changes
HWORKS-39 python env - wrong docs link
HWORKS-45 Confusing paragraph in jobs and git
HWORKS-47 Python and Docker job should show reason why it crashed also when not caused by the application program
HWORKS-93 Fix serving tests
HWORKS-94 Upgrade from 3.0.0 to master fails when applying dml sql files
HWORKS-96 Feature store connection does not reset when calling hopsworks.logout()
HWORKS-97 Ensure a default value for serving_tool in new deployments
HWORKS-101 Models saved with the latest version of joblib (1.2.0) cannot be loaded in Hopsworks
HWORKS-110 Link to Kibana logs from deployment page is wrong in App
HWORKS-114 The password recovery email send to the old ui
HWORKS-115 Deploying cluster with Managed Kubernetes fails
HWORKS-116 Cluster deployment fails when overriding Hopsworks port in the cluster definition
HWORKS-117 Doc Bug: Auth Login Page
HWORKS-118 Fix failing git tests
HWORKS-122 Revoking certificate with exact X509 Name does not work
HWORKS-123 hopsworks.login() should hide API Key after it was supplied as input
HWORKS-140 Kafka authorizer incorrectly parses x509 Subject
HWORKS-143 .condarc still templating deleted attribute
HWORKS-150 Fix failing serving tests
HWORKS-154 Execution success variable not set correctly
HWORKS-156 Deleting projects with certain names may delete hopsworks docker images from the registry
HWORKS-158 Stuck in project creation when creating a project with the same name as a recently delete project
HWORKS-159 Payara5 fixes for hopsworks-api
HWORKS-160 Can't open airflow if payara is not restarted.
HWORKS-168 Customize consul health check error exit code
HWORKS-175 Make SELinux disabling and modifying resolv.conf optional in consul-chef
HWORKS-184 onlinefs username and monitoring port not templated in onlinefs-site.xml
HWORKS-185 Template Kafka listeners even if IDs are explicitly set
HWORKS-186 Fix blocker bugs reported by sonarqube analysis
HWORKS-194 Kafka external listener port collides with alertmanager
HWORKS-203 dnsmasq systemd unit is not enabled by default in CentOS
HWORKS-205 Fix RESTCodes compilation
HWORKS-208 mgm-server-start check for existing port is too wide
HWORKS-211 Deleting a python environment in a cluster with managed kube can delete another project images.
HWORKS-212 ExperimentResultsDTO throws IllegalAnnotationException
HWORKS-214 Reorder operations to start docker
HWORKS-242 Deployment fails to start in a non-kubernetes cluster
HWORKS-249 redshift connector save fails for default driver
HWORKS-250 HSML 3.0.* does not fully work with Hopsworks 3.1
HWORKS-254 onlinefs user is hardcoded in Hopsworks
HWORKS-260 Update LDAP configuration via the rest api not working
HWORKS-268 Karamel fail during dnsmasq installation
HWORKS-271 hopsify may time out when submitting a CSR
HWORKS-273 Docker containers total memory seems wrong on grafana dashboard
HWORKS-295 Project creation error message not informative
HWORKS-296 Spark job resource docs are wrong
HWORKS-297 Add retries to package Chef resource
HWORKS-301 Connection reset is not handled by LdapRealm injected resource
HWORKS-332 LDAP Group Project mapping UI changes the name of the ldap group when the mapping is selected
HWORKS-340 Member is removed from Project in LDAP auth where no LDAP groups Project mappings are defined
HWORKS-350 Add numpy constraint to KServe Python library
HWORKS-356 HDFS copyFromLocal and chown operations eventually fail during zipping/unzipping
HWORKS-357 Fix upgrade from Glassfish 4 to Glassfish 5
HWORKS-365 Remove hardcoded yarnapp user in jupyter notebook convert script
HWORKS-371 Prometheus database alerts do not trigger if the metric is absent
HWORKS-372 explicit provenance missing community stereotype
HWORKS-383 Documentation does not mention python kernel feature flag
HWORKS-410 Pin jupyter-console in Sparkmagic
HWORKS-425 Update Grafana to address CVE-2022-39328, CVE-2022-39307 and CVE-2022-39306
Subtask
HWORKS-400 Upgrade Jupyter(Lab) to address CVE-2021-32797
Task
HWORKS-2 API Keys guide should link to hopsworks login
HWORKS-8 Which permission for the github token
HWORKS-34 Make git repositories private
HWORKS-37 git workflow docs
HWORKS-71 Increase DockerResourcesConfiguration to 2GB
HWORKS-82 Code snippet in docs should use hopsworks.login() and not hopsworks.connection()
HWORKS-95 Show resource usage in JupyterLab
HWORKS-121 Skip adding application certificates to CRL
HWORKS-125 Improve status management for deployments
HWORKS-132 Push onlinefs docker image to internal docker registry
HWORKS-134 All integration tests should use expect_status_details instead of expect_status
HWORKS-136 Docker Cgroup cpu.cfs_quota_us should account for number of cores
HWORKS-137 Add chef parameter to overwrite location of /tmp directory
HWORKS-141 Add sample yaml for Alertmanager config
HWORKS-144 Bump RonDB version to 21.04.9
HWORKS-145 Remove ROCM Code
HWORKS-146 Docker image anaconda python environment should use python 3.8
HWORKS-155 Mention hops python library as deprecated from 3.0 in docs
HWORKS-161 Add final status to Python Jobs
HWORKS-162 Don't show "service user" role if it's not app
HWORKS-163 Hopsworks python jobs api should not check if the job already exists
HWORKS-165 Disable ee_ml_pipeline spec
HWORKS-166 conda-chef ignores tmp_directory attribute when installing miniconda
HWORKS-167 Remove dev/SNAPSHOT version from hsfs, hsml and hopsworks API docs
HWORKS-169 Bump version of commons-text due to CVE-2022-42889
HWORKS-171 model version should be downloaded in temp directory instead of current working directory
HWORKS-172 Do not overwrite mysql server log files
HWORKS-173 Explicitly set JAVA_HOME when calling hash.sh in elastic-chef
HWORKS-190 Hopsworks-chef should allow installation of custom certificate for Hopsworks UI
HWORKS-191 Run unit tests, license checker and vulnerability checker on pull request review
HWORKS-192 Fix master to use latest version of HopsHadoop 3.2.0.8-SNAPSHOT
HWORKS-199 Should be possible to get python library version using .__version__
HWORKS-201 Investigate git clone failing on multi-VM setup
HWORKS-202 Add project under creation flag
HWORKS-204 Support for installing .whl and requirements.txt using hopsworks python SDK
HWORKS-207 Add Docs on how to configure LDAP from Payara UI
HWORKS-210 jupyter threads fill up executor pool
HWORKS-213 hopsworks python SDK API doc code samples should use hopsworks.login()
HWORKS-223 Remove username prefix from cloned git repository folder name
HWORKS-243 Documentation on how to export audit logs to third party platform
HWORKS-244 Documentation on how to export metrics to third party platform
HWORKS-245 Documention that new github fine-grained tokens are not supported yet
HWORKS-256 [hopsworks-chef] Deploy user supplied intermediate certificate for Glassfish
HWORKS-257 Improve data validation testing in loadtest
HWORKS-259 Sync ldap groups upon user login
HWORKS-263 Make LDAP groups Hopsworks Project synchronization more granular that hour
HWORKS-289 Add enable_git_pull_only flag
HWORKS-290 Add enable_data_science_profile flag
HWORKS-291 Update opensearch version to 1.3.6
HWORKS-292 Apply SPARK-40801 to address CVE-2022-42889
HWORKS-293 Add cookbook flag to enable remote JMX
HWORKS-299 Add enable_jupyter_python_kernel_non_k8s flag
HWORKS-303 Grafana should show all resource managers not only active ones
HWORKS-310 Disable TLS1.1 for Opensearch dashboard
HWORKS-311 Disable TLS1.0 and TLS1.1 for Docker registry
HWORKS-312 Disable insecure chipers in Opensearch
HWORKS-318 Apply HIVE-24636 to remove the memory leak
HWORKS-324 .hw_api_key should be saved in home folder, or fall back to temp
HWORKS-326 Update and fix prometheus alerting rules
HWORKS-327 Drop commons-net dependency
HWORKS-330 Create a cluster overview dashboard in Grafana
HWORKS-335 Fix failing tests
HWORKS-339 Ensure the hopsworks tutorials work with 3.1 release
HWORKS-341 Add missing Audience.JOB annotation for full hopsworks API compatibility
HWORKS-342 Check major/minor version compatibility on hopsworks.login()
HWORKS-345 Switch Livy to use log4j 1.2 bridge
HWORKS-347 Configure Spark to use Log4j2 configuration
HWORKS-349 hsml v3.1.0 fails when backend is 3.0.4
HWORKS-353 In case of an upgrade initialize Hopsworks CAs in hopsworks-ca instead of Expat
HWORKS-361 Increase TransactionInactiveTimeout to 30000
HWORKS-362 Update spark to depend on latest version of hopsfs
HWORKS-393 Enable HSFS in default deployments (without KServe)
HWORKS-413 Description for model evaluation metrics is missing
HWORKS-423 Fix links for public slack on docs and community edition