AWS SDK for pandas 3.10.0

Features

• feat: Support numpy 2.0 by @LeonLuttenberger in #2944
• feat(redshift): Automatically add new DataFrame columns to Redshift tables during write operation by @jack-dell in #2948
• feat: modify_refresh_interval flag in opensearch index_documents by @AvihaiSam in #2980
• feat: support postgresql array types by @kukushking in #2985

Bug fixes 🐛

• fix: OpenSearch import errors by @LeonLuttenberger in #2939
• fix(athena): Enable use of dataframe type, in athena2pyarrow type by @eliabrio in #2953
• fix: correct skip.header.line.count value to 1 by @jaidisido in #2977
• fix: data_api uuid typehint by @jethroguce in #2961
• fix: enforce ssl by @kukushking in #2979
• fix: handle case of str to bool env variable in config by @jaidisido in #2966
• fix: return only "current" iceberg columns by @kukushking in #2982
• fix: Athena read_sql_query cache errors for qmark style parametrized queries by @LeonLuttenberger in #2957
• fix: Prevent truncated Parquet files in S3 after failed CreateMultipartUpload (2987) by @rdwebster in #2993

Documentation 📚

• docs: use autodoc typehints by @LeonLuttenberger in #2940
• chore: Prepare 3.10.0 release by @kukushking in #2992

Security / Dependency Updates 🛡️

• chore(deps-dev): bump the development-dependencies group with 5 updates by @dependabot in #2942
• chore(deps): bump the production-dependencies group with 6 updates by @dependabot in #2941
• chore(deps-dev): bump jupyterlab from 4.2.4 to 4.2.5 by @dependabot in #2945
• chore(deps-dev): bump the development-dependencies group with 2 updates by @dependabot in #2947
• chore(deps): bump the production-dependencies group with 2 updates by @dependabot in #2946
• chore(deps): bump cryptography from 42.0.8 to 43.0.1 by @dependabot in #2949
• chore(deps): bump fiona from 1.9.6 to 1.10.0 by @dependabot in #2952
• chore(deps): bump the production-dependencies group with 2 updates by @dependabot in #2954
• chore(deps-dev): bump the development-dependencies group with 5 updates by @dependabot in #2955
• chore(deps): bump setuptools from 74.1.2 to 75.0.0 in the production-dependencies group by @dependabot in #2962
• chore(deps-dev): bump the development-dependencies group with 5 updates by @dependabot in #2963
• chore(deps-dev): bump the development-dependencies group with 5 updates by @dependabot in #2970
• chore(deps): bump the production-dependencies group with 3 updates by @dependabot in #2969
• chore(deps): bump the production-dependencies group with 3 updates by @dependabot in #2975
• chore(deps-dev): bump the development-dependencies group with 4 updates by @dependabot in #2976
• chore(deps): bump aiohttp from 3.10.8 to 3.10.9 in the production-dependencies group by @dependabot in #2988
• chore(deps-dev): bump the development-dependencies group with 4 updates by @dependabot in #2989

New Contributors

• @eliabrio made their first contribution in #2953
• @jack-dell made their first contribution in #2948
• @jethroguce made their first contribution in #2961
• @AvihaiSam made their first contribution in #2980
• @rdwebster made their first contribution in #2993

Full Changelog: 3.9.1...3.10.0

AWS SDK for pandas 3.9.1

Bug fixes 🐛

• bucketing error with newer version of Modin (0.31.0) by @LeonLuttenberger in #2896
• athena.read_sql_query failing for time columns by @LeonLuttenberger in #2895
• add an argument to control handling nulls in merge criteria by @brendan-cook-87 in #2892
• address Ray deprecation warnings by @LeonLuttenberger in #2929
• Add deprecation notice to s3.select_query by @LeonLuttenberger in #2928

Documentation 📚

• Add steps to install "using pip" to the "AWS Glue Python Shell Jobs" part of the install docs by @mc51 in #2911

Security Dependency Updates 🛡️

• bump aiohttp from 3.10.1 to 3.10.2 by @dependabot in #2930

New Contributors

• @brendan-cook-87 made their first contribution in #2892

Full Changelog: 3.9.0...3.9.1

AWS SDK for pandas 3.9.0

Enhancements 🎉

• Support ORC and CSV in redshift.copy_from_files function by @LeonLuttenberger in #2849
• Support different merge conditions in athena.to_iceberg function by @aldder in #2861
• Manage NULL values in athena.to_iceberg merge statement by @aldder in #2872
• Upgrade Ray to 2.30 by @jaidisido in #2870

Bug fixes 🐛

• Fix tz attribute check, it was checking dtype instead of dt by @sanrodari in #2855
• PyArrow backend trimming timestamp to date by @LeonLuttenberger in #2875
• Replace deprecated ray parallelism arg with override_num_blocks by @kukushking in #2876

Documentation 📚

• Adding NotebookVersion Parameter as specified in official AWS Docs by @DaxterXS in #2867
• Fix default value in doc string for athena_query_wait_polling_delay parameters by @niklasr22 in #2873

Security Dependency Updates 🛡️

• Bump urllib3 from 1.26.18 to 1.26.19 by @dependabot in #2859
• Bump certifi from 2024.6.2 to 2024.7.4 by @dependabot in #2886

New Contributors

• @aldder made their first contribution in #2861
• @DaxterXS made their first contribution in #2867
• @sanrodari made their first contribution in #2855
• @niklasr22 made their first contribution in #2873

Full Changelog: 3.8.0...3.9.0

AWS SDK for pandas 3.8.0

Enhancements 🎉

• support client-side parameter resolution in athena.create_ctas_table by @LeonLuttenberger in #2797
• add commit_transaction to postgres.to_sql by @jaidisido in #2795
• add columns parameters support by @jaidisido in #2814
• add overwrite_method to postgresql.to_sql by @LeonLuttenberger in #2820
• add upsert mode to sqlserver.py and corresponding tests by @AntonMantulo in #2835

Bug fixes 🐛

• correct athena timezone TypeError message by @jaidisido in #2796
• update license reference by @emmanuel-ferdman in #2798
• add missing index in athena.to_iceberg to_parquet by @jaidisido in #2799
• support json type in athena2pandas by @DavidKatz-il in #2806
• adapt parquet datasource to ray 2.20 by @jaidisido in #2809
• add primary keys when upserting in Postgres by @AntonMantulo in #2819
• add s3_output parameter to athena.delete_from_iceberg_table by @LeonLuttenberger in #2829
• fix s3.read_parquet_metadata not throwing proper error for empty path by @LeonLuttenberger in #2843

Documentation 📚

• fix enumerations in doc pages by @LeonLuttenberger in #2821
• fix readthedocs errors by pinning sphinx version by @LeonLuttenberger in #2822

Security Dependency Updates 🛡️

• bump werkzeug from 3.0.1 to 3.0.3 by @dependabot in #2807
• bump jinja2 from 3.1.3 to 3.1.4 by @dependabot in #2808
• bump requests from 2.31.0 to 2.32.0 by @dependabot in #2827
• bump pymysql from 1.1.0 to 1.1.1 by @dependabot in #2830

New Contributors 👏🏼

• @emmanuel-ferdman made their first contribution in #2798
• @DavidKatz-il made their first contribution in #2806
• @AntonMantulo made their first contribution in #2819

Full Changelog: 3.7.3...3.8.0

AWS SDK for pandas 3.7.3

Bug fixes 🐛

• Iceberg schema evolution fails for map, array and struct types by @LeonLuttenberger in #2755
• trickle down s3_output in athena.to_iceberg by @jaidisido in #2767
• respect order of columns in to_iceberg by @jaidisido in #2768
• add PyArrow fixed_size_binary dtype support by @jaidisido in #2775
• Opensearch serverless vector search collections - remove default _id by @kukushking in #2784
• missing keys in list_to_arrow_table by @kukushking in #2778
• prevent athena.to_iceberg overwrite to delete table in order to preserve Iceberg transactions history by @erwan-simon in #2776

Documentation 📚

• simplify README, remove AWS Glue for Ray references by @jaidisido in #2750
• fix YAML formatting in Ray Remote tutorial by @LeonLuttenberger in #2770

Security Dependency Updates 🛡️

• bump idna from 3.6 to 3.7 by @dependabot in #2772
• bump aiohttp from 3.9.3 to 3.9.4 by @dependabot in #2777

New Contributors 👋

• @erwan-simon made their first contribution in #2776

Full Changelog: 3.7.2...3.7.3

AWS SDK for pandas 3.7.2

Features/Enhancements 🚀

• Add support for DeltaLake's DynamoDB lock mechanism by @LeonLuttenberger in #2705

Bug fixes 🐛

• wr.athena.to_iceberg - Insert query has mismatched column types #2678 by @GalvFionic in #2715
• allow s3_output in athena.to_iceberg by @jaidisido in #2727
• replace deprecated np.split_array by @jaidisido in #2735
• Athena to_iceberg fails with non-lowercase column names by @LeonLuttenberger in #2736
• Support Ray 2.10 by @kukushking in #2741

New Contributors

• @GalvFionic made their first contribution in #2715

Full Changelog: 3.7.1...3.7.2

AWS SDK for pandas 3.7.1

Bug fixes 🐛

• fix breaking change in _create_table by @jaidisido in #2711
• pin pyarrow to version 8 and above by @jaidisido in #2709

Documentation 📚

• fix redshift.to_sql doc indentation error by @LeonLuttenberger in #2706

Full Changelog: 3.7.0...3.7.1

AWS SDK for pandas 3.7.0

Breaking changes 💥

Lake Formation Governed tables are being phased out and we are dropping support (#2692).

Features/Enhancements 🚀

• support parquet client encryption (#2642) by @Marwen94 in #2674

Bug fixes 🐛

• Index columns removed on s3.to_parquet by @robert-schmidtke in #2655
• Missing timezone metadata by @kukushking in #2682
• remove enforced openpyxl engine constraint by @jaidisido in #2696
• Iceberg partitioning not working with partition transform functions by @LeonLuttenberger in #2694
• remove awswrangler README from site-packages folder by @AlJohri in #2698
• indent categories in pyarrow_additional_kwargs correctly by @jaidisido in #2701

New Contributors

• @Marwen94 made their first contribution in #2674
• @AlJohri made their first contribution in #2698

Full Changelog: 3.6.0...3.7.0

AWS SDK for pandas 3.6.0

Features/Enhancements 🚀

• Enable Iceberg row deletion & add mode parameter to to_iceberg by @LeonLuttenberger in #2632
• Add support for pyarrow type large_string by @joakibo in #2663
• Add max_results to athena.list_query_executions by @LeonLuttenberger in #2665

Bug fixes 🐛

• Pyarrow 15 imports & remove unused code by @kukushking in #2649

New Contributors

• @joakibo made their first contribution in #2663

Full Changelog: 3.5.2...3.6.0

AWS SDK for pandas 3.5.2

Bug fixes 🐛

• DynamoDB key & filter expressions attribute overwrite by @kukushking in #2615
• Allow PostgreSQL reserved keywords as column names by @LeonLuttenberger in #2619
• Add to_iceberg support for filling missing columns in the DataFrame with None by @LeonLuttenberger in #2616
• Forward ignore_nulls for container types by @raaidarshad in #2636

Documentation 📚

• Add s3_additional_kwargs to docstrings by @malachi-constant in #2627
• Fix outdated hyperlinks in documentation by @LeonLuttenberger in #2634

Other 🤖

• Enable dependabot to upgrade GitHub actions by @LeonLuttenberger in #2618
• Update badges in README by @LeonLuttenberger in #2628
• Add vulnerability label to dependabot PRs with alert state by @jaidisido in #2629

New Contributors

• @raaidarshad made their first contribution in #2636

Full Changelog: 3.5.1...3.5.2