Bug Fix: Escape query in spark submit parameter

[seankao-az]

Description

Fix EMR spark submit parameter for batch FlintJob parsing issue when query contains quotes.

Ideally would like to test start job run in https://github.com/opensearch-project/sql/blob/main/spark/src/test/java/org/opensearch/sql/spark/client/EmrServerlessClientImplTest.java as well, but the AWSEMRServerless is mocked.

Manually ran end to end test connecting to a real EMR-S application for queries with nested quotes.
Example query:

CREATE INDEX test_cv
ON mys3.default.http_logs (`status`, `day`, `size`, `request` )
WITH (index_settings = '{"number_of_shards": 1 ,"number_of_replicas":1}', auto_refresh = true, checkpoint_location = 's3://test/')

This creates the index successfully.

The following tests special characters in a string get passed correctly.

CREATE INDEX test_special
ON mys3.default.http_logs (`status`, `day`, `size`, `request` )
WITH (index_settings = '{"number_of_shards": 1 ,"test ,:+\"inner\"/\|?#><":1}', auto_refresh = true, checkpoint_location = 's3://test/')

Checking the logs, we see this printed as expected:

24/03/23 16:36:10 INFO FlintSpark: Creating Flint index FlintSparkCoveringIndex(test_special,mys3.default.http_logs,Map(status -> int, day -> int, size -> int, request -> string),None,FlintSparkIndexOptions(Map(index_settings -> {"number_of_shards": 1 ,"test ,:+\"inner\"/\|?#><":1}, auto_refresh -> true, checkpoint_location -> s3://test/)),None) with ignoreIfExists false

Issues Resolved

• [BUG] Async Job query can longer than 10280 chars #2376

Check List

• New functionality includes testing.
  • All tests pass, including unit test, integration test and doctest
• New functionality has been documented.
  • New functionality has javadoc added
  • New functionality has user manual doc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.36%. Comparing base (284a0be) to head (c5c15e3).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##               main    #2587   +/-   ##
=========================================
  Coverage     95.36%   95.36%           
  Complexity     5129     5129           
=========================================
  Files           490      490           
  Lines         14419    14420    +1     
  Branches        966      966           
=========================================
+ Hits          13751    13752    +1     
  Misses          643      643           
  Partials         25       25           

Flag	Coverage Δ
sql-engine	95.36% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.x
# Create a new branch
git switch --create backport/backport-2587-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 2bcf0b8b3371558a60200c8b994ee6de64ddcf6e
# Push it to GitHub
git push --set-upstream origin backport/backport-2587-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-2587-to-2.x.

[opensearch-trigger-bot]

The backport to 2.13 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.13 2.13
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.13
# Create a new branch
git switch --create backport/backport-2587-to-2.13
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 2bcf0b8b3371558a60200c8b994ee6de64ddcf6e
# Push it to GitHub
git push --set-upstream origin backport/backport-2587-to-2.13
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.13

Then, create a pull request where the base branch is 2.13 and the compare/head branch is backport/backport-2587-to-2.13.