rgw/s3select: json output format #548
Conversation
albin-antony
force-pushed
the
json_format
branch
5 times, most recently
from
9d6af2f
to
2bb3332
Compare
|
|
||
| @pytest.mark.s3select | ||
| def test_json_column_sum_min_max(): |
There was a problem hiding this comment.
why did you make this conversion (remove JSON)?
albin-antony
force-pushed
the
json_format
branch
3 times, most recently
from
26c4133
to
c9f0e0a
Compare
| s3select_assert_result( res_s3select_like, res_s3select ) | ||
|
|
||
| @pytest.mark.s3select | ||
| def test_json_like_expressions(): |
There was a problem hiding this comment.
@albin
test_json_like_expressions is duplicated name
There was a problem hiding this comment.
Thanks. Updated the name.
albin-antony
force-pushed
the
json_format
branch
from
c9f0e0a
to
9d10291
Compare
|
the following functions they have much in common, it is possible to combine them into one function. |
| list_int = create_list_of_int( 1 , csv_obj ) | ||
| print(list_int) |
There was a problem hiding this comment.
please remove the debug printout
| res_target = min( list_int ) | ||
| print("target") |
|
@albin-antony, please review comments in the case of Parquet, it needs to make sure that RGW is correctly built with arrow-library In the JSON output format, it needs to ensure that |
albin-antony
force-pushed
the
json_format
branch
from
9d10291
to
81bd7af
Compare
|
@galsalomon66 Thanks for the review. Addressed the comments. Please take a look |
| @@ -235,30 +239,45 @@ def create_random_json_object(rows,columns,col_delim=",",record_delim="\n",csv_s | |||
|
|
|||
| return result | |||
|
|
|||
| def csv_to_json(obj, field_split=",",row_split="\n",csv_schema=""): | |||
| def create_parquet_object(parquet_size): | |||
There was a problem hiding this comment.
@albin-antony
please add other types such as string and float.
| output_serialization = {"CSV": {}} | ||
| if input == "JSON": | ||
| input_serialization = {"JSON": {"Type": "DOCUMENT"}} | ||
| output_serialization = {"JSON": {}} |
There was a problem hiding this comment.
run_s3select should support output=JSON or CSV
| elif(input == "PARQUET"): | ||
| input_serialization = {'Parquet': {}} | ||
| output_serialization = {"JSON": {}} | ||
|
|
There was a problem hiding this comment.
the same
run_s3select should support output=JSON or CSV
| res = remove_xml_tags_from_result( run_s3select(bucket_name,csv_obj_name,"select count(0) from s3object;") ).replace(",","") | ||
|
|
||
| res = remove_xml_tags_from_result( run_s3select(bucket_name,csv_obj_name,"select count(0) from s3object;","CSV") ).replace(",","") | ||
There was a problem hiding this comment.
the "CSV" parameter is default.
no?
There was a problem hiding this comment.
Yes, default is CSV. Don't need to override the same. Will remove it.
albin-antony
force-pushed
the
json_format
branch
from
81bd7af
to
62a75e3
Compare
|
@albin-antony |
Signed-off-by: Albin Antony <[email protected]>
albin-antony
force-pushed
the
json_format
branch
from
62a75e3
to
2e53973
Compare
No description provided.