Add a case to test ORC writing/reading with lots of nulls #8825
Conversation
…th lots of nulls Signed-off-by: Chong Gao <[email protected]>
|
build |
|
This case can pass, but causes this a new issue: #8826. |
| @@ -398,6 +398,10 @@ properly without it. These tests assume Delta Lake is not configured and are dis | |||
| If Spark has been configured to support Delta Lake then these tests can be enabled by adding the | |||
| `--delta_lake` option to the command. | |||
|
|
|||
| ### Enabling large data tests | |||
There was a problem hiding this comment.
Do we want to switch this over to scale testing instead? #8811
I was just thinking as a follow on this fits much better there, then it does here. If so I am happy to file a follow on issue to move the test there, rather then try and file an issue to enable these tests in a nightly build.
There was a problem hiding this comment.
Agreed. This adds the test, but I don't see any changes to CI scripts or other things that would actually run this test in practice.
There was a problem hiding this comment.
Yes, it's not running. As discussed with Tim and Gary before, will create another CI job to run this kind of tests using:
-m large_date_test --large_date_test
It's fine to move to scale test. The follow-up issue is: #8849
| sqls = ["SELECT * FROM my_large_table", | ||
| "SELECT * FROM my_large_table WHERE c2 = 5", | ||
| "SELECT COUNT(*) FROM my_large_table WHERE c3 IS NOT NULL", | ||
| "SELECT * FROM my_large_table WHERE c4 IS NULL", | ||
| "SELECT * FROM my_large_table WHERE c5 IS NULL", |
closes #8731
This is to test large number of nulls.
Signed-off-by: Chong Gao [email protected]