Skip to content

Commit

Permalink
[MINOR][PYTHON][TESTS] Reduce the python worker error log of `test_to…
Browse files Browse the repository at this point in the history
…DF_with_schema_string`

### What changes were proposed in this pull request?
Reduce the python worker error log of `test_toDF_with_schema_string`

### Why are the changes needed?
When I run the test locally
```python
python/run-tests -k --python-executables python3 --testnames 'pyspark.sql.tests.test_dataframe'
```

Two assertions in `test_toDF_with_schema_string` generate too many python worker error logs (~1k lines), which easily exceed the limitation of terminal and make it hard to debug.

So I want to reduce the number of python workers in the two assertions.

### Does this PR introduce _any_ user-facing change?
no, test only

### How was this patch tested?
manually test, the logs will be reduced to ~200 lines

### Was this patch authored or co-authored using generative AI tooling?
no

Closes #48388 from zhengruifeng/test_to_df_error.

Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
  • Loading branch information
zhengruifeng authored and HyukjinKwon committed Oct 9, 2024
1 parent f69d03e commit 6ed4bdf
Showing 1 changed file with 4 additions and 2 deletions.
6 changes: 4 additions & 2 deletions python/pyspark/sql/tests/test_dataframe.py
Original file line number Diff line number Diff line change
Expand Up @@ -506,14 +506,16 @@ def test_toDF_with_schema_string(self):

# number of fields must match.
self.assertRaisesRegex(
Exception, "FIELD_STRUCT_LENGTH_MISMATCH", lambda: rdd.toDF("key: int").collect()
Exception,
"FIELD_STRUCT_LENGTH_MISMATCH",
lambda: rdd.coalesce(1).toDF("key: int").collect(),
)

# field types mismatch will cause exception at runtime.
self.assertRaisesRegex(
Exception,
"FIELD_DATA_TYPE_UNACCEPTABLE",
lambda: rdd.toDF("key: float, value: string").collect(),
lambda: rdd.coalesce(1).toDF("key: float, value: string").collect(),
)

# flat schema values will be wrapped into row.
Expand Down

0 comments on commit 6ed4bdf

Please sign in to comment.