SNOW-1859974: Issue with SnowflakeCursor.executemany() ?

[fcremer-nl]

Python version

Python 3.10.10 (tags/v3.10.10:aad5f6a, Feb 7 2023, 17:20:36) [MSC v.1929 64 bit (AMD64)]

Operating system and processor architecture

Windows-10-10.0.19045-SP0

Installed packages

snowflake-connector-python==3.11.0
snowflake-snowpark-python==1.18.0

What did you do?

df = self.session.create_dataframe(rows, self.load_schema)
df.write.mode('append').save_as_table(self.mgt)

What did you expect to see?

I expected the data to be written to the table, but got the error "not all arguments converted during string formatting" at

File "C:\Users\504019\AppData\Roaming\Python\Python310\site-packages\snowflake\connector\cursor.py", line 1388, in executemany
    fmt % self._connection._process_params_pyformat(param, self)

Where the format (fmt) is a string, something like '(?,?,?,?,?)' and the right side the tuple with the data for the row.

If I change the line of code above to:
fmt.replace('?','%s') % self._connection._process_params_pyformat(param, self)

Then it works correctly. Similarly if I change the code original code to write row by row:

for row in rows:
    df = self.session.create_dataframe([row], self.load_schema)
    df.write.mode('append').save_as_table(self.mgt)

It also works!

Is this an issue with Snowpark or am I doing something stupid?

Can you set logging to DEBUG and collect the logs?

No

[sfc-gh-sghosh]

Hello @fcremer-nl ,

Thanks for raising the issue.

I tried to reproduce it in Windows 11 using Python 3.10 and Snowpark 1.26.0, and I am not able to reproduce it. Could you please share the full code snippet?

`load_schema = StructType([
StructField("ID", IntegerType()),
StructField("NAME", StringType())
])

rows = [
(1, "Alice"),
(2, "Bob"),
(3, "Charlie")
]

df = session.create_dataframe(rows, load_schema)

target_table = "example_table"

df.write.mode('append').save_as_table(target_table)

result_df = session.table(target_table)
result_df.show()

Output:
|"ID" |"NAME" |
|1 |Alice |
|2 |Bob |
|3 |Charlie |`

[fcremer-nl]

Hi Sujan, Thank you for looking into this issue. I have tried your example, but it does not trigger the SnowflakeCursor.executemany() function. If I use the sample code to try to insert the same data that triggered the bug, it indeed triggers the executemany() function; apparently snowpark determines which function to use based on the amount of data(?). However, the sample code with my data surprising works. I have traced the issue to the following line SnowflakeCursor.executemany() function: if self._connection.is_pyformat: The function SnowflakeConnection.is_pyformat() is a single line: return self._paramstyle in ("pyformat", "format") It turns out that _paramstyle = 'qmark' when the session is created from a config file. However, if the connection is created from a existing connection [ session = Session.builder.configs({"connection": existing_snowflake_connection}).create() ] then paramstyle='pyformat'.In that case overwriting the paramstyle [ session.connection._paramstyle = 'qmark') ] alleviates the problem. I think the assignment of paramstyle is definitely an issue with snowpark and I am not sure if this is correct behaviour. Still I am wondering if even when paramstyle='pyformat' whether the code in SnowflakeCursor.executemany() is correct. I cannot see how the [ fmt % self._connection._process_params_pyformat(param, self) ] can work when format is a string like '(?,?,?,?)' and the argument after the % is a tuple. Looking forward to hear from you. Best regards, Frank

…

-- Frank Cremer

On Fri, Dec 20, 2024 at 4:36 AM Sujan Ghosh ***@***.***> wrote: Hello @fcremer-nl <https://github.com/fcremer-nl> , Thanks for raising the issue. I tried to reproduce it in Windows 11 using Python 3.10 and Snowpark 1.26.0, and I am not able to reproduce it. Could you please share the full code snippet? `load_schema = StructType([ StructField("ID", IntegerType()), StructField("NAME", StringType()) ]) rows = [ (1, "Alice"), (2, "Bob"), (3, "Charlie") ] df = session.create_dataframe(rows, load_schema) target_table = "example_table" df.write.mode('append').save_as_table(target_table) result_df = session.table(target_table) result_df.show() ------------------------------ |"ID" |"NAME" | |1 |Alice | |2 |Bob | |3 |Charlie |` — Reply to this email directly, view it on GitHub <#2123 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BNYU4BG3C46R6FR73FNEYX32GOGFXAVCNFSM6AAAAABTYEVSY2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNJWGIZDCNRRHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>