Issue with Quotation Marks Handling in the read_csv() Function

[okankcb]

Hello,

We encountered an unexpected behavior when using the read_csv() function with Verticapy 0.12 and 1.0.3.

Problem Encountered:
When importing a CSV file (attached to the discussion) with the following structure:
produits.csv

numero,libelle
1,Produit" A
2,Service B
3,"Pack ""C"" plus"
4,Offre D
5,"Solution ""E"" ""Premium"""

• When the libelle field contains a single double quote ("), it is interpreted as a separator, replacing the default comma.
• If a line contains an even number of double quotes, that line is not imported into the Vertica table, with no error or warning message.

Temporary Solution:
To work around this issue, we set the quotechar='' parameter. By default, this parameter is set as quotechar='"'.

However, this behavior does not seem normal and can be dangerous when importing data.

Here is an example of code that reproduces the issue:

read_csv("produits.csv", 
         table_name="TEST4",
         schema=schema,
         sep=",")

Out :

To resolve the issue, we modified the code as follows:

read_csv("produits.csv", 
         table_name="TEST4",
         schema=schema,
         sep=",",
         quotechar='')

Out :

It would be desirable to receive a "WARNING" to indicate a problem in these two cases, in order to avoid potentially incorrect results without the user's knowledge.

Would it be possible to examine on your end whether it is feasible to modify this behavior?

Thank you for your attention to this matter.

Best regards,
Okan.K

[oualib]

Thanks for reporting this issue, I'll do some tests and come back to you.

[oualib]

Hi Okan,

The behavior is expected, as the default string delimiter is ". In your case, changing this delimiter to an empty one should resolve the issue.

Since the parsing occurs at the server side, managing the warning without parsing the file twice will be challenging. I don’t consider this a bug.