Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New CSV implementation #903

Draft
wants to merge 41 commits into
base: master
Choose a base branch
from
Draft

New CSV implementation #903

wants to merge 41 commits into from
+98,601 −124

Commits on Oct 1, 2024

  1. adding deprecation message for DataFrame.read(delimiter = ...) in f… 

    …avor of `readCSV`.
    @Jolanrensen
    Jolanrensen committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    ac2248f View commit details
    Browse the repository at this point in the history

  2. adding new dataframe-csv module and starting with deephaven-csv integ… 

    …ration
    @Jolanrensen
    Jolanrensen committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    54e11a6 View commit details
    Browse the repository at this point in the history

  3. Adding missing types to ColTypes, types we can already parse from Str… 

    …ing.
    @Jolanrensen
    Jolanrensen committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    fd4c8f4 View commit details
    Browse the repository at this point in the history

  4. starting with deephaven readCsvImpl

    @Jolanrensen
    Jolanrensen committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    0c938b7 View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2024

  1. adding doc preprocessor for dataframe-csv

    @Jolanrensen
    Jolanrensen committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    e91dcae View commit details
    Browse the repository at this point in the history

  2. adding Char parser

    @Jolanrensen
    Jolanrensen committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    3ed7f3e View commit details
    Browse the repository at this point in the history

  3. continuing initial csv implementation

    @Jolanrensen
    Jolanrensen committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    13f3ffa View commit details
    Browse the repository at this point in the history

  4. Merge branch 'parsing-optimization' into new-csv-implementation

    @Jolanrensen
    Jolanrensen committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    6268237 View commit details
    Browse the repository at this point in the history

  5. continuing initial csv implementation

    @Jolanrensen
    Jolanrensen committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    0a77940 View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2024

  1. removing fuel dependency

    @Jolanrensen
    Jolanrensen committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    c0d76ca View commit details
    Browse the repository at this point in the history

  2. putting char later in the parsers

    @Jolanrensen
    Jolanrensen committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    52e463f View commit details
    Browse the repository at this point in the history

  3. starting with tests

    @Jolanrensen
    Jolanrensen committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    70482b7 View commit details
    Browse the repository at this point in the history

  4. adding case for FileUrlConnection

    @Jolanrensen
    Jolanrensen committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    fdb7032 View commit details
    Browse the repository at this point in the history

  5. continuing new csv implementation, working on passing tests

    @Jolanrensen
    Jolanrensen committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    c153528 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2024

  1. Merge branch 'parsing-optimization' into new-csv-implementation 

    # Conflicts:
#	core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/impl/api/parse.kt
    @Jolanrensen
    Jolanrensen committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    fb3af5e View commit details
    Browse the repository at this point in the history

  2. Added semi-internal DoubleParser class, which includes FastDoublePars… 

    …er with locales taken into account. ParserOptions.useFastDoubleParser was added to turn this new parser on or off
    @Jolanrensen
    Jolanrensen committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    bce432a View commit details
    Browse the repository at this point in the history

  3. switching csv implementation to use the new fast double parser in :core

    @Jolanrensen
    Jolanrensen committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    843be8e View commit details
    Browse the repository at this point in the history

  4. supporting different compressions csv

    @Jolanrensen
    Jolanrensen committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    ac8a5e5 View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2024

  1. improved DoubleParser and DecimalFormatBridge such that characters sp… 

    …anning multiple bytes can now also be converted to other locales before being parsed. Added tests
    @Jolanrensen
    Jolanrensen committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    9acf660 View commit details
    Browse the repository at this point in the history

  2. adding locale-based tests

    @Jolanrensen
    Jolanrensen committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    7c90c9c View commit details
    Browse the repository at this point in the history

  3. renaming to delim

    @Jolanrensen
    Jolanrensen committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    981acf5 View commit details
    Browse the repository at this point in the history

  4. working on kdocs

    @Jolanrensen
    Jolanrensen committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    a7967a3 View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2024

  1. streamlined Compression a bit, added test for empty lines. Taking par… 

    …serOptions.skipTypes into account in deephaven
    @Jolanrensen
    Jolanrensen committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    8e157e6 View commit details
    Browse the repository at this point in the history

  2. csv documentation

    @Jolanrensen
    Jolanrensen committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    5e2b9ab View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2024

  1. Merge branch 'master' into new-csv-implementation 

    # Conflicts:
#	core/generated-sources/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api/OtherSamples.kt
#	core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api/OtherSamples.kt
    @Jolanrensen
    Jolanrensen committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    8ff35d9 View commit details
    Browse the repository at this point in the history

  2. added test for NA doubles and custom nullStrings in csv double parser

    @Jolanrensen
    Jolanrensen committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    8c2d501 View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2024

  1. adding benchmarking test and fixed some compilation issues

    @Jolanrensen
    Jolanrensen committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    920c1a0 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'refs/heads/master' into new-csv-implementation

    @Jolanrensen
    Jolanrensen committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    54f3d18 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'refs/heads/master' into new-csv-implementation

    @Jolanrensen
    Jolanrensen committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    21c91f6 View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2024

  1. testing with ConfigurableDoubleParser

    @Jolanrensen
    Jolanrensen committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    cdf47a0 View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2024

  1. wip csv docs

    @Jolanrensen
    Jolanrensen committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    b2592ec View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2024

  1. updating double parser

    @Jolanrensen
    Jolanrensen committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    e86efea View commit details
    Browse the repository at this point in the history

  2. refactoring csv files and finished kdocs

    @Jolanrensen
    Jolanrensen committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    7ee1f10 View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2024

  1. added useFastDoubleParser option to convert as well and defaulting it… 

    … to false everywhere.
    @Jolanrensen
    Jolanrensen committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    711f4b3 View commit details
    Browse the repository at this point in the history

  2. improving kdocs and tests

    @Jolanrensen
    Jolanrensen committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    cc32153 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into new-csv-implementation 

    # Conflicts:
#	core/generated-sources/src/main/kotlin/org/jetbrains/kotlinx/dataframe/api/flatten.kt
#	core/generated-sources/src/main/kotlin/org/jetbrains/kotlinx/dataframe/api/insert.kt
#	core/generated-sources/src/main/kotlin/org/jetbrains/kotlinx/dataframe/api/parse.kt
#	core/generated-sources/src/main/kotlin/org/jetbrains/kotlinx/dataframe/columns/BaseColumn.kt
#	core/generated-sources/src/main/kotlin/org/jetbrains/kotlinx/dataframe/columns/ColumnGroup.kt
#	core/generated-sources/src/main/kotlin/org/jetbrains/kotlinx/dataframe/impl/TypeUtils.kt
#	core/generated-sources/src/main/kotlin/org/jetbrains/kotlinx/dataframe/impl/api/flatten.kt
#	core/generated-sources/src/main/kotlin/org/jetbrains/kotlinx/dataframe/impl/api/insert.kt
#	core/generated-sources/src/main/kotlin/org/jetbrains/kotlinx/dataframe/impl/api/parse.kt
#	core/generated-sources/src/main/kotlin/org/jetbrains/kotlinx/dataframe/jupyter/Integration.kt
#	core/generated-sources/src/test/kotlin/org/jetbrains/kotlinx/dataframe/api/constructors.kt
#	core/generated-sources/src/test/kotlin/org/jetbrains/kotlinx/dataframe/api/flatten.kt
    @Jolanrensen
    Jolanrensen committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    639156b View commit details
    Browse the repository at this point in the history

  4. updating from master

    @Jolanrensen
    Jolanrensen committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    c811d70 View commit details
    Browse the repository at this point in the history

  5. some implementation refactoring of csv

    @Jolanrensen
    Jolanrensen committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    86dc97f View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2024

  1. Merge branch 'master' into new-csv-implementation 

    # Conflicts:
#	.github/workflows/generated-sources.yml
    @Jolanrensen
    Jolanrensen committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    eacc990 View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2024

  1. updated fastDoubleParser to 2.0.0

    @Jolanrensen
    Jolanrensen committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    4d3a7e3 View commit details
    Browse the repository at this point in the history

  2. small optimizations for the double parser

    @Jolanrensen
    Jolanrensen committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    d01e6a4 View commit details
    Browse the repository at this point in the history