Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add KafkaBatchPublisher #19

Merged
merged 12 commits into from
Oct 3, 2024
Merged

feat: add KafkaBatchPublisher #19

merged 12 commits into from
Oct 3, 2024

Commits on Oct 3, 2024

  1. feat: add kafka-ui service so I can see what goes on 

    Also:
- reshaped some data so that type checker stops complaining
    Federico Zambelli committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    2da9db5 View commit details
    Browse the repository at this point in the history

  2. test: separate batch from stream writer

    Federico Zambelli committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    aeaed3d View commit details
    Browse the repository at this point in the history

  3. feat: add chispa pkg for easy spark DF comparisons 

    See: https://github.com/MrPowers/chispa

Also:
- cleanup pytest log output
    Federico Zambelli committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    c558638 View commit details
    Browse the repository at this point in the history

  4. feat(kafkaWriter): add batch publisher

    Federico Zambelli committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    03457fe View commit details
    Browse the repository at this point in the history

  5. test(kafkaWriter): add tests for batch publisher

    Federico Zambelli committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    bf8c9e8 View commit details
    Browse the repository at this point in the history

  6. chore: ignore mypy warning for dev dependency "chispa" 

    Chispa doesn't have stubs or py.typed marker.
Mypy throws error for that. This seems unnecessarily strict for
a package that is just used for comparing dataframes in unit testing.

This commit disables the behavior for this specific package.
    Federico Zambelli committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    a96f6a2 View commit details
    Browse the repository at this point in the history

  7. fix: check if Kafka columns in writer are *exactly* {key, value} 

    Co-authored-by: Reza Khanipour <[email protected]>
    @wtfzambo @shahinism
    2 people authored and Federico Zambelli committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    33ce26e View commit details
    Browse the repository at this point in the history

  8. Update tests/conftest.py 

    Co-authored-by: Reza Khanipour <[email protected]>
    @wtfzambo @shahinism
    2 people authored and Federico Zambelli committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    d148db7 View commit details
    Browse the repository at this point in the history

  9. chore: add explanation to pytest flags

    Federico Zambelli committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    63a2851 View commit details
    Browse the repository at this point in the history

  10. chore: add logging to some tests

    Federico Zambelli committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    c108e70 View commit details
    Browse the repository at this point in the history

  11. fix: add missing colon

    Federico Zambelli committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    7791ae0 View commit details
    Browse the repository at this point in the history

  12. chore: add script to remove unused containers

    Federico Zambelli committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    f992d87 View commit details
    Browse the repository at this point in the history