Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds implementation for a Split Assigner and a Split Reader #44

Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
eb9dc8f
cleaning up the main branch for new version.
prodriguezdefino Jun 27, 2023
0695673
updates in ignore and readme files
prodriguezdefino Jun 27, 2023
e36cc44
prepping the pom addition, added parent's compliance tools
prodriguezdefino Jun 27, 2023
e167803
adding parent pom and the connector impl project pom
prodriguezdefino Jun 28, 2023
8043378
adding common functionalities
prodriguezdefino Jun 28, 2023
b455b4e
added the bigquery services wrapper and factories
prodriguezdefino Jun 28, 2023
27cd837
creates the split, its state and the enumerator state
prodriguezdefino Jun 28, 2023
4d4b60f
added configs, split reader and split assignment impls
prodriguezdefino Jun 28, 2023
0567b58
applying recommendations from sonartype-lift
prodriguezdefino Jun 28, 2023
5743292
merge changes from master (previous pom deletion resolution)
prodriguezdefino Jul 6, 2023
cab9115
fixing the package name for the schema namespace
prodriguezdefino Jul 6, 2023
3375d54
Merge branch 'common_code_source' into bq_services_wrappers
prodriguezdefino Jul 10, 2023
849f769
merged main branch and took care of few lift comments
prodriguezdefino Jul 11, 2023
fd70b95
Merge branch 'bq_services_wrappers' into source_splits
prodriguezdefino Jul 11, 2023
0f18d14
merge from source_split
prodriguezdefino Jul 11, 2023
6b08119
fixing lift recommendations and spotless
prodriguezdefino Jul 11, 2023
f473d57
addressing comments from review
prodriguezdefino Jul 27, 2023
c178f83
merge from main
prodriguezdefino Aug 1, 2023
09eaaa4
merge from master
prodriguezdefino Aug 1, 2023
def3cc4
Merge branch 'source_splits' into split_assigner_and_reader
prodriguezdefino Aug 1, 2023
ceabb12
fixed type reference Int -> Long
prodriguezdefino Aug 1, 2023
1734bac
merge from main
prodriguezdefino Aug 8, 2023
e96ff59
addressing comments from review
prodriguezdefino Aug 8, 2023
c492f02
improved hashcode and equals readability
prodriguezdefino Aug 8, 2023
9aae0af
changed tests to use google-truth instead of junit or assertj asserti…
prodriguezdefino Aug 9, 2023
edded84
addressing comments from review
prodriguezdefino Oct 9, 2023
c00b79c
resolved commments from review (new specific exception and optional o…
prodriguezdefino Oct 30, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .asf.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
github:
enabled_merge_buttons:
squash: true
merge: false
rebase: true
labels:
- flink
- bigquery
- connector
autolink_jira: FLINK
collaborators:
- flinkbot
276 changes: 276 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,276 @@
root = true

[*]
charset = utf-8
end_of_line = lf
insert_final_newline = true
max_line_length = 100
# ij_formatter_off_tag = @formatter:off
# ij_formatter_on_tag = @formatter:on
# ij_formatter_tags_enabled = false
# ij_smart_tabs = false
# ij_wrap_on_typing = false

[*.java]
indent_size = 4
indent_style = space
tab_width = 4
ij_continuation_indent_size = 8
# ij_java_align_consecutive_assignments = false
# ij_java_align_consecutive_variable_declarations = false
# ij_java_align_group_field_declarations = false
# ij_java_align_multiline_annotation_parameters = false
# ij_java_align_multiline_array_initializer_expression = false
# ij_java_align_multiline_assignment = false
# ij_java_align_multiline_binary_operation = false
# ij_java_align_multiline_chained_methods = false
# ij_java_align_multiline_extends_list = false
# ij_java_align_multiline_for = true
# ij_java_align_multiline_method_parentheses = false
# ij_java_align_multiline_parameters = true
# ij_java_align_multiline_parameters_in_calls = false
# ij_java_align_multiline_parenthesized_expression = false
# ij_java_align_multiline_records = true
# ij_java_align_multiline_resources = true
# ij_java_align_multiline_ternary_operation = false
# ij_java_align_multiline_text_blocks = false
# ij_java_align_multiline_throws_list = false
# ij_java_align_subsequent_simple_methods = false
# ij_java_align_throws_keyword = false
# ij_java_annotation_parameter_wrap = off
# ij_java_array_initializer_new_line_after_left_brace = false
# ij_java_array_initializer_right_brace_on_new_line = false
# ij_java_array_initializer_wrap = off
# ij_java_assert_statement_colon_on_next_line = false
# ij_java_assert_statement_wrap = off
# ij_java_assignment_wrap = off
ij_java_binary_operation_sign_on_next_line = true
ij_java_binary_operation_wrap = normal
# ij_java_blank_lines_after_anonymous_class_header = 0
# ij_java_blank_lines_after_class_header = 0
# ij_java_blank_lines_after_imports = 1
# ij_java_blank_lines_after_package = 1
# ij_java_blank_lines_around_class = 1
# ij_java_blank_lines_around_field = 0
# ij_java_blank_lines_around_field_in_interface = 0
# ij_java_blank_lines_around_initializer = 1
# ij_java_blank_lines_around_method = 1
# ij_java_blank_lines_around_method_in_interface = 1
# ij_java_blank_lines_before_class_end = 0
# ij_java_blank_lines_before_imports = 1
# ij_java_blank_lines_before_method_body = 0
# ij_java_blank_lines_before_package = 0
# ij_java_block_brace_style = end_of_line
# ij_java_block_comment_at_first_column = true
ij_java_call_parameters_new_line_after_left_paren = true
# ij_java_call_parameters_right_paren_on_new_line = false
ij_java_call_parameters_wrap = on_every_item
# ij_java_case_statement_on_separate_line = true
# ij_java_catch_on_new_line = false
# ij_java_class_annotation_wrap = split_into_lines
# ij_java_class_brace_style = end_of_line
ij_java_class_count_to_use_import_on_demand = 9999
# ij_java_class_names_in_javadoc = 1
# ij_java_do_not_indent_top_level_class_members = false
# ij_java_do_not_wrap_after_single_annotation = false
# ij_java_do_while_brace_force = never
# ij_java_doc_add_blank_line_after_description = true
ij_java_doc_add_blank_line_after_param_comments = true
ij_java_doc_add_blank_line_after_return = true
# ij_java_doc_add_p_tag_on_empty_lines = true
ij_java_doc_align_exception_comments = false
ij_java_doc_align_param_comments = false
ij_java_doc_do_not_wrap_if_one_line = true
ij_java_doc_enable_formatting = true
# ij_java_doc_enable_leading_asterisks = true
ij_java_doc_indent_on_continuation = true
ij_java_doc_keep_empty_lines = true
# ij_java_doc_keep_empty_parameter_tag = true
# ij_java_doc_keep_empty_return_tag = true
# ij_java_doc_keep_empty_throws_tag = true
# ij_java_doc_keep_invalid_tags = true
# ij_java_doc_param_description_on_new_line = false
ij_java_doc_preserve_line_breaks = false
# ij_java_doc_use_throws_not_exception_tag = true
# ij_java_else_on_new_line = false
# ij_java_entity_dd_suffix = EJB
# ij_java_entity_eb_suffix = Bean
# ij_java_entity_hi_suffix = Home
# ij_java_entity_lhi_prefix = Local
# ij_java_entity_lhi_suffix = Home
# ij_java_entity_li_prefix = Local
# ij_java_entity_pk_class = java.lang.String
# ij_java_entity_vo_suffix = VO
# ij_java_enum_constants_wrap = off
# ij_java_extends_keyword_wrap = off
# ij_java_extends_list_wrap = off
# ij_java_field_annotation_wrap = split_into_lines
# ij_java_finally_on_new_line = false
# ij_java_for_brace_force = never
# ij_java_for_statement_new_line_after_left_paren = false
# ij_java_for_statement_right_paren_on_new_line = false
# ij_java_for_statement_wrap = off
# ij_java_generate_final_locals = false
# ij_java_generate_final_parameters = false
# ij_java_if_brace_force = never
ij_java_imports_layout = org.apache.flink.**,|,org.apache.flink.shaded.**,|,*,|,javax.**,|,java.**,|,scala.**,|,$*
# ij_java_indent_case_from_switch = true
# ij_java_insert_inner_class_imports = false
# ij_java_insert_override_annotation = true
# ij_java_keep_blank_lines_before_right_brace = 2
# ij_java_keep_blank_lines_between_package_declaration_and_header = 2
# ij_java_keep_blank_lines_in_code = 2
# ij_java_keep_blank_lines_in_declarations = 2
# ij_java_keep_control_statement_in_one_line = true
# ij_java_keep_first_column_comment = true
# ij_java_keep_indents_on_empty_lines = false
# ij_java_keep_line_breaks = true
# ij_java_keep_multiple_expressions_in_one_line = false
# ij_java_keep_simple_blocks_in_one_line = false
# ij_java_keep_simple_classes_in_one_line = false
# ij_java_keep_simple_lambdas_in_one_line = false
# ij_java_keep_simple_methods_in_one_line = false
# ij_java_label_indent_absolute = false
# ij_java_label_indent_size = 0
# ij_java_lambda_brace_style = end_of_line
ij_java_layout_static_imports_separately = true
# ij_java_line_comment_add_space = false
# ij_java_line_comment_at_first_column = true
# ij_java_message_dd_suffix = EJB
# ij_java_message_eb_suffix = Bean
# ij_java_method_annotation_wrap = split_into_lines
# ij_java_method_brace_style = end_of_line
ij_java_method_call_chain_wrap = on_every_item
ij_java_method_parameters_new_line_after_left_paren = true
# ij_java_method_parameters_right_paren_on_new_line = false
ij_java_method_parameters_wrap = on_every_item
# ij_java_modifier_list_wrap = false
ij_java_names_count_to_use_import_on_demand = 9999
# ij_java_new_line_after_lparen_in_record_header = false
# ij_java_packages_to_use_import_on_demand = java.awt.*,javax.swing.*
# ij_java_parameter_annotation_wrap = off
# ij_java_parentheses_expression_new_line_after_left_paren = false
# ij_java_parentheses_expression_right_paren_on_new_line = false
# ij_java_place_assignment_sign_on_next_line = false
# ij_java_prefer_longer_names = true
# ij_java_prefer_parameters_wrap = false
# ij_java_record_components_wrap = normal
# ij_java_repeat_synchronized = true
# ij_java_replace_instanceof_and_cast = false
# ij_java_replace_null_check = true
# ij_java_replace_sum_lambda_with_method_ref = true
# ij_java_resource_list_new_line_after_left_paren = false
# ij_java_resource_list_right_paren_on_new_line = false
# ij_java_resource_list_wrap = off
# ij_java_rparen_on_new_line_in_record_header = false
# ij_java_session_dd_suffix = EJB
# ij_java_session_eb_suffix = Bean
# ij_java_session_hi_suffix = Home
# ij_java_session_lhi_prefix = Local
# ij_java_session_lhi_suffix = Home
# ij_java_session_li_prefix = Local
# ij_java_session_si_suffix = Service
# ij_java_space_after_closing_angle_bracket_in_type_argument = false
# ij_java_space_after_colon = true
# ij_java_space_after_comma = true
# ij_java_space_after_comma_in_type_arguments = true
# ij_java_space_after_for_semicolon = true
# ij_java_space_after_quest = true
# ij_java_space_after_type_cast = true
# ij_java_space_before_annotation_array_initializer_left_brace = false
# ij_java_space_before_annotation_parameter_list = false
# ij_java_space_before_array_initializer_left_brace = false
# ij_java_space_before_catch_keyword = true
# ij_java_space_before_catch_left_brace = true
# ij_java_space_before_catch_parentheses = true
# ij_java_space_before_class_left_brace = true
# ij_java_space_before_colon = true
# ij_java_space_before_colon_in_foreach = true
# ij_java_space_before_comma = false
# ij_java_space_before_do_left_brace = true
# ij_java_space_before_else_keyword = true
# ij_java_space_before_else_left_brace = true
# ij_java_space_before_finally_keyword = true
# ij_java_space_before_finally_left_brace = true
# ij_java_space_before_for_left_brace = true
# ij_java_space_before_for_parentheses = true
# ij_java_space_before_for_semicolon = false
# ij_java_space_before_if_left_brace = true
# ij_java_space_before_if_parentheses = true
# ij_java_space_before_method_call_parentheses = false
# ij_java_space_before_method_left_brace = true
# ij_java_space_before_method_parentheses = false
# ij_java_space_before_opening_angle_bracket_in_type_parameter = false
# ij_java_space_before_quest = true
# ij_java_space_before_switch_left_brace = true
# ij_java_space_before_switch_parentheses = true
# ij_java_space_before_synchronized_left_brace = true
# ij_java_space_before_synchronized_parentheses = true
# ij_java_space_before_try_left_brace = true
# ij_java_space_before_try_parentheses = true
# ij_java_space_before_type_parameter_list = false
# ij_java_space_before_while_keyword = true
# ij_java_space_before_while_left_brace = true
# ij_java_space_before_while_parentheses = true
# ij_java_space_inside_one_line_enum_braces = false
# ij_java_space_within_empty_array_initializer_braces = false
# ij_java_space_within_empty_method_call_parentheses = false
# ij_java_space_within_empty_method_parentheses = false
# ij_java_spaces_around_additive_operators = true
# ij_java_spaces_around_assignment_operators = true
# ij_java_spaces_around_bitwise_operators = true
# ij_java_spaces_around_equality_operators = true
# ij_java_spaces_around_lambda_arrow = true
# ij_java_spaces_around_logical_operators = true
# ij_java_spaces_around_method_ref_dbl_colon = false
# ij_java_spaces_around_multiplicative_operators = true
# ij_java_spaces_around_relational_operators = true
# ij_java_spaces_around_shift_operators = true
# ij_java_spaces_around_type_bounds_in_type_parameters = true
# ij_java_spaces_around_unary_operator = false
# ij_java_spaces_within_angle_brackets = false
# ij_java_spaces_within_annotation_parentheses = false
# ij_java_spaces_within_array_initializer_braces = false
# ij_java_spaces_within_braces = false
# ij_java_spaces_within_brackets = false
# ij_java_spaces_within_cast_parentheses = false
# ij_java_spaces_within_catch_parentheses = false
# ij_java_spaces_within_for_parentheses = false
# ij_java_spaces_within_if_parentheses = false
# ij_java_spaces_within_method_call_parentheses = false
# ij_java_spaces_within_method_parentheses = false
# ij_java_spaces_within_parentheses = false
# ij_java_spaces_within_switch_parentheses = false
# ij_java_spaces_within_synchronized_parentheses = false
# ij_java_spaces_within_try_parentheses = false
# ij_java_spaces_within_while_parentheses = false
# ij_java_special_else_if_treatment = true
# ij_java_subclass_name_suffix = Impl
# ij_java_ternary_operation_signs_on_next_line = false
# ij_java_ternary_operation_wrap = off
# ij_java_test_name_suffix = Test
# ij_java_throws_keyword_wrap = off
# ij_java_throws_list_wrap = off
# ij_java_use_external_annotations = false
# ij_java_use_fq_class_names = false
# ij_java_use_relative_indents = false
# ij_java_use_single_class_imports = true
ij_java_variable_annotation_wrap = normal
# ij_java_visibility = public
# ij_java_while_brace_force = never
# ij_java_while_on_new_line = false
# ij_java_wrap_comments = false
ij_java_wrap_first_method_in_call_chain = true
# ij_java_wrap_long_lines = false

[*.xml]
indent_style = tab
indent_size = 4

[*.scala]
indent_style = space
indent_size = 2

[*.py]
indent_style = space
indent_size = 4
9 changes: 9 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,12 @@ buildNumber.properties
# https://github.com/takari/maven-wrapper#usage-without-binary-jar
.mvn/wrapper/maven-wrapper.jar
.idea/
.DS_Store
tools/flink
tools/flink-*
tools/releasing/release
tools/japicmp-output
java.header
.java-version
nb-*
**nbproject
16 changes: 16 additions & 0 deletions NOTICE
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
Apache Flink BigQuery Connector
Copyright 2023 The Apache Software Foundation

This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).

Permission to use, copy, modify, and/or distribute this software for any purpose with or without fee is hereby
granted, provided that this permission notice appear in all copies.

THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING
ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL,
DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,
WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE
USE OR PERFORMANCE OF THIS SOFTWARE.


70 changes: 68 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,71 @@
# Flink BigQuery Connector (Under Development)
BigQuery integration to Apache Flink's Table API
# Apache Flink Google BigQuery Connector (Under Development)

[![Code Quality: Java](https://img.shields.io/lgtm/grade/java/g/GoogleCloudDataproc/flink-bigquery-connector.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/GoogleCloudDataproc/flink-bigquery-connector/context:java)
[![codecov](https://codecov.io/gh/GoogleCloudDataproc/flink-bigquery-connector/branch/master/graph/badge.svg)](https://codecov.io/gh/GoogleCloudDataproc/flink-bigquery-connector)

This repository contains the Apache Flink BigQuery connector.

## Apache Flink

Apache Flink is an open source stream processing framework with powerful stream- and batch-processing capabilities.

Learn more about Flink at [https://flink.apache.org/](https://flink.apache.org/)

## Building the Apache Flink BigQuery Connector from Source

Prerequisites:

* Unix-like environment (we use Linux, Mac OS X)
* Git
* Maven (we recommend version 3.8.6)
* Java 11

```
git clone https://github.com/GoogleCloudDataproc/flink-bigquery-connector
cd flink-connector-bigquery
mvn clean package -DskipTests
```

The resulting jars can be found in the `target` directory of the respective module.

## Developing Flink

The Flink committers use IntelliJ IDEA to develop the Flink codebase.
We recommend IntelliJ IDEA for developing projects that involve Scala code.

Minimal requirements for an IDE are:
* Support for Java and Scala (also mixed projects)
* Support for Maven with Java and Scala

### IntelliJ IDEA

The IntelliJ IDE supports Maven out of the box and offers a plugin for Scala development.

* IntelliJ download: [https://www.jetbrains.com/idea/](https://www.jetbrains.com/idea/)
* IntelliJ Scala Plugin: [https://plugins.jetbrains.com/plugin/?id=1347](https://plugins.jetbrains.com/plugin/?id=1347)

Check out our [Setting up IntelliJ](https://nightlies.apache.org/flink/flink-docs-master/flinkDev/ide_setup.html#intellij-idea) guide for details.

## Support

Don’t hesitate to ask!

Contact the developers and community on the [mailing lists](https://flink.apache.org/community.html#mailing-lists) if you need any help.

[Open an issue](https://issues.apache.org/jira/browse/FLINK) if you found a bug in Flink.

## Documentation

The documentation of Apache Flink is located on the website: [https://flink.apache.org](https://flink.apache.org)
or in the `docs/` directory of the source code.

## Fork and Contribute

This is an active open-source project. We are always open to people who want to use the system or contribute to it.
Contact us if you are looking for implementation tasks that fit your skills.
This article describes [how to contribute to Apache Flink](https://flink.apache.org/contributing/how-to-contribute.html).

## About

Apache Flink is an open source project of The Apache Software Foundation (ASF).
The Apache Flink project originated from the [Stratosphere](http://stratosphere.eu) research project.
Loading