Skip to content

Commit

Permalink
First commit for the App code
Browse files Browse the repository at this point in the history
  • Loading branch information
stheppi committed Nov 3, 2023
1 parent 6f0dd74 commit a225496
Show file tree
Hide file tree
Showing 33 changed files with 2,294 additions and 1 deletion.
84 changes: 83 additions & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
@@ -1,2 +1,84 @@
# restore-consumer-groups-offset
An application allowing to read the consumer groups offsets stored in S3 and apply them to the target Kafka cluster

An application allowing to read the consumer groups offsets stored in S3 and apply them to the target Kafka cluster.
The S3 sink stores the information under the following path: `bucket[/prefix]/group/topic/partition`, with the content
being the offset (8 bytes array).

### Configuration

The application requires the configuration file. The configuration file in HOCON format supports the following options:

| Configuration Option | Description |
|---------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `Kafka configuration` | All Kafka settings should be prefixed with `kafka.`. For example: `kafka.bootstrap.servers = "localhost:9092"`. |
| `S3 location` | - `aws.bucket`: The name of the S3 bucket where consumer group offsets are stored. |
| | - `aws.prefix` (Optional): The prefix of objects within the bucket. |
| `groups` | An optional, comma-separated list of consumer groups to restore. If not specified, all groups stored in S3 will be restored. For example: `groups = group1, group2`. |
| `AWS connection mode` | - `aws.mode`: Set to `credentials` to use provided credentials or `default` for AWS default credentials provider chain. |
| `AWS Access Key` | - `aws.access.key`: AWS access key ID (only when `aws.mode` is set to `credentials`). |
| `AWS Secret Key` | - `aws.secret.key`: AWS secret access key (only when `aws.mode` is `credentials`). |
| `AWS Region` | - `aws.region`: AWS region (only when `aws.mode` is `credentials`). |
| `AWS HTTP Retries` | - `aws.http.retries`: How many times a failed request is attempted. Default is 5 |
| `AWS HTTP Retry interval` | - `aws.http.retry.inteval`: The time in milliseconds to wait before an HTTP operation is retried. Default is 50. |


#### Examples

##### Example 1

```hocon
kafka {
bootstrap.servers = "localhost:9092"
security.protocol = "SSL"
# Add other Kafka settings here
}
aws {
bucket = "your-s3-bucket"
prefix = "optional-prefix"
}
# Optional: Specify the consumer groups to restore
groups = "group1,group2"
aws {
mode = "credentials" # or "default"
access.key = "your-access-key"
secret.key = "your-secret-key"
region = "your-aws-region"
}
```

##### Example 2

```hocon
kafka.bootstrap.servers = "localhost:9092"
aws.bucket = "your-s3-bucket"
aws.prefix = "optional-prefix"
groups = "group1,group2"
aws.mode = "credentials" # or "default"
aws.access.key = "your-access-key"
aws.secret.key = "your-secret-key"
aws.region = "your-aws-region"
```

## Running the application

It requires at least Java 8 to run.



To format the code run:

```bash
mvn com.coveo:fmt-maven-plugin:format
```

To add license header, run:

```bash
mvn license:format
```
382 changes: 382 additions & 0 deletions checkstyle/checkstyle.xml
View file

Large diffs are not rendered by default.

16 changes: 16 additions & 0 deletions checkstyle/java.header
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
27 changes: 27 additions & 0 deletions checkstyle/suppressions.xml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
<?xml version="1.0"?>

<!DOCTYPE suppressions PUBLIC
"-//Puppy Crawl//DTD Suppressions 1.1//EN"
"http://www.puppycrawl.com/dtds/suppressions_1_1.dtd">
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<suppressions>

<!-- Note that [/\\] must be used as the path separator for cross-platform support -->


</suppressions>
199 changes: 199 additions & 0 deletions pom.xml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,199 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>io.lenses</groupId>
<artifactId>restore-group-offsets-s3</artifactId>
<version>1.0.0</version>
<packaging>jar</packaging>

<properties>
<maven.compiler.source>8</maven.compiler.source>
<maven.compiler.target>8</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<java.version>1.8</java.version>
<aws.version>2.21.13</aws.version>
<hocon.version>1.4.2</hocon.version>
<kafka.version>3.5.0</kafka.version>
<mockito.version>3.12.4</mockito.version>
<headerFile>${project.basedir}/checkstyle/java.header</headerFile>
</properties>

<dependencies>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>${kafka.version}</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3</artifactId>
<version>${aws.version}</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>sts</artifactId>
<version>${aws.version}</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>apache-client</artifactId>
<version>${aws.version}</version>
</dependency>
<dependency>
<groupId>com.typesafe</groupId>
<artifactId>config</artifactId>
<version>${hocon.version}</version>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<version>RELEASE</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<version>${mockito.version}</version>
<scope>test</scope>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>com.coveo</groupId>
<artifactId>fmt-maven-plugin</artifactId>
<version>2.9</version>
<configuration>
<sourceDirectory>src/main/java</sourceDirectory>
<testSourceDirectory>src/test/java</testSourceDirectory>
<verbose>false</verbose>
<filesNamePattern>.*\.java</filesNamePattern>
<skip>false</skip>
<skipSortingImports>false</skipSortingImports>
<style>google</style>
</configuration>
<executions>
<execution>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.11.0</version>
<inherited>true</inherited>
<configuration>
<source>${java.version}</source>
<target>${java.version}</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>3.3.0</version>
<configuration>
<!-- <archive>
<manifest>
<addClasspath>true</addClasspath>
<classpathPrefix>lib</classpathPrefix>
<mainClass>io.lenses.App</mainClass>
</manifest>
</archive>-->

<archive>
<manifest>
<mainClass>io.lenses.App</mainClass>
<addDefaultImplementationEntries>true</addDefaultImplementationEntries>
<addDefaultSpecificationEntries>true</addDefaultSpecificationEntries>
</manifest>
</archive>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.6.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>
<appendAssemblyId>false</appendAssemblyId>
<descriptors>
<descriptor>src/main/assembly/zip.xml</descriptor>
</descriptors>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19.1</version>
<configuration>
<forkCount>1</forkCount>
<reuseForks>false</reuseForks>
</configuration>
</plugin>

<plugin>
<groupId>com.mycila</groupId>
<artifactId>license-maven-plugin</artifactId>
<version>4.2</version>
<configuration>
<licenseSets>
<licenseSet>
<header>${project.basedir}/src/main/resources/license-header.txt</header>
<includes>
<include>**/*.java</include>
</includes>
<excludes>
<exclude>**/README</exclude>
<exclude>${project.basedir}/src/test/resources/**</exclude>
<exclude>${project.basedir}/src/main/resources/**</exclude>
<exclude>**/generated/**/*.java</exclude>
<exclude>*.pom</exclude>
<exclude>.gitignore</exclude>
<exclude>${project.basedir}/.github/**</exclude>
<exclude>${project.basedir}/.idea/**</exclude>
<exclude>checkstyle/**</exclude>
</excludes>
</licenseSet>
</licenseSets>
<mapping>
<java>JAVADOC_STYLE</java>
</mapping>
</configuration>
</plugin>
</plugins>
</build>
</project>
42 changes: 42 additions & 0 deletions restore.sh
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
#!/bin/bash

# Set the base directory where this script is located
BASEDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"



# Initialize optional arguments
PREVIEW=false
CONFIG_FILE=""

# Create a classpath variable to include all JARs in the lib folder
CLASSPATH="${BASEDIR}/../lib/*"

# Parse command line arguments
while [[ $# -gt 0 ]]; do
key="$1"
case $key in
--preview)
PREVIEW=true
;;
*)
CONFIG_FILE="$1"
;;
esac
shift
done

# Check if a configuration file is provided
if [ -z "$CONFIG_FILE" ]; then
echo "Error: Please specify the application configuration file."
exit 1
fi

# Add optional logic for handling the --preview flag
if [ "$PREVIEW" = true ]; then
echo "Running the application in preview mode with configuration file: $CONFIG_FILE"
java -cp "$CLASSPATH" io.lenses.App --config "$CONFIG_FILE" --preview
else
echo "Running the application with configuration file: $CONFIG_FILE"
java -cp "$CLASSPATH" io.lenses.App --config "$CONFIG_FILE"
fi
Loading

0 comments on commit a225496

Please sign in to comment.