-
Notifications
You must be signed in to change notification settings - Fork 17
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Initial transfer of Equinox.Projection.Kafka
- Loading branch information
Showing
15 changed files
with
1,094 additions
and
69 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| # Changelog | ||
|
|
||
| The repo is versioned based on [SemVer 2.0](https://semver.org/spec/v2.0.0.html) using the tiny-but-mighty [MinVer](https://github.com/adamralph/minver) from [@adamralph](https://github.com/adamralph). [See here](https://github.com/adamralph/minver#how-it-works) for more information on how it works. | ||
|
|
||
| All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). | ||
|
|
||
| The `Unreleased` section name is replaced by the expected version of next release. A stable version's log contains all changes between that version and the previous stable version (can duplicate the prereleases logs). | ||
|
|
||
| ## [Unreleased] | ||
|
|
||
| ### Added | ||
| ### Changed | ||
| ### Removed | ||
| ### Fixed | ||
|
|
||
| <a name="1.0.0-preview1"></a> | ||
| ## [1.0.0-preview1] - 2019-03-05 | ||
|
|
||
| (Extracted from [Equinox Changelog](https://github.com/jet/equinox/blob/master/CHANGELOG.md) - this codebase was maintained within that repo originally) | ||
|
|
||
| ### Added | ||
|
|
||
| - `Equinox.Projection.Kafka` consumer metrics emission, see [#94](https://github.com/jet/equinox/pull/94) @michaelliao5 | ||
| - Initial release as part of `Equinox.Cosmos` projection facilities, see [#87](https://github.com/jet/equinox/pull/87) | ||
|
|
||
| <a name="1.0.0-initial"></a> | ||
| ## [1.0.0-initial] | ||
|
|
||
| (Stripped down repo for history purposes) | ||
|
|
||
| [Unreleased]: https://github.com/jet/-kafka-fsharp/compare/1.0.0-preview1...HEAD | ||
| [1.0.0-preview1]: https://github.com/jet/-kafka-fsharp/compare/1.0.0-alpha...1.0.0-preview1 | ||
| [1.0.0-initial]: https://github.com/jet/confluent-kafka-fsharp/compare/e4bc8ff53b4f4400308b09c02fe8da6fc7e61d82...1.0.0-initial |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,26 +1,35 @@ | ||
| <Project ToolsVersion="15.0"> | ||
| <PropertyGroup> | ||
| <VersionPrefix>0.9.4</VersionPrefix> | ||
| <Authors>@jet @eulerfx @eirik</Authors> | ||
| <Authors>@jet @eulerfx @eiriktsarpalis @michaelliao5 and contributors</Authors> | ||
| <Company>Jet.com</Company> | ||
| <Description>Lightweight F# wrapper for Confluent.Kafka</Description> | ||
| <PackageProjectUrl>https://github.com/jet/confluent-kafka-fsharp</PackageProjectUrl> | ||
| <RepositoryUrl>https://github.com/jet/confluent-kafka-fsharp</RepositoryUrl> | ||
| <RepositoryType>git</RepositoryType> | ||
| <PackageTags>kafka;fsharp</PackageTags> | ||
| <PackageLicenseUrl>https://github.com/jet/confluent-kafka-fsharp/blob/master/LICENSE.txt</PackageLicenseUrl> | ||
| <Copyright>Copyright © 2018</Copyright> | ||
| </PropertyGroup> | ||
| <PackageLicense>Apache-2.0</PackageLicense> | ||
| <Copyright>Copyright © 2016-9</Copyright> | ||
|
|
||
| <PropertyGroup> | ||
| <RepoRootDir>$([System.IO.Path]::GetFullPath("$(MSBuildThisFileDirectory)"))</RepoRootDir> | ||
| <NupkgsDir>$([System.IO.Path]::GetFullPath("$(MSBuildThisFileDirectory)/bin/nupkg"))</NupkgsDir> | ||
| <MonoOrEmpty Condition=" '$(OS)' != 'Windows_NT' ">mono </MonoOrEmpty> | ||
| <RepoDir>$([System.IO.Path]::GetFullPath("$(MSBuildThisFileDirectory)"))</RepoDir> | ||
|
|
||
| <!-- SourceLink related properties https://github.com/dotnet/SourceLink#using-sourcelink --> | ||
| <PublishRepositoryUrl>true</PublishRepositoryUrl> | ||
| <EmbedUntrackedSources>true</EmbedUntrackedSources> | ||
| <AllowedOutputExtensionsInPackageBuildOutputFolder>$(AllowedOutputExtensionsInPackageBuildOutputFolder);.pdb</AllowedOutputExtensionsInPackageBuildOutputFolder> | ||
| <!-- disable sourcelink on mono, to workaround https://github.com/dotnet/sourcelink/issues/155 --> | ||
| <EnableSourceLink Condition=" '$(OS)' != 'Windows_NT' AND '$(MSBuildRuntimeType)' != 'Core' ">false</EnableSourceLink> | ||
| <EnableSourceControlManagerQueries>$(EnableSourceLink)</EnableSourceControlManagerQueries> | ||
|
|
||
| <NoWarn>$(NoWarn);FS2003</NoWarn> | ||
| <!-- suppress false positive warning FS2003 about invalid version of AssemblyInformationalVersionAttribute --> | ||
| <!-- supress NU5105 triggered by trailing dotted elements such as .43 and .2 in e.g.: pr.43-rc1.2: The package version '<X>' uses SemVer 2.0.0 or components of SemVer 1.0.0 that are not supported on legacy clients. Change the package version to a SemVer 1.0.0 string. If the version contains a release label it must start with a letter. This message can be ignored if the package is not intended for older clients. --> | ||
| <NoWarn>$(NoWarn);FS2003;NU5105</NoWarn> | ||
| </PropertyGroup> | ||
|
|
||
| <!-- Workaround for https://github.com/xunit/xunit/issues/1357 --> | ||
| <PropertyGroup> | ||
| <ThisDirAbsolute>$([System.IO.Path]::GetFullPath("$(MSBuildThisFileDirectory)"))</ThisDirAbsolute> | ||
| </PropertyGroup> | ||
| </Project> | ||
| <ItemGroup> | ||
| <Content Include="$(ThisDirAbsolute)tests/xunit.runner.json" Condition=" '$(OS)' != 'Windows_NT' AND '$(IsTestProject)' != 'false'"> | ||
| <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory> | ||
| </Content> | ||
| </ItemGroup> | ||
|
|
||
| </Project> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| <Project ToolsVersion="15.0"> | ||
| <Target Name="ComputePackageVersion" AfterTargets="MinVer" Condition=" '$(BUILD_PR)' != '' AND '$(BUILD_PR)' != '%24(SYSTEM.PULLREQUEST.PULLREQUESTNUMBER)' "> | ||
| <PropertyGroup> | ||
| <PackageVersion>$(MinVerMajor).$(MinVerMinor).$(MinVerPatch)-pr.$(BUILD_PR)</PackageVersion> | ||
| <PackageVersion Condition="'$(MinVerPreRelease)' != ''">$(PackageVersion).$(MinVerPreRelease)</PackageVersion> | ||
| <PackageVersion Condition="'$(MinVerBuildMetadata)' != ''">$(PackageVersion)+$(MinVerBuildMetadata)</PackageVersion> | ||
| <Version>$(PackageVersion)</Version> | ||
| </PropertyGroup> | ||
| </Target> | ||
| <Target Name="ComputeFileVersion" AfterTargets="MinVer"> | ||
| <PropertyGroup> | ||
| <BUILD_ID Condition="'$(BUILD_ID)' == ''">0</BUILD_ID> | ||
| <FileVersion>$(MinVerMajor).$(MinVerMinor).$(MinVerPatch).$(BUILD_ID)</FileVersion> | ||
| </PropertyGroup> | ||
| </Target> | ||
| </Project> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| Microsoft Visual Studio Solution File, Format Version 12.00 | ||
| # Visual Studio Version 16 | ||
| VisualStudioVersion = 16.0.28606.126 | ||
| MinimumVisualStudioVersion = 10.0.40219.1 | ||
| Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = ".github", ".github", "{1A62C9CF-EA99-4D42-8DB3-F050981267AD}" | ||
| ProjectSection(SolutionItems) = preProject | ||
| .github\ISSUE_TEMPLATE.md = .github\ISSUE_TEMPLATE.md | ||
| EndProjectSection | ||
| EndProject | ||
| Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "tests", "tests", "{302B09C4-7F38-4CF7-93B9-1B7A6035386E}" | ||
| ProjectSection(SolutionItems) = preProject | ||
| tests\xunit.runner.json = tests\xunit.runner.json | ||
| EndProjectSection | ||
| EndProject | ||
| Project("{6EC3EE1D-3C4E-46DD-8F32-0CC8E7565705}") = "Jet.ConfluentKafka.FSharp.Integration", "tests\Jet.ConfluentKafka.FSharp.Integration\Jet.ConfluentKafka.FSharp.Integration.fsproj", "{1D205901-2742-45EA-8255-E6322B49E3BB}" | ||
| EndProject | ||
| Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution Items", "{6EADC23C-D6CA-4CBB-AB71-DA4A218AE24D}" | ||
| ProjectSection(SolutionItems) = preProject | ||
| .gitattributes = .gitattributes | ||
| .gitignore = .gitignore | ||
| azure-pipelines.yml = azure-pipelines.yml | ||
| build.proj = build.proj | ||
| CHANGELOG.md = CHANGELOG.md | ||
| Directory.Build.props = Directory.Build.props | ||
| Directory.Build.targets = Directory.Build.targets | ||
| global.json = global.json | ||
| LICENSE = LICENSE | ||
| README.md = README.md | ||
| SECURITY.md = SECURITY.md | ||
| EndProjectSection | ||
| EndProject | ||
| Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "src", "src", "{4670F7C4-A4FD-4E3F-B97C-99F9B3FC1898}" | ||
| EndProject | ||
| Project("{6EC3EE1D-3C4E-46DD-8F32-0CC8E7565705}") = "Jet.ConfluentKafka.FSharp", "src\Jet.ConfluentKafka.FSharp\Jet.ConfluentKafka.FSharp.fsproj", "{76802BE3-00C2-4B1D-96A2-95A3E2136DBE}" | ||
| EndProject | ||
| Global | ||
| GlobalSection(SolutionConfigurationPlatforms) = preSolution | ||
| Debug|Any CPU = Debug|Any CPU | ||
| Release|Any CPU = Release|Any CPU | ||
| EndGlobalSection | ||
| GlobalSection(ProjectConfigurationPlatforms) = postSolution | ||
| {1D205901-2742-45EA-8255-E6322B49E3BB}.Debug|Any CPU.ActiveCfg = Debug|Any CPU | ||
| {1D205901-2742-45EA-8255-E6322B49E3BB}.Debug|Any CPU.Build.0 = Debug|Any CPU | ||
| {1D205901-2742-45EA-8255-E6322B49E3BB}.Release|Any CPU.ActiveCfg = Release|Any CPU | ||
| {1D205901-2742-45EA-8255-E6322B49E3BB}.Release|Any CPU.Build.0 = Release|Any CPU | ||
| {76802BE3-00C2-4B1D-96A2-95A3E2136DBE}.Debug|Any CPU.ActiveCfg = Debug|Any CPU | ||
| {76802BE3-00C2-4B1D-96A2-95A3E2136DBE}.Debug|Any CPU.Build.0 = Debug|Any CPU | ||
| {76802BE3-00C2-4B1D-96A2-95A3E2136DBE}.Release|Any CPU.ActiveCfg = Release|Any CPU | ||
| {76802BE3-00C2-4B1D-96A2-95A3E2136DBE}.Release|Any CPU.Build.0 = Release|Any CPU | ||
| EndGlobalSection | ||
| GlobalSection(SolutionProperties) = preSolution | ||
| HideSolutionNode = FALSE | ||
| EndGlobalSection | ||
| GlobalSection(NestedProjects) = preSolution | ||
| {1D205901-2742-45EA-8255-E6322B49E3BB} = {302B09C4-7F38-4CF7-93B9-1B7A6035386E} | ||
| {76802BE3-00C2-4B1D-96A2-95A3E2136DBE} = {4670F7C4-A4FD-4E3F-B97C-99F9B3FC1898} | ||
| EndGlobalSection | ||
| GlobalSection(ExtensibilityGlobals) = postSolution | ||
| SolutionGuid = {DF04AF73-7412-46E5-9CC8-15CB48E3139A} | ||
| EndGlobalSection | ||
| EndGlobal |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,81 @@ | ||
| [](https://www.nuget.org/packages/Jet.ConfluentKafka.FSharp/) | ||
|
|
||
| # Jet.ConfluentKafka.FSharp 1.x | ||
| # Jet.ConfluentKafka.FSharp 1.x [](https://dev.azure.com/jet-opensource/opensource/_build/latest?definitionId=7?branchName=v1) [](https://github.com/jet/Jet.ConfluentKafka.FSharp/releases) [](https://www.nuget.org/packages/Jet.ConfluentKafka.FSharp/) [](LICENSE)  | ||
|
|
||
| F# friendly wrapper for `Confluent.Kafka` versions `>= 1.0.0`, with minimal dependencies or additional abstractions. | ||
|
|
||
| See also: [`v0` branch](https://github.com/jet/Jet.ConfluentKafka.FSharp/tree/v0), which targets `Confluent.Kafka` versions `0.9.*` and is designed for compatibility with the [Kafunk](https://github.com/jet/kafunk) API. | ||
| See also: [`v0` branch](https://github.com/jet/Jet.ConfluentKafka.FSharp/tree/v0), which targets `Confluent.Kafka` versions `0.9.*` and is designed for compatibility with the [Kafunk](https://github.com/jet/kafunk) API. | ||
|
|
||
| ## QuickStart | ||
|
|
||
| See https://github.com/jet/equinox#quickstart for examples of using this library to Project to Kafka from `Equinox.Cosmos` and/or `Equinox.EventStore` | ||
|
|
||
| ## Components | ||
|
|
||
| The components within this repository are delivered as (presently single) multi-targeted Nuget package targeting `net461` (F# 3.1+) and `netstandard2.0` (F# 4.5+) profiles | ||
|
|
||
| - [](https://www.nuget.org/packages/Jet.ConfluentKafka.FSharp/) `Jet.ConfluentKafka.FSharp`: Wraps `Confluent.Kafka` to provide efficient batched Kafka Producer and Consumer configurations, with basic logging instrumentation. ([depends](https://www.fuget.org/packages/Jet.ConfluentKafka.FSharp) on `Confluent.Kafka >= 1.0.0-beta3`, `Serilog` (but no specific Serilog sinks, i.e. you configure to emit to `NLog` etc) and `Newtonsoft.Json` (used internally to parse Statistics for logging purposes) | ||
|
|
||
| ## CONTRIBUTING | ||
|
|
||
| Where it makes sense, raise GitHub issues for any questions so others can benefit from the discussion. | ||
|
|
||
| This is an Open Source Project for many reasons, with some central goals: | ||
|
|
||
| - quality reference code (the code should be clean and easy to read; where it makes sense, components can be grabbed and cloned locally and used in altered form) | ||
| - optimal resilience and performance (getting performance right can add huge value for some systems) | ||
| - this code underpins non-trivial production systems (having good tests is not optional for reasons far deeper than having impressive coverage stats) | ||
|
|
||
| We'll do our best to be accomodating to PRs and issues, but please appreciate that [we emphasize decisiveness for the greater good of the project and its users](https://www.hanselman.com/blog/AlwaysBeClosingPullRequests.aspx); _new features [start with -100 points](https://blogs.msdn.microsoft.com/oldnewthing/20090928-00/?p=16573)_. | ||
|
|
||
| Within those constraints, contributions of all kinds are welcome: | ||
|
|
||
| - raising [Issues](https://github.com/jet/Jet.ConfluentKafka.FSharp/issues) is always welcome (but we'll aim to be decisive in the interests of keeping the list navigable). | ||
| - bugfixes with good test coverage are always welcome; in general we'll seek to move them to NuGet prerelease and then NuGet release packages with relatively short timelines (there's unfortunately not presently a MyGet feed for CI packages rigged). | ||
| - improvements / tweaks, _subject to filing a GitHub issue outlining the work first to see if it fits a general enough need to warrant adding code to the implementation and to make sure work is not wasted or duplicated_: | ||
|
|
||
| ## TEMPLATES | ||
|
|
||
| The best place to start, sample-wise is with the [QuickStart](quickstart), which walks you through sample code, tuned for approachability, from `dotnet new` templates stored [in a dedicated repo](https://github.com/jet/dotnet-templates). | ||
|
|
||
| ## BUILDING | ||
|
|
||
| Please note the [QuickStart](quickstart) is probably the best way to gain an overview - these instructions are intended to illustrated various facilities of the build script for people making changes. | ||
|
|
||
| NB The tests are reliant on a `TEST_KAFKA_BROKER` environment variable pointing to a test environment that will auto-create ephemeral Kafka Topics as required by the tests (each test run writes to a guid-named topic) | ||
|
|
||
| ### build, including tests | ||
|
|
||
| dotnet build build.proj | ||
|
|
||
| ## FAQ | ||
|
|
||
| ### What is this, why does it exist, where did it come from, is anyone using it ? | ||
|
|
||
| This code results from building out an end-to-end batteries-included set of libraries and templates as part of the [Equinox](https://github.com/jet/equinox) project. | ||
|
|
||
| Equinox has some key constraints applied to all its components:- | ||
|
|
||
| - low dependencies | ||
| - no new concepts on top of a standard base library | ||
| - thorough test coverage | ||
| - batteries-included examples of end-to-end functionality within the Equinox remit _without burdening samples with additional concepts_ | ||
|
|
||
| You can see [the development process history here](https://github.com/jet/equinox/pull/87). The bulk of the base code is taken from Jet's Warehouse systems (**NB the present version of `Confluent.Kafka` deployed in in production is `0.9.4`**) | ||
|
|
||
| ### What is Kafunk ? | ||
|
|
||
| See https://github.com/jet/kafunk. Kafunk is an F# implementation of a Kafka Client - it's presently unmaintained, and has only trickle of usage within the Jet org, mainly due to the fact that it does not support newer protocol versions. | ||
|
|
||
| ### What about `jet-confluent-kafka` and `Confluent.Kafka` v `0.9.4` support? | ||
|
|
||
| The [`v0` branch](tree/v0) continues to house the original code as previously borne by the `master` of this repo. It's not going anywhere, especially while we have a significant number of `Confluent.Kafka` v `0.9.x` based clients in operation throughout our systems. | ||
|
|
||
| - It will continue to address the need to provide an easy migration from the Kafunk API | ||
| - There are significant non-trivial changes in lifetime management in the `RdKafka` drivers accompanying `0.9.5` and `0.9.6` (with potential behavioral changes implied too) - while upgrading can likely be achieved without API changes, it does bring into play a series of changes related to how the RdKafka driver closes connections (which can result in long days chasing `AccessViolationException` and friends) | ||
| - NB Experience of [the changes necessary to accommodate the sweeping changes that the `Confluent.Kafka` v `1.0.0` API brings when compared to the `0.9.x` codebase](https://github.com/jet/equinox/pull/87) suggests it's likely to be a significant undertaking to wrap `Confluent.Kafka 1.0.0` without significant surface API changes (to go with the behavior changes from the preceding point) | ||
|
|
||
| ### Whats's involved in migrating from `Jet.ConfluentKafka.fsharp 0.9.x` to `Jet.ConfluentKafka.FSharp 1.0.x` ? | ||
|
|
||
| - The producer and consumer API wrappers provide different semantics. It's recommended to validate that they make sense for your use case before considering 'Just Porting' | ||
| - Upgrading to a new version of `Confluent.Kafka` always brings with it the knock on effect of taking an increment in the underlying `RdKafka` driver (TODO - explain key behavior and perf changes between what `1.0.0` implies vs `0.9.4`) | ||
| - You need to wire the (`Serilog`-based) logging through to your log sink (it's easy to connect it to an NLog Target etc) | ||
| - There's a transitive dependency on `Newtonsoft.Json` v `11.0.2` which should generally not be a problem to accommodate in most codebases |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| # Security Guidelines for this Project | ||
|
|
||
| ## How the Jet Security team manages security for this project | ||
|
|
||
| Jet.com takes security seriously and wants to ensure that we maintain a secure environment for our customers and that we also provide secure solutions for the open source community. To help us achieve these goals, please note the following before using this software: | ||
|
|
||
| - Review the software license to understand Jet's obligations in terms of warranties and suitability for purpose | ||
| - For any questions or concerns about security, you can reach out directly to Jet's security team at [email protected] | ||
| - We request that you work with our security team and opt for [responsible disclosure](https://corporate.walmart.com/article/responsible-disclosure-policy) using the guidelines below | ||
| - We enforce SLAs on our security team and software engineers to remediate security bugs in a timely manner | ||
| - All security related issues and pull requests you make should be tagged with "security" for easy identification | ||
| - Please monitor this repository and update your environment in a timely manner as we release patches and updates | ||
|
|
||
| ## Responsibly Disclosing Security Bugs to Jet | ||
|
|
||
| If you find a security bug in this repository, please work with Jet's security team following responsible disclosure principles and these guidelines: | ||
|
|
||
| - Do not submit a normal issue or pull request in our public repository, instead report directly to [email protected] (If you would like to encrypt, please contact us for keys) | ||
| - We will review your submission and may follow up for additional details | ||
| - If you have a patch, we will review it and approve it privately; once approved for release you can submit it as a pull request publicly in our repos (we give credit where credit is due) | ||
| - We will keep you informed during our investigation, feel free to check in for a status update | ||
| - We will release the fix and publicly disclose the issue as soon as possible, but want to ensure we do proper due diligence before releasing | ||
| - Please do not publicly blog or post about the security issue until after we have updated the public repo so that other downstream users have an opportunity to patch | ||
|
|
||
| ## Contact / Misc. | ||
|
|
||
| If you have any questions, please reach out directly to the Jet.com Security team at [email protected] |
Oops, something went wrong.