Faraday 2 support for opensearch-transport 2 (#76)

* Faraday 2 support for opensearch-transport 2

- Support Faraday 1.x and 2.x in opensearch-transport.
- Made client autodetection not break if the detected HTTP client is
  there but the associated Faraday adapter is not.
- Made the test setup require the referenced adapters.
- Made it more clear in the README that Faraday uses adapters.

[Fixes #59]

Signed-off-by: Brian Hawley <[email protected]>

* Additional requested fixes

- The Ruby < 2.7, Faraday 2+, and Typhoeus combination doesn't work.
  Adjusted the README and test setup accordingly.
- Fix the MiniTest transport test setup.
- Add extra opensearch-transport tests to the matrix for Faraday 1.

Signed-off-by: Brian Hawley <[email protected]>

* More requested fixes

- Use Gem::Version comparison.
- Fix the faraday-typhoeus links.

Signed-off-by: Brian Hawley <[email protected]>

* More fixes and requested changes

- Move the Faraday 1 tests to their own job.
- Don't reference faraday-typhoeus during Faraday 1 tests.
- Use ENV.key?('FARADAY_VERSION').

Signed-off-by: Brian Hawley <[email protected]>

* Make the version checks backwards compatible

Signed-off-by: Brian Hawley <[email protected]>

.github/workflows/main.yml

@@ -51,6 +51,40 @@ jobs:
       - name: opensearch-dsl
         run: cd opensearch-dsl && bundle exec rake test:all
 
+  test-opensearch-faraday-1:
+    env:
+      TEST_OPENSEARCH_SERVER: http://localhost:9250
+      PORT: 9250
+      FARADAY_VERSION: '~> 1.0'
+    strategy:
+      fail-fast: false
+      matrix:
+        ruby: [ 2.6, 2.7, '3.0', 3.1, jruby-9.3 ]
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - name: Increase system limits
+        run: |
+          sudo swapoff -a
+          sudo sysctl -w vm.swappiness=1
+          sudo sysctl -w fs.file-max=262144
+          sudo sysctl -w vm.max_map_count=262144
+      - uses: ./.github/actions/opensearch
+        with:
+          cluster-version: latest
+          disable-security: true
+      - uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: ${{ matrix.ruby }}
+      - name: Build and test with Rake
+        run: |
+          sudo apt-get update
+          sudo apt-get install libcurl4-openssl-dev
+          ruby -v
+          rake bundle:clean
+          rake bundle:install
+      - name: opensearch-transport
+        run: cd opensearch-transport && bundle exec rake test:all
 
   test-opensearch-security:
     env:

opensearch-transport/Gemfile

@@ -37,6 +37,9 @@ if File.exist? File.expand_path('../opensearch/opensearch.gemspec', __dir__)
   gem 'opensearch-ruby', path: File.expand_path('../opensearch', __dir__), require: false
 end
 
+ENV['FARADAY_VERSION'] = '~> 1.0' if Gem::Version.new(RUBY_VERSION) < Gem::Version.new('2.7')
+gem 'faraday', ENV['FARADAY_VERSION'], require: false if ENV.key?('FARADAY_VERSION')
+
 group :development, :test do
   gem 'rspec'
   if defined?(JRUBY_VERSION)

opensearch-transport/README.md

@@ -28,16 +28,22 @@ Features overview:
 * Node reloading (based on cluster state) on errors or on demand
 
 For optimal performance, use a HTTP library which supports persistent ("keep-alive") connections,
-such as [patron](https://github.com/toland/patron) or [Typhoeus](https://github.com/typhoeus/typhoeus).
-Just require the library (`require 'patron'`) in your code, and it will be automatically used.
+such as [Patron](https://github.com/toland/patron) or [Typhoeus](https://github.com/typhoeus/typhoeus).
+Most such HTTP libraries are used through the [Faraday](https://rubygems.org/gems/faraday) HTTP library
+and its [adapters](https://github.com/lostisland/awesome-faraday/#adapters).
+
+Include the library's gem and adapter gem, and require the library and adapter in your code, and it will be automatically used.
+If you don't use Bundler, you may need to require the library explicitly (like `require 'faraday/patron'`).
 
 Currently these libraries will be automatically detected and used:
-- [Patron](https://github.com/toland/patron)
-- [Typhoeus](https://github.com/typhoeus/typhoeus)
-- [HTTPClient](https://rubygems.org/gems/httpclient)
-- [Net::HTTP::Persistent](https://rubygems.org/gems/net-http-persistent)
+- [Patron](https://github.com/toland/patron) through [faraday-patron](https://github.com/lostisland/faraday-patron)
+- [Typhoeus](https://github.com/typhoeus/typhoeus) through [faraday-typhoeus](https://github.com/dleavitt/faraday-typhoeus) for Faraday 2 or higher, or its built-in adapter for Faraday 1.
+- [HTTPClient](https://rubygems.org/gems/httpclient) through [faraday-httpclient](https://github.com/lostisland/faraday-httpclient)
+- [Net::HTTP::Persistent](https://rubygems.org/gems/net-http-persistent) through [faraday-net_http_persistent](https://github.com/lostisland/faraday-net_http_persistent)
+
+**Note on [Typhoeus](https://github.com/typhoeus/typhoeus)**: You need to use v1.4.0 or up since older versions are not compatible with Faraday 1.0 or higher. You can't use Typhoeus with Faraday 2.0 or higher with Ruby versions less than 2.7 because [faraday-typhoeus](https://github.com/dleavitt/faraday-typhoeus) doesn't support it, so use Faraday 1.x and its built-in Typhoeus adapter.
 
-**Note on [Typhoeus](https://github.com/typhoeus/typhoeus)**: You need to use v1.4.0 or up since older versions are not compatible with Faraday 1.0.
+**Note on [Faraday](https://rubygems.org/gems/faraday)**: If you use Faraday 2.0 or higher, if the adapter is in a separate gem, you will likely need to declare that gem as well. Only the Net::HTTP adapter gem is included by default. Faraday 1.x includes most common adapter gems already.
 
 For detailed information, see example configurations [below](#transport-implementations).
 
@@ -350,10 +356,22 @@ as a transport implementation.
 It will auto-detect and use an _adapter_ for _Faraday_ based on gems loaded in your code,
 preferring HTTP clients with support for persistent connections.
 
-To use the [_Patron_](https://github.com/toland/patron) HTTP, for example, just require it:
+Faraday uses adapters, usually in separate gems, to connect to the HTTP library. You need to
+make sure that your code refers to both the HTTP library gem and the adapter gem. See this
+list of [Faraday adapters](https://github.com/lostisland/awesome-faraday/#adapters) for details.
+
+To use the [_Patron_](https://github.com/toland/patron) HTTP, for example, you need to refer to these gems:
+
+```ruby
+gem 'patron'
+gem 'faraday-patron'
+```
+
+If you don't use Bundler, you may need to require the libraries explicitly in your code:
 
 ```ruby
 require 'patron'
+require 'faraday/patron'
 ```
 
 Then, create a new client, and the _Patron_  gem will be used as the "driver":
@@ -396,8 +414,6 @@ constructor, use the `transport_options` key:
 
 To configure the _Faraday_ instance directly, use a block:
 
-    require 'patron'
-
     client = OpenSearch::Client.new(host: 'localhost', port: '9200') do |f|
       f.response :logger
       f.adapter  :patron
@@ -406,8 +422,6 @@ To configure the _Faraday_ instance directly, use a block:
 You can use any standard Faraday middleware and plugins in the configuration block. You can also initialize the transport class yourself, and pass it to the client constructor as the `transport` argument:
 
 ```ruby
-require 'patron'
-
 transport_configuration = lambda do |f|
   f.response :logger
   f.adapter  :patron

opensearch-transport/lib/opensearch/transport/client.rb

@@ -330,14 +330,23 @@ def __parse_host(host)
       # @api private
       #
       def __auto_detect_adapter
+        # Get the Faraday adapter list without initializing it.
+        if Faraday::Adapter.respond_to?(:registered_middleware) # Faraday 2.x
+          adapter = ->(name) { Faraday::Adapter.registered_middleware[name] }
+        elsif Faraday::Adapter.respond_to?(:fetch_middleware) # Faraday 1.x
+          adapter = ->(name) { Faraday::Adapter.fetch_middleware(name) }
+        else
+          adapter = {} # fallback behavior that should never happen
+        end
+        # Pick an adapter that has both the client and adapter defined.
         case
-        when defined?(::Patron)
+        when defined?(::Patron) && adapter[:patron]
           :patron
-        when defined?(::Typhoeus)
+        when defined?(::Typhoeus) && adapter[:typhoeus]
           :typhoeus
-        when defined?(::HTTPClient)
+        when defined?(::HTTPClient) && adapter[:httpclient]
           :httpclient
-        when defined?(::Net::HTTP::Persistent)
+        when defined?(::Net::HTTP::Persistent) && adapter[:net_http_persistent]
           :net_http_persistent
         else
           ::Faraday.default_adapter

opensearch-transport/opensearch-transport.gemspec

@@ -61,7 +61,7 @@ Gem::Specification.new do |s|
   s.required_ruby_version = '>= 2.4'
 
   s.add_dependency 'multi_json'
-  s.add_dependency 'faraday', '~> 1'
+  s.add_dependency 'faraday', '>= 1.0', '< 3'
 
   s.add_development_dependency 'ansi'
   s.add_development_dependency 'bundler'
@@ -70,12 +70,15 @@ Gem::Specification.new do |s|
   s.add_development_dependency 'opensearch-ruby'
   s.add_development_dependency 'hashie'
   s.add_development_dependency 'httpclient'
+  s.add_development_dependency 'faraday-httpclient'
   s.add_development_dependency 'manticore' if defined? JRUBY_VERSION
   s.add_development_dependency 'minitest'
   s.add_development_dependency 'minitest-reporters'
   s.add_development_dependency 'mocha'
   s.add_development_dependency 'net-http-persistent'
+  s.add_development_dependency 'faraday-net_http_persistent'
   s.add_development_dependency 'patron' unless defined? JRUBY_VERSION
+  s.add_development_dependency 'faraday-patron' unless defined? JRUBY_VERSION
   s.add_development_dependency 'pry'
   s.add_development_dependency 'rake', '~> 13'
   s.add_development_dependency 'require-prof' unless defined?(JRUBY_VERSION) || defined?(Rubinius)
@@ -84,6 +87,7 @@ Gem::Specification.new do |s|
   s.add_development_dependency 'simplecov'
   s.add_development_dependency 'test-unit', '~> 2'
   s.add_development_dependency 'typhoeus', '~> 1.4'
+  s.add_development_dependency 'faraday-typhoeus' if !ENV.key?('FARADAY_VERSION') && Gem::Version.new(RUBY_VERSION) >= Gem::Version.new('2.7')
   s.add_development_dependency 'yard'
 
   s.description = <<-DESC.gsub(/^    /, '')

opensearch-transport/spec/spec_helper.rb

@@ -29,6 +29,8 @@
 end
 
 require 'opensearch-transport'
+require 'faraday/httpclient'
+require 'faraday/net_http_persistent'
 require 'logger'
 require 'ansi/code'
 require 'hashie/mash'
@@ -37,6 +39,8 @@
   require 'pry-nav'
 else
   require 'pry-byebug'
+  require 'faraday/patron'
+  require 'faraday/typhoeus' if Gem::Version.new(Faraday::VERSION) >= Gem::Version.new('2')
   require 'opensearch/transport/transport/http/curb'
   require 'curb'
 end

opensearch-transport/test/integration/transport_test.rb

@@ -34,8 +34,10 @@ class OpenSearch::Transport::ClientIntegrationTest < Minitest::Test
     end
 
     should "allow to customize the Faraday adapter to Typhoeus" do
+      # Require the library so autodetection finds it.
       require 'typhoeus'
-      require 'typhoeus/adapters/faraday'
+      # Require the adapter so autodetection finds it.
+      require 'faraday/typhoeus' if Gem::Version.new(Faraday::VERSION) >= Gem::Version.new('2')
 
       transport = OpenSearch::Transport::Transport::HTTP::Faraday.new \
         :hosts => [ { host: @host, port: @port } ] do |f|
@@ -48,7 +50,10 @@ class OpenSearch::Transport::ClientIntegrationTest < Minitest::Test
     end unless jruby?
 
     should "allow to customize the Faraday adapter to NetHttpPersistent" do
+      # Require the library so autodetection finds it.
       require 'net/http/persistent'
+      # Require the adapter so autodetection finds it.
+      require 'faraday/net_http_persistent'
 
       transport = OpenSearch::Transport::Transport::HTTP::Faraday.new \
                                                                        :hosts => [ { host: @host, port: @port } ] do |f|