Searchkick

🚀 Intelligent search made easy

Searchkick learns what your users are looking for. As more people search, it gets smarter and the results get better. It’s friendly for developers - and magical for your users.

Searchkick handles:

• stemming - tomatoes matches tomato
• special characters - jalapeno matches jalapeño
• extra whitespace - dishwasher matches dish washer
• misspellings - zuchini matches zucchini
• custom synonyms - qtip matches cotton swab

Plus:

• query like SQL - no need to learn a new query language
• reindex without downtime
• easily personalize results for each user
• autocomplete
• “Did you mean” suggestions
• works with ActiveRecord and Mongoid

💬 Get handcrafted updates for new features

🍊 Battle-tested at Instacart

We highly recommend tracking queries and conversions

⚡ Searchjoy makes it easy

Get Started

Install Elasticsearch. For Homebrew, use:

brew install elasticsearch

Add this line to your application’s Gemfile:

gem 'searchkick'

For Elasticsearch 0.90, use version 0.6.3 and this readme.

Add searchkick to models you want to search.

class Product < ActiveRecord::Base
  searchkick
end

Add data to the search index.

Product.reindex

And to query, use:

products = Product.search "2% Milk"
products.each do |product|
  puts product.name
end

Searchkick supports the complete Elasticsearch Search API. As your search becomes more advanced, we recommend you use the Elasticsearch DSL for maximum flexibility.

Queries

Query like SQL

Product.search "2% Milk", where: {in_stock: true}, limit: 10, offset: 50

Search specific fields

fields: [:name, :brand]

Where

where: {
  expires_at: {gt: Time.now}, # lt, gte, lte also available
  orders_count: 1..10,        # equivalent to {gte: 1, lte: 10}
  aisle_id: [25, 30],         # in
  store_id: {not: 2},         # not
  aisle_id: {not: [25, 30]},  # not in
  user_ids: {all: [1, 3]},    # all elements in array
  or: [
    [{in_stock: true}, {backordered: true}]
  ]
}

Order

order: {_score: :desc} # most relevant first - default

All of these sort options are supported

Limit / offset

limit: 20, offset: 40

Boosting

Boost important fields

fields: ["title^10", "description"]

Boost by the value of a field

boost_by: [:orders_count] # give popular documents a little boost
boost_by: {orders_count: {factor: 10}} # default factor is 1

Boost matching documents

boost_where: {user_id: 1} # default factor is 1000
boost_where: {user_id: {value: 1, factor: 100}}

Conversions are also a great way to boost.

Get Everything

Use a * for the query.

Product.search "*"

Pagination

Plays nicely with kaminari and will_paginate.

# controller
@products = Product.search "milk", page: params[:page], per_page: 20

View with kaminari

<%= paginate @products %>

View with will_paginate

<%= will_paginate @products %>

Partial Matches

By default, results must match all words in the query.

Product.search "fresh honey" # fresh AND honey

To change this, use:

Product.search "fresh honey", operator: "or" # fresh OR honey

By default, results must match the entire word - back will not match backpack. You can change this behavior with:

class Product < ActiveRecord::Base
  searchkick word_start: [:name]
end

And to search (after you reindex):

Product.search "back", fields: [{name: :word_start}]

Available options are:

:word # default
:word_start
:word_middle
:word_end
:text_start
:text_middle
:text_end

To boost fields, use:

fields: [{"name^2" => :word_start}] # better interface on the way

Exact Matches

User.search "[email protected]", fields: [{email: :exact}, :name]

Language

Searchkick defaults to English for stemming. To change this, use:

class Product < ActiveRecord::Base
  searchkick language: "German"
end

See the list of languages

Synonyms

class Product < ActiveRecord::Base
  searchkick synonyms: [["scallion", "green onion"], ["qtip", "cotton swab"]]
end

Call Product.reindex after changing synonyms.

WordNet

Prepopulate English synonyms with the WordNet database.

Download WordNet 3.0 to each Elasticsearch server and move wn_s.pl to the /var/lib directory.

cd /tmp
curl -o wordnet.tar.gz http://wordnetcode.princeton.edu/3.0/WNprolog-3.0.tar.gz
tar -zxvf wordnet.tar.gz
mv prolog/wn_s.pl /var/lib

Tell each model to use it:

class Product < ActiveRecord::Base
  searchkick wordnet: true
end

Misspellings

By default, Searchkick handles misspelled queries by returning results with an edit distance of one. To turn off this feature, use:

Product.search "zuchini", misspellings: false # no zucchini

You can also change the edit distance with:

Product.search "zucini", misspellings: {edit_distance: 2} # zucchini

Indexing

Control what data is indexed with the search_data method. Call Product.reindex after changing this method.

class Product < ActiveRecord::Base
  def search_data
    as_json only: [:name, :active]
    # or equivalently
    {
      name: name,
      active: active
    }
  end
end

Searchkick uses find_in_batches to import documents. To eager load associations, use the search_import scope.

class Product < ActiveRecord::Base
  scope :search_import, -> { includes(:searches) }
end

By default, all records are indexed. To control which records are indexed, use the should_index? method.

class Product < ActiveRecord::Base
  def should_index?
    active # only index active records
  end
end

To Reindex, or Not to Reindex

Reindex

• when you install or upgrade searchkick
• change the search_data method
• change the searchkick method

No need to reindex

• App starts

Stay Synced

There are three strategies for keeping the index synced with your database.

1. Immediate (default)

Anytime a record is inserted, updated, or deleted

2. Asynchronous

Use background jobs for better performance

class Product < ActiveRecord::Base
  searchkick callbacks: :async
end

Supports Delayed Job only at the moment

3. Manual

Turn off automatic syncing

class Product < ActiveRecord::Base
  searchkick callbacks: false
end

Keep Getting Better

Searchkick uses conversion data to learn what users are looking for. If a user searches for “ice cream” and adds Ben & Jerry’s Chunky Monkey to the cart (our conversion metric at Instacart), that item gets a little more weight for similar searches.

The first step is to define your conversion metric and start tracking conversions. The database works well for low volume, but feel free to use Redis or another datastore.

class Search < ActiveRecord::Base
  belongs_to :product
  # fields: id, query, searched_at, converted_at, product_id
end

You do not need to clean up the search queries. Searchkick automatically treats apple and APPLES the same.

Next, add conversions to the index.

class Product < ActiveRecord::Base
  has_many :searches

  searchkick conversions: "conversions" # name of field

  def search_data
    {
      name: name,
      conversions: searches.group("query").count
      # {"ice cream" => 234, "chocolate" => 67, "cream" => 2}
    }
  end
end

Reindex and set up a cron job to add new conversions daily.

rake searchkick:reindex CLASS=Product

Personalized Results

Order results differently for each user. For example, show a user’s previously purchased products before other results.

class Product < ActiveRecord::Base

  def search_data
    {
      name: name,
      orderer_ids: orders.pluck(:user_id) # boost this product for these users
    }
  end

end

Reindex and search with:

Product.search "milk", boost_where: {orderer_ids: current_user.id}

Autocomplete

Autocomplete predicts what a user will type, making the search experience faster and easier.

Note: If you only have a few thousand records, don’t use Searchkick for autocomplete. It’s much faster to load all records into JavaScript and autocomplete there (eliminates network requests).

First, specify which fields use this feature. This is necessary since autocomplete can increase the index size significantly, but don’t worry - this gives you blazing faster queries.

class City < ActiveRecord::Base
  searchkick text_start: [:name]
end

Reindex and search with:

City.search "san fr", fields: [{name: :text_start}]

Typically, you want to use a JavaScript library like typeahead.js or jQuery UI.

Here’s how to make it work with Rails

First, add a route and controller action.

# app/controllers/cities_controller.rb
class CitiesController < ApplicationController

  def autocomplete
    render json: City.search(params[:query], fields: [{name: :text_start}], limit: 10).map(&:name)
  end

end

Then add the search box and JavaScript code to a view.

<input type="text" id="query" name="query" />

<script src="jquery.js"></script>
<script src="typeahead.js"></script>
<script>
  $("#query").typeahead({
    name: "city",
    remote: "/cities/autocomplete?query=%QUERY"
  });
</script>

Suggestions

class Product < ActiveRecord::Base
  searchkick suggest: ["name"] # fields to generate suggestions
end

Reindex and search with:

products = Product.search "peantu butta", suggest: true
products.suggestions # ["peanut butter"]

Facets

Facets provide aggregated search data.

products = Product.search "chuck taylor", facets: [:product_type, :gender, :brand]
p products.facets

By default, where conditions are not applied to facets (for backward compatibility).

Product.search "wingtips", where: {color: "brandy"}, facets: [:size]
# facets *not* filtered by color :(

Change this with:

Product.search "wingtips", where: {color: "brandy"}, facets: [:size], smart_facets: true

or set where conditions for each facet separately:

Product.search "wingtips", facets: {size: {where: {color: "brandy"}}}

Limit

Product.search "2% Milk", facets: {store_id: {limit: 10}}

Ranges

price_ranges = [{to: 20}, {from: 20, to: 50}, {from: 50}]
Product.search "*", facets: {price: {ranges: price_ranges}}

Use the stats option to get to max, min, mean, and total scores for each facet

Product.search "*", facets: {store_id: {stats: true}}

Highlight

Specify which fields to index with highlighting. [master]

class Product < ActiveRecord::Base
  searchkick highlight: [:name]
end

Highlight the search query in the results.

bands = Band.search "cinema", fields: [:name], highlight: true

Note: The fields option is required.

View the highlighted fields with:

bands.with_details.each do |band, details|
  puts details[:highlight][:name] # "Two Door <em>Cinema</em> Club"
end

To change the tag, use:

Band.search "cinema", fields: [:name], highlight: {tag: "<strong>"}

Similar Items

Find similar items.

product = Product.first
product.similar(fields: ["name"])

Geospatial Searches

class City < ActiveRecord::Base
  searchkick locations: ["location"]

  def search_data
    attributes.merge location: [latitude, longitude]
  end
end

Reindex and search with:

City.search "san", where: {location: {near: [37, -114], within: "100mi"}} # or 160km

Bounded by a box

City.search "san", where: {location: {top_left: [38, -123], bottom_right: [37, -122]}}

Inheritance

Searchkick supports single table inheritance.

class Dog < Animal
end

The parent and child model can both reindex.

Animal.reindex
Dog.reindex # equivalent

And to search, use:

Animal.search "*"                   # all animals
Dog.search "*"                      # just dogs
Animal.search "*", type: [Dog, Cat] # just cats and dogs

Note: The suggest option retrieves suggestions from the parent at the moment.

Dog.search "airbudd", suggest: true # suggestions for all animals

Debugging Queries

See how Elasticsearch tokenizes your queries with:

Product.searchkick_index.tokens("Dish Washer Soap", analyzer: "default_index")
# ["dish", "dishwash", "washer", "washersoap", "soap"]

Product.searchkick_index.tokens("dishwasher soap", analyzer: "searchkick_search")
# ["dishwashersoap"] - no match

Product.searchkick_index.tokens("dishwasher soap", analyzer: "searchkick_search2")
# ["dishwash", "soap"] - match!!

Partial matches

Product.searchkick_index.tokens("San Diego", analyzer: "searchkick_word_start_index")
# ["s", "sa", "san", "d", "di", "die", "dieg", "diego"]

Product.searchkick_index.tokens("dieg", analyzer: "searchkick_word_search")
# ["dieg"] - match!!

See the complete list of analyzers.

Deployment

Searchkick uses ENV["ELASTICSEARCH_URL"] for the Elasticsearch server. This defaults to http://localhost:9200.

Heroku

Choose an add-on: SearchBox, Bonsai, or Found.

# SearchBox
heroku addons:add searchbox:starter
heroku config:add ELASTICSEARCH_URL=`heroku config:get SEARCHBOX_URL`

# Bonsai
heroku addons:add bonsai
heroku config:add ELASTICSEARCH_URL=`heroku config:get BONSAI_URL`

# Found
heroku addons:add foundelasticsearch
heroku config:add ELASTICSEARCH_URL=`heroku config:get FOUNDELASTICSEARCH_URL`

Then deploy and reindex:

heroku run rake searchkick:reindex CLASS=Product

Other

Create an initializer config/initializers/elasticsearch.rb with:

ENV["ELASTICSEARCH_URL"] = "http://username:[email protected]"

Then deploy and reindex:

rake searchkick:reindex CLASS=Product

Performance

For the best performance, add Patron to your Gemfile.

gem 'patron'

Searchkick will automatically use it.

Note: Patron is not available for Windows.

Automatic Failover

Create an initializer config/initializers/elasticsearch.rb with multiple hosts:

Searchkick.client = Elasticsearch::Client.new(hosts: ["localhost:9200", "localhost:9201"], retry_on_failure: true)

See elasticsearch-transport for a complete list of options.

Advanced

Prefer to use the Elasticsearch DSL but still want awesome features like zero-downtime reindexing?

Create a custom mapping:

class Product < ActiveRecord::Base
  searchkick mappings: {
    product: {
      properties: {
        name: {type: "string", analyzer: "keyword"}
      }
    }
  }
end

And use the query option to search:

products = Product.search query: {match: {name: "milk"}}

View the response with:

products.response

To keep the mappings and settings generated by Searchkick, use:

class Product < ActiveRecord::Base
  searchkick merge_mappings: true, mappings: {...}
end

To modify the query generated by Searchkick, use:

query = Product.search "2% Milk", execute: false
query.body[:query] = {match_all: {}}
products = query.execute

Reference

Reindex one record

product = Product.find 10
product.reindex

Remove old indices

Product.clean_indices

Use a different index name

class Product < ActiveRecord::Base
  searchkick index_name: "products_v2"
end

Prefix the index name

class Product < ActiveRecord::Base
  searchkick index_prefix: "datakick"
end

Turn off callbacks temporarily

Product.disable_search_callbacks # or use Searchkick.disable_callbacks for all models
ExpensiveProductsTask.execute
Product.enable_search_callbacks # or use Searchkick.enable_callbacks for all models
Product.reindex

Change timeout [master]

Searchkick.timeout = 5 # defaults to 10

Change the search method name in config/initializers/searchkick.rb

Searchkick.search_method_name = :lookup

Eager load associations

Product.search "milk", include: [:brand, :stores]

Do not load models

Product.search "milk", load: false

Turn off special characters

class Product < ActiveRecord::Base
  # A will not match Ä
  searchkick special_characters: false
end

Change import batch size

class Product < ActiveRecord::Base
  searchkick batch_size: 200 # defaults to 1000
end

Reindex asynchronously

class Product < ActiveRecord::Base
  searchkick callbacks: false

  def reindex_async
    # custom code to reindex
  end

  after_commit :reindex_async
  # or for Mongoid
  # after_save :reindex_async
  # after_destroy :reindex_async
end

Reindex conditionally

Note: With ActiveRecord, use this feature with caution - transaction rollbacks can cause data inconstencies

class Product < ActiveRecord::Base
  searchkick callbacks: false

  # add the callbacks manually
  after_save :reindex, if: proc{|model| model.name_changed? } # use your own condition
  after_destroy :reindex
end

Reindex all models - Rails only

rake searchkick:reindex:all

Migrating from Tire

1. Change search methods to tire.search and add index name in existing search calls

Product.search "fruit"

should be replaced with

Product.tire.search "fruit", index: "products"

2. Replace tire mapping w/ searchkick method

class Product < ActiveRecord::Base
  searchkick
end

3. Deploy and reindex

rake searchkick:reindex CLASS=Product # or Product.reindex in the console

4. Once it finishes, replace search calls w/ searchkick calls

Upgrading

View the changelog.

Important notes are listed below.

0.6.0 and 0.7.0

If running Searchkick 0.6.0 or 0.7.0 and Elasticsearch 0.90, we recommend upgrading to Searchkick 0.6.1 or 0.7.1 to fix an issue that causes downtime when reindexing.

0.3.0

Before 0.3.0, locations were indexed incorrectly. When upgrading, be sure to reindex immediately.

Elasticsearch Gotchas

Inconsistent Scores

Due to the distributed nature of Elasticsearch, you can get incorrect results when the number of documents in the index is low. You can read more about it here. To fix this, do:

class Product < ActiveRecord::Base
  searchkick settings: {number_of_shards: 1}
end

For convenience, this is set by default in the test environment.

Thanks

Thanks to Karel Minarik for Elasticsearch Ruby and Tire, Jaroslav Kalistsuk for zero downtime reindexing, and Alex Leschenko for Elasticsearch autocomplete.

Roadmap

• Search multiple fields for different terms
• Search across models
• Search nested objects
• Add section on testing
• Much finer customization
• More transparency into generated queries (for advanced use)

Contributing

Everyone is encouraged to help improve this project. Here are a few ways you can help:

• Report bugs
• Fix bugs and submit pull requests
• Write, clarify, or fix documentation
• Suggest or add new features

To get started with development and testing:

1. Clone the repo
2. bundle
3. rake test