Skip to content

Latest commit

 

History

History
296 lines (184 loc) · 8.55 KB

README.md

File metadata and controls

296 lines (184 loc) · 8.55 KB
Raw

Ryespy

Gem Version Build Status Code Climate

Redis Sidekiq/ Resque IMAP, FTP, Amazon S3, Google Cloud Storage, Google Drive, Rackspace Cloud Files listener.

Ryespy provides an executable for listening to IMAP mailboxes, FTP folders, Amazon S3 buckets, Google Cloud Storage buckets, Google Drive accounts, or Rackspace Cloud Files containers, keeps track of what it's seen using Redis, and writes Sidekiq/Resque-compatible payloads.

Ryespy was inspired by Redimap. Yes, it's sometimes possible to inspire oneself. Ryespy with my little eye.

More sleep lost by tiredpixel.

Externals

Installation

Install using:

gem install ryespy

Listener dependencies are required dynamically. That means that it may be necessary to manually install the indicated gems if you are using that listener. If you are not using that listener, the dependencies need not be installed.

Amazon S3 Listener

gem install fog -v '~> 1.19'

Google Cloud Storage Listener

gem install fog -v '~> 1.19'

Google Drive Listener

gem install google_drive -v '~> 0.3'

Rackspace Cloud Files Listener

gem install fog -v '~> 1.19'

The default Ruby version supported is defined in .ruby-version. Any other versions supported are defined in .travis.yml.

Usage

View the available options:

ryespy --help

It is necessary to specify a listener and at least one notifier. Currently, the only notifier is --notifier-sidekiq.

To run eternally, use --eternal (no need for Cron, but you can if you prefer).

IMAP Listener

Check IMAP, queue new email UIDs, and quit:

ryespy --listener imap --imap-host mail.example.com --imap-username [email protected] --imap-password helpimacarrot --notifier-sidekiq

For non-SSL, use --no-imap-ssl. For non-INBOX or multiple mailboxes, use --imap-mailboxes INBOX,Sent.

Example Worker

class RyespyIMAPJob
  include Sidekiq::Worker
  
  sidekiq_options :queue => :ryespy
  
  def perform(mailbox, uid)
  end
end

FTP Listener

Check FTP, queue new file paths, and quit:

ryespy --listener ftp --ftp-host ftp.example.com --ftp-username [email protected] --ftp-password helpimacucumber --notifier-sidekiq

For PASSIVE mode, use --ftp-passive. For non-root or multiple directories, use --ftp-dirs /DIR1,/DIR2.

Example Worker

class RyespyFTPJob
  include Sidekiq::Worker
  
  sidekiq_options :queue => :ryespy
  
  def perform(dir, filename)
  end
end

Amazon S3 Listener

Check Amazon S3, queue new file keys, and quit:

ryespy --listener amzn-s3 --amzn-s3-access-key c/example/com --amzn-s3-secret-key helpimabroccoli --amzn-s3-bucket vegetable-box --notifier-sidekiq

For non-* or multiple key prefix filters, use --amzn-s3-prefixes virtual-dir1/,virtual-dir.

Example Worker

class RyespyAmznS3Job
  include Sidekiq::Worker
  
  sidekiq_options :queue => :ryespy
  
  def perform(filename)
  end
end

Google Cloud Storage Listener

Check Google Cloud Storage, queue new file keys, and quit:

ryespy --listener goog-cs --goog-cs-access-key d/example/com --goog-cs-secret-key helpimanasparagus --goog-cs-bucket vegetable-box --notifier-sidekiq

For non-* or multiple key prefix filters, use --goog-cs-prefixes virtual-dir1/,virtual-dir.

Example Worker

class RyespyGoogCSJob
  include Sidekiq::Worker
  
  sidekiq_options :queue => :ryespy
  
  def perform(filename)
  end
end

Google Drive Listener

Note that the Google Drive listener currently has a pre-filter maximum of 1000 files. That means that having more files than this in an account is not (yet) supported; such files might be ignored entirely, or strange behaviour might be encountered depending on the order in which the files are returned. #2
#3

Check Google Drive, queue new file resource ids, and quit:

ryespy --listener goog-drv --goog-drv-username [email protected] --goog-drv-password helpimgarlic --goog-drv-filters vegetable-box --notifier-sidekiq

Example Worker

class RyespyGoogDrvJob
  include Sidekiq::Worker
  
  sidekiq_options :queue => :ryespy
  
  def perform(resource_id)
  end
end

Rackspace Cloud Files Listener

Check Rackspace Cloud Files, queue new file keys, and quit:

ryespy --listener rax-cf --rax-cf-username vegetable --rax-cf-api-key helpimacelery --rax-cf-container vegetable-box --notifier-sidekiq

For non-DFW region, use --rax-cf-region lon. For non-US auth endpoint, use --rax-cf-endpoint uk. Is your Rackspace account in London? Fret not; combine these and use --rax-cf-endpoint uk --rax-cf-region lon. For non-* or multiple key prefix filters, use --rax-cf-prefixes virtual-dir1/,virtual-dir.

Example Worker

class RyespyRaxCFJob
  include Sidekiq::Worker
  
  sidekiq_options :queue => :ryespy
  
  def perform(filename)
  end
end

Advanced Usage

If you want to do something rather more magical such as checking multiple accounts for a listener or even multiple listeners, then you may wish to use the Ryespy library directly instead of the ryespy executable.

Depend upon the ryespy gem in a Gemfile, remembering to add any manual dependencies for listeners as detailed in Installation:

# Gemfile

gem 'ryespy'
gem 'fog' # example manual dependency

Configure Ryespy Redis and require Ryespy:

require 'redis'
require 'redis/namespace'

Redis.current = Redis::Namespace.new('ryespy',
  :redis => Redis.connect(:url => nil) # Redis default
)

require 'ryespy'

Create the notifiers:

require 'ryespy/notifier/sidekiq'

notifiers = []
notifiers << Ryespy::Notifier::Sidekiq.new(
  :url       => nil, # Redis default
  :namespace => 'resque'
)

For each listener, configure like in ryespy --help but without the prefix and with - changed to _ (e.g. --amzn-s3-access-key => :access_key). Pass in an array of notifiers. Note that the check() argument varies per listener, meaning IMAP mailbox, FTP directory, Google Drive filter, or storage key prefix.

require 'ryespy/listener/amzn_s3'

Ryespy::Listener::AmznS3.new(
  :access_key => 'ACCESS_KEY',
  :secret_key => 'SECRET_KEY',
  :bucket     => 'BUCKET',
  :notifiers  => notifiers
) do |listener|
  listener.check('prefix/')
end

That's about the size of it.

Stay Tuned

We have a Librelist mailing list! To subscribe, send an email to [email protected]. To unsubscribe, send an email to [email protected]. There be archives. That was easy.

You can also become a watcher on GitHub. And don't forget you can become a stargazer if you are so minded. :D

Contributions

Contributions are embraced with much love and affection! Please fork the repository and wizard your magic, preferably with plenty of fairy-dust sprinkled over the tests. ;) Then send me a pull request. Simples! If you'd like to discuss what you're doing or planning to do, or if you get stuck on something, then just wave. :)

Do whatever makes you happy. We'll probably still like you. :)

Tests are written using minitest, which is included by default in Ruby 1.9 onwards. To run all tests:

rake test

When using the ryespy executable in development, you'll probably want to set --debug mode so debug-level messages are logged and stack traces raised.

We render grateful thanks unto our contributors.

Blessing

May you find peace, and help others to do likewise.

Licence

© tiredpixel and others 2013 - 2015. It is free software, released under the MIT License, and may be redistributed under the terms specified in LICENSE.txt.