Skip to content

Latest commit

 

History

History
45 lines (30 loc) · 2.92 KB

README.md

File metadata and controls

45 lines (30 loc) · 2.92 KB
Raw

Sketchy #Overview#

What is Sketchy?

Sketchy is a task based API for taking screenshots and scraping text from websites.

What is the Output of Sketchy?

Sketchy's capture model contains all of the information associated with screenshotting, scraping, and storing html files from a provided URL. Screenshots (sketches), text scrapes, and html files can either be stored locally or on an S3 bucket. Optionally, token auth can be configured for creating and retrieving captures. Sketchy can also perform callbacks if required.

How Does Sketchy Do It?

Sketchy utilizes PhantomJS with lazy-rendering to ensure Ajax heavy sites are captured correctly. Sketchy also utilizes Celery task management system allowing users to scale Sketchy accordingly and manage time intensive captures.

Release History

Version 1.1.1 - June 16, 2015

This minor release addresses a few bugs and some new configuration features:

  • A new configuration option PHANTOMJS_TIMEOUT allows setting how long to wait for a capture to render before terminating the subprocess
  • Celery retry functionality was added when PhatomJS fails to render a screenshot before the PhantomJS timeout occurs
  • An incremental PhantomJS timeout was introduced to improve PhantomJS success at generating very large screenshots. Each time PhantomJS retries to render a screenshot 5 seconds will be added to the previous PHANTOMJS_TIMEOUT configuration option.
  • A number of typos have been fixed and comments have been added.

Version 1.1 - December 4, 2014

A number of improvements and bug fixes have been made:

  • A new model and API endpoint called "Static" was created. This allows users to send Sketchy a static HTML file for text scraping and screenshoting. See the Wiki for usage information.
  • New PhantomJS script called 'static.js' for creating screenshots of static html files.
  • Creation of a new endpont: api/v1.0/capture/last which shows the last capture that was taken.
  • Creation of a new endpont: api/v1.0/static/last which shows the last static capture that was taken.
  • API list view is now reverse sorted so most recent capture is listed on the top of the page.
  • For callback requests, capture status is now updated
  • Task retry has been optimitzed to only retry on ConnectionErrors. This should speedup errors that would never succeed during a retry.
  • A new configuration setting "SSL_HOST_VALIDATION" can be set to scrape/screenshot webpages with SSL errors.
  • A new configuration setting "CAPTURE_ERRORS" can be used to scrape/screenshot webpages that have 4xx or 5xx http status codes.

#Documentation#

Documentation is maintained in the Github Wiki

#Docker# Sketchy is also available as a Docker container.