Modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites

Kimurai Scraping Framework Note about v1.0.0 version: The code was massively refactored for a support to run spiders multiple times from inside a single process. Now it's possible to run Kimurai spiders usin

Related Repos



felipecsl Wombat Web scraper with an elegant DSL that parses structured data from web pages. Usage: gem install wombat Scraping a page: The simplest way to use Wombat is by calling Wombat.crawl and passing it
 

propublica Upton Upton is a framework for easy web-scraping with a useful debug mode that doesn't hammer your target's servers. It does the repetitive parts of writing scrapers, so you only have to write the unique parts for each site.
 

sparklemotion Mechanize ¶ ↑ docs.seattlerb.org/mechanize github.com/sparklemotion/mechanize Description¶ ↑ The Mechanize library is used for automating interaction with websites. Mechanize automatically stores and sends coo
 

joenorton [RubyRetriever] (http://softwarebyjoe.com/rubyretriever/) By Joe Norton RubyRetriever is a Web Crawler, Scraper & File Harvester. Available as a command-line executable and as a crawling framework. RubyRetriever (RR) use
 

vifreefly Kimurai Scraping Framework Note about v1.0.0 version: The code was massively refactored for a support to run spiders multiple times from inside a single process. Now it's possible to run Kimurai spiders usin
 

mgleon08 Instagram Crawler The easiest way to download instagram photos, posts and videos. Instagram Crawler is a ruby gem to crawl instagram photos, posts and videos for download. Installation $ gem i