Scrapy, a fast high-level web crawling & scraping framework for Python.

Scrapy Overview Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes,

Related Repos

snjoer Broad Crawler Overview This is a project aiming to crawl a variety of web pages(especially pages of news) with a spider, a.k.a. broad crawler. Features The broad crawler should support the following fea

howie6879 Google search results crawler, get google search results that you need

LinuxTerminali A Terminal based program to follow live cricket score by scraping

hellysmile fake-useragent info: Up to date simple useragent faker with real world database Features grabs up to date useragent from randomize with real world statistic

gaojiuli Web crawling framework for everyone. Written with asyncio, uvloop and aiohttp. Requirements Python3.5+ Installation pip install gain pip install uvloop (Only linux) Usage Write

untwisted Sukhoi Minimalist and powerful Web Crawler. Sukhoi is built on top of the concept of miners, it is similar to what happens with scrapy and its spiders. However, in sukhoi the miners can be placed in structures like lists or dict

pythad Selenium extensions Tools that will make writing tests, bots and scrapers using Selenium much easier Free software: MIT license Documentation: Install

jullrich pcap2curl Read a packet capture, extract HTTP requests and turn them into cURL commands for replay. See This is a simple (too simple?) Python script that will read a pcap, find HTTP