Home / Python / Web Crawling

Broad Crawler

Broad Crawler Overview This is a project aiming to crawl a variety of web pages(especially pages of news) with a spider, a.k.a. broad crawler. Features The broad crawler should support the following fea
Information
Category: Python / Web Crawling
Watchers: 2
Star: 2
Fork: 2
Last update: May 12, 2022
Resource links
https://github.com/snjoer/Broad_Crawler

Broad Crawler

Build Status Python Version License Type

Overview

This is a project aiming to crawl a variety of web pages(especially pages of news) with a spider, a.k.a. broad crawler.

Features

The broad crawler should support the following features:

  • Title Extractor
  • URL Extractor
  • Date Extractor
  • Main Content Extractor

Requirements

  • Python 2.7
  • Scrapy 1.3.3
  • beautifulsoup4 4.5.3
  • scrapy-redis 0.6.8
  • PyMySQL 0.7.11
  • redis 2.10.5
  • virtualenv (Optional)

Usage

License

GPL license

2 Open More issues
Closed

改进了获取时间的算法

opened Apr 20, 2017 by cheersberry 0
Closed

MySQL is too hard T_T ; SQLite : )

opened Apr 19, 2017 by cheersberry 0
Closed

Merge pull request #2 from Rafael-Cheng/master

Merge pull request #5 from black-line/master

opened Apr 18, 2017 by cheersberry 0
Closed

获取时间函数放到另一个文件里

opened Apr 16, 2017 by cheersberry 0
Closed

add new feature: catch time

分布式不会用 T_T

opened Apr 8, 2017 by cheersberry 0
Closed

:cn: Update README.md

bukuishidalaoboke666 @Rafael-Cheng

opened Mar 26, 2017 by cheersberry 0
Open

修改匹配时间的正则表达式以匹配换行符

然而并没有什么卵用....

opened Apr 8, 2017 by cheersberry 0
Open

py3 version crawl news.sina.com.cn

编码问题坑爹啊...先来个py3的。。。

opened Apr 2, 2017 by cheersberry 0

Contents

    spacenick Lightsabing Node/PhantomJS crawler; scrape dynamic content : without the hassle
    antivanov Web crawler for Node.JS
    LitoMore Bing wallpaper crawler
    CrawlScript WebCollector is an open source web crawler framework based on Java.It provides some simple interfaces for crawling the Web,you can setup a multi-threaded web crawler in less than 5 minutes.
    code4craft A scalable web crawler framework for Java.
    indrajithi A tiny web crawler in Python
    dadoonet Elasticsearch File System Crawler (FS Crawler)
    yasserg Open Source Web Crawler for Java
    BruceDone A collection of awesome web crawler,spider in different languages
    wcong open source, distributed, restful crawler engine in golang
    wspl :paw_prints: Creeper - The Next Generation Crawler Framework (Go)
    PuerkitoBio Polite, slim and concurrent web crawler.
    shirosaidev File system crawler, disk space usage, file search engine and file system analytics powered by Elasticsearch
    rivermont The simple, easy to use command line web crawler.
    yujiosaka Distributed crawler powered by Headless Chrome
    alinebastos Web Crawler created with Node.js and Puppeteer
    pop-os A configurable parallel web crawler, designed to crawl a website for content.
    USCDataScience Spark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.
    reactor-joy Simple content crawler for joyreactor.cc
    guilatrova Google Maps crawler using Selenium
    EkkoG a WWDC 2015 session video's download url crawler
    Share this repo

    Related Repos


    Web Crawling
    6
    SpotifyToMp3 - downloading spotify playlists from youtube
    ivg3n1 SpotifyToMp3 - downloading spotify playlists from youtube
     
    Web Crawling
    17
    Dungeon Crawler System
    Alexangelj GM my christmas notes basically my entire process Maybe more than two players have to play against the level. One player stakes, the other bets agains
     
    Web Crawling
    31
    Downloader Middleware to support Playwright in Scrapy & Gerapy
    Gerapy Gerapy Playwright This is a package for supporting Playwright in Scrapy, also this package is a module in Gerapy. Installation pip3 install gerapy-pla
     
    Web Crawling
    5
    Examine.com supplement research scraper!
    tytydraco ExamineScraper Examine.com supplement research scraper! Why I want to be able to search pages for a specific term. For example, I want to be able to s
     
    Web Crawling
    3
    A lightweight web crawler for your daily needs
    grkek Anonymous A minimal web scraper implementation inspired by Crawly Installation Add the dependency to your shard.yml: dependencies: anonymous: gi
     
    Web Crawling
    4
    Google Maps crawler using Selenium
    guilatrova Google Maps Crawler using Selenium Built as part of the Antifragile Dev Project Selenium crawler that browses Google Maps as a regular user and stores
     
    Web Crawling
    1
    A scraper for CoinMarketCap. It returns most important data about any cryptocurrency
    elia-orsini coinmarketcap-scraper A scraper for CoinMarketCap. It returns the most important data about any cryptocurrency listed. usage With the requirements ins
     
    Web Crawling
    4
    Daily scraper (and mebbe more in the future) for CISA's Known Exploited Vulnerabilities li...
    hrbrmstr cisa-known-exploited-vulns Code with daily GH action to scrape CISA's KEV Catalog (https://www.cisa.gov/known-exploited-vulnerabilities-catalog) Each
     
    Web Crawling
    5
    An Automated udemy coupons scraper which scrapes coupons and autopost the result in blogsp...
    gokulapap Autoscraper-n-blogger An Automated udemy coupons scraper which scrapes coupons and autopost the result in blogspot post and notifies via Telegram bot
     
    Web Crawling
    15
    Add your favorite songs to this coding playlist this is a practice project for helping beg...
    bhargav794 hacktoberfest-practice-music-playlists Note:-This repo is not included in the Hacktoberfest event, as it is for practice only. If you have any trouble
     
    Web Crawling
    19
    OSINT tool to export to excel/csv someone's instagram followers and/or following with deta...
    novitae S T E R R A X C Y L A python program that allows you to put in an excel/csv table followers and/or following of an aimed account, with detailed infos
     
    Web Crawling
    39
    Script that allows to download portable installers of different versions Adobe software fo...
    Drovosek01 What is this and for what This is a script that allows you to download portable installers of programs from Adobe for macOS with different versions. T
     
    Web Crawling
    27
    An hcaptcha-solving discord account generator; capable of randomizing names, profile pictu...
    AcierP discord-account-generator An hcaptcha-solving discord account generator; capable of randomizing names, profile pictures, and verifying phone numbers.
     
    Web Crawling
    12
    Data from popular CS:GO website hltv.org
    dchoruzy Welcome to hltv-data 👋 🎮 Data from popular CS:GO website hltv.org Install pip install hltv-data Usage The public methods can be reached using HLTVCl
     
    Web Crawling
    30
    UdemyBot - A Simple Udemy Free Courses Scrapper
    gautamajay52 UdemyBot - A Simple Udemy Free Courses Scrapper