Web Crawling

Libraries to automate web scraping.

Newest releases

enesusta github-trending-crawler is a basic rest API that crawls github.com/trending.
 

ayoubeddafali A minimal framework to automate web Actions/Plans, and run them in a containerized fashion.
 

samuelm2 Simple, quick to set up stock notification bot for Nvidia 3080 that I used to get my 3080. Less than 250 lines of code.
 

philippnormann 🎯 Autonomously buy Nvidia Founders Edition GPUs as soon as they become available
 

AlteredSecurity 365-Stealer is the tool written in python3 which steals data from victims office365 by using access_token which we get by phishing. It steals outlook mails, attachments, OneDrive files, OneNote notes and injects macros.
 

nehalist Bot for crawling stock availability of RTX 3000 cards and tweeting about it
 

alirezamika AutoScraper: A Smart, Fast and Lightweight Automatic Web Scraper for Python
 

Lumorti A dungeon crawler designed for a quantum computer as a series of 17000 quantum gates.
 

kangvcar 支持数据源包括GitHub、QQ邮箱、网易邮箱、阿里邮箱、新浪邮箱、Hotmail邮箱、Outlook邮箱、京东、淘宝、支付宝、中国移动、中国联通、中国电信、知乎、哔哩哔哩、网易云音乐、QQ好友、QQ群、生成朋友圈相册、浏览器浏览历史、12306、博客园、CSDN博客、开源中国博客、简书。
 

codingforentrepreneurs Scrape websites asynchronously with Python 3.8+, Asyncio, & arsenic (aka Selenium for Async).
 

SamPom100 scans every ticker on the market, gets their last 5 months of volume history, and alerts you when a stock's volume exceeds 10 standard deviations from the mean within the last 3 days
 

pirate 🗃 The open source self-hosted web archive. Takes browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
 

micha3lb3n Tested environments: Windows, MAC, linux, and windows subsystem for linux (WSL)
 

piyx A Script which adds all songs from youtube playlist to a new spotify playlist.
 

lemonpaul Simple Python script, that allow to import favorite tracks, playlists, albums and artists from Yandex.Music to Spotify
 

Gerapy This is a package for supporting pyppeteer in Scrapy, also this package is a module in Gerapy.
 

gusdnd852 A collection of useful Korean crawlers (always updated) 🌐
 

anoopjangra Library to extract realtime and historical data from NSE website.EOD data like bhavcopy and option chain are also saved to directory. First run will create directories for storing the data and will download the index symbols.
 

hausa-han A spider coded to get the hot_Comment of WangYiYun
 

neelsomani Scrape public filings of the buy + sell orders of U.S. senators and calculate their returns
 

Destaq Download the hottest 100 images from r/wallpaper and set them as your cycling Desktop background.
 

SXKDZ A simple configurable bot for sending arXiv article alert by mail
 

wbt5 获取斗鱼&虎牙&哔哩哔哩&抖音&快手等26个直播平台的真实流媒体地址(直播源)和弹幕,直播源可在PotPlayer、flv.js等播放器中播放。
 

adbar Trafilatura is Python package and command-line tool which seamlessly downloads, parses, and scrapes web page data: it can extract metadata, main body text and comments while preserving part of the text formatting and page structur
 

twintproject An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.
 

knudmoeller This is a scraper for the daily press releases announcing the current Corona/COVID-19 case numbers for Berlin, as issued by the Senatsverwaltung für Gesundheit, Pflege und Gleichstellung (Senate Department for Health, Care and Equ
 

psalias2006 Google2Csv is a simple google scraper that saves the results on a csv/xlsx/jsonl file
 

mps-youtube Python library to download YouTube content and retrieve metadata
 

chris-hamberg Scraper that downloads Springers Free COVID-19 English books.
 

sameera-madushan SBOX - Subtitle Box SBOX is a python script to download subtitles for your movies from SubDB database using their API. SubDB is a free, centralized subtitle database intended to be used only by opensource and non-commer
 

dvingerh Python script to download videos from a TikTok profile without any watermarks.
 

Jodagito YouTube Downloader V1.1 Requirements python>=3.4 pytube3==9.5.13 Installation Open a terminal and execute git clone https://github.com/Jodagito/YoutubeDownloader on the destination folder. On
 

luisvonmuller Spotify Playlist Downloader Downloads songs as listed on a spotify playlist from youtube. Uses scrapy, selenium (need chrome webdriver) and youtube_dl. It also uses a library called ffmpeg to convert from webm (without