Home / Swift / Network

A Swift Web Crawler 🕷

Selenops 🕷 Welcome to Selenops, a stupid simple Swift Web Crawler. This repository contains three ways to use Selenops: A Swift Playground A Swift Script A 🏃 Marathon Script To know more about ev
Information
Category: Swift / Network
Watchers: 4
Star: 241
Fork: 18
Last update: May 22, 2022
Resource links
https://github.com/zntfdr/Selenops

Selenops 🕷

Swift Web Crawler in action

Welcome to Selenops, a stupid simple Swift Web Crawler.

This repository contains three ways to use Selenops:

  • A Swift Playground
  • A Swift Script
  • A 🏃 Marathon Script

To know more about everything above, please refer to my blog here.

Installation 🕸

Download or clone the repo:

$ git clone [email protected]:zntfdr/Selenops.git

How to Run Selenops 🕷

Swift Web Crawler in action

  • Swift Playground
    Just open the .playground package with Xcode.

  • Swift Script
    Make Script/Selenops.swift executable:
    $ chmod +x Script/Selenops.swift
    Run it:
    $ Script/Selenops.swift.

  • 🏃 Marathon Script
    Run it as any other Marathon script:
    $ marathon run Script/Selenops

Author 🕸

Federico Zanetello (@zntfdr)

0 Open More issues
Closed

Added support for command line arguments and help message describing them.

Added support for command line arguments (all optional, use two (startUrl and searchWord) or three (additionally maxPagesToVisit)). Also added -help/help command line option.

opened Apr 13, 2017 by grysvn 4
Closed

Adopt ArgumentParser library

opened Feb 28, 2020 by zntfdr 0
Closed

add shouldVisit optional crawler function, Improve CLI UI

This PR:

  • adds new crawler(_:, shouldVisitUrl:) in CrawlerDelegate
  • introduces TSCUtility's NinjaProgressAnimation use for the cli tool
opened Feb 28, 2020 by zntfdr 0
Closed

Make URL ArgumentKind conformation follow TSC style

opened Feb 25, 2020 by zntfdr 0
Closed

Replace custom error type with ArgumentConversionError

opened Feb 25, 2020 by zntfdr 0
Closed

Revamp code, separate library and command line tool, add SPM support

Update readme, improve documentation, add SwiftLint CI, etc.

opened Feb 18, 2020 by zntfdr 0

2.0.1(Jan 17, 2021)

Library

  • add func crawler(_ crawler: Crawler, shouldVisitUrl url: URL) -> Bool method to CrawlerDelegate
  • make some CrawlerDelegate methods implementation optional
  • make crawler case-insensitive

Executable

  • Migrated to ArgumentParser
  • Better animations

2.0.0(Feb 18, 2020)

This release is a big revamp of the project:

  • replace script/playground code with one Swift package
  • separate Selenops in one library and one executable
  • add SwiftLint CI
  • use SwiftSoup for much better parsing
  • improve command line tool with better parsing and help
  • improve documentation

1.1.0(Feb 18, 2020)

1.0.0(Feb 18, 2020)

Contents

    antivanov Web crawler for Node.JS
    CrawlScript WebCollector is an open source web crawler framework based on Java.It provides some simple interfaces for crawling the Web,you can setup a multi-threaded web crawler in less than 5 minutes.
    spacenick Lightsabing Node/PhantomJS crawler; scrape dynamic content : without the hassle
    LitoMore Bing wallpaper crawler
    indrajithi A tiny web crawler in Python
    snjoer Broad Crawler
    code4craft A scalable web crawler framework for Java.
    dadoonet Elasticsearch File System Crawler (FS Crawler)
    yasserg Open Source Web Crawler for Java
    rivermont The simple, easy to use command line web crawler.
    PuerkitoBio Polite, slim and concurrent web crawler.
    BruceDone A collection of awesome web crawler,spider in different languages
    alinebastos Web Crawler created with Node.js and Puppeteer
    pop-os A configurable parallel web crawler, designed to crawl a website for content.
    USCDataScience Spark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.
    wspl :paw_prints: Creeper - The Next Generation Crawler Framework (Go)
    wcong open source, distributed, restful crawler engine in golang
    shirosaidev File system crawler, disk space usage, file search engine and file system analytics powered by Elasticsearch
    yujiosaka Distributed crawler powered by Headless Chrome
    reactor-joy Simple content crawler for joyreactor.cc
    guilatrova Google Maps crawler using Selenium
    Share this repo

    Related Repos


    Network
    5
    Swift async/await OAuth 2.0 HTTP request library.
    Colaski SwAuth SwAuth is an OAuth 2.0 HTTP request library written in Swift for iOS 15.0+, macOS 12.0+, watchOS 8.0+, and tvOS 15.0+. Features Requirements In
     
    Network
    3
    Advanced Networking Layer Using Alamofire with Unit Testing
    Ali-Fayed Advanced Swift Networking Layer Hi there, This is Advanced Networking Layer Using Alamofire with Unit Testing 👋 Networking Layer Features: Generic AP
     
    Network
    24
    Backward compatible async/await for URLSession!
    mochidev URLSessionBackport URLSessionBackport aims to make it possible to use URLSession's new async/await syntax on older OSs, namely iOS 13 or macOS 10.15 a
     
    Network
    47
    📡 Lightweight yet powerful HTTP Client for Swift ages - Declarative + Combine + Decodable...
    immobiliare RealHTTP RealHTTP is a lightweight yet powerful client-side HTTP library. Our goal is make an easy to use and effortless http client for Swift. Featur
     
    Network
    13
    DNS tunnel for iOS based off of Iodine
    osy Purple Haze A DNS tunnel client for iOS based on Iodine. A paid Apple Developer account is required to build because of the entitlements needed for Ne
     
    Network
    21
    gRPC implementation for Common Lisp
    qitab gRPC Client Library in Common Lisp Overview This package defines a gRPC client library for Common Lisp. It wraps gRPC core functions with CFFI calls a
     
    Network
    22
    Generic Network Layer created using Swift.
    Shubham0812 Generic Network Layer created using Swift.
     
    Network
    117
    A macOS application to test APIs with HTTP methods (Decluttered Postman)
    PraneetNeuro A macOS application to test APIs with HTTP methods (Decluttered Postman), built with SwiftUI.
     
    Network
    4.2k
    Pulse is a powerful logging system for Apple Platforms. Native. Built with SwiftUI.
    kean Network Inspector for Apple Platforms
     
    Network
    10
    Lightweight OAuth 2.0 Client with PKCE
    d-date Lightweight OAuth 2.0 Client with PKCE
     
    Network
    10
    iOS tweak companion for WebMessage client
    sgtaziz iOS tweak companion for WebMessage client.
     
    Network
    19
    DNS over TLS and DNS over HTTPS Configuration for iOS
    snbl DNS over TLS and DNS over HTTPS Configuration for iOS
     
    Network
    12
    VPN daemon utilized by nitoTV to deliver VPN access on tvOS
    lechium VPN daemon utilized by nitoTV to deliver VPN access on tvOS
     
    Network
    1.5k
    The Swift language implementation of gRPC.
    grpc It is intended for use with Apple's SwiftProtobuf support for Protocol Buffers. Both projects contain code generation plugins for protoc, Google's Protocol Buffer compiler, and both contain libraries of supporting code that is needed to build and run the generated code.
     
    Network
    67
    Log iOS network traffic without a proxy
    evilpenguin NetworkSniffer will log ALL traffic for any iOS application. This includes WKWebView and UIWebView.