A Python library for automating interaction with websites.

Home page https://mechanicalsoup.readthedocs.io/ Overview A Python library for automating interaction with websites. MechanicalSoup automatically stores and sends cookies, follows redirects, and can follow links

Information

Category: Python / Testing
Watchers: 109
Star: 4.4k
Fork: 390
Last update: May 26, 2023

Resource links

https://github.com/MechanicalSoup/MechanicalSoup

Home page

https://mechanicalsoup.readthedocs.io/

Overview

A Python library for automating interaction with websites. MechanicalSoup automatically stores and sends cookies, follows redirects, and can follow links and submit forms. It doesn't do JavaScript.

MechanicalSoup was created by M Hickford, who was a fond user of the Mechanize library. Unfortunately, Mechanize was incompatible with Python 3 until 2019 and its development stalled for several years. MechanicalSoup provides a similar API, built on Python giants Requests (for HTTP sessions) and BeautifulSoup (for document navigation). Since 2017 it is a project actively maintained by a small team including @hemberger and @moy.

Installation

PyPy and PyPy3 are also supported (and tested against).

Download and install the latest released version from PyPI:

pip install MechanicalSoup

Download and install the development version from GitHub:

pip install git+https://github.com/MechanicalSoup/MechanicalSoup

Installing from source (installs the version in the current working directory):

python setup.py install

(In all cases, add --user to the install command to install in the current user's home directory.)

Documentation

The full documentation is available on https://mechanicalsoup.readthedocs.io/. You may want to jump directly to the automatically generated API documentation.

Example

From examples/expl_qwant.py, code to get the results from a Qwant search:

"""Example usage of MechanicalSoup to get the results from the Qwant
search engine.
"""

import re
import mechanicalsoup
import html
import urllib.parse

# Connect to duckduckgo
browser = mechanicalsoup.StatefulBrowser(user_agent='MechanicalSoup')
browser.open("https://lite.qwant.com/")

# Fill-in the search form
browser.select_form('#search-form')
browser["q"] = "MechanicalSoup"
browser.submit_selected()

# Display the results
for link in browser.page.select('.result a'):
    # Qwant shows redirection links, not the actual URL, so extract
    # the actual URL from the redirect link:
    href = link.attrs['href']
    m = re.match(r"^/redirect/[^/]*/(.*)$", href)
    if m:
        href = urllib.parse.unquote(m.group(1))
    print(link.text, '->', href)

More examples are available in examples/.

For an example with a more complex form (checkboxes, radio buttons and textareas), read tests/test_browser.py and tests/test_form.py.

Development

Instructions for building, testing and contributing to MechanicalSoup: see CONTRIBUTING.rst.

Common problems

Read the FAQ.

35 Open More issues

Closed

Submit an empty file when leaving a file input blank

This is in regards to issue #250

For the tests, I followed @moy 's train of thought :

they are basically a copy+paste without the creation of a temp file
assert value["doc"] == "" checks that the response contains an empty file

Thought a different test definition was necessary, was I right to assume so ?

In browser.py, I changed the continue around line 179 to something similar to what has been done in test__request_file here

There are 2 Add no file input submit test commits : the second one is simply a clean up of some commented code. Will avoid it next time !

I was unable to run test_browser.py due to some weird Import module error on modules that are installed, so I'm kind of Pull Requesting blindly. Does it matter that I say I'm confident in the changes though ?

opened Feb 13, 2019 by senabIsShort 27

Closed

MechanicalSoup logo

In the Roadmap, some artwork is requested. I asked an artistic friend to try to interpret this request, and this is what they came up with. I would love to use this as our logo (in both the README, as per the roadmap, and perhaps also as our organization icon). Before I make a PR, I just wanted to see if this was what you were going for.

opened Oct 12, 2017 by hemberger 20

Closed

Adds logo to readme and documentation

opened Aug 24, 2018 by mkormendy 18

Closed

Tests randomly hanging on Travis-CI

Every couple of Travis builds, I see one of the sub-builds hang. It happens frequently enough that I feel like I have to babysit Travis, which is not a good situation to be in. From what I can tell, this occurs under two conditions:

httpbin.org is under heavy load (this occurs infrequently, but can occur for extended periods of time)
flake8 hangs for some unknown reason (seems arbitrary, and rerunning almost always fixes it)

I really want to understand 2), because for 1) we could simply rely on httpbin.org a bit less if necessary.

opened Oct 24, 2017 by hemberger 18

Closed

Remove `name` attribute from all unused buttons on form submit

I ran into a site with forms including buttons of type "button" with name attributes. Because Form.choose_submit() was only removing name from buttons of type "submit", the values for the "button" buttons were being erroneously sent on POST, thereby breaking my submission. This patch fixes the issue, even when a submit button isn't explicitly chosen.

Note that all buttons that aren't of type "button" or "reset" function as "submit" in all major browsers and should therefore be choosable.

opened Feb 27, 2018 by blackwind 16

Closed

Do not submit disabled <input> elements

https://www.w3.org/TR/html52/sec-forms.html#element-attrdef-disabledformelements-disabled

The disabled attribute is used to make the control non-interactive and to prevent its value from being submitted.

MechanicalSoup ignores disabled attributes which should be fixed.

Some additional notes: (from https://www.wufoo.com/html5/disabled-attribute/)

If the disabled attribute is set on a <fieldset>, the descendent form controls are disabled.

A disabled field can’t be modified, tabbed to, highlighted, or have its contents copied. Its value is also ignored when the form goes thru constraint validation.

The disabled value is Boolean, and therefore doesn’t need a value. But, if you must, you can include disabled="disabled".

Setting the value of the disabled attribute to null does not remove the effects of the attribute. Instead use removeAttribute('disabled').

You can target elements that are disabled with the :disabled pseudo-class. Or, if you want to specifically target the presence of the attribute, you can use input[disabled]. Similarly, you can use :enabled and input:not([disabled]) to target elements that are not disabled.

You do not need to include aria-disabled="true" when including the disabled attribute because disabled is already well supported. However, if you are programmatically disabling an element that is not a form control and therefore the disabled attribute does not apply, include aria-disabled="true".

The disabled attribute is valid for all form controls including all <input> types, <textarea>, <button>, <select>, <fieldset>, and <keygen>.

opened Nov 11, 2018 by 5j9 14

Open

Add to the `browser.follow_link(...)` docstring what the valid kwargs are

Currently the docstring says something to the effect of "kwargs are passed to browser.links(...)". When coding in an IDE like VSCode, which shows the docstring as you're typing, this isn't super helpful as you have to go look up the docs for browser.links(...). It would be great if it just showed the valid args right there, either by having them as named args to browser.follow_links(...), or as a description.

opened May 8, 2023 by DeflateAwning 0

Open

Add a way to fork a browser

I want a way to fork a Browser state, by calling something like browser_copy = browser.copy(), and then being able to work from browser_copy, while eventually going back to the un-touched browser.

Why I want it

I was writing a scraper for a file browser site. To download files, you first navigate to a specific folder, then you click on each file. I wanted to fork/copy the browser before clicking on each file download link, but couldn't.

What I tried

import copy
browser_copy = copy.copy(browser) # works on the first use, but not after
browser_copy = deepcopy.copy(browser) # crashes here, as it can't copy recursive elements

What I want

I want you to add a browser.copy() or browser.fork() method, which returns a fork of the browser with the same active page, same selected from, same session, etc., which can then be navigated independently from there. This would be similar to duplicating a tab in a GUI web browser, presumably.

enhancement

opened May 8, 2023 by DeflateAwning 3

Open

[Snyk] Security upgrade certifi from 2021.10.8 to 2022.12.7

Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

Changes included in this PR

Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
- tests/requirements.txt

⚠️ Warning

pytest-flake8 1.1.1 has requirement pytest>=7.0, but you have pytest 4.6.11.
pytest-flake8 1.1.1 has requirement flake8>=4.0, but you have flake8 3.9.2.

Vulnerabilities that will be fixed

By pinning:

Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- | 554/1000
Why? Has a fix available, CVSS 6.8 | Insufficient Verification of Data Authenticity
SNYK-PYTHON-CERTIFI-3164749 | certifi:
2021.10.8 -> 2022.12.7
| No | No Known Exploit

(*) Note that the real score may have changed since the PR was raised.

Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.

Check the changes in this PR to ensure they won't cause issues with your project.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

For more information: 🧐 View latest project report

🛠 Adjust project settings

Learn how to fix vulnerabilities with free interactive lessons:

🦉 Learn about vulnerability in an interactive lesson of Snyk Learn.

opened Feb 14, 2023 by snyk-bot 2

Open

python cleanup for 3.11

converted strings to f-strings
sorted imports
added trailing commas (as per 3.5, 3.6+)
fixed some docstrings
fixed spacings
made some quotes consistent

opened Jan 22, 2023 by marksmayo 1

Open

browser.links() should return an empty list if self.page is None

I was writing a fuzzer for a cybersecurity assignment, and it crashed when it tried to find the links on a PDF file. I think it would make more sense to return that there are no links, if the page fails to parse. This seems relatively straightforward to implement.

opened Nov 29, 2022 by npetrangelo 1

Open

Add typehints

null

opened May 7, 2022 by gophra 4

v1.2.0(Sep 17, 2022)

Main changes

Added support for Python 3.10.
Added support for HTML form-associated elements (i.e. input elements that are associated with a form by a form attribute, but are not a child element of the form). [#380]

Bug fixes

When uploading a file, only the filename is now submitted to the server. Previously, the full file path was being submitted, which exposed more local information than users may have been expecting. [#375]

Source code(tar.gz)
Source code(zip)

v1.1.0(May 29, 2021)

Main changes

Dropped support for EOL Python versions: 2.7 and 3.5.
Increased minimum version requirement for requests from 2.0 to 2.22.0 and beautifulsoup4 from 4.4 to 4.7.
Use encoding from the HTTP request when no HTML encoding is specified. [#355]
Added the put method to the Browser class. This is a light wrapper around requests.Session.put. [#359]
Don't override Referer headers passed in by the user. [#364]
StatefulBrowser methods follow_link and download_link now support passing a dictionary of keyword arguments to requests, via requests_kwargs. For symmetry, they also support passing Beautiful Soup args in as bs4_kwargs, although any excess **kwargs are sent to Beautiful Soup as well, just as they were previously. [#368]

Many thanks to the contributors who made this release possible!

Source code(tar.gz)
Source code(zip)

v1.0.0(Jan 5, 2021)

This is the last release that will support Python 2.7. Thanks to the many contributors that made this release possible!

Main changes:

Added support for Python 3.8 and 3.9.
StatefulBrowser has new properties page, form, and url, which can be used in place of the methods get_current_page, get_current_form and get_url respectively (e.g. the new x.page is equivalent to x.get_current_page()). These methods may be deprecated in a future release. [#175]
StatefulBrowser.form will raise an AttributeError instead of returning None if no form has been selected yet. Note that StatefulBrowser.get_current_form() still returns None for backward compatibility.

Bug fixes

Decompose <select> elements with the same name when adding a new input element to a form. [#297]
The params and data kwargs passed to submit will now properly be forwarded to the underlying request for GET methods (whereas previously params was being overwritten by data). [#343]

Source code(tar.gz)
Source code(zip)

v0.12.0(Aug 27, 2019)

Main changes:

Changes in official python version support: added 3.7 and dropped 3.4.
Added ability to submit a form without updating StatefulBrowser internal state: submit_selected(..., update_state=False). This means you get a response from the form submission, but your browser stays on the same page. Useful for handling forms that result in a file download or open a new tab.

Bug fixes

Improve handling of form enctype to behave like a real browser. [#242]
HTML type attributes are no longer required to be lowercase. [#245]
Form controls with the disabled attribute will no longer be submitted to improve compliance with the HTML standard. If you were relying on this bug to submit disabled elements, you can still achieve this by deleting the disabled attribute from the element in the Form object directly. [#248]
When a form containing a file input field is submitted without choosing a file, an empty filename & content will be sent just like in a real browser. [#250]
<option> tags without a value attribute will now use their text as the value. [#252]
The optional url_regex argument to follow_link and download_link was fixed so that it is no longer ignored. [#256]
Allow duplicate submit elements instead of raising a LinkNotFoundError. [#264]

Our thanks to the many new contributors in this release!

Source code(tar.gz)
Source code(zip)

v0.11.0(Sep 11, 2018)

This release focuses on fixing bugs related to uncommon HTTP/HTML scenarios and on improving the documentation.

Bug fixes

Constructing a Form instance from a bs4.element.Tag whose tag name is not form will now emit a warning, and may be deprecated in the future. [#228]
Breaking Change: LinkNotFoundError now derives from Exception instead of BaseException. While this will bring the behavior in line with most people's expectations, it may affect the behavior of your code if you were heavily relying on this implementation detail in your exception handling. [#203]
Improve handling of button submit elements. Will now correctly ignore buttons of type button and reset during form submission, since they are not considered to be submit elements. [#199]
Do a better job of inferring the content type of a response if the Content-Type header is not provided. [#195]
Improve consistency of query string construction between MechanicalSoup and web browsers in edge cases where form elements have duplicate name attributes. This prevents errors in valid use cases, and also makes MechanicalSoup more tolerant of invalid HTML. [#158]

Source code(tar.gz)
Source code(zip)

v0.10.0(Feb 3, 2018)

Main changes:

Added StatefulBrowser.refresh() to reload the current page with the same request. [#188]
StatefulBrowser.follow_link, StatefulBrowser.submit_selected() and the new StatefulBrowser.download_link now sets the Referer: HTTP header to the page from which the link is followed. [#179]
Added method StatefulBrowser.download_link, which will download the contents of a link to a file without changing the state of the browser. [#170]
The selector argument of Browser.select_form can now be a bs4.element.Tag in addition to a CSS selector. [#169]
Browser.submit and StatefulBrowser.submit_selected accept a larger number of keyword arguments. Arguments are forwarded to requests.Session.request. [#166]

Internal changes:

StatefulBrowser.choose_submit will now ignore input elements that are missing a name-attribute instead of raising a KeyError. [#180]
Private methods Browser._build_request and Browser._prepare_request have been replaced by a single method Browser._request. [#166]

Source code(tar.gz)
Source code(zip)

A Python library for automating interaction with websites.

automating AWS with Python using boto3 library

Mechanize is a ruby library that makes automated web interaction easy.

Catalogue of models of interaction. Part of the trilogy on models of computers/interaction/computation

React native swipe cards interaction

Simple & light weight (3kb minified & zipped) vanilla javascript plugin to create smooth & beautiful animations when you scrolllll! Harness the power of the most intuitive interaction and make your websites come alive!

Tabbar Component For React-Native

Lux is a Python library that makes data science easier by automating aspects of the data exploration process

Tensorflow Implementation of Interaction Networks for Learning about Objects, Relations and Physics

This's an implementation of deepmind Visual Interaction Networks paper using pytorch

Development-environment CLI program providing the native-like interaction with a Dockerized application.

Automating Web Performance testing with Puppeteer 🎪

Websites within websites

Filters to block and remove copycat-websites from DuckDuckGo and Google. Specific to dev websites like StackOverflow or GitHub.

A tool for automating the installation of the Microsoft Windows operating system on various device platforms.

Milano is a tool for automating hyper-parameters search for your models on a backend of your choice.

idb is a flexible command line interface for automating iOS simulators and devices

Run IDA Pro disassembler in Docker containers for automating, scaling and distributing the use of IDAPython scripts.

A plugin-oriented tool for automating the investigation of broken hosts and services.

A process for automating Docker container base image updates.

obtain free SSL certificates from letsencrypt ACME server Suitable for automating the process on remote servers.

Share this repo

Related Repos

Testing

A simple Python script I wrote that scrapes NASA's James Webb Space Telescope tracker webs...

This script requires Selenium to run. To install it, run the command: pip install selenium The path for geckodriver (if you have Firefox installed) i

Testing

Fuzz Introspector

Fuzz introspector Fuzz introspector is a tool to help fuzzer developers to get an understanding of their fuzzer’s performance and identify any potenti

Testing

Selenium webdriver step library for use with the behave BDD testing framework

behave-webdriver behave-webdriver is a step library intended to allow users to easily write selenium webdriver tests with the behave BDD testing frame

Testing

Callisto is an open-source Kubernetes-native implementation of Selenium Grid.

Callisto Callisto is an open-source Kubernetes-native implementation of Selenium Grid. Architecture Callisto architecture Components Nginx proxies req

Testing

748

A curated list of delightful Selenium resources.

Awesome Selenium A curated list of delightful Selenium resources. Inspired by the awesome list thing. Resources Tools CSS Regression Testing Container

Testing

An extremely simple, yet extensible framework to evade AV with obfuscated payloads under W...

An extremely simple, yet extensible framework to evade AV with obfuscated payloads under Windows.

Testing

Generates realistic traffic for load testing tile servers.

Realistic load testing for tile servers

Testing

120

A framework designed to test authentication for web applications.

Web authentication testing framework

Testing

Activate your test enviroment, so you can use your test dependencies in the REPL

This is a 1-function package: TestEnv.activate. It lets you activate the test enviroment from a given package. Just like Pkg.activate lets you activate it's main enviroment.