MechanicalSoup was created by M Hickford, who was a fond user of the Mechanize library. Unfortunately, Mechanize was incompatible with Python 3 until 2019 and its development stalled for several years. MechanicalSoup provides a similar API, built on Python giants Requests (for HTTP sessions) and BeautifulSoup (for document navigation). Since 2017 it is a project actively maintained by a small team including @hemberger and @moy.
PyPy and PyPy3 are also supported (and tested against).
Download and install the latest released version from PyPI:
pip install MechanicalSoup
Download and install the development version from GitHub:
pip install git+https://github.com/MechanicalSoup/MechanicalSoup
Installing from source (installs the version in the current working directory):
python setup.py install
(In all cases, add
--user to the
install command to install in the current user's home directory.)
The full documentation is available on https://mechanicalsoup.readthedocs.io/. You may want to jump directly to the automatically generated API documentation.
From examples/expl_qwant.py, code to get the results from a Qwant search:
"""Example usage of MechanicalSoup to get the results from the Qwant search engine. """ import re import mechanicalsoup import html import urllib.parse # Connect to duckduckgo browser = mechanicalsoup.StatefulBrowser(user_agent='MechanicalSoup') browser.open("https://lite.qwant.com/") # Fill-in the search form browser.select_form('#search-form') browser["q"] = "MechanicalSoup" browser.submit_selected() # Display the results for link in browser.page.select('.result a'): # Qwant shows redirection links, not the actual URL, so extract # the actual URL from the redirect link: href = link.attrs['href'] m = re.match(r"^/redirect/[^/]*/(.*)$", href) if m: href = urllib.parse.unquote(m.group(1)) print(link.text, '->', href)
More examples are available in examples/.
For an example with a more complex form (checkboxes, radio buttons and textareas), read tests/test_browser.py and tests/test_form.py.
Instructions for building, testing and contributing to MechanicalSoup: see CONTRIBUTING.rst.
Read the FAQ.