BeakerX: Beaker extensions for Jupyter
BeakerX is a collection of JVM kernels and interactive widgets for plotting, tables, autotranslation, and other extensions to Jupyter Notebook.
This README is for developers. Users should see the documentation on the homepage for how to install and run BeakerX.
Build and Install (linux and mac)
conda env create -n beakerx -f configuration.yml conda activate beakerx # For conda versions prior to 4.6, run: source activate beakerx (cd beakerx; pip install -r requirements.txt --verbose) beakerx install beakerx_databrowser install
Build and Install (win)
conda env create -n beakerx -f configuration.yml activate beakerx cd beakerx pip install -r requirements.txt --verbose cd .. beakerx install beakerx_databrowser install
Build and Install for Jupyter Lab
conda env create -n labx -f configuration.yml conda activate labx # For conda versions prior to 4.6, run: source activate labx conda install -y -c conda-forge jupyterlab=1 (cd beakerx; pip install -r requirements.txt --verbose) beakerx install jupyter labextension install @jupyter-widgets/jupyterlab-manager --no-build (cd js/lab; jupyter labextension install . --no-build) (cd js/lab-theme-dark; jupyter labextension install . --no-build) (cd js/lab-theme-light; jupyter labextension install . --no-build) jupyter lab build
Running with Docker
docker run -p 8888:8888 beakerx/beakerx
Update after Java change
The kernels are installed to run out of the repo, so just a local build should suffice:
(cd kernel; ./gradlew build)
Update after JS change
The notebook extensions are installed to run out of the repo, so just a local build should suffice:
(cd js/notebook; yarn run build) beakerx install
Run TypeScript Unit Tests
The Java and TypeScript unit tests are run with every build. See test/README.md for how to run the e2e tests.
Run Python Unit Tests
(cd beakerx; python -m unittest) (cd beakerx_tabledisplay; python -m unittest)
Groovy with Interactive Plotting:
Table Display from conda package
conda create --name beakerx_td source activate beakerx_td conda install beakerx_tabledisplay
conda create --name labx_td source activate labx_td conda install -y -c conda-forge jupyterlab conda install beakerx_tabledisplay beakerx_tabledisplay install --lab
Architecture and Code Overview
BeakerX is a collection of kernels and extensions for Jupyter. The code is organized into subdirectories as follows:
beakerx The Python packages. The main beakerx package has:
a customized KernelSpec to allow BeakerX to configure the JVMs that run the kernels,
a server extension for the javadoc, settings, version, and autotranslation endpoints,
the beakerx command line program, which has the bkr2ipynb converter, the py4j server, utilities, install, and uninstall functions.
the Python API for the runtime (tables, plots, easyform), including automatically installing a displayer for pandas tables, and autotranslation;
the compiled Java JARs of each of the kernels, and a directory of shared JARs.
There is a separate python package (beakerx_magics) for the
%%groovymagic so it can be loaded without loading the regular beakerx package (which would turn on display of pandas tables with our table widget).
BeakerX configures ipython to automatically load the magics in the beakerx_magics package,
%load_extis not required.
The groovy magic uses the standard Jupyter API, jupyter_client.manager.KernelManager to start the kernel. It then proxies Comm into the inner kernel.
This package also has the py4j support for the
%%pythonmagic. In order for the JVM kernels to be able to start Jupyter kernels they need to be able to call into Python. There is a
beakerx py4j_serversubcommand for this purpose (for internal use, not for the user). It calls into the groovy magic with its Comm proxy, implemented in Python.
BeakerX implements autotranslation with a server extension and metaprogramming glue in each language. The glue makes the
The autotranslation server has a separate thread of control from Jupyter, and it manages its own port, which it protects by accepting only local connections and requiring a secure password (which is passed to the kernels via an environment variable). The extra thread is necessary to avoid deadlock in tornado. This might be better done with a queue, as explained in #5039.
See #7577 for the plan to improve this architecture with shared memory and Apache Arrow.
docker configuration files for creating the Docker image. There is a subdirectory docker/base for an image with BeakerX's dependencies (the Ubuntu and conda packages). The main image is built by compiling BeakerX and installing BeakerX in the base image.
The lab subdirectory has the extension for Jupyter Lab (distributed by npm). Notebook has two extensions, one for the widgets (which are included in Lab as well, and are also separately distributed with npm for embedded applications such as nbviewer), and one for the notebook application. This adds a tab to the tree view with our options panel.
And for regular notebook pages the extension handles: running initialization cells, publication, autotranslation, the getCodeCells and runByTag APIs, callbacks for table and plot actions, UI customizations such as changing the fonts, allowing wide code cells, and disabling autosave.
kernel The Java implementation of the kernels is here. The main directory is kernel/base which has generic code for all the languages. The base kernel has classes for Jupyter's Comm protocol (a layer over ZMQ), magics, the classpath (including loading from maven), and the kernel parts of the widget APIs.
There is also a subdirectory for each language which has the evaluator for that language. Scala has wrappers for the widgets so they have native types.
BeakerX contains and depends on many projects including:
The kernel is originally derived from lappsgrid, but has been rewritten in Java and refactored and expanded.
The Java support uses Adrian Witas' org.abstractmeta.toolbox.
ANTLR Copyright (c) 2012 Terence Parr and Sam Harwell
d3 Copyright (c) 2010-2015, Michael Bostock
IPython Copyright (c) 2008-2014, IPython Development Team Copyright (c) 2001-2007, Fernando Perez Copyright (c) 2001, Janko Hauser Copyright (c) 2001, Nathaniel Gray
The table of contents and init cells extensions come from: IPython-contrib Copyright (c) 2013-2015, IPython-contrib Developers
Scala Copyright (c) 2002-2015 EPFL Copyright (c) 2011-2015 Typesafe, Inc.
Guava Copyright (C) 2012 The Guava Authors
Apache Spark Copyright (C) 2014 and onwards The Apache Software Foundation.
H2 database engine This software contains unmodified binary redistributions for H2 database engine (http://www.h2database.com/), which is dual licensed and available under the MPL 2.0 (Mozilla Public License) or under the EPL 1.0 (Eclipse Public License). An original copy of the license agreement can be found at: http://www.h2database.com/html/license.html