Home / Python / Data Analysis

A Python library for creating fast, repeatable and self-documenting data analysis pipelines.

proof is a Python library for creating optimized, repeatable and self-documenting data analysis pipelines. proof was designed to be used with the agate data analysis library, but can be used with numpy, pandas or any other method of proces
Information
Category: Python / Data Analysis
Watchers: 11
Star: 223
Fork: 20
Last update: Oct 15, 2021 
proof is a Python library for creating optimized, repeatable and self-documenting data analysis pipelines.

proof was designed to be used with the agate data analysis library, but can be used with numpy, pandas or any other method of processing data.

Important links:

* Documentation:    http://proof.rtfd.org
* Repository:       https://github.com/wireservice/proof
* Issues:           https://github.com/wireservice/proof/issues
11 Open More issues
Closed

Analysis.run should return final state of data, for output printing that always happens

opened Sep 2, 2015 by onyxfish 2
Closed

Why not proof.py?

Any reason the library is a package and not a single Python module?

opened Sep 10, 2015 by NiklasRosenstein 2
Closed

Don't load Analysis cache data unless it is needed

Formerly onyxfish/agate#184

opened Aug 31, 2015 by onyxfish 1
Closed

A `never_cache` flag?

Might make sense as a way of allowing "print blocks"

opened Sep 2, 2015 by onyxfish 1
Closed

Fix spelling of "propagated"

opened Sep 6, 2015 by jean 1
Closed

Can't hash unicode data 

File "/Users/tylerfisher/.virtualenvs/amazon-data/lib/python2.7/site-packages/proof/analysis.py", line 133, in then
    _trace=self._trace
  File "/Users/tylerfisher/.virtualenvs/amazon-data/lib/python2.7/site-packages/proof/analysis.py", line 92, in __init__
    self._cache_path = os.path.join(self._cache_dir, '%s.cache' % self._fingerprint())
  File "/Users/tylerfisher/.virtualenvs/amazon-data/lib/python2.7/site-packages/proof/analysis.py", line 108, in _fingerprint
    hasher.update(source.encode('utf-8'))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 106: ordinal not in range(128)
opened Sep 24, 2015 by TylerFisher 0
Open

A built-in way to define a process as the last one to be executed

Hello,

Would be great if there was a built-in way to define that a process will be the last one executed. Something like:

works as expected...


analysis = proof.Analysis(initialize_data, last_process=last_process)
analysis.then(process_0)
analysis.then(process_1)
analysis.then(process_2)
analysis.then(last_process)

raises an exception


analysis = proof.Analysis(initialize_data, last_process=last_process)
analysis.then(process_0)
analysis.then(process_1)
analysis.then(process_2)
analysis.then(last_process)
analysis.then(process_3)
# Exception(Other process was added after the one specified as the last one.)

raises an exception


analysis = proof.Analysis(initialize_data, last_process=last_process)
analysis.then(process_0)
analysis.then(process_1)
analysis.then(process_2)
# Exception(It's missing the last one!)

In the last example, instead an exception, the call of the method then (with the last process) could be implicit, but you know... "Explicit is better than implicit."

What do you think ?

opened Jul 11, 2016 by alexandre 0
Open

Make it possible to retrieve the final data object from an executed analysis

This would make it easier to use Proof to load and clean your data, then to switch to exploratory analysis with another tool, such as a Jupyter notebook.

opened Mar 27, 2016 by aplayford 2
Open

Better handling of callables

Hello,

Instead to use a function, I prefer to write a class aiming a better description of my analysis rules. So, I defined a callable:

class MyCallable(object):
    def __init__(self):
        # proof.Analysis will look for a dunder name attribute (duck typing)
        self.__name__ = self.__class__.__name__

    def __call__(self, data):
        # analysis rules

Then, I use it in the Analysis.then method:

import proof

analysis_one = MyCallable()
myproof = proof.Analysis(initial_data)

myproof.then(analysis_one)

The proof.Analisys._fingerprint is considering only functions and instead any callable like.

TL;DR I got this exception: https://github.com/python/cpython/blob/master/Lib/inspect.py#L624

I think this PR could be an initial step to a better way to handle callable in this method (_fingerprint).

P.S - I'm sorry if I broke some PR rule/template. I would be happy to discuss this point, improve this PR and write more tests if necessary.

opened Feb 18, 2016 by alexandre 4
Open

Make cache a parameter to Analysis constructor

There are often cases where multiple reporters, with different skillsets, will be working on analysis in parallel. Often the common language is a SQL database. It would be cool to be able to replace the disk-based Pickle cache with a cache that is storing the data to SQL tables. This is outside the scope of proof, but parameterizing the cache would let people write their own cache implementations.

opened Jan 11, 2016 by ghing 2
Open

deepcopy bug

I think this is a proof issue?

Sample code:

import agate
import proof

def load_data(data):
    data['responses'] = agate.Table.from_csv('responses.csv')

def unweighted_totals(data):
    data['first'] = data['responses'].group_by('first')

if __name__ == '__main__':

    data_loaded = proof.Analysis(load_data)
    data_loaded.then(unweighted_totals)

Result:

File "tally.py", line 31, in <module>
    data_loaded.run()
  File "/Users/tylerfisher/.virtualenvs/listeners2015/lib/python2.7/site-packages/proof/analysis.py", line 204, in run
    analysis.run(refresh=refresh, _parent_cache=self._cache)
  File "/Users/tylerfisher/.virtualenvs/listeners2015/lib/python2.7/site-packages/proof/analysis.py", line 192, in run
    local_data = _parent_cache.get()
  File "/Users/tylerfisher/.virtualenvs/listeners2015/lib/python2.7/site-packages/proof/analysis.py", line 50, in get
    return deepcopy(self._data)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 237, in _deepcopy_tuple
    y.append(deepcopy(a, memo))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 298, in _deepcopy_inst
    state = deepcopy(state, memo)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 174, in deepcopy
    y = copier(memo)
TypeError: cannot deepcopy this pattern object

If I remove proof from the equation:

import agate

data = {}

def load_data():
    data['responses'] = agate.Table.from_csv('responses.csv')

def unweighted_totals():
    data['first'] = data['responses'].group_by('first')

if __name__ == '__main__':
    load_data()
    unweighted_totals()

It all works, and I can do things with my grouped data.

opened Dec 11, 2015 by TylerFisher 3
Open

Factor analysis step caching into separate package

I like the idea behind proof, but find the step chaining cumbersome in some cases. I know this isn't very constructive feedback and I'm still working out exactly which cases these are. Rather than feature requesting and making the tool more opinionated about workflow, I'm wondering if the caching logic/API could be factored out into a separate package, allowing the persisting of data and passing it between steps to work using a different task runner, such as invoke.

Factoring this out into a separate package could also parameterize this operation, one day allowing power users to address #11 by serializing to a different format assuming their data is easily serialized as JSON (or something else).

opened Nov 23, 2015 by ghing 1

Contents
    luontola Testing framework for Go. Allows writing self-documenting tests/specifications, and executes them concurrently and safely isolated. [UNMAINTAINED]
    calebwin Pipelines is a language for high-level data analysis.
    tuplex Tuplex is a parallel big data processing framework that runs data science pipelines written in Python
    EntilZha Python library for creating data pipelines with chain functional programming
    cgarciae Concurrent data pipelines in Python >>>
    combust MLeap: Deploy Spark Pipelines to Production
    avito-tech Framework for creating efficient data processing pipelines
    timoschick This repository contains the code for "Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP".
    c8r Tools for developing, documenting, and testing React component libraries
    dongsuo A Data Analysis Board in Vue.
    databricks Deep Learning Pipelines for Apache Spark
    acln0 Zero-copy I/O primitives and pipelines for Go. Linux-specific.
    nickslevine Data analysis library for JavaScript built with Ramda
    formatc1702 WireViz is a tool for easily documenting cables, wiring harnesses and connector pinouts.
    SaraSoueidan This is where I will be documenting & sharing the process behind the refactor and redesign of my personal Web site.
    yardenshafir Documenting system information classes and their uses
    fluxcd Documenting the GitOps principles & other GitOps WG
    alii A detailed guide documenting the setup process of Next.js + Discord OAuth & TypeScript
    gitops-working-group Documenting the GitOps principles & other GitOps WG
    tensorflow TFX is an end-to-end platform for deploying production ML pipelines
    pandas-dev Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more
    Resource links
    http://proof.rtfd.org
    https://github.com/wireservice/proof
    Share this repo

    Related Repos


    Data Analysis
    1k
    tidytext: Text mining using tidy tools
    juliasilge tidytext: Text mining using tidy tools
     
    Data Analysis
    163
    Efficient Python Tricks and Tools for Data Scientists
    khuyentran1401 Efficient Python Tricks and Tools for Data Scientists
     
    Data Analysis
    41
    Framework for creating efficient data processing pipelines
    avito-tech Framework for creating efficient data processing pipelines.
     
    Data Analysis
    94
    Set of Mindmaps providing a detailed overview of the different #Windows auditing capacitie...
    mdecrevoisier Set of Mindmaps providing a detailed overview of the different #Windows auditing capacities and event log files.
     
    Data Analysis
    17
    numberwang will convert floating point numbers (and integers) to their word representation...
    coolbutuseless numberwang will convert floating point numbers (and integers) to their word representations, and vice versa.
     
    Data Analysis
    4.7k
    🦔 PostHog provides open-source product analytics that you can self host.
    posthog PostHog is an open-source product analytics suite, built for developers. Automate the collection of every event on your website or app, with no need to send data to 3rd parties.
     
    Data Analysis
    31
    Mapillary Street-level Sequences (MSLS) is a large-scale long-term place recognition datas...
    mapillary Mapillary Street-level Sequences (MSLS) is a large-scale long-term place recognition dataset that contains 1.6M street-level images.
     
    Data Analysis
    25
    DenseClus is a Python module for clustering mixed type data using UMAP and HDBSCAN.
    awslabs DenseClus is a Python module for clustering mixed type data using UMAP and HDBSCAN. Allowing for both categorical and numerical data, DenseClus makes it possible to incorporate all features in clustering.
     
    Data Analysis
    12
    PreallocationTools.jl is a set of tools for helping build non-allocating pre-cached functi...
    SciML PreallocationTools.jl is a set of tools for helping build non-allocating pre-cached functions for high-performance computing in Julia.
     
    Data Analysis
    12
    Pyspark Spotify ETL
    Amaguk2023 This is my first Data Engineering project, it extracts data from the user's recently played tracks using Spotify's API, transforms data and then loads it into Postgresql using SQLAlchemy engine. Data is shown as a Spark Dataframe before loading and the whole ETL job is scheduled with crontab.
     
    Data Analysis
    454
    Official data on the COVID-19 epidemic in Malaysia.
    MoH-Malaysia Official data on the COVID-19 epidemic in Malaysia. Powered by CPRC, CPRC Hospital System, MKAK, and MySejahtera.
     
    Data Analysis
    16
    Data exploration done quick.
    ryxcommar Implementation of Stata's tabulate command in Pandas for extremely easy to type one-way and two-way tabulations.
     
    Data Analysis
    52
    To provide 100 datatable exercises over different sections structured as a course or tutor...
    vopani To provide 100 datatable exercises over different sections structured as a course or tutorials to teach and learn for beginners, intermediates as well as experts.
     
    Data Analysis
    34
    Implementation in R of the Black Scholes formula and some greeks.
    daleroberts Implementation in R of the Black Scholes formula and some greeks.
     
    Data Analysis
    261
    Official data on Malaysia's National Covid-​19 Immunisation Programme (PICK).
    CITF-Malaysia Official data on Malaysia's National Covid-​19 Immunisation Programme (PICK). Powered by MySejahtera.