proof is a Python library for creating optimized, repeatable and self-documenting data analysis pipelines. proof was designed to be used with the agate data analysis library, but can be used with numpy, pandas or any other method of processing data. Important links: * Documentation: http://proof.rtfd.org * Repository: https://github.com/wireservice/proof * Issues: https://github.com/wireservice/proof/issues
A Python library for creating fast, repeatable and self-documenting data analysis pipelines.
proof is a Python library for creating optimized, repeatable and self-documenting data analysis pipelines. proof was designed to be used with the agate data analysis library, but can be used with numpy, pandas or any other method of proces
11 Open
More issues
Closed
opened Sep 2, 2015 by onyxfish 2
Closed
Any reason the library is a package and not a single Python module?
opened Sep 10, 2015 by NiklasRosenstein 2
Closed
Formerly onyxfish/agate#184
opened Aug 31, 2015 by onyxfish 1
Closed
Might make sense as a way of allowing "print blocks"
opened Sep 2, 2015 by onyxfish 1
Closed
opened Sep 6, 2015 by jean 1
Closed
File "/Users/tylerfisher/.virtualenvs/amazon-data/lib/python2.7/site-packages/proof/analysis.py", line 133, in then
_trace=self._trace
File "/Users/tylerfisher/.virtualenvs/amazon-data/lib/python2.7/site-packages/proof/analysis.py", line 92, in __init__
self._cache_path = os.path.join(self._cache_dir, '%s.cache' % self._fingerprint())
File "/Users/tylerfisher/.virtualenvs/amazon-data/lib/python2.7/site-packages/proof/analysis.py", line 108, in _fingerprint
hasher.update(source.encode('utf-8'))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 106: ordinal not in range(128)
opened Sep 24, 2015 by TylerFisher 0
Open
Hello,
Would be great if there was a built-in way to define that a process will be the last one executed. Something like:
works as expected...
analysis = proof.Analysis(initialize_data, last_process=last_process)
analysis.then(process_0)
analysis.then(process_1)
analysis.then(process_2)
analysis.then(last_process)
raises an exception
analysis = proof.Analysis(initialize_data, last_process=last_process)
analysis.then(process_0)
analysis.then(process_1)
analysis.then(process_2)
analysis.then(last_process)
analysis.then(process_3)
# Exception(Other process was added after the one specified as the last one.)
raises an exception
analysis = proof.Analysis(initialize_data, last_process=last_process)
analysis.then(process_0)
analysis.then(process_1)
analysis.then(process_2)
# Exception(It's missing the last one!)
In the last example, instead an exception, the call of the method then (with the last process) could be implicit, but you know... "Explicit is better than implicit."
What do you think ?
opened Jul 11, 2016 by alexandre 0
Open
This would make it easier to use Proof to load and clean your data, then to switch to exploratory analysis with another tool, such as a Jupyter notebook.
opened Mar 27, 2016 by aplayford 2
Open
Hello,
Instead to use a function, I prefer to write a class aiming a better description of my analysis rules. So, I defined a callable:
class MyCallable(object):
def __init__(self):
# proof.Analysis will look for a dunder name attribute (duck typing)
self.__name__ = self.__class__.__name__
def __call__(self, data):
# analysis rules
Then, I use it in the Analysis.then method:
import proof
analysis_one = MyCallable()
myproof = proof.Analysis(initial_data)
myproof.then(analysis_one)
The proof.Analisys._fingerprint is considering only functions and instead any callable like.
TL;DR I got this exception: https://github.com/python/cpython/blob/master/Lib/inspect.py#L624
I think this PR could be an initial step to a better way to handle callable in this method (_fingerprint).
P.S - I'm sorry if I broke some PR rule/template. I would be happy to discuss this point, improve this PR and write more tests if necessary.
opened Feb 18, 2016 by alexandre 4
Open
There are often cases where multiple reporters, with different skillsets, will be working on analysis in parallel. Often the common language is a SQL database. It would be cool to be able to replace the disk-based Pickle cache with a cache that is storing the data to SQL tables. This is outside the scope of proof, but parameterizing the cache would let people write their own cache implementations.
opened Jan 11, 2016 by ghing 2
Open
I think this is a proof issue?
Sample code:
import agate
import proof
def load_data(data):
data['responses'] = agate.Table.from_csv('responses.csv')
def unweighted_totals(data):
data['first'] = data['responses'].group_by('first')
if __name__ == '__main__':
data_loaded = proof.Analysis(load_data)
data_loaded.then(unweighted_totals)
Result:
File "tally.py", line 31, in <module>
data_loaded.run()
File "/Users/tylerfisher/.virtualenvs/listeners2015/lib/python2.7/site-packages/proof/analysis.py", line 204, in run
analysis.run(refresh=refresh, _parent_cache=self._cache)
File "/Users/tylerfisher/.virtualenvs/listeners2015/lib/python2.7/site-packages/proof/analysis.py", line 192, in run
local_data = _parent_cache.get()
File "/Users/tylerfisher/.virtualenvs/listeners2015/lib/python2.7/site-packages/proof/analysis.py", line 50, in get
return deepcopy(self._data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 237, in _deepcopy_tuple
y.append(deepcopy(a, memo))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 298, in _deepcopy_inst
state = deepcopy(state, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 174, in deepcopy
y = copier(memo)
TypeError: cannot deepcopy this pattern object
If I remove proof from the equation:
import agate
data = {}
def load_data():
data['responses'] = agate.Table.from_csv('responses.csv')
def unweighted_totals():
data['first'] = data['responses'].group_by('first')
if __name__ == '__main__':
load_data()
unweighted_totals()
It all works, and I can do things with my grouped data.
opened Dec 11, 2015 by TylerFisher 3
Open
I like the idea behind proof, but find the step chaining cumbersome in some cases. I know this isn't very constructive feedback and I'm still working out exactly which cases these are. Rather than feature requesting and making the tool more opinionated about workflow, I'm wondering if the caching logic/API could be factored out into a separate package, allowing the persisting of data and passing it between steps to work using a different task runner, such as invoke.
Factoring this out into a separate package could also parameterize this operation, one day allowing power users to address #11 by serializing to a different format assuming their data is easily serialized as JSON (or something else).
opened Nov 23, 2015 by ghing 1
Information
| Category: Python / Data Analysis |
| Watchers: 11 |
| Star: 225 |
| Fork: 20 |
| Last update: Jul 25, 2021 |
Contents
Share this repo
Most popular
Related Repos
A tiered chat app based on reddit account age for all wall street bets users.
4.6k
This Repository Consists of Free Resources needed for a person to learn Datascience from the beginning to end. This repository is divided into Four main Parts.
2.5k
A helpful 4-page data science cheatsheet to assist with exam reviews, interview prep, and anything in-between.
3.2k
This repositary is a combination of different resources lying scattered all over the internet. The reason for making such an repositary is to combine all the valuable resources in a sequential manner, so that it helps every beginners who are in a search of free and structured learning resource for Data Science. For Constant Updates Follow me in Twitter.
Use advanced feature engineering strategies and select the best features from your data set fast with a single line of code.
489
CORTX Community Object Storage is 100% open source object storage uniquely optimized for mass capacity storage devices.
🌀 What's really in your Discord Data package?
172
Graph-Based Data Science: an abstraction layer in Python for building knowledge graphs, integrated with popular graph libraries – atop Pandas, RDFlib, pySHACL, RAPIDS, NetworkX, iGraph, PyVis, pslpython, pyarrow, etc.