An implementation of chunked, compressed, N-dimensional arrays for Python.

Zarr Latest Release Package Status License Build Status Coverage Downloads Gitter Citation What is it? Zarr is a Python package providing an implement

Information

Category: Python / Deep Learning
Watchers: 42
Star: 837
Fork: 142
Last update: Mar 5, 2022

Resource links

http://zarr.readthedocs.io/

https://github.com/zarr-developers/zarr-python

Zarr

Latest Release

Package Status
License
Build Status
Coverage
Downloads
Gitter
Citation

What is it?

Zarr is a Python package providing an implementation of compressed, chunked, N-dimensional arrays, designed for use in parallel computing. See the documentation for more information.

Main Features

Create N-dimensional arrays with any NumPy dtype.
Chunk arrays along any dimension.
Compress and/or filter chunks using any NumCodecs codec.
Store arrays in memory, on disk, inside a zip file, on S3, etc...
Read an array concurrently from multiple threads or processes.
Write to an array concurrently from multiple threads or processes.
Organize arrays into hierarchies via groups.

Where to get it

Zarr can be installed from PyPI using pip:

pip install zarr

or via conda:

conda install -c conda-forge zarr

For more details, including how to install from source, see the installation documentation.

212 Open More issues

Closed

Refactoring WIP

This PR contains work on refactoring to achieve separation of concerns between compression/decompression, storage, synchronization and array/chunk data management. Primarily motivated by #21.

Still TODO:

[x] Improve test coverage
[x] Rework persistance/storage documentation to generalise beyond directory store
[x] Improve docstrings, add examples where appropriate
[x] Do some benchmarking/profiling
[x] Fall back to pure python install

opened Apr 19, 2016 by alimanfoo 103

Closed

async in zarr

I think there are some places where zarr would benefit immensely from some async capabilities when reading and writing data. I will try to illustrate this with the simplest example I can.

Let's consider a zarr array stored in a public S3 bucket, which we can read with fsspec's HTTPFileSystem interface (no special S3 API needed, just regular http calls).

import fsspec
url_base = 'https://mur-sst.s3.us-west-2.amazonaws.com/zarr/time'
mapper = fsspec.get_mapper(url_base)
za = zarr.open(mapper)
za.info

Note that this is a highly sub-optimal choice of chunks. The 1D array of shape (6443,) is stored in chunks of only (5,) items, resulting in over 1000 tiny chunks. Reading this data takes forever, over 5 minutes

%prun tdata = za[:]

         20312192 function calls (20310903 primitive calls) in 342.624 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1289  139.941    0.109  140.077    0.109 {built-in method _openssl.SSL_do_handshake}
     2578   99.914    0.039   99.914    0.039 {built-in method _openssl.SSL_read}
     1289   68.375    0.053   68.375    0.053 {method 'connect' of '_socket.socket' objects}
     1289    9.252    0.007    9.252    0.007 {built-in method _openssl.SSL_CTX_load_verify_locations}
     1289    7.857    0.006    7.868    0.006 {built-in method _socket.getaddrinfo}
     1289    1.619    0.001    1.828    0.001 connectionpool.py:455(close)
   930658    0.980    0.000    2.103    0.000 os.py:674(__getitem__)
...

I believe fsspec is introducing some major overhead by not reusing a connectionpool. But regardless, zarr is iterating synchronously over each chunk to load the data:

https://github.com/zarr-developers/zarr-python/blob/994f2449b84be544c9dfac3e23a15be3f5478b71/zarr/core.py#L1023-L1028

As a lower bound on how fast this approach could be, we bypass zarr and fsspec and just fetch all the chunks with requests:

import requests
s = requests.Session()

def get_chunk_http(n):
    r = s.get(url_base + f'/{n}')
    r.raise_for_status()
    return r.content

%prun all_data = [get_chunk_http(n) for n in range(za.shape[0] // za.chunks[0])]

         12550435 function calls (12549147 primitive calls) in 98.508 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     2576   87.798    0.034   87.798    0.034 {built-in method _openssl.SSL_read}
       13    1.436    0.110    1.437    0.111 {built-in method _openssl.SSL_do_handshake}
   929936    1.042    0.000    2.224    0.000 os.py:674(__getitem__)

As expected, reusing a connection pool sped things up, but it still takes 100 s to read the array.

Finally, we can try the same thing with asyncio

import asyncio
import aiohttp
import time

async def get_chunk_http_async(n, session):
    url = url_base + f'/{n}'
    async with session.get(url) as r:
        r.raise_for_status()
        data = await r.read()
    return data

async with aiohttp.ClientSession() as session:
    tic = time.time()
    all_data = await asyncio.gather(*[get_chunk_http_async(n, session)
                                    for n in range(za.shape[0] // za.chunks[0])])
    print(f"{time.time() - tic} seconds")

# > 1.7969944477081299 seconds

This is a MAJOR speedup!

I am aware that using dask could possibly help me here. But I don't have big data here, and I don't want to use dask. I want zarr to support asyncio natively.

I am quite new to async programming and have no idea how hard / complicated it would be to do this. But based on this experiment, I am quite sure there are major performance benefits to be had, particularly when using zarr with remote storage protocols.

Thoughts?

cc @cgentemann

opened Feb 14, 2020 by rabernat 93

Closed

Sharding Prototype I: implementation as translating Store

This PR is for an early prototype of sharding support, as described in the corresponding issue #877. It serves mainly to discuss the overall implementation approach for sharding. This PR is not (yet) meant to be merged.

This prototype

allows to specify shards as the number of chunks that should be contained in a shard (e.g. using arr.zeros((20, 3), chunks=(3, 3), shards=(2, 2), …)). One shard corresponds to one storage key, but can contain multiple chunks:
ensures that this setting is persisted in the .zarray config and loaded when opening an array again, adding two entries:
- "shard_format": "indexed" specifies the binary format of the shards and allows to extend sharding with other formats later
- "shards": [2, 2] specifies how many chunks are contained in a shard,
adds a IndexedShardedStore class that is used to wrap the chunk-store when sharding is enabled. This store handles the grouping of multiple chunks to one shard and transparently reads and writes them via the inner store in a binary format which is specified below. The original store API does not need to be adapted, it just stores shards instead of chunks, which are translated back to chunks by the IndexedShardedStore.
adds a small script chunking_test.py for demonstration purposes, this is not meant to be merged but servers to illustrate the changes.

The currently implemented file format is still up for discussion. It implements "Format 2" @jbms describes in https://github.com/zarr-developers/zarr-python/pull/876#issuecomment-973462279.

Chunks are written successively in a shard (unused space between them is allowed), followed by an index referencing them. The index holding an offset, length pair of little-endian uint64 per chunk, the chunks-order in the index is row-major (C) order, e.g. for (2, 2) chunks per shard an index would look like:

| chunk (0, 0)    | chunk (0, 1)    | chunk (1, 0)    | chunk (1, 1)    |
| offset | length | offset | length | offset | length | offset | length |
| uint64 | uint64 | uint64 | uint64 | uint64 | uint64 | uint64 | uint64 |

Empty chunks are denoted by setting both offset and length to 2^64 - 1. All the index always has the full shape of all possible chunks per shard, even if they are outside of the array size.

For the default order of the actual chunk-content in a shard I'd propose to use Morton order, but this can easily be changed and customized, since any order can be read.

If the overall direction of this PR is pursued, the following steps (and possibly more) are missing:

Functionality
- [ ] Use a default write-order (Morton) of chunks in a shard and allow customization
- [ ] Support deletion in the ShardedStore
- [ ] Group chunk-wise operations in Array where possible (e.g. in digest & _resize_nosync)
- [ ] Consider locking mechanisms to guard against concurrency issues within a shard
- [ ] Allow partial reads and writes when the wrapped store supports them
- [ ] Add support for prefixes before the chunk-dimensions in the storage key, e.g. for arrays that are contained in a group
- [ ] Add warnings for inefficient reads/writes (might be configured)
- [ ] Maybe use the new partial read method on the Store also for the current PartialReadBuffer usage (to detect if this is possible and reading via it)
Tests
- [ ] Add unit tests and/or doctests in docstrings
- [ ] Test coverage is 100% (Codecov passes)
Documentation
- [ ] also document optional optimization possibilities on the Store or BaseStore class, such as getitems or partial reads
- [ ] Add docstrings and API docs for any new/modified user-facing classes and functions
- [ ] New/modified features documented in docs/tutorial.rst
- [ ] Changes documented in docs/release.rst

changed 2021-12-07: added file format description and updated TODOs

opened Nov 18, 2021 by jstriebel 69

Closed

Consolidate zarr metadata into single key

A simple possible way of scanning all the metadata keys ('.zgroup'...) in a dataset and copying them into a single key, so that on systems where there is a substantial overhead to reading small files, everything can be grabbed in a single read. This is important in the context of xarray, which traverses all groups during opening the dataset, to find the various sub-groups and arrays.

The test shows how you could use the generated key. We could contemplate automatically looking for the metadata key when opening.

REF: https://github.com/pangeo-data/pangeo/issues/309

TODO:

[x] Add unit tests and/or doctests in docstrings
[x] Unit tests and doctests pass locally under Python 3.6 (e.g., run tox -e py36 or pytest -v --doctest-modules zarr)
[x] Unit tests pass locally under Python 2.7 (e.g., run tox -e py27 or pytest -v zarr)
[x] PEP8 checks pass (e.g., run tox -e py36 or flake8 --max-line-length=100 zarr)
[x] Add docstrings and API docs for any new/modified user-facing classes and functions
[x] New/modified features documented in docs/tutorial.rst
[x] Doctests in tutorial pass (e.g., run tox -e py36 or python -m doctest -o NORMALIZE_WHITESPACE -o ELLIPSIS docs/tutorial.rst)
[x] Changes documented in docs/release.rst
[x] Docs build locally (e.g., run tox -e docs)
[x] AppVeyor and Travis CI passes
[ ] Test coverage to 100% (Coveralls passes)

opened Jun 26, 2018 by martindurant 66

Closed

WIP: google cloud storage class

First, apologies for submitting an un-solicited pull request. I know that is against the contributor guidelines. I thought this idea would be easier to discuss with a concrete implementation to look at.

In my highly opinionated view, the killer feature of zarr is its ability to efficiently store array data in cloud storage. Currently, the recommended way to do this is via outside packages (e.g. s3fs, gcsfs), which provide a MutableMapping that zarr can store things in.

In this PR, I have implemented an experimental google cloud storage class directly within zarr.

Why did I do this? Because, in the pangeo project, we are now making heavy use of the xarray -> zarr -> gcsfs -> cloud storage stack. I have come to the conclusion that a tighter coupling between zarr and gcs, via the google.cloud.storage API, may prove advantageous.

In addition to performance benefits and easier debugging, I think there are social advantages to having cloud storage as a first-class part of zarr. Lots of people want to store arrays in the cloud, and if zarr can provide this capability more natively, it could increase adoption.

Thoughts?

These tests require GCP credentials and the google cloud storage package. It is possible add encrypted credentials to travis, but I haven't done that yet. Tests are (mostly) working locally for me.

TODO:

[x] Add unit tests and/or doctests in docstrings
[ ] Unit tests and doctests pass locally under Python 3.6 (e.g., run tox -e py36 or pytest -v --doctest-modules zarr)
[ ] Unit tests pass locally under Python 2.7 (e.g., run tox -e py27 or pytest -v zarr)
[ ] PEP8 checks pass (e.g., run tox -e py36 or flake8 --max-line-length=100 zarr)
[ ] Add docstrings and API docs for any new/modified user-facing classes and functions
[ ] New/modified features documented in docs/tutorial.rst
[ ] Doctests in tutorial pass (e.g., run tox -e py36 or python -m doctest -o NORMALIZE_WHITESPACE -o ELLIPSIS docs/tutorial.rst)
[ ] Changes documented in docs/release.rst
[ ] Docs build locally (e.g., run tox -e docs)
[ ] AppVeyor and Travis CI passes
[ ] Test coverage to 100% (Coveralls passes)

opened Mar 29, 2018 by rabernat 52

Closed

Add N5 Support

This adds support to read and write from and to N5 containers. The N5Store handling the conversion between the zarr and N5 format will automatically be selected whenever the path for a container ends in .n5 (similar to how the ZipStore is used for files ending in .zip).

The conversion is done mostly transparently, with one exception being the N5ChunkWrapper: This is a Codec with id n5_wrapper that will automatically be wrapped around the requested compressor. For example, if you create an array with a zlib compressor, in fact the n5_wrapper codec will be used that delegates to the zlib codec internally. The additional codec was necessary to introduce N5's chunk headers and ensure big endian storage.

In a related note, gzip compressed N5 arrays can at the moment not be read, since numcodecs treats zlib and gzip as synonyms, which they aren't (their compression headers differ). PR https://github.com/zarr-developers/numcodecs/pull/87 solves this issue.

See also https://github.com/zarr-developers/zarr/issues/231

TODO:

[x] Add unit tests and/or doctests in docstrings
[x] Unit tests and doctests pass locally under Python 3.6 (e.g., run tox -e py36 or pytest -v --doctest-modules zarr)
[x] Unit tests pass locally under Python 2.7 (e.g., run tox -e py27 or pytest -v zarr)
[x] PEP8 checks pass (e.g., run tox -e py36 or flake8 --max-line-length=100 zarr)
[x] Add docstrings and API docs for any new/modified user-facing classes and functions
[x] New/modified features documented in docs/tutorial.rst
[x] Doctests in tutorial pass (e.g., run tox -e py36 or python -m doctest -o NORMALIZE_WHITESPACE -o ELLIPSIS docs/tutorial.rst)
[x] Changes documented in docs/release.rst
[x] Docs build locally (e.g., run tox -e docs)
[x] AppVeyor and Travis CI passes
[x] Test coverage to 100% (Coveralls passes)

opened Oct 18, 2018 by funkey 45

Open

Simple API for specifying each benchmark workload

In today's Benchmarking meeting, we all agreed that it'd be great to get a simple benchmarking solution implemented ASAP, so we can get on with the fun work of trying to speed things up 🙂. To start with, we just need something simple that'll allow us to run benchmark workloads on our machines, figure out how close the current code is to the theoretical IO performance of the hardware, and compare performance between git commits.

Here's an early attempt to specify a very simple framework to allow people to specify workloads. To implement a new workload, you'd implement a class which inherits from Workload, and you'd just override prepare_dataset and run_workload. Does this look OK? Is there anything missing?

The (very simple) framework would then take care of discovering & running the workloads, whilst also recording relevant metrics to disk as JSON, and printing a summary.

The framework isn't ready yet! But the idea is that the MVP will expose a very simple CLI that allows you to run all benchmarks, or a specific benchmark. It'll automatically record metadata about the system and the git commit. And it'll make it easy to compare performance between different git commits.

The idea is to make it super-easy for anyone to submit a new workload via a PR. And easy for us to share "human-readable" metrics ("My new PR speeds up Zarr-Python on workload x by 10x on my machine! Hurray!") and share machine-readable metrics (JSON).

I'd imagine moving this code to zarr-developers if folks agree that the approach is OK.

(See https://github.com/zarr-developers/zarr-python/issues/1527 for a discussion of why I'm currently leaning towards the idea of implementing our own benchmarking tool. But feel free to make the case for using an existing tool! I'm just not sure that any existing tool would allow us to measure IO performance.)

opened Sep 28, 2023 by JackKelly 0

Open

What to measure during benchmarking?

The plan is to implement a benchmarking tool which automatically runs a suite of "Zarr workloads" across a range of compute platforms, storage media, chunk sizes, and Zarr implementations.

What would we like to measure for each workload?

Existing benchmarking tools only measure the runtime of each workload. That doesn't feel sufficient for Zarr because one of our main questions during benchmarking is whether the Zarr implementation is able to saturate the IO subsystem, and how much CPU and RAM is required to saturate the IO.

I'd propose that it'd be great to measure these parameters each time each workload is run:

Total execution time of the workload
Total bytes read / written for disk / network
Total IO operations
Total bytes in final numpy array
Average CPU utilization (per CPU).
Max RAM usage during the execution of the workload
CPU cache hit ratio

(Each run would also capture a bunch of metadata about the environment such as the compute environment, storage media, chunk sizes, Zarr implementation name and version, etc.)

I had previously gotten over-excited and starting thinking about capturing a full "trace" during the execution of each workload, e.g. capturing a timeseries of the IO utilization every 100 milliseconds. This might be useful, but makes the benchmarking code rather more complex, and maybe doesn't tell us much more than the "totals per workload" tell us. And some benchmark workloads might run for less than 100 ms. And psutil's documentation states that some of its counters aren't reliable when polled more frequently than 10 times a second.

What do you folks think? Do we need to record a full "trace" during each workload? Or is it sufficient to just capture totals per workload? Are there any changes you'd make to the list of parameters I proposed above?

opened Sep 27, 2023 by JackKelly 1

Open

Bump azure-storage-blob from 12.16.0 to 12.18.2

Bumps azure-storage-blob from 12.16.0 to 12.18.2.

Release notes

Sourced from azure-storage-blob's releases.

azure-storage-blob_12.18.2

12.18.2 (2023-09-25)

Bugs Fixed

Fixed an issue where user_agent was being ignored on upload_blob or download_blob if client was configured for encryption.

azure-storage-blob_12.18.1

12.18.1 (2023-09-13)

Bugs Fixed

Fixed breaking KeyError: 'sdk_moniker' in create_configuration. NOTE: This is not an exported method and therefore should not be imported/called directly.

azure-storage-blob_12.18.0

12.18.0 (2023-09-12)

Features Added

Stable release of features from 12.18.0b1

azure-storage-blob_12.18.0b1

12.18.0b1 (2023-08-08)

Features Added

Added support for service versions 2023-05-03 and 2023-08-03.

Added version_id as a client constructor parameter to BlobClient. This change makes BlobClients version-aware, such that all APIs that accept version_id will operate on the version ID provided during client construction by default.

Added optional keyword version_id to get_blob_client APIs which, if provided, will result in a version-aware BlobClient in which all APIs that accept version_id will operate on the version ID provided to the get_blob_client API call by default.

Commits

9bb963a [Storage] Prepare for blob/queue patch release (#32194)
eeacc10 [Storage] Fix user_agent on method calls with encryption (#32183)
dda42ab [Storage] Make hack to run changefeed tests locally opt-in via comments (#32002)
57cdeaa [Storage] Improve query_blob docs (#32048)
b7e4b4a Bump (#32085)
80e2aa4 [Storage] [Hotfix] Fix breaking change in create_configuration and prepare ...
c4bec7a GA prep (#32038)
e8ee96f pin crypto to resolve regular storage-blob-changfeed envs (#31960)
668f410 Remove vcrpy completely (#32022)
f1f3836 Refactor Conda Pipeline (#31804)
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR
@dependabot recreate will recreate this PR, overwriting any edits that have been made to it
@dependabot merge will merge this PR after your CI passes on it
@dependabot squash and merge will squash and merge this PR after your CI passes on it
@dependabot cancel merge will cancel a previously requested merge and block automerging
@dependabot reopen will reopen this PR if it is closed
@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
@dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies python needs release notes

opened Sep 26, 2023 by dependabot[bot] 0

Open

Pick benchmarking tool

We'd like to benchmark the performance of existing Zarr implementations, starting with Zarr-Python.

We've identified 5 benchmarking frameworks:

airspeed velocity (also known as ASV. Hat tip to @MSanKeys963 for telling us about ASV :slightly_smiling_face:). ASV has been around since 2013. It's used by lots of python projects such as astropy, numpy, scipy, and pandas. But it's built for one very specific use-case: tracking performance of compute-bound code over multiple releases. Some of the lead developers for a range of highly influential Python packages have collated a long list of concerns with ASV. Their conclusion is that ASV is clunky or impossible to use for other use-cases (like checking if local code modifications affect performance), and the code is hard to modify.
pytest-benchmark (Hat tip to @ap-- for telling us about pytest-benchmark :slightly_smiling_face:). A plugin for pytest that provides a benchmark fixture. Lots of stats. JSON export.
pyperf. Used by the pyperformance project.
codespeed. "a web application to monitor and analyze the performance of your code. Known to be used by CPython, PyPy, Twisted and others."
conbench. "Language-independent Continuous Benchmarking (CB) Framework... The Apache Arrow project is using Conbench for Continuous Benchmarking.".

My current sense is that none of these packages are a perfect fit. The task is to decide if any of these packages fit well enough to be useful. I plan to more rigorously compare these 5 packages against our requirements.

The first 4 benchmarking frameworks are very focused on measuring the execution time of CPU-bound tasks, and detecting performance regressions. They answer questions like "does the latest release of the code reduce the runtime of the matrix multiplication function?". None of first 4 benchmarking frameworks are particularly interested in IO behaviour, or in comparing the behaviour of different projects.

I'm probably biased but I'd like our benchmarks to help answer questions like:

Does Zarr-Python currently sustain 100% IO utilization across a range of different storage mediums (local SSD, cloud bucket, etc.)? How does this behaviour change if we use Dask with Zarr to submit many IO operations in parallel? Does bandwidth drop away if we submit too many IO ops in parallel? And does that limit depend on the storage medium and/or the number of CPU cores?
How does bandwidth & latency vary as a function of the read pattern (e.g. reading the entire Zarr dataset versus reading crops on chunk boundaries versus reading random crops which cross chunk boundaries)? How does that vary with different storage mediums?
How does Zarr-Python's IO behaviour compare to TensorStore's IO behaviour and Zarrita's IO behaviour?
What's the peak memory utilisation whilst reading?
If IO isn't the bottleneck then what is? Are all CPU cores constantly saturated?
What's the maximum number of IO operations per second that Zarr-Python can request?

I'm almost certain that none of the 5 benchmarking frameworks listed above can help us answer these questions. So I'm wondering if we might be better rolling our own benchmarking framework. (Which shouldn't be too hard. psutil can measure utilisation of CPU(s), IO, etc. We could persist the benchmarks as JSON. And use something like streamlit to build a web UI.)

Or maybe I'm getting over-excited and we should just use an existing benchmarking framework and be happy with just measuring execution time :slightly_smiling_face:. We almost certainly don't want to try to answer all my questions for every PR! Maybe the automated benchmark suite should just measure execution time. And then we can do one-off, detailed, manual analyses to answer the questions above. Although i do think it could be extremely powerful to be able to share detailed, interactive analysis of Zarr's performance across a wide range of compute platforms, storage media, and Zarr implementations :slightly_smiling_face:. And some of the questions above can only be answered after collecting a lot of performance data. So it might be nice to at least collect (but not analyse) lots of data on every benchmark run.

opened Sep 19, 2023 by JackKelly 1

Open

feat(zep-6): Implement ZOM dataclasses for V2 and V3 objects

This (experimental) PR adds a set of Zarr Object Models in the form of datalcasses. These models are intended to be independent representations of the core Zarr classes (e.g. Array), allowing for creation of metadata-only hierarchies (groups and arrays). These models are intended to be used internally to represent the metadata of groups and arrays.

See also: https://github.com/zarr-developers/zeps/pull/46

TODO:

[ ] Add unit tests and/or doctests in docstrings
[ ] Add docstrings and API docs for any new/modified user-facing classes and functions
[ ] New/modified features documented in docs/tutorial.rst
[ ] Changes documented in docs/release.rst
[ ] GitHub Actions have all passed
[ ] Test coverage is 100% (Codecov passes)

cc @d-v-b

needs release notes

opened Sep 18, 2023 by jhamman 3

Open

Bump numpy from 1.24.3 to 1.26.0

Bumps numpy from 1.24.3 to 1.26.0.

Release notes

Sourced from numpy's releases.

v1.26.0

NumPy 1.26.0 Release Notes

The NumPy 1.26.0 release is a continuation of the 1.25.x release cycle with the addition of Python 3.12.0 support. Python 3.12 dropped distutils, consequently supporting it required finding a replacement for the setup.py/distutils based build system NumPy was using. We have chosen to use the Meson build system instead, and this is the first NumPy release supporting it. This is also the first release that supports Cython 3.0 in addition to retaining 0.29.X compatibility. Supporting those two upgrades was a large project, over 100 files have been touched in this release. The changelog doesn't capture the full extent of the work, special thanks to Ralf Gommers, Sayed Adel, Stéfan van der Walt, and Matti Picus who did much of the work in the main development branch.

The highlights of this release are:

Python 3.12.0 support.

Cython 3.0.0 compatibility.

Use of the Meson build system

Updated SIMD support

f2py fixes, meson and bind(x) support

Support for the updated Accelerate BLAS/LAPACK library

The Python versions supported in this release are 3.9-3.12.

New Features

Array API v2022.12 support in numpy.array_api

numpy.array_api now full supports the v2022.12 version of the array API standard. Note that this does not yet include the optional fft extension in the standard.

(gh-23789)

Support for the updated Accelerate BLAS/LAPACK library

Support for the updated Accelerate BLAS/LAPACK library, including ILP64 (64-bit integer) support, in macOS 13.3 has been added. This brings arm64 support, and significant performance improvements of up to 10x for commonly used linear algebra operations. When Accelerate is selected at build time, the 13.3+ version will automatically be used if available.

(gh-24053)

meson backend for f2py

f2py in compile mode (i.e. f2py -c) now accepts the

... (truncated)

Commits

d35cd07 Merge pull request #24722 from charris/prepare-1.26.0-release
2f7c195 REL: Prepare for the NumPy 1.26.0 release
c550abf Merge pull request #24718 from rgommers/doc-update
4893b24 DEV: improve spin docs command
44fe37c DOC: fix two cross-references to SciPy docs in "building from source"
f2121dc DOC: Fix doc build warning for random.
ca97802 Merge pull request #24707 from charris/backport-24705
92aab8c TYP: Use collections.abc.Buffer in the npt.ArrayLike definition
56e580d TYP: Use collections.abc.Buffer in the Array API
5ef53c6 TYP: Add annotations for the py3.12 buffer protocol
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR
@dependabot recreate will recreate this PR, overwriting any edits that have been made to it
@dependabot merge will merge this PR after your CI passes on it
@dependabot squash and merge will squash and merge this PR after your CI passes on it
@dependabot cancel merge will cancel a previously requested merge and block automerging
@dependabot reopen will reopen this PR if it is closed
@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
@dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies python needs release notes

opened Sep 18, 2023 by dependabot[bot] 0

v2.16.1(Aug 18, 2023)

See release notes: https://zarr.readthedocs.io/en/stable/release.html#release-2-16-1

What's Changed

Add "Generate release notes" to release process by @joshmoore in https://github.com/zarr-developers/zarr-python/pull/1476
Require setuptools_scm version 1.5.4+ by @jakirkham in https://github.com/zarr-developers/zarr-python/pull/1477
Delete trailing spaces by @DimitriPapadopoulos in https://github.com/zarr-developers/zarr-python/pull/1424
Ensure that chunks is tuple of ints upon array creation by @hanslovsky in https://github.com/zarr-developers/zarr-python/pull/1470
Add docs requirements to pyproject.toml by @jakirkham in https://github.com/zarr-developers/zarr-python/pull/1494
zarr.group now accept the meta_array argument by @madsbk in https://github.com/zarr-developers/zarr-python/pull/1489
Bump pypa/gh-action-pypi-publish from 1.8.7 to 1.8.10 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1498
LRUStoreCache: cache "contains" by contains checks by @madsbk in https://github.com/zarr-developers/zarr-python/pull/1499
Prepare for 2.16.1 release by @rabernat in https://github.com/zarr-developers/zarr-python/pull/1503

New Contributors

@hanslovsky made their first contribution in https://github.com/zarr-developers/zarr-python/pull/1470

Full Changelog: https://github.com/zarr-developers/zarr-python/compare/v2.16.0...v2.16.1

Source code(tar.gz)
Source code(zip)

v2.16.0(Jul 20, 2023)

See release notes https://zarr.readthedocs.io/en/stable/release.html#release-2-16-0

What's Changed

Bump pytest-doctestplus from 0.12.1 to 0.13.0 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1429
Bump pytest from 7.3.1 to 7.3.2 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1432
Bump fsspec from 2023.5.0 to 2023.6.0 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1433
Bump h5py from 3.8.0 to 3.9.0 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1440
chore: update pre-commit hooks by @pre-commit-ci in https://github.com/zarr-developers/zarr-python/pull/1438
Bump pymongo from 4.3.3 to 4.4.0 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1441
Bump pytest-cov from 4.0.0 to 4.1.0 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1418
test: replace pkg_resources with packaging.version for version parsing/comparison by @jhamman in https://github.com/zarr-developers/zarr-python/pull/1450
Bump pypa/gh-action-pypi-publish from 1.8.6 to 1.8.7 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1451
Bump ipywidgets from 8.0.6 to 8.0.7 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1452
Add contains method to KVStore by @cgohlke in https://github.com/zarr-developers/zarr-python/pull/1454
Support Block (Chunk) Indexing by @tasansal in https://github.com/zarr-developers/zarr-python/pull/1428
[V3] Allow for incomplete codec metadata using numcodecs.get_codec by @jhamman in https://github.com/zarr-developers/zarr-python/pull/1447
Refactor core tests, round 2 by @d-v-b in https://github.com/zarr-developers/zarr-python/pull/1462
Small typo in v2 spec by @ways in https://github.com/zarr-developers/zarr-python/pull/1468
expose codecov token in ci by @d-v-b in https://github.com/zarr-developers/zarr-python/pull/1473
Updated release notes for 2.16.0 by @rabernat in https://github.com/zarr-developers/zarr-python/pull/1471

New Contributors

@cgohlke made their first contribution in https://github.com/zarr-developers/zarr-python/pull/1454
@ways made their first contribution in https://github.com/zarr-developers/zarr-python/pull/1468

Full Changelog: https://github.com/zarr-developers/zarr-python/compare/v2.15.0...v2.16.0

Source code(tar.gz)
Source code(zip)

v2.15.0(Jun 14, 2023)

See release notes https://zarr.readthedocs.io/en/latest/release.html#release-2-15-0

What's Changed

Bump pytest from 7.2.1 to 7.2.2 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1361
Bump fsspec from 2023.1.0 to 2023.3.0 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1360
chore: update pre-commit hooks by @pre-commit-ci in https://github.com/zarr-developers/zarr-python/pull/1351
Bump azure-storage-blob from 12.14.1 to 12.15.0 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1357
Bump actions/setup-python from 4.3.0 to 4.5.0 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1318
More extensive orthogonal indexing in get/setitem by @AndreasAlbertQC in https://github.com/zarr-developers/zarr-python/pull/1333
Don't use relative fixture path by @joshmoore in https://github.com/zarr-developers/zarr-python/pull/1364
Bump pypa/gh-action-pypi-publish from 1.6.4 to 1.7.1 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1365
One more fix for missing directories by @joshmoore in https://github.com/zarr-developers/zarr-python/pull/1367
Add API reference for V3 Implementation in the docs by @MSanKeys963 in https://github.com/zarr-developers/zarr-python/pull/1345
Bump pypa/gh-action-pypi-publish from 1.7.1 to 1.8.1 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1369
Bump redis from 4.5.1 to 4.5.3 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1373
Bump ipywidgets from 8.0.4 to 8.0.6 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1379
Bump pypa/gh-action-pypi-publish from 1.8.1 to 1.8.3 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1376
Getitems: support meta_array by @madsbk in https://github.com/zarr-developers/zarr-python/pull/1131
Update release.rst for the upcoming release - 2.15.0 by @MSanKeys963 in https://github.com/zarr-developers/zarr-python/pull/1378
Remove codecov by @jakirkham in https://github.com/zarr-developers/zarr-python/pull/1391
fix for readonly error in _normalize_store_arg_v3 by @jhamman in https://github.com/zarr-developers/zarr-python/pull/1383
Update release.rst for 2.15.0 by @MSanKeys963 in https://github.com/zarr-developers/zarr-python/pull/1392
Bump actions/setup-python from 4.5.0 to 4.6.0 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1399
Bump pytest from 7.2.2 to 7.3.1 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1393
Bump azure-storage-blob from 12.15.0 to 12.16.0 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1403
Bump numpy from 1.24.2 to 1.24.3 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1402
Mark for 2.15.0a1 pre-release by @joshmoore in https://github.com/zarr-developers/zarr-python/pull/1404
Bump fsspec from 2023.3.0 to 2023.4.0 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1387
Avoid deprecated product in numpy=1.25 by @jrbourbeau in https://github.com/zarr-developers/zarr-python/pull/1405
Fix normalize_fill_value for structured arrays by @alanhdu in https://github.com/zarr-developers/zarr-python/pull/1397
Merge isinstance calls by @DimitriPapadopoulos in https://github.com/zarr-developers/zarr-python/pull/1409
Bump pypa/gh-action-pypi-publish from 1.8.3 to 1.8.6 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1412
Bump lmdb from 1.4.0 to 1.4.1 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1384
chore: update pre-commit hooks by @pre-commit-ci in https://github.com/zarr-developers/zarr-python/pull/1366
Bump redis from 4.5.3 to 4.5.4 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1380
Bump fsspec from 2023.4.0 to 2023.5.0 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1411
test(ci): Run tests against python 3.11 by @jhamman in https://github.com/zarr-developers/zarr-python/pull/1415
Bump redis from 4.5.4 to 4.5.5 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1413
chore: update pre-commit hooks by @pre-commit-ci in https://github.com/zarr-developers/zarr-python/pull/1416
open_array(): adding the meta_array argument by @madsbk in https://github.com/zarr-developers/zarr-python/pull/1396
Fix typo introduced by dcce26e by @DimitriPapadopoulos in https://github.com/zarr-developers/zarr-python/pull/1420
2.15.0: Remove pre-release warning by @joshmoore in https://github.com/zarr-developers/zarr-python/pull/1419

New Contributors

@AndreasAlbertQC made their first contribution in https://github.com/zarr-developers/zarr-python/pull/1333
@MSanKeys963 made their first contribution in https://github.com/zarr-developers/zarr-python/pull/1345
@alanhdu made their first contribution in https://github.com/zarr-developers/zarr-python/pull/1397

Full Changelog: https://github.com/zarr-developers/zarr-python/compare/v2.14.2...v2.15.0

Source code(tar.gz)
Source code(zip)

v2.15.0a2(May 25, 2023)

See release notes https://zarr.readthedocs.io/en/latest/release.html#release-2-15-0

Source code(tar.gz)
Source code(zip)

v2.15.0a1(May 2, 2023)

See release notes https://zarr.readthedocs.io/en/latest/release.html#release-2-15-0

What's Changed

Bump pytest from 7.2.1 to 7.2.2 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1361
Bump fsspec from 2023.1.0 to 2023.3.0 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1360
chore: update pre-commit hooks by @pre-commit-ci in https://github.com/zarr-developers/zarr-python/pull/1351
Bump azure-storage-blob from 12.14.1 to 12.15.0 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1357
Bump actions/setup-python from 4.3.0 to 4.5.0 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1318
More extensive orthogonal indexing in get/setitem by @AndreasAlbertQC in https://github.com/zarr-developers/zarr-python/pull/1333
Don't use relative fixture path by @joshmoore in https://github.com/zarr-developers/zarr-python/pull/1364
Bump pypa/gh-action-pypi-publish from 1.6.4 to 1.7.1 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1365
One more fix for missing directories by @joshmoore in https://github.com/zarr-developers/zarr-python/pull/1367
Add API reference for V3 Implementation in the docs by @MSanKeys963 in https://github.com/zarr-developers/zarr-python/pull/1345
Bump pypa/gh-action-pypi-publish from 1.7.1 to 1.8.1 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1369
Bump redis from 4.5.1 to 4.5.3 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1373
Bump ipywidgets from 8.0.4 to 8.0.6 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1379
Bump pypa/gh-action-pypi-publish from 1.8.1 to 1.8.3 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1376
Getitems: support meta_array by @madsbk in https://github.com/zarr-developers/zarr-python/pull/1131
Update release.rst for the upcoming release - 2.15.0 by @MSanKeys963 in https://github.com/zarr-developers/zarr-python/pull/1378
Remove codecov by @jakirkham in https://github.com/zarr-developers/zarr-python/pull/1391
fix for readonly error in _normalize_store_arg_v3 by @jhamman in https://github.com/zarr-developers/zarr-python/pull/1383
Update release.rst for 2.15.0 by @MSanKeys963 in https://github.com/zarr-developers/zarr-python/pull/1392
Bump actions/setup-python from 4.5.0 to 4.6.0 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1399
Bump pytest from 7.2.2 to 7.3.1 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1393
Bump azure-storage-blob from 12.15.0 to 12.16.0 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1403
Bump numpy from 1.24.2 to 1.24.3 by @dependabot in https://github.com/zarr-developers/zarr-python/pull/1402
Mark for 2.15.0a1 pre-release by @joshmoore in https://github.com/zarr-developers/zarr-python/pull/1404

New Contributors

@AndreasAlbertQC made their first contribution in https://github.com/zarr-developers/zarr-python/pull/1333
@MSanKeys963 made their first contribution in https://github.com/zarr-developers/zarr-python/pull/1345

Full Changelog: https://github.com/zarr-developers/zarr-python/compare/v2.14.2...v2.15.0a1

Source code(tar.gz)
Source code(zip)

v2.14.2(Feb 24, 2023)

See release notes https://zarr.readthedocs.io/en/stable/release.html#release-2-14-2

Source code(tar.gz)
Source code(zip)

A columnar data container that can be compressed.

N-D labeled arrays and datasets in Python

Theano was a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently

A Python application to view n-dimensional hypercubes and rotate them in n-dimensional space.

arr_macro - Initialize arrays with ease!

A flexible view for providing a limited rect window into a large data set,just like a two-dimensional RecyclerView. It different from RecyclerView is that it's two-dimensional(just like a Panel) and it pin the itemView of first row and first column in their original location.

This is a simple projection of four-dimensional objects into three-dimensional space, and then the projection objects are presented on the 2D screen.

Encoder and decoder for HTTP chunked transfer coding (RFC 7230 § 4.1)

Build configuration arrays from config files and env vars.

Rendering algorithms implemented in raw WebGL 2.

Use a Structure of Arrays like an Array of Structures

Some useful methods for JavaScript Arrays.

HiPlot makes understanding high dimensional data easy

Compressed Volumetric Heatmaps for Multi-Person 3D Pose Estimation

Pure golang package for reading and writing xz-compressed files

Succinct, compact, and compressed data structures for data-intensive applications

🔍NEW ultra fast grep with interactive query UI: search file systems, source code, text, binary files, archives (cpio/tar/pax/zip), compressed files (gz/Z/bz2/lzma/xz/lz4), documents, fuzzy search, and more. A faster, user-friendly replacement for GNU/BSD grep.

Ship React Native .jsbundles compressed by Brotli algorithm.

woftool is a proof-of-concept utility for creating WOF-compressed files

ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training

A mini (1.6kb compressed and 1.1kb gziped!) high-performance animation library. Provide promise-based API.

Share this repo

Related Repos

Deep Learning

837

An implementation of chunked, compressed, N-dimensional arrays for Python.

Zarr Latest Release Package Status License Build Status Coverage Downloads Gitter Citation What is it? Zarr is a Python package providing an implement

Deep Learning

317

Serverless proxy for Spark cluster

Hydrosphere Mist Hydrosphere Mist is a serverless proxy for Spark cluster. Mist provides a new functional programming framework and deployment model f

Deep Learning

2.5k

Distributed Tensorflow, Keras and PyTorch on Apache Spark/Flink & Ray

Distributed TensorFlow, PyTorch, Keras and BigDL on Apache Spark & Ray Analytics Zoo is an open source Big Data AI platform, and includes the followin

Deep Learning

Black-Scholes StarkNet Library

Cairo Black-Scholes Library Black-Scholes library implemented as a Cairo smart contract. All inputs, outputs, and internal calculations use 27-digit f

Deep Learning

Motoko library to help create an append-only logger actor

IC Logger This motoko library provides a module to help create an append-only logger actor. Usage You can use this library with the vessel package man

Deep Learning

LeetCode_Solution

LeetCode_Solution LeetCode: https://leetcode.com/problemset/all/ # Title Difficulty Colab Code 1 Two Sum Easy 2 Add Two Numbers Medium 3 Longest Subst

Deep Learning

OCR-D wrapper for detectron2 based segmentation models

ocrd_detectron2 OCR-D wrapper for detectron2 based segmentation models Introduction Installation Usage OCR-D processor interface ocrd-detectron2-segm

Deep Learning

134

Check for LDAP protections regarding the relay of NTLM authentication

LDAP Relay Scan A tool to check Domain Controllers for LDAP server protections regarding the relay of NTLM authentication. If you're interested in the

Deep Learning

Fuzzware's main repository. Start here to install.

Fuzzware Fuzzware is a project for automated, self-configuring fuzzing of firmware images. The idea of this project is to configure the memory ranges

Deep Learning

Time Weighted Asset Mints

TWAM • Time Weighted Asset Mints A minting harness enabling time-weighted assets to determine minting prices. How it works Requirements: maxMintingAmo

Deep Learning

A complete, self-contained example for training ImageNet at state-of-the-art speed with FF...

ffcv ImageNet Training A minimal, single-file PyTorch ImageNet training script designed for hackability. Run train_imagenet.py to get... ...high accur

Deep Learning

เอกสารประกอบการสอนทุกเนื้อหาในช่องยูทูป KongRuksiam Official

📖 เอกสารประกอบการสอน (PDF) 🔏 Blockchain & Smart Contract Blockchain เบื้องต้น Smart Contract & Solidity เบื้องต้น 💻 Programming เขียนโปรแกรมภาษา Ja

Deep Learning

Omnivore A Single Model for Many Visual Modalities

Omnivore: A Single Model for Many Visual Modalities [paper][website] OMNIVORE is a single vision model for many different visual modalities. It learns

Deep Learning

Python implementation of the Lox language from Robert Nystrom's Crafting Interpreters

pylox Python implementation of the Lox language from Robert Nystrom's Crafting Interpreters. https://craftinginterpreters.com. This only implements th

Deep Learning

ParaGen is a PyTorch deep learning framework for parallel sequence generation.

ParaGen ParaGen is a PyTorch deep learning framework for parallel sequence generation. Apart from sequence generation, ParaGen also enhances various N

An implementation of chunked, compressed, N-dimensional arrays for Python.

Zarr

What is it?

Main Features

Where to get it

azure-storage-blob_12.18.2

12.18.2 (2023-09-25)

Bugs Fixed

azure-storage-blob_12.18.1

12.18.1 (2023-09-13)

Bugs Fixed

azure-storage-blob_12.18.0

12.18.0 (2023-09-12)

Features Added

azure-storage-blob_12.18.0b1

12.18.0b1 (2023-08-08)

Features Added

v1.26.0

NumPy 1.26.0 Release Notes

New Features

Array API v2022.12 support in numpy.array_api

Support for the updated Accelerate BLAS/LAPACK library

meson backend for f2py

v2.16.1(Aug 18, 2023)

What's Changed

New Contributors

v2.16.0(Jul 20, 2023)

What's Changed

New Contributors

v2.15.0(Jun 14, 2023)

What's Changed

New Contributors

v2.15.0a2(May 25, 2023)

v2.15.0a1(May 2, 2023)

What's Changed

New Contributors

v2.14.2(Feb 24, 2023)

Related Repos

Array API v2022.12 support in `numpy.array_api`

`meson` backend for `f2py`