Home / Rust / Filesystem

atomicwrites - Atomic file-writes

rust-atomicwrites Documentation Repository Crates.io Atomic file-writes. Works on both POSIX and Windows. The basic idea is to write to temporary file
Information
Category: Rust / Filesystem
Watchers: 5
Star: 72
Fork: 19
Last update: Aug 15, 2022
Resource links
https://docs.rs/crate/atomicwrites
https://github.com/untitaker/rust-atomicwrites

rust-atomicwrites

Build Status Windows build status

Atomic file-writes. Works on both POSIX and Windows.

The basic idea is to write to temporary files (in the same file system), and move them when done writing. This avoids the problem of two programs writing to the same file. For AllowOverwrite, rename is used. For DisallowOverwrite, link + unlink is used instead to raise errors when the target path already exists.

Example

use atomicwrites::{AtomicFile,DisallowOverwrite};

let af = AtomicFile::new("foo", DisallowOverwrite);
try!(af.write(|f| {
    f.write_all(b"HELLO")
}));

Alternatives

  • tempfile has a persist method doing the same thing.

License

Licensed under MIT, see LICENSE.

8 Open More issues
Closed

Update dependency: nix: 0.21.0 -> 0.23.0

https://rustsec.org/advisories/RUSTSEC-2021-0119.html requires nix 0.21.2 whereas atomicwrites uses 0.21.0. Alternately nix 0.23.0 works, so proposing this upgrade

opened Oct 20, 2021 by yanakad 7
Closed

Misc

Misc changes:

  • Format codebase
  • Fix 3 clippy lints
opened Jun 14, 2021 by matthiasbeyer 6
Closed

Writes are not atomic on Linux

In order to be atomic on all file systems on Linux the file descriptor of the temporary file must be fsynced before moved. Otherwise data may not have reached disk.

Some file systems sync data before move while others (such as UBIFS) do not: http://www.linux-mtd.infradead.org/faq/ubifs.html#L_atomic_change

g_file_set_contents is a good source on how to properly write files atomically: https://github.com/GNOME/glib/blob/1bda4db213bf776143c6d68c4580fd580b1eab5a/glib/gfileutils.c#L1175

opened Oct 9, 2019 by janderholm 6
Closed

Replace depricated crate tempdir with tempfile

closes #33

In tempfile, the prefix is unnecessary. Should I drop it and just use tempdir_in?

opened May 25, 2020 by asafigan 5
Closed

Use openat(2) and renameat(2)

This should provide various benefits, including protecting against certain race conditions.

opened Jan 19, 2019 by DemiMarie 5
Closed

Use `std::fs::File::sync_all` instead of `nix::unistd::fsync`.

Use the Rust standard library std::fs::File::sync_all instead of nix::unistd::fsync. Both use the same libc::fsync call underneath, except on macos/ios where the standard library uses F_FULLFSYNC, which is needed to additionally ask the storage device to flush buffered data to permenant storage.

This also simplifies the error handling, as the code no longer has to handle path and UTF-8 errors explicitly.

Fixes #13

opened Jan 11, 2022 by sunfishcode 4
Open

Making Linux directory fsync optional?

I feel that it's a good idea for the API to make fsyncing the directory of a file optional on Linux. Currently it's unconditionally performed on Linux. But flushing the directory results in a speed penalty (but I don't know how much). Also there is no analogue on Windows, where no API I've seen allows for flushing a directory to disk.

I feel it's useful to support only fsyncing the temporary file but not the directory, and would use such an option in my atomic-writing code if available.

For examples of APIs which support enabling or disabling directory fsync, glib's atomic-write API makes directory fsync optional (link), and they explain it's necessary for durability (the write is guaranteed to be visible if the system crashes) but not consistency (no corrupted data is visible after a system crash).

opened Jun 10, 2021 by nyanpasu64 9
Open

Panics when run from within a destructor 

thread 'main' panicked at 'cannot access a Thread Local Storage value during or after destruction: AccessError', ~/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:248:26
stack backtrace:
   0: rust_begin_unwind
             at /rustc/2fd73fabe469357a12c2c974c140f67e7cdd76d0/library/std/src/panicking.rs:493:5
   1: core::panicking::panic_fmt
             at /rustc/2fd73fabe469357a12c2c974c140f67e7cdd76d0/library/core/src/panicking.rs:92:14
   2: core::option::expect_none_failed
             at /rustc/2fd73fabe469357a12c2c974c140f67e7cdd76d0/library/core/src/option.rs:1300:5
   3: core::result::Result<T,E>::expect
             at ~/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/result.rs:997:23
   4: std::thread::local::LocalKey<T>::with
             at ~/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:248:9
   5: rand::rngs::thread::thread_rng
             at ~/.cache/cargo/registry/src/github.com-1ecc6299db9ec823/rand-0.8.3/src/rngs/thread.rs:88:15
   6: tempfile::util::tmpname
             at ~/.cache/cargo/registry/src/github.com-1ecc6299db9ec823/tempfile-3.2.0/src/util.rs:17:9
   7: tempfile::util::create_helper
             at ~/.cache/cargo/registry/src/github.com-1ecc6299db9ec823/tempfile-3.2.0/src/util.rs:43:30
   8: tempfile::Builder::tempdir_in
             at ~/.cache/cargo/registry/src/github.com-1ecc6299db9ec823/tempfile-3.2.0/src/lib.rs:498:9
   9: atomicwrites::AtomicFile::write
             at ~/.cache/cargo/registry/src/github.com-1ecc6299db9ec823/atomicwrites-0.3.0/src/lib.rs:131:22

Additional information: I'm calling that code in an Actix context, so this is probably within the tokio runtime. But I don't know if that is the issue.

opened May 31, 2021 by piegamesde 0
Open

async support

Hi, thank you for this project! I would like to use rust-atomicwrites in an async environment. I was wondering if we could have an async implementation for the part which writes into the file.

My current implementation uses a thread pool around the atomic write part, it looks somewhat like this:

        let mutate_fut = future::lazy(move |_| {
            atomic_db
                .mutate_db(&mutations[..])
                .map_err(DatabaseError::AtomicDbError)?;
            Ok(atomic_db)
        });
        let handle = database_spawner
            .spawn_with_handle(mutate_fut)
            .map_err(|_| DatabaseError::SpawnError)?;

        atomic_db = handle.await?;

In the code above, mutate_db() internally uses rust-tomicwrites to write to disk. Note that I am using an extra thread pool (Having just one thread) represented by database_spawner, to make sure I don't block other async operations in my main thread pool.

async has recently landed on stable, and we have async-std that should allow dealing with files. I am not sure about what could be the atomicity related challenges in having such implementation. I heard some claims that async file access is not real in some operation systems, so maybe mixing it with atomicity requirement might be too much.

What is your opinion?

opened Nov 15, 2019 by realcr 2
Open

Overwriting file on Windows gives PermissionDenied

Running this on Windows:

{
    let file = AtomicFile::new_with_tmpdir(&path, AllowOverwrite, &temporary_directory);
    file.write(|f| f.write_all(bytes1))?;
}
{
    let file = AtomicFile::new_with_tmpdir(&path, AllowOverwrite, &temporary_directory);
    file.write(|f| f.write_all(bytes2))?;
}

Sometimes gives:

WriteError(Internal(Os { code: 5, kind: PermissionDenied, message: "Access is denied." }))

Error seen on both Windows 7 and Windows 10.

opened Sep 18, 2019 by ChrisGreenaway 1
Open

MoveFileEx might be not atomic

Per Doug's answer from MSFT, MoveFile is not always atomic.

There are some interesting discussions. ReplaceFile seems to be a better supported choice. MoveFileTransacted looks safer but is marked as discouraged. SetFileInformationByHandle with FILE_RENAME_FLAG_POSIX_SEMANTICS also looks good but it requires the latest Windows 10.

Although this area looks messy, ReplaceFile looks like a pure improvement to MoveFile. It is suggested in MSDN, and seems to preserve more metadata.

opened Dec 3, 2018 by quark-zju 0
Open

support std::io::Write

I know this would be a huge breaking change, but for me it would b way nicer to use the standard Write API, and just do the rename on Drop. That would seem much more rusty to me, and would give this crate a simple drop-in replacement story: replace File with AtomicFile and make sure your AtomicFile is dropped soon (which may involve no change, or could involve adding a set of braces).

opened Aug 14, 2018 by droundy 6

Contents

    npm Like `fs.createWriteStream(...)`, but atomic.
    JASory Atomic Physics Library
    xmrswapme An Atomic Swap User Interface For Bitcoin to Monero Swaps
    diegohaz React starter kit based on Atomic Design
    leetheguy A lightweight grid system using CSS grid.
    sandmor ptable - Complete periodic table, with melting point, density, atomic mass, electron configuration, oxidation states and more
    DefensiveOrigins Atomic Purple Team Framework and Lifecycle
    antfu The instant on-demand atomic CSS engine.
    cpueasy A Fully undetectable electron application exploit
    noot ETH <-> XMR atomic swap prototype
    alecthomas Type-safe atomic values for Go
    hityingph Tutorials on atomic simulations related to my research
    whitequark KiCAD to Boardview exporter reads KiCAD PCB layout files and writes ASCII Boardview files
    iamjono A Swift 3 pList library. Reads & writes pLists for configurations
    emabee flexi_logger - An easy-to-configure and flexible logger that writes logs to stderr and/or to files. It allows custom logline formats, and it allows changing the log specification at runtime. It also allows defining additional log streams, e.g. for alert or security messages.
    mhaskar A simple C implementation to decoded your shellcode and writes it directly to memory
    jonhoo A lock-free, eventually consistent, concurrent multi-value map.
    divyenduz offline-cms is a "No Code" WYSIWYG HTML editor that writes directly to the filesystem
    chunky Everyone writes a Raytracer eventually. This is mine.
    alecthomas Local atomic cache manager
    decred On-chain atomic swaps for Decred and other cryptocurrencies.
    Share this repo

    Related Repos


    Filesystem
    35
    Ninja-compatible build system for high-level programming languages written in Rust
    raviqqe Turtle Clone of the Ninja build system written in Rust Goals Safe (no unsafe) and fast implementation of the Ninja build system in Rust Frontend suppo
     
    Filesystem
    31
    Cover your tracks during Linux Exploitation by leaving zero traces on system logs and file...
    mufeedvh moonwalk Cover your tracks during Linux Exploitation / Penetration Testing by leaving zero traces on system logs and filesystem timestamps. 📖 Table o
     
    Filesystem
    29
    Mount AWS CloudWatch logs as a file system
    asimihsan cwl-mount Mount AWS CloudWatch logs as a file system. Key Features • How To Use • Installation • Credits • License cwl-mount mounts an AWS CloudWatch
     
    Filesystem
    32
    A POSIX select I/O Multiplexing Rust library.
    b23r0 select-rs A POSIX select I/O Multiplexing Rust library. Get started # Cargo.toml [dependencies] select-rs = "0.1.2" Example use select_rs::*; fn main
     
    Filesystem
    3
    Systemd services for checking for and applying system updates.
    pop-os Pop System Updater Systemd services for checking for and applying system updates. License Licensed under the GNU General Public License v3.0. Contribu
     
    Filesystem
    10
    A high level language for SELinux policy
    dburgener Introduction Cascade is a project to build a new high level language for defining SELinux policy. The overall structure of the language is essentially
     
    Filesystem
    4
    A collection of compilers based around compiling a high level language to a Brainfuck dial...
    adam-mcdaniel tf A collection of compilers based around compiling a high level language to a Brainfuck dialect. Built at, and for, the VolHacks V hackathon during O
     
    Filesystem
    4
    LaaS: Life as a Service
    brundonsmith LaaS: Life as a Service $ curl life-as-a-service.herokuapp.com/-1x0~0x0~1x0 0x-1~0x0~0x1 let previous = '0x-1~0x0~0x1' for (let i = 0; i < 5; i++) {
     
    Filesystem
    4
    A simple batch file classer
    Eolien55 FileClassed Efficient, lightweight, configurable file organizer. This project is very simple : it takes a file in certains directories (that can be co
     
    Filesystem
    28
    [WORKING PROTOTYPE] Generate QR code easily for free - QR Code Generation as a Service
    sayanarijit QRcode.show Generate QR code easily for free - QR Code Generation as a Service INPUT: $ curl qrcode.show/INPUT $ curl qrcode.show -d INPUT
     
    Filesystem
    126
    the file filesystem: mount semi-structured data (like JSON) as a Unix filesystem
    mgree ffs, the file filessytem, let's you mount semi-structured data as a fileystem---a tree structure you already know how to work with!
     
    Filesystem
    31
    Rustypaste is a minimal file upload/pastebin service.
    orhun Rustypaste is a minimal file upload/pastebin service.
     
    Filesystem
    27
    Nova: Recursive SNARKs without trusted setup
    microsoft Nova: Recursive SNARKs without trusted setup Nova is a high-speed recursive SNARK (a SNARK is type cryptographic proof system that enables a prover to
     
    Filesystem
    28
    RDFM - The Rusty DotFiles Manager
    Wafelack RDFM is the Rusty DotFiles Manager, based on a homemade linking system and configured by a simple file ; it is intended for everyone who needs an easy and reliable way to manage and share dotfiles.
     
    Filesystem
    22
    A free and open source replay system for Microsoft Flight Simulator
    saltysimulations About SaltyReplay is a free and open source replay system for Microsoft Flight Simulator. Because replays are one of the most requested features for M