Home / Rust / Encoding Binary

cross-platform, zero-copy, and endian-aware binary parsing

libgoblin Documentation https://docs.rs/goblin/ changelog Usage Goblin requires rustc 1.36.0. Add to your Cargo.toml [dependencies]
Information
Category: Rust / Encoding Binary
Watchers: 23
Star: 1k
Fork: 141
Last update: Sep 23, 2023
Resource links
https://github.com/m4b/goblin

libgoblin

Actions crates.io version

say the right words

Documentation

https://docs.rs/goblin/

changelog

Usage

Goblin requires rustc 1.36.0.

Add to your Cargo.toml

[dependencies]
goblin = "0.2"

Features

  • awesome crate name
  • zero-copy, cross-platform, endian-aware, ELF64/32 implementation - wow!
  • zero-copy, cross-platform, endian-aware, 32/64 bit Mach-o parser - zoiks!
  • PE 32/64-bit parser - bing!
  • a Unix and BSD style archive parser (latter courtesy of @willglynn) - huzzah!
  • many cfg options - it will make your head spin, and make you angry when reading the source!
  • fuzzed - "I am happy to report that goblin withstood 100 million fuzzing runs, 1 million runs each for seed 1~100." - @sanxiyn
  • tests

libgoblin aims to be your one-stop shop for binary parsing, loading, and analysis.

Use-cases

Goblin primarily supports the following important use cases:

  1. Core, std-free #[repr(C)] structs, tiny compile time, 32/64 (or both) at your leisure.

  2. Type punning. Define a function once on a type, but have it work on 32 or 64-bit variants - without really changing anything, and no macros! See examples/automagic.rs for a basic example.

  3. std mode. This throws in read and write impls via Pread and Pwrite, reading from file, convenience allocations, extra methods, etc. This is for clients who can allocate and want to read binaries off disk.

  4. Endian_fd. A truly terrible name 😆 this is for binary analysis like in panopticon or falcon which needs to read binaries of foreign endianness, or as a basis for constructing cross platform foreign architecture binutils, e.g. cargo-sym and bingrep are simple examples of this, but the sky is the limit.

Here are some things you could do with this crate (or help to implement so they could be done):

  1. Write a compiler and use it to generate binaries (all the raw C structs have Pwrite derived).
  2. Write a binary analysis tool which loads, parses, and analyzes various binary formats, e.g., panopticon or falcon.
  3. Write a semi-functioning dynamic linker.
  4. Write a kernel and load binaries using no_std cfg. I.e., it is essentially just struct and const defs (like a C header) - no fd, no output, no std.
  5. Write a bin2json tool, because why shouldn't binary formats be in JSON?

Cfgs

libgoblin is designed to be massively configurable. The current flags are:

  • elf64 - 64-bit elf binaries, repr(C) struct defs
  • elf32 - 32-bit elf binaries, repr(C) struct defs
  • mach64 - 64-bit mach-o repr(C) struct defs
  • mach32 - 32-bit mach-o repr(C) struct defs
  • pe32 - 32-bit PE repr(C) struct defs
  • pe64 - 64-bit PE repr(C) struct defs
  • archive - a Unix Archive parser
  • endian_fd - parses according to the endianness in the binary
  • std - to allow no_std environments

Contributors

Thank you all ❤️ !

In lexicographic order:

Contributing

  1. Please prefix commits with the affected binary component; the more specific the better, e.g., if you only modify relocations in the elf module, then do "elf.reloc: added new constants for Z80"
  2. Commit messages must explain their change, no generic "changed", or "fix"; if you push commits like this on a PR, be aware @m4b or someone will most likely squash them.
  3. If you are making a large change to a module, please raise an issue first and lets discuss; I don't want to waste your time if its not a good technical direction, or etc.
  4. If your PR is not getting attention, please respond to all relevant comments raised on the PR, and if still no response, ping @m4b, @philipc, or @willglynn in github and also feel free to email @m4b.
  5. Please add tests if you are adding a new feature. Feel free to add tests even if you are not, tests are awesome and easy in rust.
  6. Once cargo format is officially released, please format your patch using the default settings.
71 Open More issues
Closed

Add alloc feature

This allows building the endian_fd feature without std, although it does still require the alloc crate and a nightly rustc.

Fixes #75

opened Jan 31, 2018 by philipc 22
Closed

Replace unsafe bits using crate `plain`.

Wrapped in a separate tiny crate this time, because it's just that bit of functionality that totally should be in core. It is possible to go one step further and implement #[derive(Plain)], with static checking of the prescribed requirements, but I'm not yet guru enough to do that.

opened Apr 4, 2017 by le-jzr 21
Closed

Goblin panics when reading certain kinds of UPXed binaries

I was investigating my options for parsing EXE files to determine what environment to auto-fill in my experimental game launcher (ie. DOSBox, Wine, Wine+qemu-user, Mono, etc.) and I managed to trigger some panics in goblin.

====================
TESTING WITH GOBLIN:
====================
unknown magic: ./hello_owatcom_com.com
Parse error: ./hello_owatcom_os2v2.exe => Invalid magic number: 0x1
pe: ./hello_pacific.exe
Parse error: ./hello_owatcom_dos.upx.exe => requested range [309100590..309100594) from object of len 6881
Parse error: ./hello_owatcom_dos4g.exe => Invalid magic number: 0x1
Parse error: ./hello_owatcom_windows.exe => Invalid magic number: 0x0
pe: ./hello_mingw32.exe
pe: ./hello_csharp_exe_itanium.exe
pe: ./hello_owatcom_win95.exe
Parse error: ./hello_owatcom_dos4g.upx.exe => Invalid magic number: 0x1
elf: ./hello_gcc.x86
unknown magic: ./hello_djgpp.upx.coff.exe
unknown magic: ./hello_owatcom_com.upx.com
unknown magic: ./hello_dev86.upx.com
unknown magic: ./hello_dev86.com
pe: ./hello_mingw64.exe
Parse error: ./hello_djgpp.exe => Invalid magic number: 0x0
PANICKED on hello_mingw32.upx.exe
Parse error: ./hello_owatcom_dos.exe => Invalid magic number: 0x20
PANICKED on hello_owatcom_win95.upx.exe
pe: ./hello_owatcom_nt.exe
Parse error: ./hello_owatcom_win386.exe => Invalid magic number: 0x0
Parse error: ./hello_djgpp.upx.exe => Invalid magic number: 0x0
Parse error: ./hello_owatcom_dos4gnz.exe => Invalid magic number: 0x1
PANICKED on hello_mingw64.upx.exe
Parse error: ./hello_owatcom_os2.exe => Invalid magic number: 0x0
pe: ./hello_csharp_exe_arm.exe
pe: ./hello_csharp_exe_x64.exe
pe: ./hello_csharp_exe_x86.exe
elf: ./hello_gcc.x86_64
PANICKED on hello_owatcom_nt.upx.exe
Parse error: ./hello_pacific.upx.exe => requested range [309100590..309100594) from object of len 4527

Here's a backtrace which appears to represent all of the panics:

ssokolow@monolith test_exes [rusty-core] % RUST_BACKTRACE=1 ./pe-test/target/debug/goblin-test hello_owatcom_nt.upx.exe
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', /checkout/src/libcore/option.rs:329
stack backtrace:
   0: std::sys::imp::backtrace::tracing::imp::unwind_backtrace
             at /checkout/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::_print
             at /checkout/src/libstd/sys_common/backtrace.rs:71
   2: std::panicking::default_hook::{{closure}}
             at /checkout/src/libstd/sys_common/backtrace.rs:60
             at /checkout/src/libstd/panicking.rs:355
   3: std::panicking::default_hook
             at /checkout/src/libstd/panicking.rs:371
   4: std::panicking::rust_panic_with_hook
             at /checkout/src/libstd/panicking.rs:549
   5: std::panicking::begin_panic
             at /checkout/src/libstd/panicking.rs:511
   6: std::panicking::begin_panic_fmt
             at /checkout/src/libstd/panicking.rs:495
   7: rust_begin_unwind
             at /checkout/src/libstd/panicking.rs:471
   8: core::panicking::panic_fmt
             at /checkout/src/libcore/panicking.rs:69
   9: core::panicking::panic
             at /checkout/src/libcore/panicking.rs:49
  10: <core::option::Option<T>>::unwrap
             at /checkout/src/libcore/macros.rs:21
  11: goblin::pe::import::SyntheticImportDirectoryEntry::parse
             at /home/ssokolow/.cargo/registry/src/github.com-1ecc6299db9ec823/goblin-0.0.10/src/pe/import.rs:125
  12: goblin::pe::import::ImportData::parse
             at /home/ssokolow/.cargo/registry/src/github.com-1ecc6299db9ec823/goblin-0.0.10/src/pe/import.rs:158
  13: goblin::pe::PE::parse
             at /home/ssokolow/.cargo/registry/src/github.com-1ecc6299db9ec823/goblin-0.0.10/src/pe/mod.rs:80
  14: goblin::parse
             at /home/ssokolow/.cargo/registry/src/github.com-1ecc6299db9ec823/goblin-0.0.10/src/lib.rs:276
  15: goblin_test::run
             at ./pe-test/src/goblin.rs:17
  16: goblin_test::main
             at ./pe-test/src/goblin.rs:36
  17: __rust_maybe_catch_panic
             at /checkout/src/libpanic_unwind/lib.rs:98
  18: std::rt::lang_start
             at /checkout/src/libstd/panicking.rs:433
             at /checkout/src/libstd/panic.rs:361
             at /checkout/src/libstd/rt.rs:57
  19: main
  20: __libc_start_main
  21: <unknown>

While this renders it unsuitable for my project (the mere fact that Goblin is capable of dying at an unwrap (when the other PE parser I've tried so far simply used Result to indicate a parse failure) indicates that using it in my project would cause me more worry than simply writing my own MZ/NE/PE parser with Nom), I thought you'd want to know so you can fix the problem for others.

If you want to re-create my test binaries, the source materials are in the test_exes folder of ssokolow/game_launcher and build.sh contains instructions for the simplest, easiest way to install the requisite packages on a *buntu Linux 14.04 LTS machine like mine.

To reiterate what build.sh says, all compilers are optional, so producing just the binaries which caused panics here should only require apt-get install upx-ucl mingw-w64 and then downloading and unpacking OpenWatcom.

bug 
opened Aug 5, 2017 by ssokolow 20
Closed

pe: Fix resolution of redirect unwind info

Fixes the resolution of redirected unwind information in the PE exception table.

The issue originated in https://github.com/m4b/goblin/pull/136. Runtime function entries usually specify an offset of a UnwindInfo struct, which is always 4-byte aligned. However, sometimes the last bit of this offset is flipped, with indicates that instead another RuntimeFunction must be resolved at the offset (with the bit flipped back). This logic is handled by get_unwind_info.

There was a bug in the initial implementation, that treated the offset as relative, where in fact it is absolute. To fix this, get_function_by_offset now expects an absolute offset.

opened Oct 2, 2019 by jan-auer 19
Closed

Refactor `elf::gnu_hash::GnuHash`

Closes #149 .

opened Apr 8, 2019 by lzutao 18
Closed

Get proper name for over-8-character sections

Fixes #99

An early PR to get some reviews, as there are probably better ways to do this. I need to implement the base64 kind of section table names, and fix the error handling in some places.

opened Aug 6, 2018 by roblabla 15
Open

Resolve clippy lints

Sorry if this PR is just noise, but I saw some other commits cleaning clippy warnings up.

I'm happy to squash this into 1 commit too, if that's the preference.

edit: the clippy output is saved in each commit

opened May 19, 2020 by connorkuehl 0
Open

Safety invariants for GnuHash::from_raw_table should be documented

I can't tell what I need to do to use this safely. https://docs.rs/goblin/0.2.1/goblin/elf64/gnu_hash/struct.GnuHash.html

opened Apr 17, 2020 by leo60228 1
Open

Reading the ELF of the current executable without copying it into memory a second time.

Hi,

This is more of a question that a bug.

I have a somewhat unusual use-case. I want to inspect the symbol table of the currently running executable.

The goblin example does this:

            let buffer = fs::read(path)?;

which I think loads the entire binary into memory. In the case of the binary being the same as the one currently running, that means we load most of the stuff in a second time.

Is there a more efficient way? If I were to mmap() the binary into memory, would most kernels be smart enough to share the pages with those already loaded?

Thanks

opened Apr 1, 2020 by vext01 6
Open

Is an elf file with DT_GNU_HASH buckets_num == 0 legal?

My lld on Linux produces some elf files that Goblin complains about being invalid.

If bucket_len is zero, Goblin currently returns an error stating that DT_GNU_HASH is invalid. Patching this to return 0 results in the binary loading just fine.

I can't find any documentation if DT_GNU_HASH intends for bucket_len to be zero. Is this an issue in lld or Goblin?

opened Mar 31, 2020 by dbartussek 4
Open

Build failure when specifying elf32 feature without elf64

Disable default-features and specify only elf32:

[dependencies]
goblin = { version = "0.2", default-features = false, features = ["std", "elf32"] }

Build fails:

   Compiling goblin v0.2.0
error[E0432]: unresolved import `crate::elf64`
   --> /Users/parasyte/.cargo/registry/src/github.com-1ecc6299db9ec823/goblin-0.2.0/src/elf/header.rs:207:17
    |
207 |             use crate::elf64;
    |                 ^^^^^^^^^^^^ no `elf64` in the root

error[E0433]: failed to resolve: maybe a missing crate `elf64`?
   --> /Users/parasyte/.cargo/registry/src/github.com-1ecc6299db9ec823/goblin-0.2.0/src/elf/dynamic.rs:704:35
    |
704 |     elf_dyn_std_impl!(u64, crate::elf64::program_header::ProgramHeader);
    |                                   ^^^^^ maybe a missing crate `elf64`?

error: aborting due to 2 previous errors

Some errors have detailed explanations: E0432, E0433.
For more information about an error, try `rustc --explain E0432`.
error: could not compile `goblin`.

Looking at the code, specifying the opposite (elf64 without elf32) will also fail for similar reasons.

opened Feb 15, 2020 by parasyte 1
Open

mach relocation parsing doesn't handle big endian

The relocation fields are bitfields, which makes the ordering all weird. Currently we only use the little endian ordering, but that won't work for big endian files.

LLVM has code to handle this difference, see here.

Also related, we don't currently parse scattered relocations.

opened Jan 6, 2020 by philipc 3

Contents

    nrc zero — zero-allocation parsing of binary data
    rrrfff Lightweight ARMv8-A(ARM64, AArch64, Little-Endian) Inline Hook Library for Android C/C++
    andylokandy byte - A low-level, zero-copy and panic-free serializer and deserializer for binary
    taoyds scripts and baselines for Spider: Yale complex and cross-domain semantic parsing and text-to-SQL challenge
    ramonszo A javascript library that lets you add stories EVERYWHERE.
    egoist Vue directive to copy to clipboard. (1kB)
    ghostdevv A svelte action to copy text on click!
    richardartoul Molecule is a Go library for parsing protobufs in an efficient and zero-allocation manner.
    m4b scroll - A suite of powerful, extensible, generic, endian-aware Read/Write traits for byte buffers
    fox-it A no-nonsense c-like structure parsing library for Python
    njones Simple Cross Network, Cross OS, Cross Application Copy-Paste
    MayloIFERR RACF (Recency Aware Collaborative Filtering) is the implementation of the "Recency Aware Collaborative Filtering for Next Basket Recommendation" original paper. which is a set of collaborative filtering approaches for solving the problem of predicting the Next Basket Recommendation.
    RoganDawes P4wnP1 is a highly customizable USB attack platform, based on a low cost Raspberry Pi Zero or Raspberry Pi Zero W.
    ShaohuaDong2021 Edge-aware Guidance Fusion Network for RGB-Thermal Scene Parsing
    wavesoft Javascript Binary Bundle - A binary format for packing data structures for the web
    asweigart Pyperclip is a cross-platform Python module for copy and paste clipboard functions
    sonnylazuardi Cross-platform colour copy paste from phone's camera to web, figma and sketch plugin
    ms-jpq Cross platform clipboard | networkless! remote copy
    bcmi CaGNet: Context-aware Feature Generation for Zero-shot Semantic Segmentation
    nicokoch reflink - copy-on-write mechanism on supported file systems
    Nike-Inc Elevate is a JSON parsing framework that leverages Swift to make parsing simple, reliable and composable.
    Share this repo

    Related Repos


    Encoding Binary
    511
    encodes and decodes base64 as bytes or utf8
    marshallpierce encodes and decodes base64 as bytes or utf8
     
    Encoding Binary
    1k
    cross-platform, zero-copy, and endian-aware binary parsing
    m4b libgoblin Documentation https://docs.rs/goblin/ changelog Usage Goblin requires rustc 1.36.0. Add to your Cargo.toml [dependencies]
     
    Encoding Binary
    2.3k
    bincode — A binary encoder/decoder in Rust
    bincode-org Bincode A compact encoder / decoder pair that uses a binary zero-fluff encoding scheme. The size of the encoded object will be the same or smaller tha
     
    Encoding Binary
    39
    nue — I/O and binary data encoding for Rust
    arcnmx nue A collection of tools for working with binary data and POD structs in Rust. pod is an approach at building a safe interface for transmuting POD st