Home / Golang / Miscellaneous

Quickly cross-compile your C code

WORK IN PROGRESS: Viceroy is a very early work in progress and is subject to breaking changes. It's also subject to not really working all that well y
Information
Category: Golang / Miscellaneous
Watchers: 1
Star: 3
Fork: 1
Last update: Jan 19, 2022
Resource links
https://github.com/rfratto/viceroy

Viceroy logo

WORK IN PROGRESS: Viceroy is a very early work in progress and is subject to breaking changes. It's also subject to not really working all that well yet. Submit issues if you find bugs!

Viceroy makes cross-compiling C code fast and easy by proxying build commands to Docker containers.

It was designed for use with Go applications which need to use Cgo, but it can be used generally anywhere where you may need to compile C code.

Using Viceroy with Go can be much faster than using a dedicated Docker build container because Docker is only used to build what the host machine can't. Preliminary testing of Viceroy showed that cross-compiling Grafana Agent on an M1 Max MacBook Pro goes from a 10 minute build to an 80 second one.

Goals

  • Improve iteration speed of Go projects that need Cgo
  • Compile C and C++ code for many OS and architectures
  • Install C dependencies in build workers
  • Usable in standalone mode inside of Docker/Dockerfiles

TODO

  • Support for crosscompiling to darwin
  • Support for crosscompiling to freebsd
  • Worker container images on Dockerhub
  • Installers (Homebrew, etc.)
  • Managing packages installed on workers
  • pkg-config support

Supported Platforms

Viceroy examines the following environment variables to determine which target system to cross-compile for:

  • VICEROYOS (or GOOS)
  • VICEROYARCH (or GOARCH)
  • VICEROYARM (or GOARM)

These environment variables determine which compiler toolchain to use and some defaults for environment variables (such as LD_LIBRARY_PATH). The environment variables will default to values appropriate for the worker container's environment when unspecified.

The following target platforms are currently supported:

linux darwin freebsd windows
amd64 x x
386 x x
armv5 x
armv6 x
armv7 x
arm64 x

Support for darwin and freebsd will come in future commmits.

Installation

Host machine

Building from source is currently required. Installers for various systems will eventually be made available as the project matures.

Go 1.17 is required to install from source. Run make install to install the binaries.

Until pushed images are available on Dockerhub, you must also run make FULL_BASE_IMAGE=1 container to build the worker container locally.

viceroyd may be run with viceroyd -listen-addr tcp://0.0.0.0:12194. It is currently left as an exercise to the reader for how to configure this as a permanent system service.

Docker Container

Viceroy can be installed on Debian-based systems in "standalone mode," which enables viceroycc to function completely locally without the need for daemons. This is useful when you want to unify your build process by also using Viceroy from within Dockerfiles.

TODO

Usage

Usage is the same regardless of Viceroy being installed on your host machine or inside of a Docker container:

  1. Install Viceroy
  2. export CC=viceroycc
  3. Compile!

If using Cgo, your build command would look similar to CGO_ENABLED=1 CC_FOR_TARGET=viceroycc GOOS=<target> GOARCH=<target> go build <package>

Installing packages on worker containers

TODO

Legal note

OSX/Darwin/Apple builds: Please ensure you have read and understood the Xcode license terms before continuing.

0 Open More issues
Closed

Find a way to expose project-specific build dependencies

The default libraries installed in Viceroy worker containers may not be enough to build any given project. There is currently no way for projects to expose their extra build dependencies in a way Viceroy can use.

Viceroy needs a way to allow projects to define extra build dependencies, which may be need for any combination of Linux, macOS, Windows, and FreeBSD builds.

Security is the biggest concern here; Viceroy allows access to the host filesystem, and we'd want users to be able to audit the custom containers used for building.

So far, I can think of two ways to do this:

  1. Allow projects to specify custom build containers. Exposing security auditing capabilities would be likely very difficult here.

  2. Allow some kind of project manifest to list extra Deb repositories, dependencies, files/folders (with a checksum) to add into the worker containers. This one is harder to implement, but easier to audit from a security perspective.

opened Mar 21, 2022 by rfratto 1
Closed

Remove extra mount for /proc

The workaround added for darwin support requires mounting /proc on top of the chroot:

https://github.com/rfratto/viceroy/blob/f99e9f9ec7d65392f698db47d78ac76d30a1245b/cmd/viceroyfs/main.go#L240-L267

Aside from feeling hacky, it's likely prone to deadlocking if the process only has one thread available to it (Go expecting all syscalls to support async I/O causes issues with FUSE, which is also why viceroyfs is launched as a child process of viceroyworkerd)

We currently need /proc because OSXCross expects it to be mounted. Unless I'm missing something, it should be possible to remove the requirement by adding in various fallbacks similar to the ones found in Clang.

At the moment, I don't see a reason to need /proc available in the viceroy chroot outside of this issue. It seems worthwhile to have OSXCross support chroots rather than finding a permanent way to add these extra mounts on top of the viceroy overlay chroot.

opened Mar 15, 2022 by rfratto 1
Closed

Update version of macOS and FreeBSD

Opening this up early since I tested it and everything works, but it takes time to do a release so I'll merge tomorrow.

Closes #4.

opened May 16, 2023 by rfratto 0
Closed

Upgrade to version 12 of the MacOS SDK

Version 11 of the SDK doesn't work for libraries such as [email protected]. This version of go-m1cpu is used by the OpenTelemetry Collector. We could work around this by using version v0.1.6, but it would be even better to just update the SDK.

The reason we cannot use v.0.1.4 is that apparently "the kIOMainPortDefault variable is only available on MacOS 12.0+ (Monterey); before that, it was named kIOMasterPortDefault.".

opened May 16, 2023 by ptodev 0
Closed

Add support for arm64 image

null

opened Aug 19, 2022 by rfratto 0

v0.4.0(May 18, 2023)

Enhancements

  • Update macOS SDK to macOS 12.3.
  • Update FreeBSD SDK to 12.4.

The Docker image is available as rfratto/viceroy:v0.4.0.

v0.3.0(May 18, 2023)

Enhancements

  • Support s390x for Linux.

The Docker image is available as rfratto/viceroy:v0.3.0.

v0.2.1(Oct 23, 2022)

Bug fixes

  • Invocations of viceroycc for amd64 Darwin will now work properly.

The Docker image is available as rfratto/viceroy:v0.2.1.

v0.2.0(Aug 20, 2022)

NOTE: As of this release, per-architecture images are pushed as <release>-<arch>. These images are artifacts of the build process and can be ignored in favor of the <release> image, which will be a manifest containing all supported architectures.

Features

  • Include ARM64 images.

See the repository README for which platforms are supported for compilation targets.

The Docker image is available as rfratto/viceroy:v0.2.0.

v0.1.0(Aug 19, 2022)

Initial release!

Features

  • Expose viceroycc binary which proxies build commands to one of many C compilers depending on values of the VICEROYOS/GOOS, VICEROYARCH/GOARCH, and VICEROYARM/GOARM environment variables.

See the repository README for which platforms are supported.

The Docker image is available as rfratto/viceroy:v0.1.0.

Contents

    cobbr SharpGen is a .NET Core console application that utilizes the Rosyln C# compiler to quickly cross-compile .NET Framework console applications or libraries.
    keiichiw Compile-time C Compiler implemented as C++14 constant expressions
    NyaMisty Cross Platform Hook Library based on Detours
    zhlynn Maybe is the most quickly codesign alternative for iOS12+ in the world, cross-platform ( Linux & macOS ), more features. sign!
    mitchellh A dead simple, no frills Go cross compile tool
    johnpapa Subtly change the color of your Visual Studio Code workspace. Ideal when you have multiple VS Code instances, use VS Live Share, or use VS Code's Remote features, and you want to quickly identify your editor.
    kovetskiy Compile Go programs 7x faster
    tilpner tauri_includedir_codegen - Include a whole directory tree at compile time! - Compile time part
    peter-winter Compile Time Parser Generator is a C++ single header library which takes a language description as a C++ code and turns it into a LR1 table parser with a deterministic finite automaton lexical analyzer, all in compile time.
    Julien-R44 FastSSH is a TUI that allows you to quickly connect to your services by navigating through your SSH config.
    kitech Qt binding for Go (Golang) aims get Go's compile speed again.
    rep-movsd A C++ HTML template engine that uses compile time HTML parsing
    psiayn Compile your project within neovim
    Ogeon rust-on-raspberry-piRaspberry Pi — instructions for how to cross compile Rust projects for the Raspberry Pi .
    moroneyt Compile-time linear algebra in C++
    fastred Collection of advice on optimizing compile times of Swift projects.
    SephDB Compile-time printf-like string formatting
    hjwdzh Lightweight Cuda Renderer with Python Wrapper.
    PaulBernier JavaScript to Lua compiler with runtime library.
    Redcrafter This project will compile verilog (a hardware description language) into factorio blueprints.
    IchMagBier Simple example to show how x64 Assembler and Raylib work together
    Share this repo

    Related Repos


    Miscellaneous
    6.7k
    Fast and Simple Serverless Functions for Kubernetes
    fission Fission: Serverless Functions for Kubernetes Fission is a fast serverless framework for Kubernetes with a focus on developer productivity and high per
     
    Miscellaneous
    4.3k
    Kubernetes-based, scale-to-zero, request-driven compute
    knative Knative Serving Knative Serving builds on Kubernetes to support deploying and serving of applications and functions as serverless containers. Serving
     
    Miscellaneous
    21
    Test your application on Kubernetes in a brand new simple way⚡
    teamcode-inc Like KubeOrbit idea? ⭐ Give us a GitHub Star! ⭐ KubeOrbit is an open-source abstraction layer library that turns easy apps testing&debuging on Kuberne
     
    Miscellaneous
    16
    Load ssp dll golang implementation
    timwhitez Doge-AddSSP Load ssp dll golang implementation Administrator/System Privilege Usage: .\Doge-AddSSP.exe D:\ssp.dll //must use full path
     
    Miscellaneous
    34
    Scalable Feature Store
    bgokden Veri Veri is a Distributed Feature Store optimized for Search and Recommendation tasks. Feature Label store allows storing features as keys and labels
     
    Miscellaneous
    99
    Ngram index for golang
    Lazin go-ngram N-gram index for Go. Key features Unicode support. Append only. Data can't be deleted from index. GC friendly (all strings are pooled and com
     
    Miscellaneous
    24
    A battle-tested recommendation library written in Go.
    rlouf Birdland is a famous Jazz club. It is also a recommendation library. Birdland is a collaborative filtering algorithm in two steps: exploration and rec
     
    Miscellaneous
    85
    A tool for creating hidden accounts using the registry.
    wgpsec CreateHiddenAccount 创建隐藏账号 中文 | EN Tool Introduction There are two common ways to create a hidden account. One is to add the $ sign directly after the
     
    Miscellaneous
    7
    go library for complex struct mapping
    dranikpg Dto mapper dto-mapper is an easy-to-use library for complex struct mapping. It's intended for the creation of data transfer objects, hence the name. W
     
    Miscellaneous
    4
    Official codebase and releases for the DEXbot Client application
    DEXbotLLC The sidecar to your cryptocurrency wallet Create an account on DEXbot.io » Read the Docs to Get Started » Check Out The Whitepaper » Introduction DEXb
     
    Miscellaneous
    3
    💃 MPRIS implementation for osu!
    TorchedSammy Opera 💃 MRPIS implementation for osu! Opera is a program to send currently playing osu! maps/songs over DBus. It implements the MPRIS specification a
     
    Miscellaneous
    3
    A Go library for building mongo queries with factory functions
    ahmetcanozcan FET Build query dynamically A Go library for building mongo queries with factory functions. What is FET? you can build queries with factory functions.
     
    Miscellaneous
    3
    Send SSE when files change
    tomberek ssentr Send SSE when files change. Setup: $ cat Caddyfile http://localhost:8085 route /reload { header Access-Control-Allow-Origin * header Access-C
     
    Miscellaneous
    3
    Quickly cross-compile your C code
    rfratto WORK IN PROGRESS: Viceroy is a very early work in progress and is subject to breaking changes. It's also subject to not really working all that well y
     
    Miscellaneous
    3
    A Socket Framework Built Using Golang
    golangpros gosocket About: A Simple Go Socket Framework Installation: $ go get -u github.com/golangpros/gosocket Tutorial: import "github.com/golangpros/gosocke