Ngx - Neural network based visual generator and mixer
Ngx is an attempt at utilizing a neural network for VJing. It implements pix2pix (image-to-image translation with cGAN) as an ad-hoc next-frame prediction model that is trained with pairs of consecutive frames extracted from a video clip, so that it can generate an image sequence for an infinite duration just by repeatedly feeding frames back. It also has functionality for mixing (crossfading) two pix2pix models that gives unexpected variation and transition to generated video.
(The gif on the right is from a tweet by @chaosgroove.)
|Model Name||Original clip||Vimeo link|
|Beeple 1||FIBER OPTICAL||https://vimeo.com/238083470|
|Beeple 8||shifting pains||https://vimeo.com/48818536|
The original video clips are licensed under a Creative Commons Attribution license (CC-BY 3.0). These pre-trained models are also attributed to the original author.
How to control
Ngx can be controlled with the on-screen controller or a MIDI controller.
|Mix||Crossfading between model 1 and 2||77|
|Noise 1||Noise injection (low density)||78|
|Noise 2||Noise injection (medium density)||79|
|Noise 3||Noise injection (high density)||80|
|Feedback||Feedback rate||81, 82|
|FColor||False color effect||83|
|Hue Shift||Amount of hue shift||55, 56|
|Model Select 1||41, 42, 43, 44, 57, 58, 59, 60|
|Model Select 2||73, 74, 75, 76, 89, 90, 91, 92|
The default MIDI mapping is optimized for Novation Launch Control XL: You can change model with the track buttons and control parameters with the track faders. The 7th and 8th panning knobs are used to tweak the hue shift values.
How to open the project in Unity
This repository uses Git submodules to manage dependent packages. To check these submodules out, execute
git submodule init && git submodule update from the command line, or specify
--recursive option when initially cloning.
After cloning the repository,
.pict (pix2pix weight data) files should be manuyally downloaded and copied into the Streaming Assets directory because these files are excluded from the repository to save bandwidth and storage usage. Please download the pict file package and extract it into
How to train pix2pix models
Prerequisites: Basic experience on a machine learning framework. It's recommended to have a look into the pix2pix tutorial from Machine Learning for Artists if you haven't tried pix2pix or any similar GAN model.
The followings are Colaboratory notebooks I used to train the default models. These notebooks are written to use Google Drive as a dataset storage.
- Video preprocess: Converts a video clip into AtoB image pairs.
- pix2pix-tensorflow colab
- pix2pix export: Exports a
.pictfile from a checkpoint of a trained model.
Frequently asked questions
What are the hardware recommendations?
I used a Windows system with GeForce GTX 1070 for a live performance. It runs at around 21 fps; this might be the minimum practical spec for the system.