Neural GPU block compression

Neural texture block decoding, trained with Evolution Strategies (ES), 3,000 iterations.

This example uses 2 source latent texture: A full-res 512x512 at 3-bpp, and 1/4 res (128x128) with 4 channels at 8 bpp=32 bpp. Latent is sampled using nearest sampling, i.e. it's just a trained block decoder. It uses 5.0 bpp of latent data amortized across the entire texture stack (in this example just 1 texture, but it supports up to 4).

Neural net (MLP leaky ReLU, sigmoid on the output): 7→36→ 36→3, 1,731 weights

MLP inputs: 5 latent values + local block UV texel coordinate as [-1,1]

The full-res latent texture can be 4-bpp, which looks noticeably better, or 2-bpp etc. Further lossy or lossless compression can be applied to the latent data.

This is a practical approach and it works on PBR materials (i.e. across multiple related textures in a correlated texture array). Prior art disclosure repo is next.

I have a variant that adds in-loop deblocking using a 5 tap filter which ES factors in before computing loss, which boosts quality by ~0.9 dB in one 4x4 test.

Side by side - right is compressed:

Top-level (full resolution) 3-bit latent data, visualized as a 2D texture:

The top-level can have X channels, each a different number of bits, down to even 1-bit per channel. This roughly corresponds to dual or single plane modes in ASTC/BC7.

Second level (quarter resolution) "control" latent data, visualized as a 2D texture (4 channels, 8-bits per channel):

The "control" level can have a varying number of channels, not just 4, depending on the complexity of the texture or material.

Note this per-block data can be stored into compressed block packets, not textures: much like a classic GPU texture.

Same settings, but using a 4-bit top level latent texture:

I have some ideas on how to fix the BC1-like block artifacts on chroma blocks.

One especially interesting consequence of using ES for training and inference on load (i.e. transcoding to a compressed texture like BC1-7/ASTC): the GPU texture transcoder itself can be inside the optimization loop. Since we're using derivative-free ES, the objective could literally be:

neural bits → neural decode → BC7 encoder → BC7 decode → compare against source

The BC7 encoder can contain discrete mode decisions, bit quantization, endpoint selection, partition search, etc. None of that needs to be differentiable.