Notes on NNTC vs. neural texturing

NNTC (here on GitHub) uses a bilinear/degree-2 polynomial decoder fitted to each PBR material (i.e. a degree-2 polynomial whose only quadratic terms are the products between the two latents). Other solutions use full non-linear neural networks (MLP's).

In my testing, MLP's are usually but not always stronger, but not by much (low PSNR difference, like ~1-3 dB). MLP's also make it harder to get filtered samples - NNTC easily leverages existing GPU texture filtering hardware.

However, eventually as I optimized the NNTC-specific CUDA encoder, and made it BC4/BC5 aware, it started to beat my neural network based solution on raw PSNR. 

Training MLP's is a lot more expensive, and a harder problem to solve - and IMHO unnecessary for PBR textures if you configure the latent textures correctly. Encoding high frequency details into low-res feature channels and training MLP's to decode them is going to be quite expensive. The alternative is to do what GPU texture formats like BC1-7/PVRTC1/ASTC/etc. have been using for ages: use high resolution weight planes, and optionally (for more efficient distribution) supercompress the data in some way using RDO+LZ, DCT etc.

One of NNTC's current limits: it only supports up to 4 channels on each of the two latent planes. I may expand this to up to ~6-8 channels in the future. If I hadn't implemented a neural net prototype first, I couldn't have found a way to NNTC.

NNTC also encodes in seconds with CUDA. Even with backprop, I'm skeptical a neural solution can be competitive there.

NNTC repo now on GitHub

NNTC ("Non-Neural Texture Compression") is a two-latent PBR texture representation where the encoder fits the GPU sampling path, not a texel-only reconstruction. The GitHub repo is here:

https://github.com/richgel999/nntc/

It's a PBR texture encoder which outputs 2-3 standard .DDS files and a JSON file containing fitted coefficients. It doesn't use neural networks to decode correlated PBR material textures (between 3-18 channels, or 1-6 3-channel RGB textures). Instead it uses plain linear operations to decode (or remap, or expand the dimensionality) of the output channels after standard hardware texture sampling/filtering.

I got this working after implementing neural texture compression using ES (Evolution Strategies), then realizing (after taking inspiration from the old PVRTC1 texture format) the problem can be easily changed so no neural networks were needed to decode the material channels. The end result is far simpler and faster to encode and sample.

The first latent the tool encodes is at full texture resolution and uses BC4's or BC5's. The encoder's backend is BC4/BC5 aware. The second latent is at 1/4 resolution and uses 8-bits/channel, and is 8,16,24 or 32 bits. The encoder supports mipmapping, and the resulting latent textures can be sampled normally (with standard filtering). 

The encoder supports 1-4 channels per latent and up to 6 RGB textures.

The encoder supports Windows and Linux and currently requires CUDA. There's a D3D11 viewer (works on any GPU) showing how to sample and then decode by applying the fitted matrix coefficients. The 2nd latent texture (which is at 1/4 resolution) needs to be sampled with a MipLODBias setting of +2.

This allows PBR textures to be stored in VRAM at roughly ~2.5 bpp, not counting mipmaps. The actual bitrate in memory depends on the channel settings and the material size in textures (it's a matrix of possible bitrates).

The more correlated (or lower dimensional) the textures in the material, the better this method works. The less correlated the textures, the more latent channels are needed. 

Also see this excellent series here:

"Dimensionality reduction for image and texture set compression"

https://bartwronski.com/2020/05/21/dimensionality-reduction-for-image-and-texture-set-compression/

This repo also includes a Prior Art Disclosure, dated September 13, 2026:

https://github.com/richgel999/nntc/blob/main/PRIOR_ART_DISCLOSURE.md

BC4/BC5 could be transcoded near-losslessly to ASTC LDR 4x4 fairly easily for mobile/tablet usage. I may add this in the future.

A variant of this idea with lower in-memory bitrates, that doesn't support native GPU texture sampling but is more flexible (supporting more channels, with quantization aware encoding), is possible. This variant would still only use plain linear operations (i.e. no MLP's for decoding), but would require the developer to roll their own filtering.

Neural texture decoders can suppress DCT ringing artifacts within latent features

This is mirrored from my X post.

This is very image codec specific: Normally a DCT-based (JPEG style) codec has mosquito/ringing noise especially on text. But within a neural texture/image codec that places a quantized DCT in the training loop (via ES), the neural net can learn how to suppress these artifacts quite effectively. This is a libjpeg Q=5 AC quantization matrix, applied XUASTC/XUBC7 style on the level 0 latent's spatial values - super low DCT quality.

The first image is the full texel resolution IDCT decoded level 0 latent, and the second is the fully decompressed image (the output from the ~500 weight neural network, after decoding the 2 latents and local "cell" UV as inputs). 

The latent textures:

A latent can tolerate quantization artifacts that would be catastrophic in image space because joint training can rotate/warp the useful representation so those artifacts lie largely in low-sensitivity directions of a tiny learned decoder.

New GitHub repo release: "neural_block_textures"

My latest work in this space is here:

https://github.com/richgel999/neural_block_textures

This is a time stamped Prior Art release dated: September 8, 2026, which will be mirrored to various archives.

Neural block textures vs. GPU textures (like BC7) at Equal Bitrates

The GitHub release is here.

The existing neural texture work has made the entire space overly complicated.

The first step beyond GPU textures is to essentially keep the current GPU texture-style latent, but replace the fixed hardware generative decoder with a configurable MLP, and adopt PVRTC1-like block (which is at 1/4 resolution in PVRTC1) endpoint bilinear sampling:

Replace the classical GPU texture decoder (ASTC/BC1-7 etc.) with a neural net (MLP). Now you get as many output channels as needed. Then bilinear filter the "color" endpoints (like PVRTC1), and use a flexible (say 2-4) # of channels. 

I've been testing the "colors" latent (which is typically 2-4 channels) at 1/4, 1/6, 1/8 and 1/16 sizes relative to the texture's resolution.

Per-texel "selectors"/"weights" can be unchanged: use 1 channel for single textures, 2-3 for materials. I've tested 1-5 bits per channel, with potentially a different number of bits on each channel. 

The "encoder" is the training step which can use backprop or ES (Evolution Strategies) etc. It's amazingly flexible and looks remarkable. ES on the per-texel or lower res latents is easily optimized, and at this tiny MLP size it's not a big deal.

Even on a single texture (not a material), the bitrate and quality is roughly competitive vs. transform methods on GPU texture latents. Except this method scales easily to materials and inference (decoding) cost is amortized across the material. The more channels the MLP outputs, the lower the effective bitrate.

Around 250-1200 MLP weights seems like a good range for this method. Tiny networks. I don't see why this can't be implemented directly in GPU hardware.

For inference on load (on the CPU) this approach seems entirely reasonable, using threading and SIMD to accelerate inference. No sidebands (like NTC uses) are needed for fast full-format BC7 encoding - the math is simple, and is already solved by bc7f. 

None of this is theoretical - I have this working now using CUDA for training (with a CPU SIMD fallback if needed). It's exciting.

Some hard numbers, comparing this neural texture design vs. BC7, are from this X post:

A single complex 1080x1080 photo, neural block texture vs. two BC7 encoders at matched 8.0 bpp bitrates (neural was 8.02, including quantized latents+all MLP weights as FP16):

Neural block texture: 43.15 dB
bc7f analytical: 43.78 dB
bc7e_scalar level 0: 41.03 dB
bc7e_scalar level 1: 44.85 dB
bc7e_scalar level 2: 44.67 dB
bc7e_scalar level 3: 45.18 dB
bc7e_scalar level 6 (slowest): 45.59 dB

Neural config: Per-texel "selectors" latent: 2 channels, 4+2 bits (6 bits total)
1/4 resolution "colors" latent: 4 channels, 8-bit channels (32 bits total)
MLP: 1,083 weights
Training: Quant aware, ES+CD, 8k iterations, 106 secs on a 5090

This is a sanity check: at equal very high bitrates, can this format and my trainer match a highly tuned widely deployed BC7 encoder? Yes it can, or very close.

Unlike any GPU texture format, this design is highly configurable/flexible. It easily scales down to low bitrates, or scales wider to handle multi-channel PBR textures with the same latents.

Neural block textures stored inside KTX2: a near-perfect fit

A neural block texture codec can be part of the Khronos KTX2 texture format. The initial plan is a lowest common denominator approach: inference on load with SIMD+threading, ~250-500 MLP weights (up to 1k), real-time encode to BC1-7/ASTC with no sidebands (using basisu's existing analytical real-time encoders).

It would compete vs. transform domain codecs (ours and others that are surely coming). Valuable for single textures/photos, PBR material sets, correlated geospatial tiles. At first training will be focused on CUDA with a slow CPU fallback. Target bitrate would be ~1-2.5 bpp (so XUBC7 class, not XUASTC which goes down to ~0.35 bpp). Inference cost is highly amortized across 2+ correlated textures (the more the better).

KTX2 global data can be used to hold the MLP inference weights, and the container format already supports mipmaps, texture arrays, seek tables, etc.

The existing Basis Universal transcoding API would also be a great match. The initial R&D is done.

Neural block GPU texture compression: Prior Art disclosure

Prior art disclosure: neural block texturing, a hybrid of compressed GPU textures and neural texturing. Updated Sept. 6, 2026. Added my latest findings.

https://github.com/richgel999/neural_tex_block_compression