Notes on NNTC vs. neural texturing

NNTC (here on GitHub) uses a bilinear/degree-2 polynomial decoder fitted to each PBR material (i.e. a degree-2 polynomial whose only quadratic terms are the products between the two latents). Other solutions use full non-linear neural networks (MLP's).

In my testing, MLP's are usually but not always stronger, but not by much (low PSNR difference, like ~1-3 dB). MLP's also make it harder to get filtered samples - NNTC easily leverages existing GPU texture filtering hardware.

However, eventually as I optimized the NNTC-specific CUDA encoder, and made it BC4/BC5 aware, it started to beat my neural network based solution on raw PSNR. 

Training MLP's is a lot more expensive, and a harder problem to solve - and IMHO unnecessary for PBR textures if you configure the latent textures correctly. Encoding high frequency details into low-res feature channels and training MLP's to decode them is going to be quite expensive. The alternative is to do what GPU texture formats like BC1-7/PVRTC1/ASTC/etc. have been using for ages: use high resolution weight planes, and optionally (for more efficient distribution) supercompress the data in some way using RDO+LZ, DCT etc.

One of NNTC's current limits: it only supports up to 4 channels on each of the two latent planes. I may expand this to up to ~6-8 channels in the future. If I hadn't implemented a neural net prototype first, I couldn't have found a way to NNTC.

NNTC also encodes in seconds with CUDA. Even with backprop, I'm skeptical a neural solution can be competitive there.