Neural block texturing example

Neural block texturing, bilinear sampled "colors" latent+full res GPU texture-style "selector" latent (sampled with nearest): 4 textures, 2.24 bpp/texture. Top level ("selector") latent contains 2 channels/4 bits each, and the 1/6 res "colors" level (sampled bilinearly) contains 4 channels/8 bits each. Trained in 56 secs (5090).

• Material: normal, roughness, albedo, AO (PavingStones070, 512×512 padded to 516×516)
• Selectors: 516×516×2, 4 + 4 bits per pixel, shared by all four maps, exact search each iteration
• Block latent: 86×86×4 (6×6 blocks), 8 bits/value, sampled bilinearly, ES-trained
• Decoder: 8→27→27→12 MLP, 1335 weights, decodes all four maps from one record - 8000 iterations, ES (256 pairs) then central finite differences, lr annealed, no backprop
• 32.27 dB weighted; normal 28.94 / roughness 35.40 / albedo 32.52 / AO 36.09 dB
• 320 bits per 6×6 block for the whole stack: 2.24 bpp per texture raw, 2.16 entropy-coded
• Selectors: 516×516×2, 4 + 4 bits per pixel, shared by all four maps, exact search each iteration
• Block latent: 86×86×4 (6×6 blocks), 8 bits/value, sampled bilinearly, ES-trained
• Decoder: 8→27→27→12 MLP, 1335 weights, decodes all four maps from one record
• 8000 iterations, ES (256 pairs) then central finite differences, lr annealed, no backprop
• 32.27 dB weighted; normal 28.94 / roughness 35.40 / albedo 32.52 / AO 36.09 dB
• 320 bits per 6×6 block for the whole stack: 2.24 bpp per texture raw, 2.16 entropy-coded

Decoded 512x512 material:

Latents: top full-resolution level has 2 channels (4-bits each), second 1/6 res level has 4 channels (8-bits each):

Using bilinear sampling on the "colors" latent results in noticeably higher quality and less block artifacts, but the data for each block is no longer independent. It's like PVRTC-like, not BC7-like.