Fitting a neural texture decoder with ES (no backprop)

This is something I prototyped for fun in an evening: Toy neural texture compression, trained entirely with Evolution Strategies (no backprop, no derivatives). A 128×128×8 latent texture + a 1035-weight MLP decoder reconstruct a 512×512 image (kodim23 crop) at 32.2 dB PSNR. Latent quantizes to 8 bits with only 0.04 dB loss: ~4 bpp raw, ~3.3 bpp entropy coded. Trained in 150 s on CPU. Single C++ file, ~600 lines, stb_image for I/O.

The trick that makes ES viable: each latent texel only affects a small pixel footprint through bilinear filtering, so one full-image decode pair yields a gradient estimate for every texel at once.

Rate/distortion sweep, all 3,000 iters, 8-bit latent:
64×64×4 → 26.9 dB @ 0.47 bpp
64×64×8 → 28.2 dB @ 0.87 bpp
128×128×4 → 30.3 dB @ 1.65 bpp
128×128×8 → 32.2 dB @ 3.27 bpp

Output:

The latent:

Notes:

• No training framework. Dependencies are stb_image, stb_image_write, and OpenMP. The encoder and decoder share the same forward function.
• MLP training: antithetic ES with 32 perturbation pairs per step, each pair evaluated on the same random 4096-pixel minibatch. The estimated gradient goes through Adam.
• Latent training: all 131,072 latent values perturbed at once, two full-image decodes per pair, 4 pairs per step. Each pixel's loss change is credited only to the 4 texels its bilinear tap reads, so every texel gets its own local estimate from one decode pair.
• Quantization is post-hoc.
• Trained for 3,000 iterations, each iteration one ES update of the MLP followed by one ES update of the latent. The curve was still rising slowly at the end; 6000 iterations gained another ~0.3 dB.
• Decoder: 14 → 24 → 24 → 3 MLP, leaky ReLU hidden layers, sigmoid on the RGB output, 1,035 weights. Inputs are the 8 bilinearly sampled latent channels, u, v, and sin/cos of 2πu and 2πv.
• I’m using OpenAI-style ES: Gaussian parameter perturbations with antithetic $+\epsilon/-\epsilon$ evaluations to estimate an update direction.