ETC1S Chroma Artifact Filtering, Real-Time Encoders and Transcoding Format Graph Expansion

ETC1S is the original universal supercompression texture format supported by the open source Basis Universal library. It was designed to support all LDR GPU texture formats like ETC1, BC1-7, ASTC, PVRTC1, etc. and be fast to transcode in basic WASM (no threading, no SIMD needed), and also have compelling bitrates via codebooks applied to the individual components of the ETC1S latent. 

It decodes slightly faster than libjpeg. In various domains, like in very large scale geospatial use cases, it's quite popular, but using the ETC1S format resulted in quality tradeoffs. My previous open source "crunch" library (used by some game engines and by various geospatial vendors) had higher quality, but also higher bitrates and could only target the older BC1-5 formats (originally).

We didn't announce this ETC1S specific feature when it was first released over a year ago: over time we've slowly moved to real-time filtering and encoding of some ETC1S blocks when targeting some formats like BC7. Developers can disable this via transcoder "decode" flag, but by default we now apply a chroma space filter during transcoding ETC1S.

Here's an example: ETC1S supercompression at 1.5 bpp, 6x pixel zoom, BC7 target. We use a real-time encoder (mode 5 BC7) selectively if doing so will reduce artifacts. The filter works on the CoCg components of the YCoCg colorspace.

Crop 6x pixel zoom, chroma filter enabled:

Crop 6x pixel zoom, chroma filter disabled:

Fitting a neural texture decoder with ES (no backprop)

This is something I prototyped for fun in an evening, and now the idea is Prior Art:

https://github.com/richgel999/neural_texture_es

See GitHub commit f486380. Also see the time stamped X post.

Toy neural texture compression, trained entirely with Evolution Strategies (no backprop, no derivatives). A 128×128×8 latent texture + a 1035-weight MLP decoder reconstruct a 512×512 image (kodim23 crop) at 32.2 dB PSNR. Latent quantizes to 8 bits with only 0.04 dB loss: ~4 bpp raw, ~3.3 bpp entropy coded. Trained in 150 s on CPU. Single C++ file, ~600 lines, stb_image for I/O.

The trick that makes ES viable: each latent texel only affects a small pixel footprint through bilinear filtering, so one full-image decode pair yields a gradient estimate for every texel at once.

Rate/distortion sweep, all 3,000 iters, 8-bit latent:
64×64×4 → 26.9 dB @ 0.47 bpp
64×64×8 → 28.2 dB @ 0.87 bpp
128×128×4 → 30.3 dB @ 1.65 bpp
128×128×8 → 32.2 dB @ 3.27 bpp

Output:

The latent:

Notes:

• No training framework. Dependencies are stb_image, stb_image_write, and OpenMP. The encoder and decoder share the same forward function.
• MLP training: antithetic ES with 32 perturbation pairs per step, each pair evaluated on the same random 4096-pixel minibatch. The estimated gradient goes through Adam.
• Latent training: all 131,072 latent values perturbed at once, two full-image decodes per pair, 4 pairs per step. Each pixel's loss change is credited only to the 4 texels its bilinear tap reads, so every texel gets its own local estimate from one decode pair.
• Quantization is post-hoc.
• Trained for 3,000 iterations, each iteration one ES update of the MLP followed by one ES update of the latent. The curve was still rising slowly at the end; 6000 iterations gained another ~0.3 dB.
• Decoder: 14 → 24 → 24 → 3 MLP, leaky ReLU hidden layers, sigmoid on the RGB output, 1,035 weights. Inputs are the 8 bilinearly sampled latent channels, u, v, and sin/cos of 2πu and 2πv.
• I’m using OpenAI-style ES: Gaussian parameter perturbations with antithetic $+\epsilon/-\epsilon$ evaluations to estimate an update direction.

XUBC7 Low-Level Format Document

GPU texture blocks (ASTC, BC7, ETC1/2, etc.) can serve as a latent representation for a modern intra codec. GPU texture formats aren't merely final compressed bitstreams; they can be treated as intermediate learned-by-hand representations upon which another serious codec can operate. So far we've shipped two latent-space GPU texture codecs in the Basis Universal library: XUASTC LDR and XUBC7.

The low-level format of the XUBC7 v1 codec (which, to my knowledge, is the first DCT-based BC7 codec) is being documented here:

https://github.com/BinomialLLC/basis_universal/wiki/XUBC7-Low-Level-Format

More updates to this document are coming today.

It supports lossless or lossy coding of the entire BC7 format (all modes, all mode features, all partition patterns, all mode options), absolute DCT, residual DCT using dozens of predictors (like a modern image codec such as WebP), threaded decompression, all-integer deterministic decompression, and SIMD is optional (so a good fit for WASM).

XUBC7's lossless bitrate is ~3.5-5.6 bpp, and ~1.2-2.5 bpp lossy. (Plain BC7 is 8.0 bpp.) XUBC7 in lossless mode (relative to BC7) is so effective it can hold its own against lossy RDO BC7+LZ codecs, i.e. without sacrificing any quality. v2 will adaptively use 2x2-4x4 macroblocks, pushing its lossy bitrate much lower (below 1.0 bpp).

Note XUBC7 is also universal: we can transcode the BC7 latent directly to ASTC LDR 4x4's just by converting the BC7 block config to ASTC's (which is way more flexible than BC7's). In those rare cases where the BC7 partition pattern index isn't directly mappable to ASTC, we can fall back to the bc7f real-time analytical encoder running in a mode that outputs ASTC.

In-loop deblocking of GPU textures

I posted this thread on X last night - mirroring it here:

On GPU textures the block lattice is a function of UV + known block size. This makes GPU texture seam artifacts entirely predictable, so filtering them is easy. This applies to all GPU texture formats, but is the most valuable on ASTC which supports up to 12x12 pixel block sizes.

No extra metadata is required. The geometry of GPU texture block artifacts is known before you even look at the compressed bits. It's also easy to apply the seam filter while preserving mipmapping, trilinear filtering, etc.:

https://github.com/BinomialLLC/basis_universal/wiki/Shader-and-In-Loop-Deblocking

For normal maps: you can apply the seam filter (which only kicks in near/on block edges) then renormalize the results.

Encoders can optimize the GPU texture block parameters taking into account the reconstruction operator (the seam filter) during compression (in-loop). We use a form of SCD: Stochastic Coordinate Descent.

Basis Universal v2.50 has shipped with ASTC in-loop deblocking, XUBC7, and universal .DDS transcoding

The documentation is still coming online. This wiki page was just added:

https://github.com/BinomialLLC/basis_universal/wiki/Shader-and-In-Loop-Deblocking

Basis Universal library .DDS support coming in v2.5

The Basis Universal GPU texture library v2.5 will have full .DDS file format support: 2D/cubemaps/texarrays+mips, BC1-7 and uncompressed, are usable on any GPU device/API. It uses near-lossless transcoding to ASTC LDR 4x4, and in cases when it can't do a format latent to latent transcode it falls back to real-time full-featured analytical encoding for other LDR formats using bc7f, etc1f etc. It supports real-time encoding to BC1-7, ETC1, ETC2 (various formats), and PVRTC1 4bpp.

It supports totally standard .DDS files written by AMD Compressonator, Microsoft DirectXTex "texconv" tool, etc. This allows desktop content to be easily deployed on mobile/tablets with low (or even no) loss. (BC1->ASTC 4x4 can be transcoded losslessly in almost all cases, and BC7 mode 6->ASTC 4x4 is near-lossless.)

This doesn't add much code to the existing transcoder because it leverages existing functionality we need anyway.

XUBC7/XBC7: Trellis quantization added

We just added Trellis quantization to XUBC7 (supercompressed universal BC7) via AC truncation, as long as the PSNR drop is within a configurable channel-weighted PSNR window, and the PSNR doesn't fall below a lower limit. This helps at higher Q (DCT quality) levels.

Trellis quant is another video method we've ported successfully into GPU texture supercompression.

https://en.wikipedia.org/wiki/Trellis_quantization

This new encoder also supports several forms of block-level RDO (all optionally) for surprising gains.