Existing BC7 codecs don't handle textures with decorrelated alpha channels well

Developers aren't getting the alpha quality they could be getting if they had better BC7 codecs. I noticed while working on our new non-RDO BC7 codec that existing BC7 codecs don't handle textures with decorrelated alpha signals well. They wind up trashing the alpha channel when the A signal doesn't resemble the signal in RGB. I didn't have time to investigate the issue until now. I'm guessing most developers either don't care, or they use simple (correlated) alpha channels, or multiple textures.

Some codecs allow the user to specify individual RGBA channel weightings. (ispc_texcomp isn't one of them.) This doesn't work well in practice, and users will rarely fiddle with the weightings anyway. You have to weight A so highly that RGB gets trashed.

Here's an example using a well-known CPU BC7 codec:

RGB image: kodim18
Alpha image: kodim17

Encoded using Intel's ispc_texcomp to BC7 profile alpha_slow:

RGB Average Error: Max:  40, Mean: 1.868, MSE: 7.456, RMSE: 2.731, PSNR: 39.406
Luma        Error: Max:  26, Mean: 1.334, MSE: 3.754, RMSE: 1.938, PSNR: 42.386
Alpha       Error: Max:  36, Mean: 1.932, MSE: 7.572, RMSE: 2.752, PSNR: 39.339

Encoded RGB:

Encoded A:

Experimental RDO BC7 codec (quantization disabled) with support for decorrelated alpha. Uses only modes 4, 6, and 7:

M4: 72.432457%, M6: 17.028809%, M7: 10.538737%
RGB Average Error: Max:  65, Mean: 2.031, MSE: 8.871, RMSE: 2.978, PSNR: 38.651
Luma        Error: Max:  34, Mean: 1.502, MSE: 4.887, RMSE: 2.211, PSNR: 41.241
Alpha       Error: Max:  29, Mean: 1.601, MSE: 5.703, RMSE: 2.388, PSNR: 40.570

Encoded RGB:

Encoded A:

Zoomed in comparison:

This experimental codec separately measures per-block RGB average and alpha PSNR. It prefers mode 4, and switches to modes 6 or 7 using this logic:

const float M7_RGB_THRESHOLD = 1.0f;
const float M7_A_THRESHOLD = 40.0f;
const float M7_A_DERATING = 12.0f;

const float M6_RGB_THRESHOLD = 1.0f;
const float M6_A_THRESHOLD = 40.0f;
const float M6_A_DERATING = 7.0f;

if ((m6_rgb_psnr > (math::maximum(m4_rgb_psnr, m7_rgb_psnr) + M6_RGB_THRESHOLD)) && (m6_a_psnr > M6_A_THRESHOLD) && (m6_a_psnr > math::maximum(m4_a_psnr, m7_a_psnr) - M6_A_DERATING))
{
block_modes[block_index] = 6;
}
else if ((m7_rgb_psnr > (m4_rgb_psnr + M7_RGB_THRESHOLD)) && (m7_a_psnr > (m4_a_psnr - M7_A_DERATING)) && (m7_a_psnr > M7_A_THRESHOLD))
{
block_modes[block_index] = 7;
}
else 
{
block_modes[block_index] = 4;
}

What this basically does: we only use mode 6 or 7 when the RGB PSNR is greater than mode 4's RGB PSNR plus a threshold (1 dB). But we only do this if we don't degrade alpha quality too much (either 12 dB or 7 dB), and if alpha quality is above a minimum threshold (40 dB).

PSNR doesn't really capture the extra distortion being introduced here. It can help to load the alpha images into Photoshop or whatever and compare them closely as layers.

RGB PSNR went down a little with my experimental RDO BC7 codec, but alpha went up. Visually, alpha is greatly improved. My ispc non-RDO BC7 code currently has the same issue as Intel's ispc_texcomp with decorrelated alpha.