I've been tuning the fixed Deflate model in bc7enc_rdo. In this test I varied the # of literal bits from 8 to 14. Higher values push the system to prefer matches vs. literals.
The orange line was yesterday's encoder, all other lines are for today's encoder. Today's encoder has several improvements, such as lazy parsing and mode 6 endpoint match trails.
Some minor observations about Lagrangian multiplier based RDO (with BC7 RDO+Deflate or LZ4):
We're optimizing to find lowest t (sometimes called j), given many hundreds/thousands of ways of encoding a BC7 block:
float t = trial_mse * smooth_block_error_scale + trial_lz_bits * lambda;
For each trial block, we compute its MSE and estimate its LZ bits using a simple Deflate/LZ4-like model.
If we already have a potential solution for a block (the best found so far), given the trial block's MSE and the current best_t we can compute how many bits (maximum) a new trial encoding would take to be an improvement. If the number of computed threshold bits is ridiculous (like negative, or just impossible to achieve with Deflate on a 128-bit block input), we can immediately throw out that trial block:
threshold_trial_lz_bits = (best_t - trial_mse * smooth_block_error_scale ) / lambda
Same for MSE: if we already have a solution, we can compute the MSE threshold where it's impossible for a trial to be an improvement:
threshold_trial_mse = (best_t - (trial_lz_bits * lambda)) / smooth_block_error_scale
This seems less valuable because running the LZ simulator to compute trial_lz_bits is likely more expensive than computing a trial block's MSE. We could plug in a lowest possible estimate for trial_lz_bits, and use that as a threshold MSE. Another interesting thing about this: trials are very likely to always have an MSE >= than the best found encoding for a block.
Using simple formulas like this results in large perf. improvements (~2x).
"The output is fv.bmp with the given size in pixels, which visually displays where matching substrings of various lengths and offsets are found. A pixel at x, y is (black, red, green, blue) if the last matching substring of length (1, 2, 4, 8) at x occurred y bytes ago. x and y are scaled so that the image dimensions match the file length. The y axis is scaled log base 10."Tool source:
It's relatively easy to reduce the output entropy of BC7 by around 5-10%, without slowing down encoding or even speeding it up. I'll be adding this stuff to the bc7e ispc encoder soon. I've been testing these tricks in bc7enc_rdo:
- Weight the mode errors: For example weight mode 1 and 6's errors way lower than the other modes. This shifts the encoder to use modes 1 and 6 more often, which reduces the output data's entropy. This requires the other modes to make a truly significant difference in reducing distortion before the encoder switches to using them.
- Biased p-bits: When deciding which p-bits to use (0 vs. 1), weight the error from using p-bit 1 slightly lower (or the opposite). This will cause the encoder to favor one of the p-bits more than the other, reducing the block output data's entropy.
- Partition pattern weighting: Weight the error from using the lower frequency partitions [0,15] or [0,33] slightly lower vs. the other patterns. This reduces the output entropy of the first or second byte of BC7 modes with partitions.
- Quantize mode 6's endpoints and force its p-bits to [0,1]: Mode 6 uses 7-bit endpoint components. Use 6-bits instead, with fixed [0,1] p-bits. You'll need to do this in combination with reducing mode 6's error weight, or a multi-mode encoder won't use mode 6 as much.
- Don't use mode 4/5 component rotations, or the index flag.
In practice these options aren't particularly useful, and just increase the output entropy. The component rotation feature can also cause odd looking color artifacts.
- Don't use mode 0,2,3, possibly 4: These modes are less useful, at least on albedo/specular/etc. maps, sRGB content, and photos/images. Almost all BC7 encoders, including ispc_texcomp's, can't even handle mode 0 correctly anyway.
Mode 4 is useful on decorrelated alpha. If your content doesn't have much of that, just always use mode 5.
