Download Times Matter
Sean Cooper, Ryan Inselmann and I have been building a custom lossless archiver designed specifically for Unity asset bundle files. The archiver itself uses several well-known techniques in the hardcore archiving/game repacking world. This post is mostly about how we've begun to tune the LZMA settings used by the archiver to be more effective on Unity asset bundle data. We'll cover the actual archiver in a later post.
LZMA has several knobs you can turn to potentially improve compression:
http://stackoverflow.com/questions/3057171/lzma-compression-settings-details
The LZHAM codec (my faster to decode alternative to LZMA) isn't easily tunable in the same way as LZMA yet, which is a flaw. I totally regret this because tuning these options works:
Total asset bundle asset data (iOS): 197,514,230
http://stackoverflow.com/questions/3057171/lzma-compression-settings-details
The LZHAM codec (my faster to decode alternative to LZMA) isn't easily tunable in the same way as LZMA yet, which is a flaw. I totally regret this because tuning these options works:
Total asset bundle asset data (iOS): 197,514,230
Unity's built-in compression: 91,692,327
Our archiver, un-tuned LZMA: 76,049,530Our archiver, tuned LZMA (48 option trials): 75,231,764
Our archiver, tuned LZMA (225 option trials): 74,154,817
LZ codecs with untunable models/settings are much less interesting to me now. (Yet one more thing to work on in LZHAM.)
Here are the optimal settings we've found for each of our Unity asset classes on iOS. So for example, textures are compressed using different LZMA options (3, 3, 3) vs. animation clips (1, 2, 2).
Best LZMA settings after trying all 225 options. (In case of ties the compressor just selects the lowest lc,lp,pb settings - I just placed a triply nested for() loop around calling LZMA and it chooses the first best as the "best".)
Class (id): lc lp pb
GameObject (1): 8 0 0
Light (108): 2 2 2
Animation (111): 8 2 2
MonoScript (115): 2 0 0
LineRenderer (120): 0 0 0
SphereCollider (135): 0 0 0
SkinnedMeshRenderer (137): 8 4 2
AssetBundle (142): 0 2 2
WindZone (182): 0 0 0
ParticleSystem (198): 0 2 3
ParticleSystemRenderer (199): 8 3 3
Camera (20): 0 2 2
Material (21): 3 0 1
SpriteRenderer (212): 0 0 0
MeshRenderer (23): 8 4 2
Texture2D (28): 8 2 3
MeshFilter (33): 8 4 1
Transform (4): 6 2 2
Mesh (43): 0 0 1
MeshCollider (64): 1 4 1
BoxCollider (65): 7 4 2
AnimationClip (74): 0 2 2
AudioSource (82): 0 0 0
AudioClip (83): 8 0 0
Avatar (90): 2 0 2
AnimatorController (91): 2 0 2
? (95): 7 4 1
TrailRenderer (96): 0 0 0
Sorry, I missed your comment in my email stream:
ReplyDeleteKarl Schmidt has left a new comment on your post "Improved Unity asset bundle file compression":
"I'm curious what got you started down the path of looking to improve the compression of Unity asset bundles. I can understand the curiosity, craftsmanship, challenge and even passion - but I know if I had ever thought about it, my thinking would be something like "well the people at Unity are super smart so I'm sure it's probably close to being as good as it could be. Will delving into this help me ship my game faster or at a significantly higher quality bar? No, ok on to the next thing." I'm very interested to learn about your line of thinking on this. I'm also very excited to watch the progress of this.."
Instead of legos I work on codecs for the enjoyment of it. There's a lot of practical computer science, algorithms, and optimization involved so it strikes at the core of why I love programming. I've been working on compression since around 1991, it's one of my specialties and passions. I've got codecs shipped in the AoE, Forza, Halo, Titanfall, and the Planetside series, which I think is cool for something that is mostly a hobby.
Lossless codecs are a niche with relatively little competition (and few patent worries relative to video/audio), and the end results are universally applicable to many shipping products outside of games. LZMA is too slow, expensive, and power hungry for use on many of the devices I want to ship games on which is why I'm targeting it, otherwise it's a great codec.
About Unity: I've done an in-depth investigation into Unity's built-in asset bundle compression system and a lot of improvements are possible in v4.6. They've made some good improvements for the next major release, but we're shipping with v4.6 and it's unlikely we'll be switching to v5.0 soon.
I'm at a company making products in a space where app store and in-game download sizes and times actually matters to the bottom line. In our alpha, we have data collected from thousands of customers in the field showing we've been losing ~30% of our customers in our app's loading and download screens, especially with customers on slow connections (~500kbps or so). We've been optimizing every aspect of our app that reduces retention, and switching to a codec+bundle compression system that allows faster downloads, and one that doesn't incur a ~35 second CPU cost to decompress our game data is valuable to us. (We're also adding streaming, and we can't afford to bog down the 1-2 CPU's we have with LZMA, but that's another story.)