Wednesday, April 15, 2020

.basis file format specification

[This is a work in progress, and the formatting isn't ideal. It will be copied & pasted into the Basis Universal wiki once it's done, and deleted from here.]

The Basis Universal GPU texture codec supports reading and writing ".basis" files. Currently the file format supports ETC1S or UASTC 4x4 texture data:

  • ETC1S is a simplified subset of ETC1.

The mode is always differential (diff bit=1), the Rd, Gd, and Bd color deltas are always (0,0,0), and the flip bit is always set. ETC1S texture data is fully 100% compliant with all existing software and hardware ETC1 decoders. Existing encoders can be easily modified to limit their output to ETC1S.

  • UASTC 4x4 is a 19 mode subset of the ASTC texture format. Its specification is here. UASTC texture data can always be losslessly transcoded to ASTC.

At a high level, a typical .basis file consists of multiple sections:

  • The file header
  • Optional ETC1S compressed endpoint/selector codebooks
  • Optional ETC1S Huffman table information
  • A required "slice" description array describing the resolutions and file offset/compressed sizes of each texture slice present in the file
  • 1 or more slices containing ETC1S or UASTC compressed texture data. 
  • For future expansion, the format supports an "extended" header which may be located anywhere in the file. This section contains .PNG-like chunked data. 

Apart from the header, which must always be present at the start of the file, the other sections can appear in any order.


Enums

// basis_file_header::m_tex_type
enum basis_texture_type
{
  cBASISTexType2D = 0,
  cBASISTexType2DArray = 1,
  cBASISTexTypeCubemapArray = 2,
  cBASISTexTypeVideoFrames = 3,
  cBASISTexTypeVolume = 4,
  cBASISTexTypeTotal
};

// basis_slice_desc::flags
enum basis_slice_desc_flags
{
  cSliceDescFlagsHasAlpha = 1,
  cSliceDescFlagsFrameIsIFrame = 2
};

// basis_file_header::m_tex_format enum basis_tex_format
{
  cETC1S = 0,
  cUASTC4x4 = 1
};

// basis_file_header::m_flags enum basis_header_flags
{
  cBASISHeaderFlagETC1S = 1.
  cBASISHeaderFlagYFlipped = 2,
  cBASISHeaderFlagHasAlphaSlices = 4
};



File Structures


All individual values in all file structures are byte aligned and always little endian. The structs have no padding (i.e. they are declared with #pragma pack(1)).


Struct basis_file_header

The file header must always be at the beginning of the file.

struct basis_file_header
{
  uint16      m_sig;              // 2 byte file signature
  uint16      m_ver;              // File version
  uint16      m_header_size;      // Header size in bytes, sizeof(basis_file_header) or 0x4D
  uint16      m_header_crc16;     // CRC16/genibus of the remaining header data

  uint32      m_data_size;        // The total size of all data after the header
  uint16      m_data_crc16;       // The CRC16 of all data after the header

  uint24      m_total_slices;     // The number of compressed slices 
  uint24      m_total_images;     // The total # of images
          
  byte        m_tex_format;       // enum basis_tex_format
  uint16      m_flags;            // enum basis_header_flags
  byte        m_tex_type;         // enum basis_texture_type
  uint24      m_us_per_frame;     // Video: microseconds per frame

  uint32      m_reserved;         // For future use
  uint32      m_userdata0;        // For client use
  uint32      m_userdata1;        // For client use

  uint16      m_total_endpoints;          // ETC1S: The number of endpoints in the endpoint codebook 
  uint32      m_endpoint_cb_file_ofs;     // ETC1S: The compressed endpoint codebook's file offset relative to the header
  uint24      m_endpoint_cb_file_size;    // ETC1S: The compressed endpoint codebook's size in bytes

  uint16      m_total_selectors;          // ETC1S: The number of selectors in the selector codebook 
  uint32      m_selector_cb_file_ofs;     // ETC1S: The compressed selector codebook's file offset relative to the header
  uint24      m_selector_cb_file_size;    // ETC1S: The compressed selector codebook's size in bytes

  uint32      m_tables_file_ofs;          // ETC1S: The file offset of the compressed Huffman codelength tables.
  uint32      m_tables_file_size;         // ETC1S: The file size in bytes of the compressed Huffman codelength tables.

  uint32      m_slice_desc_file_ofs;      // The file offset to the slice description array, usually follows the header

  uint32      m_extended_file_ofs;        // The file offset of the "extended" header and compressed data, for future use
  uint32      m_extended_file_size;       // The file size in bytes of the "extended" header and compressed data, for future use
};


Details:
  • m_sig is always 'B' * 256 + 's', or 0x4273.
  • m_ver is currently always 0x10.
  • m_header_size is sizeof(basis_file_header). It's always 0x4D.
  • m_header_crc16 is the CRC-16 of the remaining header data. See the "CRC-16" section for more information.
  • m_data_size, m_data_crc16: The size of all data following the header, and its CRC-16.
  • m_total_slices: The total number of slices, from [1,2^24-1]
  • m_total_images: The total number of images (where one image can contain multiple mipmap levels, and each mipmap level is a different slice).
  • m_tex_format: basis_tex_format. Either cETC1S (0), or cUASTC4x4 (1).
  • m_flags: A combination of flags from the basis_header_flags enum.
  • m_tex_type: The texture type, from enum basis_texture_type
  • m_us_per_frame: Microseconds per frame, only valid for cBASISTexTypeVideoFrames texture types.
  • m_total_endpoints, m_endpoint_cb_file_ofs, m_endpoint_cb_file_size: Information about the compressed ETC1S endpoint codebook: The total # of entries, the offset to the compressed data, and the compressed data's size.
  • m_total_selectors, m_selector_cb_file_ofs, m_selector_cb_file_size: Information about the compressed ETC1S selector codebook: The total # of entries, the offset to the compressed data, and the compressed data's size.
  • m_tables_file_ofs, m_tables_file_size: The file offset and size of the compressed Huffman tables for ETC1S format files. 
  • m_slice_desc_file_ofs: 
    The file offset to the array of slice description structures. There will be m_total_slices structures at this file offset.
  • m_extended_file_ofs, m_extended_file_size: The "extended" header, for future expansion. Currently unused.


Struct basis_slice_desc

struct basis_slice_desc
{
    uint24 m_image_index;  
    uint8 m_level_index;   
    uint8 m_flags;         

    uint16 m_orig_width;   
    uint16 m_orig_height;  

    uint16 m_num_blocks_x; 
    uint16 m_num_blocks_y; 

    uint32 m_file_ofs;     
    uint32 m_file_size;    

    uint16 m_slice_data_crc16; 
};


Details:
  • m_image_index: The index of the source image provided to the encoder (will always appear in order from first to last, first image index is 0, no skipping allowed)
  • m_level_index: The mipmap level index (mipmaps will always appear from largest to smallest)
  • m_flags: enum basis_slice_desc_flags
  • m_orig_width: The original image width (may not be a multiple of 4 pixels)
  • m_orig_height: The original image height (may not be a multiple of 4 pixels)
  • m_num_blocks_x: The slice's block X dimensions. Each block is 4x4 pixels. The slice's pixel resolution may or may not be a power of 2.
  • m_num_blocks_y: The slice's block Y dimensions. 
  • m_file_ofs: Offset from the header to the start of the slice's data
  • m_file_size: The size of the compressed slice data in bytes
  • m_slice_data_crc16: The CRC16 of the compressed slice data, for extra-paranoid use cases


CRC-16 Function

.basis files use CRC-16/genibus(aka CRC-16 EPC, CRC-16 I-CODE, CRC-16 DARC) format CRC-16's. Here's an example function in C++:

uint16_t crc16(const void* r, size_t size, uint16_t crc)
{
  crc = ~crc;
  const uint8_t* p = static_cast<const uint8_t*>(r);
  for ( ; size; --size)
  {
    const uint16_t q = *p++ ^ (crc >> 8);
    uint16_t k = (q >> 4) ^ q;
    crc = (((crc << 8) ^ k) ^ (k << 5)) ^ (k << 12);
  }
  return static_cast<uint16_t>(~crc);
}

This function is called with 0 in the final "crc" parameter when computing CRC-16's of file data.


No comments:

Post a Comment