docs/DirectX/DXContainer.rst

=================
DirectX Container
=================

.. contents::
   :local:

.. toctree::
   :hidden:

Overview
========

The DirectX Container (DXContainer) file format is the binary file format for
compiled shaders targeting the DirectX runtime. The file format is also called
the DXIL Container or DXBC file format. Because the file format can be used to
include either DXIL or DXBC compiled shaders, the nomenclature in LLVM is simply
DirectX Container.

DirectX Container files are read by the compiler and associated tools as well as
the DirectX runtime, profiling tools and other users. This document serves as a
companion to the implementation in LLVM to more completely document the file
format for its many users.

Basic Structure
===============

A DXContainer file begins with a header, and is then followed by a sequence of
"parts", which are analogous to object file sections. Each part contains a part
header, and some number of bytes of data after the header in a defined format.

DX Container data structures are encoded little-endian in the binary file.

The LLVM versions of all data structures described and/or referenced in this
file are defined in
`llvm/include/llvm/BinaryFormat/DXContainer.h
<https://github.com/llvm/llvm-project/blob/main/llvm/include/llvm/BinaryFormat/DXContainer.h>`_.
Some pseudo code is provided in blocks below to ease understanding of this
document, but reading it with the header available will provide the most
clarity.

File Header
-----------

.. code-block:: c

  struct Header {
    uint8_t Magic[4];
    uint8_t Digest[16];
    uint16_t MajorVersion;
    uint16_t MinorVersion;
    uint32_t FileSize;
    uint32_t PartCount;
  };

The DXContainer header matches the pseudo-definition above. It begins with a
four character code (magic number) with the value ``DXBC`` to denote the file
format.

The ``Digest`` is a 128bit hash digest computed with a proprietary algorithm and
encoded in the binary by the bytecode validator.

The ``MajorVersion`` and ``MinorVersion`` encode the file format version
``1.0``.

The remaining fields encode 32-bit unsigned integers for the file size and
number of parts.

Following the part header is an array of ``PartCount`` 32-bit unsigned integers
specifying the offsets of each part header.

Part Data
---------

.. code-block:: c

  struct PartHeader {
    uint8_t Name[4];
    uint32_t Size;
  }

Each part begins with a part header. A part header includes the 4-character part
name, and a 32-bit unsigned integer specifying the size of the part data. The
part header is followed by ``Size`` bytes of data comprising the part. The
format does not explicitly require 32-bit alignment of parts, although LLVM does
implement this restriction in the writer code (because it's a good idea). The
LLVM object reader code does not assume inputs are correctly aligned to avoid
undefined behavior caused by misaligned inputs generated by other compilers.

Part Formats
============

The part name indicates the format of the part data. There are 24 part headers
used by DXC and FXC. Not all compiled shaders contain all parts. In the list
below parts generated only by DXC are marked with †, and parts generated only by
FXC are marked with \*.

#. `DXIL`_† - Stores the DXIL bytecode.
#. `HASH`_† - Stores the shader MD5 hash.
#. ILDB† - Stores the DXIL bytecode with LLVM Debug Information embedded in the module.
#. ILDN† - Stores shader debug name for external debug information.
#. `ISG1`_ - Stores the input signature for Shader Model 5.1+.
#. ISGN\* - Stores the input signature for Shader Model 4 and earlier.
#. `OSG1`_ - Stores the output signature for Shader Model 5.1+.
#. OSG5\* - Stores the output signature for Shader Model 5.
#. OSGN\* - Stores the output signature for Shader Model 4 and earlier.
#. PCSG\* - Stores the patch constant signature for Shader Model 5.1 and earlier.
#. PDBI† - Stores PDB information.
#. PRIV - Stores arbitrary private data (Not encoded by either FXC or DXC).
#. `PSG1`_ - Stores the patch constant signature for Shader Model 6+.
#. `PSV0`_ - Stores Pipeline State Validation data.
#. RDAT† - Stores Runtime Data.
#. RDEF\* - Stores resource definitions.
#. RTS0 - Stores compiled root signature.
#. `SFI0`_ - Stores shader feature flags.
#. SHDR\* - Stores compiled DXBC bytecode.
#. SHEX\* - Stores compiled DXBC bytecode.
#. DXBC\* - Stores compiled DXBC bytecode.
#. SRCI† - Stores shader source information.
#. STAT† - Stores shader statistics.
#. VERS† - Stores shader compiler version information.

DXIL Part
---------
.. _DXIL:

The DXIL part is comprised of three data structures: the ``ProgramHeader``, the
``BitcodeHeader`` and the bitcode serialized LLVM 3.7 IR Module.

The ``ProgramHeader`` contains the shader model version and pipeline stage
enumeration value. This identifies the target profile of the contained shader
bitcode.

The ``BitcodeHeader`` contains the DXIL version information and refers to the
start of the bitcode data.

HASH Part
---------
.. _HASH:

The HASH part contains a 32-bit unsigned integer with the shader hash flags, and
a 128-bit MD5 hash digest. The flags field can either have the value ``0`` to
indicate no flags, or ``1`` to indicate that the file hash was computed
including the source code that produced the binary.

Program Signature (SG1) Parts
-----------------------------
.. _ISG1:
.. _OSG1:
.. _PSG1:

.. code-block:: c

  struct ProgramSignatureHeader {
    uint32_t ParamCount;
    uint32_t FirstParamOffset;
  }

The program signature parts (ISG1, OSG1, & PSG1) all use the same data
structures to encode inputs, outputs and patch information. The
``ProgramSignatureHeader`` includes two 32-bit unsigned integers to specify the
number of signature parameters and the offset of the first parameter.

Beginning at ``FirstParamOffset`` bytes from the start of the
``ProgramSignatureHeader``, ``ParamCount`` ``ProgramSignatureElement``
structures are written. Following the ``ProgramSignatureElements`` is a string
table of null terminated strings padded to 32-byte alignment. This string table
matches the DWARF string table format as implemented by LLVM.

Each ``ProgramSignatureElement`` encodes a ``NameOffset`` value which specifies
the offset into the string table. A value of ``0`` denotes no name. The offsets
encoded here are from the beginning of the ``ProgramSignatureHeader`` not the
beginning of the string table.

The ``ProgramSignatureElement`` contains several enumeration fields which are
defined in `llvm/include/llvm/BinaryFormat/DXContainerConstants.def <https://github.com/llvm/llvm-project/blob/main/llvm/include/llvm/BinaryFormat/DXContainerConstants.def>`_.
These fields encode the D3D system value, the type of data and its precision
requirements.

PSV0 Part
---------
.. _PSV0:

The Pipeline State Validation data encodes versioned runtime information
structures. These structures use a scheme where in lieu of encoding a version
number, they encode the size of the structure and each new version of the
structure is additive. This allows readers to infer the version of the structure
by comparing the encoded size with the size of known structures. If the encoded
size is larger than any known structure, the largest known structure can validly
parse the data represented in the known structure.

In LLVM we represent the versions of the associated data structures with
versioned namespaces under the ``llvm::dxbc::PSV`` namespace (e.g. ``v0``,
``v1``). Each structure in the ``v0`` namespace is the base version, the
structures in the ``v1`` namespace inherit from the ``v0`` namespace, and the
``v2`` structures inherit from the ``v1`` structures, and so on.

The high-level structure of the PSV data is:

#. ``RuntimeInfo`` structure
#. Resource bindings
#. Signature elements
#. Mask Vectors (Output, Input, InputPatch, PatchOutput)

Immediately following the part header for the PSV0 part is a 32-bit unsigned
integer specifying the size of the ``RuntimeInfo`` structure that follows.

Immediately following the ``RuntimeInfo`` structure is a 32-bit unsigned integer
specifying the number of resource bindings. If the number of resources is
greater than zero, another unsigned 32-bit integer follows to specify the size
of the ``ResourceBindInfo`` structure. This is followed by the specified number
of structures of the specified size (which infers the version of the structure).

For version 0 of the data this ends the part data.

PSV0 Signature Elements
~~~~~~~~~~~~~~~~~~~~~~~

The signature elements are conceptually a single concept but the data is encoded
in three different blocks. The first block is a string table, the second block
is an index table, and the third block is the elements themselves, which in turn
are separeated by input, output and patch constant or primitive elements.

Signature elements capture much of the same data captured in the :ref:`SG1
<ISG1>` parts. The use of an index table allows de-duplciation of data for a more
compact final representation.

The string table begins with a 32-bit unsigned integer specifying the table
size. This string table uses the DXContainer format as implemented in LLVM. This
format prefixes the string table with a null byte so that offset ``0`` is a null
string, and pads to 32-byte alignment.

The index table begins with a 32-bit unsigned integer specifying the size of the
table, and is followed by that many 32-bit unsigned integers representing the
table. The index table may or may not deduplicate repeated sequences (both DXC
and Clang do). The indices signify the indices in the flattened aggregate
representation which the signature element describes. A single semantic may have
more than one entry in this table to denote the different attributes of its
members.

For example given the following code:

.. code-block:: c

  struct VSOut_1
  {
      float4 f3 : VOUT2;
      float3 f4 : VOUT3;
  };


  struct VSOut
  {
      float4 f1 : VOUT0;
      float2 f2[4] : VOUT1;
      VSOut_1 s;
      int4 f5 : VOUT4;
  };

  void main(out VSOut o1 : A) {
  }

The semantic ``A`` gets expanded into 5 output signature elements. Those
elements are:

.. note::

  In the example below, it is a coincidence that the rows match the indices, in
  more complicated examples with multiple semantics this is not the case.

#. Index 0 starts at row 0, contains 4 columns, and is float32. This represents
   ``f1`` in the source.
#. Index 1, 2, 3, and 4 starts at row 1, contains two columns and is float32.
   This represents ``f2`` in the source, and it spreads across rows 1 - 4.
#. Index 5 starts at row 5, contains 4 columns, and is float32. This represents
   ``f3`` in the source.
#. Index 6 starts at row 6, contains 3 columns, and is float32. This represents
   ``f4``.
#. Index 7 starts at row 7, contains 4 columns, and is signed 32-bit integer.
   This represents ``f5`` in the source.

The LLVM ``obj2yaml`` tool can parse this data out of the PSV and present it in
human readable YAML. For the example above it produces the output:

.. code-block:: YAML

  SigOutputElements:
    - Name:            A
      Indices:         [ 0 ]
      StartRow:        0
      Cols:            4
      StartCol:        0
      Allocated:       true
      Kind:            Arbitrary
      ComponentType:   Float32
      Interpolation:   Linear
      DynamicMask:     0x0
      Stream:          0
    - Name:            A
      Indices:         [ 1, 2, 3, 4 ]
      StartRow:        1
      Cols:            2
      StartCol:        0
      Allocated:       true
      Kind:            Arbitrary
      ComponentType:   Float32
      Interpolation:   Linear
      DynamicMask:     0x0
      Stream:          0
    - Name:            A
      Indices:         [ 5 ]
      StartRow:        5
      Cols:            4
      StartCol:        0
      Allocated:       true
      Kind:            Arbitrary
      ComponentType:   Float32
      Interpolation:   Linear
      DynamicMask:     0x0
      Stream:          0
    - Name:            A
      Indices:         [ 6 ]
      StartRow:        6
      Cols:            3
      StartCol:        0
      Allocated:       true
      Kind:            Arbitrary
      ComponentType:   Float32
      Interpolation:   Linear
      DynamicMask:     0x0
      Stream:          0
    - Name:            A
      Indices:         [ 7 ]
      StartRow:        7
      Cols:            4
      StartCol:        0
      Allocated:       true
      Kind:            Arbitrary
      ComponentType:   SInt32
      Interpolation:   Constant
      DynamicMask:     0x0
      Stream:          0

The number of signature elements of each type is encoded in the
``llvm::dxbc::PSV::v1::RuntimeInfo`` structure. If any of the element count
values are non-zero, the size of the ``ProgramSignatureElement`` structure is
encoded next to allow versioning of that structure. Today there is only one
version. Following the size field is the specified number of signature elements
in the order input, output, then patch constant or primitive.

Following the signature elements is a sequence of mask vectors encoded as a
series of 32-bit integers. Each 32-bit integer in the mask encodes values for 8
input/output/patch or primitive elements. The mask vector is filled from least
significant bit to most significant bit with each added element shifting the
previous elements left. A reader needs to consult the total number of vectors
encoded in the ``RuntimeInfo`` structure to know how to read the mask vector.

If the shader has ``UsesViewID`` enabled in the ``RuntimeInfo`` an output mask
vector will be included. The output mask vector is four arrays of 32-bit
unsigned integers. Each of the four arrays corresponds to an output stream.
Geometry shaders have a maximum of four output streams, all other shader stages
only support one output stream. Each bit in the mask vector identifies one
column of an output from the output signature depends on the ViewID.

If the shader has ``UsesViewID`` enabled, it is a hull shader, and it has patch
constant or primitive vector elements, a patch constant or primitive vector mask
will be included. It is identical in structure to the output mask vector. Each
bit in the mask vector identifies one column of a patch constant output which
depends on the ViewID.

The next series of mask vectors are similar in structure to the output mask
vector, but they contain an extra dimension.

The output/input map is encoded next if the shader has inputs and outputs. The
output/input mask encodes which outputs are impacted by each column of each
input. The size for each mask vector is the size of the output max vector * the
number of inputs * 4 (for each component). Each bit in the mask vector
identifies one column of an output and a column of an input. A value of 1 means
the output is impacted by the input.

If the shader is a hull shader, and it has inputs and patch outputs, an input to
patch map will be included next. This is identical in structure to the
output/input map. The dimensions are defined by the size of the patch constant
or primitive vector mask * the number of inputs * 4 (for each component). Each
bit in the mask vector identifies one column of a patch constant output and a
column of an input. A value of 1 means the output is impacted by the input.

If the shader is a domain shader, and it has outputs and patch outputs, an
output patch map will be included next. This is identical in structure to the
output/input map. The dimensions are defined by the size of the patch constant
or primitive vector mask * the number of outputs * 4 (for each component). Each
bit in the mask vector identifies one column of a patch constant input and a
column of an output. A value of 1 means the output is impacted by the primitive
input.

SFI0 Part
---------
.. _SFI0:

The SFI0 part encodes a 64-bit unsigned integer bitmask of the feature flags.
This denotes which optional features the shader requires. The flag values are
defined in `llvm/include/llvm/BinaryFormat/DXContainerConstants.def <https://github.com/llvm/llvm-project/blob/main/llvm/include/llvm/BinaryFormat/DXContainerConstants.def>`_.