Binary Data Definition Language

• It must be non-procedural, simple and human readable.
• User must be able to describe virtually any file format.
• Language must support any data structure you can possibly meet in a file.

MiraplacidBinaryDataDefinitionFileInfo {
version <version_number>;
credits <text>;
license <text>;
comments <text>;
}

<annotation>
document | struct <name> {
<item>
:
<item>
 [AcceptRule = <expression>)];
}

<annotation> : '[' name[=["]value["]] , name[=["]value["]] , : ']'

• helpstring. Defines help string for schema object. Can be defined for structures, documents and items. Binary Viewer uses it for tooltips.
• mime_type. Defines mime type for a document. Helps to recognize document type. Value string contains mime type values separated by comma.
• extension. Defines extensions for a document. Helps to recognize document type. Value string contains extension values (without leading dot, i.e. "jpg" rather than ".jpg") separated by comma.
• alterview. Value contains a name of view object which will be used as an alternate view. Can be used with structures, documents and items.
• pointer. Value contains a name directory used in directories and pointers mechanism. Can be used with single numeric data items only.
• dataheap. This annotation marks a data item used as a data source in directories and pointers mechanism. Can be used with structures, documents and items.

<annotation>
<type>['['dimension']'] name [optional implicit allowempty restrictions {expression}];

• uint8 or Byte - byte
• int8 or SByte - sbyte
• uint16 or UInt16 - ushort
• int16 or Int16 - short
• uint32 or UInt32 - uint
• int32 Int32 - int
• uint64 or UInt64 - ulong big endian
• int64 or Int64 - long big endian
• uint16be or UInt16BE - ushort big endian
• int16be or Int16BE - short big endian
• uint32be Uint32BE - uint big endian
• int32be or Int32BE - int big endian
• uint64be or UInt64BE - ulong big endian
• int64be or Int64BE - long big endian
• char or Char - 8-bit character (ANSI)
• wchar or UChar - 16-bit character (Unicode)
• float or Float - single
• double or Double - double

(type1|type2|type3)

• Efficient implementation of enumerated type items. If some type in enumeration does not have an AcceptRule, it will be accepted and further variants will not be considered.
• Building arrays of structures. Array continues as long as each parsed element is accepted. As soon as some array element is not accepted, array is finished and parser / validator will proceed with the next item in data definition.
• Building arrays of enumerated types. Mix of the options listed above. This array will end when no variants in enumeration can be accepted in the current array element.
• File validation and other validation purposes - for example, to ensure that the "magic" byte in the file header is really 0xAA.

namespace <name> {
    type DotNet|COM|DLL;
    location <location>;
}

dictionary <name> {
key=value,
...
key=value
}

view <name> {
source <source_structure_name>;
type <item_type>[<dimension_expression>];
ContentEncoding <enconding_handler_specification>;
ConversionRule <rule>;
}

namespace::name

or 

namespace

1. Execute ConversionRule and get the result. The system assumes that result will be a collection of descendant elements of current node. When BinaryDOM creates an alternate view, the system connects it to the parent of current node. So, it is possible to use almost any axes in expressions.
   If ConversionRule is not defined for the view, the result is current node itself.
2. Convert the result of previous step to a Stream, call filter plug-in and store result of its work in a Stream.
   If this step is not present, the resulting Stream is a stream generated from a previous result directly.
3. Create a BinNode described as type and initialize it from the resulting Stream from previous step.

• When document has a very complex structure, and for your practical purposes it is not necessary to parse all the structure deep down.
• When document contains a whole another document inside. It may be convenient to parse internal document as an array of bytes and convert this array to inner document only when necessary. For example, Windows spool file contains EMF files as images of its pages.
• When document contain a structure (array) which can be represented as an array of structures, and number of these structures is undefined - up to the end of enclosing item.
  In this case, only alterview mechanism will allow to parse the file correctly and naturally limiting the inner array.
  Since array of structures disallowed to be a type in a view, we have to wrap structures array into another structure.
  Example of such a tricky file format is AVI.
• When file has two structural levels: outer level is a set of transport-oriented data, wrapped around inner set of structures with document content. Reader of such formats must receive "transport" packets, check CRC, combine contents of the document, decode the document and do something with it.
  With a view, we can write a ConversionRule which will collect collection of data arrays refined from outer headers, footers and CRC codes. Our document content is scattered in these arrays. Then, we can use decoding filter plug-in (ContentEncoding) which will decode collected content. If document format is known, and alternate view with this document format is added to the binary data definition (.bdd) file, decoded document can be parsed just like if it would loaded from a hard drive.
  A good example of a nested format of this kind is OGG audio file.

[pointer=MainPointer]
uint32 ptr;

directory <name> {
PointerRule <rule>
type <item_type>[<dimension_expression>];
AcceptRule <rule>
}

• If we have raw data area which should be converted to a set of various data structures. This area shall be declared as an array of bytes unless there is a strong reason to use some other data type.
  Parser will convert this array into a set of data structures according to types provided by directory specifications. Pointers are allowed to point inside of the array.
  Let's say, pointer points to byte 100, and expect to find structure of size 50 there. Parser will split source array into 3 elements: byte array of size 100, structure of size 50, and byte array with all the bytes after 150. If directory-based document does not contain "holes", all automatically created arrays will be converted into specific data structures.
  Structures created during conversion process may contain pointers and data heaps. As soon as parser knows where they are and wat is their type, it can process them.
• If we need to convert some common type structure or primitive type nodes into a different data structure.
  This is type refinement driven by pointers which allows refining data types with help of directories. Pointers are not allowed to point inside these nodes and refined nodes must have the same size as original nodes.
  New data structures are allowed to have pointers and data heaps.

[dataheap]
uint8[*] body;

or

[dataheap]
struct Sector {
...
}

struct SomeOuterStructure {
   [alterview=SomeStructure]
   uint8[31] varArray;
   uint8 count;
}

view SomeStructure {
   type SomeStructure;
}

struct SomeStructure {
   uint8[parent.count] array1;
   uint8[31 - sizeof(array1)] filler;
}

include <filename>

• Language structures syntax
• Existence of referenced structures, documents, views, namespaces
• Validity of BinPath expressions
• Validity of reference paths elements used in expressions, and validity of function arguments

• If some structure contains only single nested structure, and the inner structure is implicit, items of the inner structure will be included into outer structure. AcceprRules of these structures will be united with && operation. This is a kind of inheritance.
  Please note, that you must use in AcceptRule inner structure item names (references) as if they are items of outer structure.
• In addition, if inner structure contains only one item with restriction rule, and item containing this structure has its own restrictions, these rules will be united with && operation.
• If one structure includes another structure, and the inner structure is implicit and has no AcceptRules, items of the inner structure will be included into the outer structure.

• Don't over-use implicit attribute. It might slow down schema parsing process (if this implicit status will not be optimized during compilation process), and you might lose important metainformation. For example, structure name and char[*] tells us more than just char[*] string.
• Use mandatory attribute only when you have a very regular binary structure and you must have all items despite the fact that empty arrays and structures are allowed in that format.
• Try to avoid items with enumerated types and arrays of enums. Use these constructions only if absolutely any value from the enumeration could appear in any order. Otherwise, try to make the data definition more specific.
• If some structure item is marked as optional and it was rejected for some reason, parser/validator will try to fill and validate the next schema item with the same data.
• Structures/documents will be rejected if:
  • It did not pass AcceptRule expression check
  • One or more non-optional items rejected, or missed when end-of-file occurred
  • Structure/document contains no items and it is not marked as mandatory
  • Restrictions rule failed in item representing this structure/document
  Usually, you don't have to verify each nested structure with AcceptRule. Some structures cannot be verified, and that's Ok as long as you can verify the outer structure. The outer structure will be rejected as soon as any its non-optional item is rejected.
• Enumeration item will be rejected only after all the variants do not pass verification against the current data.
• Array of primitive types will be rejected if it defined with non-unlimited length and actual length is not equal to the calculated length (end of file occurred or restrictions rule failed on some element), or if array is empty and it is not marked as mandatory.
• Primitive types will be rejected if end of file occurred or restrictions rule failed.
• If you use enums, take into account that parser will try types from left to right in the list of enumeration. You got to place types in enumeration in the correct order, typically, from more strict types (i.e., with stricter AcceptRule) to less strict. For example, PDF file might contain Dictionary structure beginning with "<<" or HexString structure beginning from "<". It is obvious that in enumeration Dictionary must placed before HexString. Type with no restrictions should be placed last.
• Arrays of structures or enums of unlimited length [*] will end when the next array element will be rejected. Parser/validator will try to fill/analyse the next item after that array. If the array is empty and it is not mandatory, an error will be thrown. However, if the array contains at least one element, array will be successfully accepted and parser will try the next item.
  
  How to troubleshoot a data definition if it fails on or around arrays of structures or enums?
  Here are some tips:
  • Use as small document as possible, even create a very simple artificial one to isolate the problem.
  • If you have an unlimited array in your data definition file, try to temporarily replace it with fixed-size array, where array size matched the data in your data file, and see whether it helps.
  • Transform your array or enumeration array into a set of specific structures included into enclosing structure, to match your data file, and see whether it helps.
• When you use unlimited arrays, always think how will they end. For example, unlimited array of bytes will be read up to the end of file and will be accepted successfully. For arrays of structures and enums, the limit is AcceptRule and restrictions rule aplied for each item.
  For arrays of primitive types, this is a restrictions rule applied for each item.
• If your structure has AcceptRule, it will not be accepted until AcceptRule executed and succeeded. AcceptRule will be executed when the last item which referenced in the expression will be filled with data.
  For example, your structure is included into enumeration and has several items (for example, the first and the last one) in AcceptRule. So, parser will pump data into this structure up to the last item, and here AcceptRule rejects the structure and parser will try the next enum variant. Although, it was possible to reject the structure from the first item. In some cases like the one described above, it makes sense to let parser reject variants as early as possible - just apply the restriction rules to the first item and exclude this condition from structure's AcceptRule.
• When working with textual data, if possible, use iswhitepsace(), isletter(), etc functions instead of "manual" comparison.
• If function has default parameters, and they are what you need, do not pass arguments.
• If document structure defined with pointers to some location, it does not necessarily mean that you have to use pointers in binary data definition file. Sometimes they are provided only for convenience, and the document itself is quite regular. Try to avoid using pointers, because they will cause an unnecessary second parsing pass over the document data.

• About Miraplacid Binary and Text DOM Library
• Binary Data Definition Language
• BinPath Expression Language
• Text Data Definition Language
• BinaryDOM and TextDOM API
• Binary Schema Object Model (BinSOM) classes
• Binary Document Object Model (BinDOM) classes
• BinPath classes
• Binary Top-Level classes
• Text Schema Object Model (TextSOM) classes
• Text Document Object Model (TextDOM) classes
• Text Navigation classes
• Text Top-Level classes
• BinaryDOM / TextDOM Common classes
• Low-Level classes
• Extension Components
• BinaryDOM PowerShell Snapin
• BinaryDOM SDK examples
• TextDOM SDK examples
• Miraplacid Binary DOM SDK Main Page