OPALS Format Definition

When importing/exporting data from/to files, OPALS uses certain default values for e.g. the LAS file format version on export. To override these defaults, users must specify an OPALS Format Definition (OFD). For the import/export of generic text and binary formats, the specification of an OFD is mandatory in any case. For exemplary OFD usage, see e.g. the import, export LAS, and export text examples. Please note that Module Info can be used to generate specific OFD files based on given ODM or vector files (see section Generation of OPALS Format Definition (OFD) and the corresponding example for more details).

OFD as Parameter

There are two ways to pass an OFD as option value to OPALS modules:

  1. As file path. The file contents must be XML that adheres to the OFD XML Schema . Examples can be found in $OPALS_ROOT/addons/formatdef/*.xml.
  2. As inline XML text. Before the text is validated against the OFD XML Schema , it is enclosed in an <opalsFormatDefinition> element, and abbreviations of element and attribute names are resolved in a case insensitive manner. This allows for a very compact specification.

Supported File Formats

OFDs must specify exactly one of the following file formats:

Supported File Formats
File Format XML element tag
Generic Text <text>
Generic Binary <binary>
LAS <las>
Riegl RDB <rdb>
Esri shapefile <shape>

Supported Data Types

As described in this table, the OPALS Datamanager supports various data types. When specifying data types in OFD files, entries in its column "Name" are to be used. The same names must be used when explicitly defining the data types of custom attributes with the Generic filter's assignment operator.

OFD XML Schema

OFD file contents must adhere to the OFD XML Schema below, which can be found at $OPALS_ROOT/addons/formatdef/opalsFormatDefinition_1_2.xsd .

<!--
OPALS format definition schema.
Do not edit!
For examples that conform to this schema, have a look at the format definitions that come with OPALS.
-->
<xs:schema xmlns:xs='http://www.w3.org/2001/XMLSchema'>
<!-- Definition of various simple types -->
<!-- Strings supported by OPALS for coordinates and attributes.
Starts with a letter or '_', followed by zero or more arbitrary characters (including space).
Must not end with a space character.
Must not contain newlines or carriage returns. -->
<xs:simpleType name='Name'>
<xs:restriction base='xs:string'>
<xs:pattern value='[a-zA-Z_](.*\S)?' />
</xs:restriction>
</xs:simpleType>
<!-- Data types supported by OPALS for coordinates and attributes. -->
<xs:simpleType name='Type'>
<xs:union>
<xs:simpleType>
<xs:restriction base='xs:string'>
<xs:enumeration value='bool'/>
<xs:enumeration value='int8'/>
<xs:enumeration value='uint8'/>
<xs:enumeration value='int16'/>
<xs:enumeration value='uint16'/>
<xs:enumeration value='int32'/>
<xs:enumeration value='uint32'/>
<xs:enumeration value='int64'/>
<xs:enumeration value='float'/>
<xs:enumeration value='float32'/>
<xs:enumeration value='double'/>
<xs:enumeration value='float64'/>
<xs:enumeration value='string'/>
<xs:enumeration value='quotedString'/>
</xs:restriction>
</xs:simpleType>
<xs:simpleType>
<xs:restriction base='xs:string'>
<xs:pattern value='cstr\([1-9][0-9]*\)'/>
</xs:restriction>
</xs:simpleType>
</xs:union>
</xs:simpleType>
<!-- A single character. -->
<xs:simpleType name='Char'> <xs:restriction base='xs:string'> <xs:length value='1'/> </xs:restriction> </xs:simpleType>
<xs:simpleType name='Format'>
<xs:restriction base='xs:decimal'>
<xs:minInclusive value='0.0' />
</xs:restriction>
</xs:simpleType>
<!-- Scale: either a numerical value, or 'min'. -->
<xs:simpleType name='Scale'>
<xs:union memberTypes='xs:decimal'>
<xs:simpleType>
<xs:restriction base='xs:string'>
<xs:enumeration value='Min'/>
</xs:restriction>
</xs:simpleType>
</xs:union>
</xs:simpleType>
<!-- A coordinate or attribute to be im-/exported from/to binary and text formats. -->
<xs:attributeGroup name='CoordinateOrAttribute'>
<!-- Either a coordinate name ('x', 'y', or 'z'), or the name of a predefined or custom attribute to be used internally. -->
<xs:attribute name='name' type='Name' use='required' />
<!-- Import of custom attributes only: the data type to be used internally. -->
<xs:attribute name='type' type='Type' />
<!-- The data type to be used on file.
Default: @type for custom attributes, otherwise the predefined type. -->
<xs:attribute name='typeFile' type='Type' />
<!-- The value to be considered as invalid on file. Must be convertible to @typeFile.
Import: not supported yet!
Export: store @invalidValue if the coordinate or attribute is not defined.
Concerning numeric types, there exist strings that are interpreted specially:
'Min' -> minimum finite representable value (for real number types: minimum positive value)
'Max' -> maximum finite representable value
'NaN' -> quiet (non-signaling) not-a-number (real number types only)
'Inf' -> infinity (real number types only) -->
<xs:attribute name='invalidValue' type='xs:string' />
<!-- Import only: transformation of values on file to internal ones.
Useful for applying unit changes, e.g. if angles on file are given in degrees, while OPALS expects them in radians.
Must follow the generic filter grammar, where the value on file is accessed via 'x'.
Example:
'x*Pi/180' (Converts from degrees on file to radians.) -->
<xs:attribute name='toOpals' type='xs:string' />
<!-- Export only: transformation of internal values to those on file.
Useful for applying unit changes, e.g. if OPALS angles should be stored in degrees on file.
Must follow the generic filter grammar, where the OPALS value is accessed via 'x'.
Example:
'x/Pi*180' (Converts from radians to degrees on file.) -->
<xs:attribute name='toFile' type='xs:string' />
</xs:attributeGroup>
<!-- The root element of every Opals format definition -->
<xs:element name='opalsFormatDefinition'>
<xs:complexType>
<xs:sequence>
<!-- Optional textual description of file format -->
<xs:element name='description' type='xs:string' minOccurs='0' />
<!-- Choose the format to be one of text, binary, LAS, ... -->
<xs:choice>
<!-- ===================== Binary Section ===================== -->
<xs:element name='binary'>
<xs:complexType>
<xs:sequence>
<!-- Optional header section -->
<xs:element name='header' minOccurs='0'>
<xs:complexType>
<!-- Import: at the file's beginning, skip/ignore as many bytes (~characters) as @text consists of.
Export: start files with @text. -->
<xs:attribute name='text' type='xs:string' />
<!-- Import only: skip/ignore the first @skipBytes bytes. Mutually exclusive with @text. -->
<xs:attribute name='skipBytes' type='xs:positiveInteger' />
</xs:complexType>
</xs:element>
<!-- The sequence of coordinates and attributes to be im-/exported. The order is relevant! -->
<xs:choice minOccurs='1' maxOccurs='unbounded'>
<!-- A coordinate or attribute to be im-/exported. Supports all attributes defined by CoordinateOrAttribute above. -->
<xs:element name='segment'>
<xs:complexType>
<xs:attributeGroup ref='CoordinateOrAttribute' />
</xs:complexType>
</xs:element>
<!-- Import only: skip/ignore this many bytes at this point -->
<xs:element name='skip'>
<xs:complexType>
<xs:attribute name='count' type='xs:positiveInteger' default='1' />
</xs:complexType>
</xs:element>
</xs:choice>
</xs:sequence>
<!-- Import only: throw if X- or Y- coordinates are unspecified. -->
<xs:attribute name='throwIfXYMissing' type='xs:boolean' default='true' />
<!-- Endianness, advanced feature:
Import: assume the data to be stored in this representation.
Export: store the data in the given representation. -->
<xs:attribute name='endian' default='native'>
<xs:simpleType>
<xs:restriction base='xs:string'>
<xs:enumeration value='native'/> <!-- Use whatever representation is native on your system. -->
<xs:enumeration value='little'/> <!-- Force little endian byte order. -->
<xs:enumeration value='big'/> <!-- Force big endian byte order. -->
</xs:restriction>
</xs:simpleType>
</xs:attribute>
</xs:complexType>
</xs:element>
<!-- ===================== Text Section ===================== -->
<xs:element name='text'>
<xs:complexType>
<xs:sequence>
<!-- Optional header section -->
<xs:element name='header' minOccurs='0'>
<xs:complexType>
<!-- Import: at the beginning of files, skip as many lines as @text consists of.
Export: start files with @text. -->
<xs:attribute name='text' type='xs:string' />
<!-- Import only: skip the first @skipLines lines. Mutually exclusive with @text. -->
<xs:attribute name='skipLines' type='xs:positiveInteger' />
</xs:complexType>
</xs:element>
<!-- The sequence of coordinates and attributes to be im-/exported. The order is relevant! -->
<xs:choice minOccurs='1' maxOccurs='unbounded'>
<!-- A coordinate or attribute to be im-/exported. -->
<xs:element name='column'>
<xs:complexType>
<!-- Supports all attributes defined by CoordinateOrAttribute above. -->
<xs:attributeGroup ref='CoordinateOrAttribute' />
<!-- Export only: the format to use for numeric data types. Syntax: {width}.{precision} -->
<xs:attribute name='format' type='Format' default='.6' />
<!-- Export only: if externalType=quotedString, then quote the string with this character. -->
<xs:attribute name='quote' type='Char' default='"' />
</xs:complexType>
</xs:element>
<!-- Import only: skip/ignore this many columns at this point -->
<xs:element name='skip' >
<xs:complexType>
<xs:attribute name='count' type='xs:positiveInteger' default='1' />
</xs:complexType>
</xs:element>
</xs:choice>
</xs:sequence>
<!-- Specify the decimal separator. -->
<xs:attribute name='decimalSeparator' type='Char' default='.' />
<!-- Specify the character set to be considered as separating data entries (default: space (0x20) or horizonal tabulator (0x09)).
Import: match any of the given characters.
Export: use first character. -->
<xs:attribute name='columnSeparators' type='xs:string' default=' &#09;' />
<!-- Import only: skip lines that begin with this string. -->
<xs:attribute name='commentInitiator' type='xs:string' default='' />
<!-- Import only: ignore white space.. -->
<xs:attribute name='skipWhiteSpace' type='xs:boolean' default='true' />
<!-- Import only: throw if X- or Y- coordinates are unspecified. -->
<xs:attribute name='throwIfXYMissing' type='xs:boolean' default='true' />
</xs:complexType>
</xs:element>
<!-- ===================== LAS Section ===================== -->
<xs:element name='las'>
<xs:complexType>
<xs:sequence>
<!-- Export only: set the scale of LAS point coordinates. Default: 0.001 -->
<xs:element name='scaleCoordinates' minOccurs='0'>
<xs:complexType>
<xs:attribute name='all' type='Scale' />
<xs:attribute name='x' type='Scale' /> <!-- Overrules @all -->
<xs:attribute name='y' type='Scale' /> <!-- Overrules @all -->
<xs:attribute name='z' type='Scale' /> <!-- Overrules @all -->
</xs:complexType>
</xs:element>
<!-- Export only: set the offset of LAS point coordinates. Defaults to the center of the bounding box. -->
<xs:element name='offsetCoordinates' minOccurs='0'>
<xs:complexType>
<xs:attribute name='all' type='xs:decimal' />
<xs:attribute name='x' type='xs:decimal' /> <!-- Overrules @all -->
<xs:attribute name='y' type='xs:decimal' /> <!-- Overrules @all -->
<xs:attribute name='z' type='xs:decimal' /> <!-- Overrules @all -->
</xs:complexType>
</xs:element>
<!-- Specification of extraBytes in the LAS-file. -->
<xs:element name='extraBytes' minOccurs='0' maxOccurs='unbounded'>
<xs:complexType>
<!-- The internal name of (the first element of) the extraBytes-entry: the name of a predefined or custom attribute. -->
<xs:attribute name='name' type='Name' use='required' />
<!-- The internal names of eventual 2nd and 3rd elements of the extraBytes-entry: the names of predefined or custom attributes. -->
<xs:attribute name='name2' type='Name' />
<xs:attribute name='name3' type='Name' />
<!-- Custom attributes only: the data type to be used internally. -->
<xs:attribute name='type' type='Type' />
<!-- The name of the extraBytes-entry in the LAS-file. Default: @name-->
<xs:attribute name='lasName'>
<xs:simpleType>
<xs:restriction base='xs:string'> <xs:maxLength value='32'/> </xs:restriction>
</xs:simpleType>
</xs:attribute>
<!-- Export only: The description of the extraBytes-entry in the LAS-file. -->
<xs:attribute name='lasDescription'>
<xs:simpleType>
<xs:restriction base='xs:string'> <xs:maxLength value='32'/> </xs:restriction>
</xs:simpleType>
</xs:attribute>
<!-- Export only: the data type to be used in the LAS file. default: @type -->
<xs:attribute name='lasType' type='Type' />
<!-- Export only: the scale to be used. Default: choose a scale such that all values can be stored and the least resolution is lost. -->
<xs:attribute name='scale' type='xs:decimal' />
<!-- Export only: the offset to be used. Default: choose an offset such that all values can be stored and the least resolution is lost. -->
<xs:attribute name='offset' type='xs:decimal' />
<!-- Export only: the (numeric) value to be written to file if the attribute is not present (ignoring scale and offset). -->
<xs:attribute name='invalidValue'>
<xs:simpleType>
<xs:union memberTypes='xs:decimal'>
<xs:simpleType>
<!-- These strings that are interpreted specially: -->
<xs:restriction base='xs:string'>
<xs:enumeration value='Max'/> <!-- Maximum finite representable value. -->
<xs:enumeration value='Min'/> <!-- Minimum finite representable value (for real number types: minimum positive value). -->
<xs:enumeration value='NaN'/> <!-- Quiet (non-signaling) not-a-number (real number types only). -->
<xs:enumeration value='Inf'/> <!-- Infinity (real number types only). -->
</xs:restriction>
</xs:simpleType>
</xs:union>
</xs:simpleType>
</xs:attribute>
</xs:complexType>
</xs:element>
</xs:sequence>
<!-- Export only: set the LAS minor version. -->
<xs:attribute name='versionMinor' default='1'>
<xs:simpleType>
<xs:restriction base='xs:nonNegativeInteger'> <xs:maxInclusive value='4'/> </xs:restriction>
</xs:simpleType>
</xs:attribute>
<!-- Export only: set the LAS point data record format. -->
<xs:attribute name='pointDataRecordFormat' default='0'>
<xs:simpleType>
<xs:restriction base='xs:nonNegativeInteger'> <xs:maxInclusive value='10'/> </xs:restriction>
</xs:simpleType>
</xs:attribute>
</xs:complexType>
</xs:element>
<!-- ===================== SHAPE Section ===================== -->
<xs:element name='shape'>
<xs:complexType>
<xs:sequence>
<!-- Specification of attributes/columns in the DBF file. -->
<xs:element name='attribute' minOccurs='0' maxOccurs='unbounded'>
<xs:complexType>
<!-- The internal name of the DBF entry: the name of a predefined or custom attribute.
Export only: Use $IdCounter for an automatic counter that is increased for each exported object -->
<xs:attribute name='name' use='required'>
<xs:simpleType>
<xs:restriction base='xs:string'>
<xs:pattern value='[a-zA-Z_](.*\S)?|$IdCounter' />
</xs:restriction>
</xs:simpleType>
</xs:attribute>
<!-- The name of the entry in the DBF file (max name length: 32. names will be clipped if need). Default: @name. -->
<xs:attribute name='dbfName'>
<xs:simpleType>
<xs:restriction base='Name'> <xs:maxLength value='32'/> </xs:restriction>
</xs:simpleType>
</xs:attribute>
<!-- Export only: column type describution used in the DBF file. default: appropriate value for predefined / internal type -->
<xs:attribute name='dbfType'>
<xs:simpleType>
<xs:restriction base='xs:string'>
<xs:enumeration value='string'/>
<xs:enumeration value='integer'/>
<xs:enumeration value='double'/>
<xs:enumeration value='boolean'/>
</xs:restriction>
</xs:simpleType>
</xs:attribute>
<!-- Export only: the total field width and precision to use for this DBF attribute: {width}.{precision}.
Mind that DBF stores all values as text.
Precision is the number of decimals and is relevant only for @dbfType=='double'.
If width and / or precision are not given, choose values appropriate for @dbfType. -->
<xs:attribute name='format' type='Format' default='.' />
</xs:complexType>
</xs:element>
</xs:sequence>
<!-- Export only: set geometry type for SHAPE file -->
<xs:attribute name='geometryType' default='auto'>
<xs:simpleType>
<xs:restriction base='xs:string'>
<xs:enumeration value='auto'/>
<xs:enumeration value='point'/> <!-- Each point is exported as separat entry supporting attribute dbf columns for each point -->
<xs:enumeration value='pointSet'/> <!-- Points are exported in chunks (requires less disk space than type 'point'). dbf attributs are availble for each chunk only -->
<xs:enumeration value='polyline'/>
<xs:enumeration value='polygon'/>
<!-- same as above but with 2d coordinates only -->
<xs:enumeration value='point2d'/>
<xs:enumeration value='pointSet2d'/>
<xs:enumeration value='polyline2d'/>
<xs:enumeration value='polygon2d'/>
</xs:restriction>
</xs:simpleType>
</xs:attribute>
<!-- Export only: SHAPE can store geometries of a single type only.
OPALS will raise an error if a geometry of a type different from @geometryType would be exported, unless @skipUnmatchingTypes is true. -->
<xs:attribute name='skipUnmatchingTypes' type='xs:boolean' default='false' />
</xs:complexType>
</xs:element>
<!-- ===================== RDB Section ===================== -->
<xs:element name='rdb'>
<xs:complexType>
<xs:sequence minOccurs='0' maxOccurs='1'>
<!-- pose defines the transformation between project coordinate system and the ecef coordinate system.
it can be either defined by a 4x4 trafo matrix or indirectly by a reference point
on the ellipsoid (which is internally converted to the 4x4 trafo matrix). -->
<xs:element name='pose' minOccurs='0' maxOccurs='1'>
<xs:complexType>
<!-- trafo reference point needs to be specified in lat [degree], lon [degree] and ellipsoidal height [m] -->
<xs:attribute name='point' use='optional'>
<xs:simpleType>
<xs:restriction>
<xs:simpleType>
<xs:list itemType='xs:double'></xs:list>
</xs:simpleType>
<xs:maxLength value='3'/>
<xs:minLength value='3'/>
</xs:restriction>
</xs:simpleType>
</xs:attribute>
<!-- 4x4 trafo matrix in row major order -->
<xs:attribute name='matrix' use='optional'>
<xs:simpleType>
<xs:restriction>
<xs:simpleType>
<xs:list itemType='xs:double'></xs:list>
</xs:simpleType>
<xs:maxLength value='16' />
<xs:minLength value='16' />
</xs:restriction>
</xs:simpleType>
</xs:attribute>
</xs:complexType>
<!-- Validate either point or matrix attribute specified (but not both). -->
<xs:key name="PointOrMatrixPresent">
<xs:selector xpath="." />
<xs:field xpath="@point | @matrix" />
</xs:key>
</xs:element>
<!-- Export only: min, max and scale value for the coordinates -->
<xs:element name='primaryAttribute' minOccurs='0' maxOccurs='1'>
<xs:complexType>
<xs:attribute name='min' type='xs:double' use='required'/>
<xs:attribute name='max' type='xs:double' use='required'/>
<xs:attribute name='scale' type='xs:double' use='optional'/>
</xs:complexType>
</xs:element>
<!-- Specification of attributes for importing -->
<xs:element name='attribute' minOccurs='0' maxOccurs='unbounded'>
<xs:complexType>
<!-- The internal name of the attribute entry: the name of a predefined or custom attribute. -->
<xs:attribute name='name' type='Name' use='required' />
<!-- Import of custom attributes only: the data type to be used internally. -->
<xs:attribute name='type' type='Type' use='optional'/>
<!-- The name of the unique (for queries) attribute entry in the RDB file. Default: @name -->
<xs:attribute name='rdbName' type='xs:string' use='required'/>
<!-- Export only: The rdb attribute title (for display). -->
<xs:attribute name='rdbTitle' type='xs:string' use='optional' />
<!-- Export only: The rdb attribute description (for display). -->
<xs:attribute name='rdbDescription' type='xs:string' use='optional' />
<!-- Export only: The physical rdb unit symbol (e.g. "m", "rad", "K"). -->
<xs:attribute name='unit' type='xs:string' use="optional" />
<!-- Export only: The expected value scale/resolution to select the appropriate rdb storage size.-->
<xs:attribute name='scale' type='xs:double' use="optional" />
<!-- Export only: The theoretical minimum value to select the appropriate rdb storage size. -->
<xs:attribute name='minValue' type='xs:double' use="optional" />
<!-- Export only: The theoretical maximum value to select the appropriate rdb storage size -->
<xs:attribute name='maxValue' type='xs:double' use="optional" />
<!-- Export only: The default value (minimum <= default <= maximum). -->
<xs:attribute name='defaultValue' type='xs:double' use="optional" />
<!-- Export only: The invalid value (minimum <= invalid <= maximum, use "not-a-number" if there is no invalid value). -->
<xs:attribute name='invalidValue' type='xs:double' use="optional" />
<!-- Import only: transformation of values on file to internal ones.
Useful for applying unit changes, e.g. if angles on file are given in degrees, while OPALS expects them in radians.
Must follow the generic filter grammar, where the value on file is accessed via 'x'.
Example:
'x*Pi/180' (Converts from degrees on file to radians.) -->
<xs:attribute name='toOpals' type='xs:string' />
<!-- Export only: transformation of internal values to those on file.
Useful for applying unit changes, e.g. if OPALS angles should be stored in degrees on file.
Must follow the generic filter grammar, where the OPALS value is accessed via 'x'.
Example:
'x/Pi*180' (Converts from radians to degrees on file.) -->
<xs:attribute name='toFile' type='xs:string' />
</xs:complexType>
</xs:element>
</xs:sequence>
<!-- defines which coordinates should be read/written -->
<xs:attribute name='coordinateSystem' default='proj'>
<xs:simpleType>
<xs:restriction base='xs:string'>
<xs:enumeration value='proj'/> <!-- project/application coordinate system (default) -->
<xs:enumeration value='ecef'/> <!-- earth centered earth fixed coordinate system -->
<xs:enumeration value='scs'/> <!-- import only: sensor coordinate system -->
<xs:enumeration value='map'/> <!-- import only: map projection coordinate system -->
</xs:restriction>
</xs:simpleType>
</xs:attribute>
</xs:complexType>
</xs:element>
</xs:choice>
</xs:sequence>
<xs:attribute name='version' type='xs:string' fixed='1.2' />
</xs:complexType>
</xs:element>
</xs:schema>
Author
wk
Date
05.05.2023