Module Translate
See Also
opals::ITranslate

Aim of module

Performs format conversion for vector data files with additional read support of GDAL rasters.

General description

Module Translate is a tool providing data format conversion functionality. Up to now, it supports the reading/writing of vector data formats (LAS, ESRI Shapefile, ODM, ASCII /binary, etc.). In the longer perspective Module Translate will also support raster file conversion and coordinate system transformations.

The main parameters are a single or multiple input file names (inFile) and the output file name (outFile). The respective data format is automatically estimated based on the file contents (input) and/or extension (output) but can also be defined explicitly (iFormat, oFormat). In both cases, well defined standard formats as described in Supported data formats as well as arbitrary ascii or binary data formats based on OPALS Format Definition XML files are supported. In case that no output format is specified, the output file is created using the file format and structure of the input file. This is especially useful if only a subset of the data should be written to the output by specifying a filter. For details on filters and how they are defined please refer to the Filter manual. Please note that, in contrast to Module Import only a single input format is supported. Thus, if multiple input files are specified, all files must share the same data format.

An additional functionality of Module Translate is the application of a 3D affine transformation (trafo) in the course of file format conversion. The transformation parameters are typically computed by other OPALS modules like Module LSM, Module ICP or Module GeorefApprox.

Splitting file content

Furthermore, Module Translate supports splitting of the input file contents into separate output files based on a specific attribute of the dataset splitbyAttribute. This feature is, e.g., useful to extract the original flight strips from tiled datasets via a strip identifier (e.g. PointSourceId, cf. Example 5:) or to divide a multi-return dataset into 1st, 2nd, ..., nth echo data files (EchoNumber, cf. Example 4:). While the feature is supported for all numeric types, integer-valued attributes constitute the standard usage case. For each distinct attribute value, a separate file is created. Please note that real-valued attributes (float, double) are automatically casted to integer.

Coordinate Reference System transformation

As a general rule within OPALS, data are expected in the same Coordinate Reference System (CRS) whenever multiple file inputs are supported. This applies to modules accepting file lists (e.g. inFile in Module Import , Module Algebra ...) and modules offering multiple (input) file parameters (e.g. inFile and trjFile in Module Import or inFile and gridFile in Module Snellius ...). Module Translate enables CRS transformations via the parameters inCRS, outCRS, and operation, available within the crsTrafo group. The implementation is based on the respective part of the GDAL/OGR library, which in turn internally uses the PROJ library.

In case the input datasets do not contain CRS information, the input reference system can either be provided via inCRS as a WKT string, a PROJ string, or an EPSG code. The same applies to the output reference system (outCRS), which is the only mandatory parameter for performing coordinate system transformations. Please refer to the CRS page for more information about reference system definition.

Please further note that the global parameter coord_ref_system is considered, if CRS information is neither stored on the data file nor explicitly set via inCRS. In case of multiple definitions, the following precedence is enforced: (i) inCRS, (ii) CRS information stored on the data file, (iii) coord_ref_system.

Starting from version 6.0, the PROJ library supports a flexible framework for constructing user-defined transformation pipelines. Built around this so-called "CoordinateOperation" feature, Module Translate offers a generic way for transforming data between different CRS including datum shift, change of height system, and gidi-shift based correction of inhomogeneous reference systems. The possible transformation paths are depicted in the following Figure 1.

Translate_trafoPipeline.png
Figure 1: Generic transformation pipeline

In Figure 1, the dashed and solid boxes denote source and target reference systems including potential intermediate systems, the thick arrows indicate possible transformation routes, and the green dashed boxes finally represent the available atomic coordinate operations between two systems. Some of these operations require further parameters as detailed in the table below. Syntactically, a transformation pipeline (operation) is a list of strings, each containing the textual, function-style description of an individual operation. Each string consits of the name of the operation (i.e. labels of green boxes in Figure 1) followed by 0..N arguments in key=value enclosed in parentheses and multiple arguments are separated by commas.

operation(argument1=value1, argument2=value2, ..., argumentN=valueN

The individual operations, their specific meaning and required parameters are listed in the table below:

Table 1: Coordinate operations
operation arguments dim. description
toEllipHeight hgtShift=<float>, hgtCorr=<path> | invHgtCorr=<path>, undulation=<path> 1D Converts custom heights h' (e.g. orthometric/normal/dynamic/tidal/etc. heights) to ellipsoidal heights H. The transformation may include a a constant offset (hgtShift) as well as the application of correction grids to account for inhomogenities of the source height system (hgtCorr = -invHgtCorr) and (geoid) undulations (undulation). Cf. Figure 1 for the transformation formulae. The height correction and undulation files are expected in geographic coordinates (lon/lat) of the source system and in <a href="https://vdatum.noaa.gov/docs/gtx_info.html>GTX file format</a).
toCustomHeight undulation=<path>, hgtCorr=<path> | invHgtCorr=<path>, hgtShift=<float> 1D Converts ellipsoidal heights H to custom heights h. The same rules as above apply. The height correction and undulation files are expected in geographic coordinates (lon/lat) of the target system.
mapProjection 2D Converts geographic coordinates (lon/lat) to map projection coordinates (x/y or easting/northing, respectively). No arguments required as the necessary parameters are stored with the CRS definitions (WKT, proj, EPSG code)
invMapProjection 2D Converts map projection coordinates (x/y or easting/northing, respectively) to geographic coordinates (lon/lat). No arguments required (as above).
toCartesian 3D Converts 2D-geographic coordinates + 1D-ellipsoidal heigths (lon/lat/H) to 3D Earth-Centered-Earth-Fixed (ECEF) cartesian coordinates (X/Y/Z). No arguments required (as above).
toGeographic 3D Converts ECEF cartesian coordinates (X/Y/Z) to 2D-geographic coordinates + 1D-ellipsoidal heigths (lon/lat/H). No arguments required (as above).
paramDatumTransform float*0 | float*3 | float*7 | float*10, inverse = 0 | 1 3D Converts ECEF cartesian coordinates (X/Y/Z) from one geodetic datum to another using a parametric transformation: 3P: shifts only, 7P: Spatial similarity transformation (Helmert: shifts, rotations, scale), 10P: Molodensky=Spatial similarity transformation with reduction point (3 shifts, 3 rotations, 1 scale, 3 reduction point shifts). The optional inverse key can be used, if the transformation paraemters are known in the opposite transformation direction. In this case, parameter inversion is performed internally. If no parameters are defined (i.e, float*0), the parameters are automatically extracted from the CRS definion strings (e.g.: WGS: TOWGS84).
gridShiftDatumTransform gridShift=<path>, inverse = 0 | 1 2D Converts geographic coordinates (lon/lat) from source datum to target datum via application of (ntV2) grid shift file (gridShift). Please use invGridShift, if the grid shift files denote the opposite transformation direction.
gridShiftHomogen gridShift=<path>, inverse = 0 | 1 2D Performs homogenization of geographic coordinates (lon/lat) via application of (ntV2) grid shift file (gridShift). Please note that no datum transformation is performed in this step but the grid shift values (dLon/dLat) are only intended to correct potential inhomogenities of the source CRS (gridShift=-invGridShift).
gridShiftInvHomogen gridShift=<path>, inverse = 0 | 1 2D Performs inverse homogenization of geographic coordinates (lon/lat) via application of (ntV2) grid shift file (gridShift). Please note that no datum transformation is performed in this step but the grid shift values (dLon/dLat) in the grid shift file are only intended to restore the inhomogenities of the target CRS (gridShift=-invGridShift).

Please find concrete examples here.

Parameter description

-inFileinput file(s)
Type: opals::Vector<opals::Path>
Remarks: mandatory
Import vector data file.
-outFileoutput file name
Type: opals::Path
Remarks: estimable
Estimation rule: The current path and the name (body) of the (first) input file is used as file name basis (stem). Depending on the number of input files and whether of not -splitByAttribute is specified, the output file is named as follows:
nr. inFiles==1 and splitByAttribute=no : stem + '.translatedTo' + oFormat extension
nr. inFiles >1 and splitByAttribute=no : stem + '.multipleInputsTranslatedTo' + oFormat extension
splitByAttribute=yes : stem + '_' + name-of-attribute name + '_' + value-of-attribute + oFormat extension
In the latter case, value-of-attribute denotes the unique values of the splitting attribute found in the input datasets.
-iFormatfile format [auto, xyz, bxyz, wnp, bwnp, las, sdw, fwf, odm, <opals format def. xml file>]
Type: opals::String
Remarks: default=auto
Input file format. If no format is specified the file content is used to recognise the file format.
-oFormatoutput format [xyz, bxyz, wnp, bwnp, las, sdw, fwf, odm, <opals format def. xml file>]
Type: opals::String
Remarks: optional
Defines the output format. If not specified, the output file written in the very same format as the input file.
-filtermodify the input using a (tree of) filter(s)
Type: opals::String
Remarks: optional
A filter string in EBNF syntax as described in section 'Filter syntax' can be passed to restrict the input data set (e.g. to consider last echoes only)
-trafogeometrically transform the data during import
Type: opals::TrafPars3dAffine
Remarks: optional
An eventually passed affine 3-d transformation is applied during data import. The passed value is converted internally into a single filter tree for parameters 'filter' and 'trafo'. If both are to be used, make sure to mark the position where in the filter's string 'trafo' shall be introduced, using curly brackets {}
-splitByAttributesplit attribute
Type: opals::String
Remarks: optional
If specified, the output is split into individual files based on the respective attribute. This can, e.g., be used to separate flight strips from tiled data using the PointSourceId attribute or to separate the 1st, 2nd, etc. echoes based on the EchoNumber attribute. Please note that this works only for numerical attributes but not for strings. Furthermore, all numerical attributes are automatically converted (i.e. casted) to integer.
-crsTrafoIGroup: Options for crs transformation
to come

.inCRSsource coordinate reference system (CRS)
Type: opals::String
Remarks: optional
In case of coordinate transformation the source CRS may be specified either as OGC Well-Known-Text (WKT) string, as proj4 definition string or as EPSG code. This is not needed if the source system is stored within the input file, but can be overwritten by using this parameter.
.outCRStarget coordinate reference system (CRS)
Type: opals::String
Remarks: optional
To apply a coordinate transformation specify the target CRS as OGC Well-Known-Text (WKT) string, as proj4 definition string or as EPSG code. In case the input file doesn't contain CRS information, it has to be provided by the inCRS parameter.
.operationcoordinate transformation pipline
Type: opals::Vector<opals::String>
Remarks: optional
to come
-tileSizetile size length for ODM files
Type: double
Remarks: optional
This parameter is only relevant if translated to ODM(s) and the automatic tile size estimation should be overruled. Inappropriate tile sizes can lead to memory problems or slow performance in later processing steps. On average, 100,000 to 200,000 points should be stored within one tile (check index statistics after export).

Examples

The data used in the following examples can be found in the $OPALS_ROOT/demo/ directory.

Example 1:

opalsTranslate -inFile strip31.laz -oFormat fwf

The above command converts the file strip31.laz from compressed LAS file format to the ASCII file strip31.laz.convertedTo.fwf. The attributes Amplitude (i.e. LAS intensity), EchoNumber, and NrOfEchos are preserved as they are contained in the predefined FWF format. Please note that the name of the output file is automatically estimated based on the respective estimation rule.

Example 2:

opalsTranslate -inFile strip31.laz -iFormat LAS_1.2_ew.xml -oFormat fwf

Providing the proper input format definition XML file LAS_1.2._ew.xml which maps the LAS extry byte attribute PulseWidth to the predefined OPALS Datamanager attribute EchoWidth additionally preserves the echo width information in the ASCII output file.

Example 3:

opalsTranslate -inFile strip31.laz -outFile strip31-intermed.bxyz -filter echo[intermediate]

This command reads the LAS file, analyses the echo attributes via the specified filter, and writes all intermediate echo points to a binary XYZ file.

Example 4:

This example demonstrates the usage of splitByAttribute to extract the 1st, 2nd, ..., nth echo from multiple input files.

opalsTranslate -inFile strip?1.laz -outFile echo.las -splitByAttr EchoNumber

As a result, seven files named echo_1.las to echo_7.las are created corresponding to the unique echo numbers of the input datasets. Please note that it is not necessary to know the attribute range beforehand, as the uniqueness is checked for each point and new output files are automatically created if needed.

Example 5:

This example illustrates the usage of splitByAttribute to extract the flight strips from an entire flight block dataset based on the attribute PointSourceId.

opalsImport -inFile strip?1.laz -outFile strips.odm -tileSize 25
opalsTranslate -inFile strips.odm -splitByAttr PointSourceId

Please note that in this example no output file name is specified. Module Translate automatically composes the output file names from the name of the input file (strips_), the attribute name (PointSourceId_) and the unique PointSourceId values (11, 21, 31). The following files are hence exported.

strips_PointSourceId_11.odm
strips_PointSourceId_21.odm
strips_PointSourceId_31.odm

Example 6:

This example illustrates the usage of trafo to do 3D affine transformation on a raster dataset based on the result of alignment from Module LSM.

opalsImport -inFile strip11.laz
opalsImport -inFile strip21.laz
opalsGrid -inFile strip11.odm -outFile strip11.tif -interpolation movingPlanes
opalsGrid -inFile strip21.odm -outFile strip21.tif -interpolation movingPlanes
opalsLSM -inFile strip11.tif strip21.tif -outParamFile lsm_output.xml
opalsTranslate -inFile strip21.tif -inParamFiles lsm_output.xml -paramMapping "trafo=opalsLSM.outTrafPars"

Module Translate automatically composes the output file name from the name of the input file.

Example 6:

This example shows how to perform a simple CRS transformation.

opalsTranslate -inFile strip11.laz -outFile strip11_gkm34.laz -outCRS EPSG:31256

In this example, only the output CRS is set while the input CRS (ETRS89/UTM33) is automatically determined from the file header of the input dataset (strip11.laz). As no user-defined transformation pipeline is specified, the transformation path is automatically determined based on the given CRS definitions. In particular, the TOWGS84 specification of the output CRS (EPSG:31256, MGI, Gauss-Krueger East) is used for performing the datum transition from ETRS89->MGI via a 7P (spatial similarity transformation). Please notethat, as both system definitions are 2D (PROJCS), the transformations is only performed in 2D and consequently only the x,y, coordinates change but not the elevations.

strip11.laz (EPSG: 25833)
Nr. points 378712
Minimum X-Y-Z 529570.001 5338590.000 223.063
Maximum X-Y-Z 529860.000 5338769.999 303.948
strip11_gkm34.laz (EPSG:31256)
Minimum X-Y-Z -69444.545 340581.203 223.063
Maximum X-Y-Z -69151.405 340766.225 303.948

Example 7:

This example shows how to use custom transformation pipelines (operation) to transform global coordinates in UTM33/ETRS89 coordinate frame to the Austrian national system MGI in Gauss-Krueger projection.

opalsTranslate -inFile etrs89_utm33.xyz -outFile mgi_gkm34.xyz -inCRS etrs89_utm33.prj -outCRS mgi_gkm34.prj -operation "invMapProjection" "toCartesian" "paramDatumTransform" "toGeographic" "mapProjection"

Current restrictions:

Currently only point datasets are supported, when running Module Translate with splitByAttribute and with CRS transformations.

Author
G. Mandlburger, J. Otepka
Date
17.05.2018