Module Translate
See also
opals::ITranslate

Aim of module

Performs format conversion for vector data files with additional read support of GDAL rasters.

General description

Module Translate is a tool providing data format conversion functionality. Up to now, it supports the reading/writing of vector data formats (LAS, ESRI Shapefile, ODM, ASCII /binary, etc.). In the longer perspective Module Translate will also support raster file conversion and coordinate system transformations.

The main parameters are a single or multiple input file names (inFile) and the output file name (outFile). The respective data format is automatically estimated based on the file contents (input) and/or extension (output) but can also be defined explicitly (iFormat, oFormat). In both cases, well defined standard formats as described in Supported data formats as well as arbitrary ascii or binary data formats based on OPALS Format Definition XML files are supported. In case that no output format is specified, the output file is created using the file format and structure of the input file. This is especially useful if only a subset of the data should be written to the output by specifying a filter. For details on filters and how they are defined please refer to the Filter manual. Please note that, in contrast to Module Import only a single input format is supported. Thus, if multiple input files are specified, all files must share the same data format.

An additional functionality of Module Translate is the application of a 3D affine transformation (trafo) in the course of file format conversion. The transformation parameters are typically computed by other OPALS modules like Module LSM, Module ICP or Module GeorefApprox.

Splitting file content

Furthermore, Module Translate supports splitting of the input file contents into separate output files based on a specific attribute of the dataset splitbyAttribute. This feature is, e.g., useful to extract the original flight strips from tiled datasets via a strip identifier (e.g. PointSourceId, cf. Example 5: Extracting original flight strips from tiled data) or to divide a multi-return dataset into 1st, 2nd, ..., nth echo data files (EchoNumber, cf. Example 4: Splitting file contents by echo number). While the feature is supported for all numeric types, integer-valued attributes constitute the standard usage case. For each distinct attribute value, a separate file is created. Please note that real-valued attributes (float, double) are automatically casted to integer.

Coordinate Reference System transformation

As a general rule within OPALS, data are expected in the same Coordinate Reference System (CRS) whenever multiple file inputs are supported. This applies to modules accepting file lists (e.g. inFile in Module Import , Module Algebra ...) and modules offering multiple (input) file parameters (e.g. inFile and trjFile in Module Import or inFile and gridFile in Module Snellius ...). Module Translate enables CRS transformations via the parameters inCRS, outCRS, and operation, available within the crsTrafo group. The implementation is based on the respective part of the GDAL/OGR library, which in turn internally uses the PROJ library.

In case the input datasets do not contain CRS information, the input reference system can either be provided via inCRS as a WKT string, a PROJ string, or an EPSG code. The same applies to the output reference system (outCRS), which is the only mandatory parameter for performing coordinate system transformations. Please refer to the CRS page for more information about reference system definition.

Please further note that the global parameter coord_ref_system is considered, if CRS information is neither stored on the data file nor explicitly set via inCRS. In case of multiple definitions, the following precedence is enforced: (i) inCRS, (ii) CRS information stored on the data file, (iii) coord_ref_system.

Starting from version 6.0, the PROJ library supports a flexible framework for constructing user-defined transformation pipelines. Built around this so-called "CoordinateOperation" feature, Module Translate offers a generic way for transforming data between different CRS including datum shift, change of height system, and gidi-shift based correction of inhomogeneous reference systems. The possible transformation paths are depicted in the following Figure 1.

Figure 1: Generic transformation pipeline

In Figure 1, the dashed and solid boxes denote source and target reference systems including potential intermediate systems, the thick arrows indicate possible transformation routes, and the green dashed boxes finally represent the available atomic coordinate operations between two systems. Some of these operations require further parameters as detailed in the table below. Syntactically, a transformation pipeline (operation) is a list of strings, each containing the textual, function-style description of an individual operation. Each string consits of the name of the operation (i.e. labels of green boxes in Figure 1) followed by 0..N arguments in key=value enclosed in parentheses and multiple arguments are separated by commas.

operation(argument1=value1, argument2=value2, ..., argumentN=valueN

The individual operations, their specific meaning and required parameters are listed in the table below:

Table 1: Coordinate operations
operation arguments dim.description
projectedToGeodetic arguments 2D Converts map projection coordinates (x/y or easting/northing, respectively) to geodetic coordinates (lon/lat). No arguments required (cf. above).
geodeticToProjected arguments 2D Converts geodetic coordinates (lon/lat) to map projection coordinates (x/y or easting/northing, respectively). No arguments required as the necessary parameters are stored in the CRS definitions (WKT, proj, EPSG code).
customToOrthoH hgtShift=<float>, hgtScale=<float>, hgtCorr | invHgtCorr=<path> 1D Converts custom heights h' (e.g. normal/tidal/etc. heights) to orthometric heights h. The transformation may include a a constant offset (hgtShift) and scale (hgtScale) as well as the application of a correction grids to account for inhomogenities of the source height system (hgtCorr = -invHgtCorr). Cf. Figure above for the transformation formulae. Georeferencing of the height correction grid file is expected in geodetic coordinates (lon/lat; unit: deg) of the source system. File format: GTX.
orthoToCustomH hgtShift=<float>, hgtScale=<float>, hgtCorr | invHgtCorr=<path> 1D Converts orthometric heights h to custom heights h'. The same rules as above apply. Georeferening of the height correction file is expected in geodetic coordinates (lon/lat; unit: deg) of the target system.
ellipToOrthoH undulation=<path> 1D Converts ellipsoidal heights H to orthometric heights h via application of a correction grid containing geoid undulations N (H=h+N). Georeferencing of the undulation grid file is expected in geodetic coordinates (lon/lat; unit: deg). File format: GTX.
orthoToEllipH undulation=<path> 1D Converts orthometric heights h to ellipsoidal heights H. The same rules as above apply.
geodeticToGeocentric arguments 3D Converts 2D geodetic coordinates + 1D-ellipsoidal heigths (lon/lat/H) to 3D Earth-Centered-Earth-Fixed (ECEF) cartesian coordinates (X/Y/Z). No arguments required (cf. above).
geocentricToGeodetic arguments 3D Converts 3D ECEF cartesian coordinates (X/Y/Z) to 2D-geodetic coordinates + 1D-ellipsoidal heigths (lon/lat/H). No arguments required (cf. above).
paramDatTrans float*0 | float*3 | float*7 | float*10, inverse = 0 | 1 3D Converts 3D ECEF cartesian coordinates (X/Y/Z) from one geodetic datum to another via parametric transformation: 3P: shifts only, 7P: Spatial similarity transformation (Helmert: shifts[m], rotations[arc sec], scale[ppm]), 10P: Molodensky=Spatial similarity transformation with reduction point (shifts[m], rotations[arc sec], scale[ppm], reduction point shifts[m]). The optional inverse key can be used, if the transformation parameters are known in the opposite transformation direction. In this case, parameter inversion is performed internally. If no parameters are defined (i.e, float*0), the parameters are automatically extracted from the CRS definion strings (e.g.: WKT: TOWGS84).
gridShiftDatTrans gridShift | invGridShift=<path> 2D Performs datum transformation of geodetic coordinates (lon/lat) potentially including homogenization due to irragularities of either CRS applying grid shifts (dLon, dLat) provided as an ntV2 correction grid (gridShift). Georeferencing of the grid shift file is expected in geodetic coordinates (lon/lat; unit: deg). Please use invGridShift, if the grid shift files denote the opposite transformation direction.
gridShiftHomogen gridShift | invGridShift=<path> 2D Performs homogenization of geodetic coordinates (lon/lat) applying (ntV2) grid shifts. Please note that no datum transformation is performed in this step but the grid shift values (dLon/dLat) only correct potential inhomogenities of the source CRS (gridShift=-invGridShift). Georeferencing of the grid shift file is expected in geodetic coordinates (lon/lat; unit: deg).
gridShiftInvHomogen gridShift | invGridShift=<path> 2D Performs inverse homogenization of geodetic coordinates (lon/lat) applying (ntV2) grid shifts. Please note that no datum transformation is performed in this step but the grid shift values (dLon/dLat) only restore potential inhomogenities of the target CRS (gridShift=-invGridShift). Georeferencing of the grid shift file is expected in geodetic coordinates (lon/lat; unit: deg).

Please find specific examples here.

Parameter description

-inFileinput file(s)
Type: opals::Vector<opals::VectorOrRasterFile>
Remarks: mandatory
Import vector data file.
VectorOrRasterFile: File path of a vector or raster dataset.
See Supported vector formats, OPALS Datamanager, and Supported grid/raster formats
-outFileoutput file name
Type: opals::Path
Remarks: estimable
Estimation rule: The current path and the name (body) of the (first) input file is used as file name basis (stem). Depending on the number of input files and whether of not -splitByAttribute is specified, the output file is named as follows:
nr. inFiles==1 and splitByAttribute=no : stem + '.translatedTo' + oFormat extension
nr. inFiles >1 and splitByAttribute=no : stem + '.multipleInputsTranslatedTo' + oFormat extension
splitByAttribute=yes : stem + '_' + name-of-attribute name + '_' + value-of-attribute + oFormat extension
In the latter case, value-of-attribute denotes the unique values of the splitting attribute found in the input datasets.
-iFormatfile format [auto, xyz, bxyz, wnp, bwnp, las, sdw, fwf, odm, <opals format def. xml file>]
Type: opals::Vector<opals::String>
Remarks: default=auto
Input file format. If no format is specified the file content is used to recognise the file format.
-oFormatoutput format [xyz, bxyz, wnp, bwnp, las, sdw, fwf, odm, <opals format def. xml file>]
Type: opals::String
Remarks: optional
Defines the output format. If not specified, the output file written in the very same format as the input file.
-filtermodify the input using a (tree of) filter(s)
Type: opals::FilterWithPlaceHolders<DM::IFilter::ReadAccess::coordinatesAttributes, DM::IFilter::WriteAccess::coordinates | attributes | delayedResults>
Remarks: optional
Reduce and/or transform the input data set: import last echoes only, transform coordinates, modify attributes, etc.
FilterWithPlaceHolders<coordinatesAttributes, coordinates | attributes | delayedResults>: has read access to: coordinatesAttributes; has write access to: coordinates | attributes | delayedResults;
See Filters
-trafogeometrically transform the data during import
Type: opals::TrafPars3dAffine
Remarks: optional
An eventually passed affine 3-d transformation is applied during data import. Internally, the passed value is incorporated into 'filter'. If both are to be used, then mark with {trafo} the position where 'trafo' shall be introduced into 'filter'.
TrafPars3dAffine: Describes a relative or absolute 3D-affine transformation. This class contains the transformation parameters of an affine 3D (i.e. 12-parameter) transformation, given as 12-array
-splitByAttributesplit attribute
Type: opals::String
Remarks: optional
If specified, the output is split into individual files based on the respective attribute. This can, e.g., be used to separate flight strips from tiled data using the PointSourceId attribute or to separate the 1st, 2nd, etc. echoes based on the EchoNumber attribute. Please note that this works only for numerical attributes but not for strings. Furthermore, all numerical attributes are automatically converted (i.e. casted) to integer.
-crsTrafoIGroup: Options for crs transformation
to come
.inCRSsource coordinate reference system (CRS)
Type: opals::String
Remarks: optional
In case of coordinate transformation the source CRS may be specified either as OGC Well-Known-Text (WKT) string, as proj4 definition string or as EPSG code. This is not needed if the source system is stored within the input file, but can be overwritten by using this parameter.
.outCRStarget coordinate reference system (CRS)
Type: opals::String
Remarks: optional
To apply a coordinate transformation specify the target CRS as OGC Well-Known-Text (WKT) string, as proj4 definition string or as EPSG code. In case the input file doesn't contain CRS information, it has to be provided by the inCRS parameter.
.operationcoordinate transformation pipline
Type: opals::Vector<opals::String>
Remarks: optional
to come
-tileSizetile size length for ODM files
Type: double
Remarks: optional
This parameter is only relevant if translated to ODM(s) and the automatic tile size estimation should be overruled. Inappropriate tile sizes can lead to memory problems or slow performance in later processing steps. On average, 100,000 to 200,000 points should be stored within one tile (check index statistics after export).

Short examples

LAS -> XYZ

opalsImport -inf G111.las
opalsGrid -inf G111.odm
from opals.Translate import Translate
Translate(inFile='G111.laz').run()
import os.path
from pyDM import isValidODM
assert isValidODM('G111.odm')
assert os.path.exists('G111_z.tif')

Examples

The data used in the following examples can be found in the $OPALS_ROOT/demo/ directory.

Example 1: Simple format conversion

opalsTranslate -inFile strip31.laz -oFormat fwf

The above command converts the file strip31.laz from compressed LAS file format to the ASCII file strip31.laz.convertedTo.fwf. The attributes Amplitude (i.e. LAS intensity), EchoNumber, and NrOfEchos are preserved as they are contained in the predefined FWF format. Please note that the name of the output file is automatically estimated based on the respective estimation rule.

Example 2: Format conversion with custom input format

opalsTranslate -inFile strip31.laz -iFormat LAS_1.2_ew.xml -oFormat fwf

Providing the proper input format definition XML file LAS_1.2._ew.xml which maps the LAS extry byte attribute PulseWidth to the predefined OPALS Datamanager attribute EchoWidth additionally preserves the echo width information in the ASCII output file.

Example 3: Format conversion with on-the-fly filtering

opalsTranslate -inFile strip31.laz -outFile strip31-intermed.bxyz -filter echo[intermediate]

This command reads the LAS file, analyses the echo attributes via the specified filter, and writes all intermediate echo points to a binary XYZ file.

Example 4: Splitting file contents by echo number

This example demonstrates the usage of splitByAttribute to extract the 1st, 2nd, ..., nth echo from multiple input files.

opalsTranslate -inFile strip?1.laz -outFile echo.las -splitByAttr EchoNumber

As a result, seven files named echo_1.las to echo_7.las are created corresponding to the unique echo numbers of the input datasets. Please note that it is not necessary to know the attribute range beforehand, as the uniqueness is checked for each point and new output files are automatically created if needed.

Example 5: Extracting original flight strips from tiled data

This example illustrates the usage of splitByAttribute to extract the flight strips from an entire flight block dataset based on the attribute PointSourceId.

opalsImport -inFile strip?1.laz -outFile strips.odm -tileSize 25
opalsTranslate -inFile strips.odm -splitByAttr PointSourceId

Please note that in this example no output file name is specified. Module Translate automatically composes the output file names from the name of the input file (strips_), the attribute name (PointSourceId_) and the unique PointSourceId values (11, 21, 31). The following files are hence exported.

strips_PointSourceId_11.odm
strips_PointSourceId_21.odm
strips_PointSourceId_31.odm

Example 6: 3D transformation

This example illustrates the usage of trafo to do 3D affine transformation on a raster dataset based on the result of alignment from Module LSM.

opalsImport -inFile strip11.laz
opalsImport -inFile strip21.laz
opalsGrid -inFile strip11.odm -outFile strip11.tif -interpolation movingPlanes
opalsGrid -inFile strip21.odm -outFile strip21.tif -interpolation movingPlanes
opalsLSM -inFile strip11.tif strip21.tif -outParamFile lsm_output.xml
opalsTranslate -inFile strip21.tif -inParamFiles lsm_output.xml -paramMapping "trafo=opalsLSM.outTrafPars"

Module Translate automatically composes the output file name from the name of the input file.

Example 7: Simple coordinate reference system transformation

This example shows how to perform a simple CRS transformation.

opalsTranslate -inFile strip11.laz -outFile strip11_gkm34.laz -outCRS EPSG:31256

In this example, only the output CRS is set while the input CRS (ETRS89/UTM33) is automatically determined from the file header of the input dataset (strip11.laz). As no user-defined transformation pipeline is specified, the transformation path is automatically determined based on the given CRS definitions. In particular, the TOWGS84 specification of the output CRS (EPSG:31256, MGI, Gauss-Krueger East) is used for performing the datum transition from ETRS89->MGI via a 7P (spatial similarity transformation). Please notethat, as both system definitions are 2D (PROJCS), the transformations is only performed in 2D and consequently only the x,y, coordinates change but not the elevations.

strip11.laz (EPSG: 25833)
Nr. points 378712
Minimum X-Y-Z 529570.001 5338590.000 223.063
Maximum X-Y-Z 529860.000 5338769.999 303.948
strip11_gkm34.laz (EPSG:31256)
Minimum X-Y-Z -69444.545 340581.203 223.063
Maximum X-Y-Z -69151.405 340766.225 303.948

Example 7b: Using coordinate reference system transformations in Python

In Module Translate , the parameters for the specification of coordinate reference systems are located within the option group crsTrafo. While it is not necessary to specify the option group on the command line, this is required in Python. The following code example illustrates the usage:

## @package python.demo.translateCrsDemo
#
# @brief: Transforms coordinates from EPSG:25833 to EPSG:31256.
#
# Demo script demonstrating the use of opalsTranslate with coordinate transformations in Python.
# The script reads the sample data file strip11.laz georeferenced in ETRS89/UTM33 and
# transforms the coordinates to the national Austrian system MGI/Gauss-Krüger M34 (EPSG:31256).
#
import opals
from opals import Translate
trl = Translate.Translate()
trl.inFile = 'strip11.laz'
trl.outFile = 'strip11_gkm34_py.laz'
trl.crsTrafo.outCRS = 'EPSG:31256'
trl.run()

The script can be executed via:

opals translateCrsDemo

Example 8: Coordinate reference system transformation with simple custom pipeline

This example shows how to use custom transformation pipelines (operation) to transform trans-national coordinates in UTM33/ETRS89 coordinate frame to the Austrian national system MGI in Gauss-Krueger projection.

opalsTranslate -inFile etrs89_utm33.xyz -outFile mgi_gkm34_ellH.xyz -inCRS etrs89_utm33.prj -outCRS mgi_gkm34.prj -operation "projectedToGeodetic" "geodeticToGeocentric" "paramDatTrans" "geocentricToGeodetic" "geodeticToProjected"

Please note that no additional parameters are set in the individual transformation operations, especially not in the datum transformation step (paramDatTrans). Instead, the TOWGS84 parameters defined the WKT files (*.prj) are used. In the above example, the following line in mgi_gkm34.prj is considered:

TOWGS84[577.326,90.129,463.919,5.137,1.474,5.297,2.4232],

These seven parameters of a spatial similarity transformation (3 shifts, 3 rotations, 1 scale) provide the datum transition from ETRS89 to MGI with an accuracy of around 1m in the resulting MGI coordinates for the entire area of Austria. Furthermore, as the input height system is ellipsoidal and no height transformations are provided, the resulting Z-component constitutes ellipsoidal heights above the underlying BESSEL ellipsoid.

The first three resulting MGI/Gauss-Krueger coordinates are:

-20087.330 5300137.211 837.664
-18790.537 5300159.799 522.529
-19582.704 5300455.166 835.869

Example 9: Coordinate reference system transformation with complex custom pipeline

For the same input as in the above example (ETRS89/UTM33) this example illustrates the full capabilities of the custom transformation pipeline concept including grid shift (ntV2) based datum transition and transformation of the height component from ellipsoidal heights above the GRS80 ellipsoid in the input to MGI (custom) heights..

opalsTranslate -inFile etrs89_utm33.xyz -outFile mgi_gkm34_customH.xyz -inCRS etrs89_utm33.prj -outCRS mgi_gkm34.prj -operation projectedToGeodetic "ellipToOrthoH(undulation=GEOID_GRS80_Oesterreich.gtx)" "gridShiftDatTrans(gridShift=AT_GIS_GRID.gsb,inverse=1)" "orthoToCustomH(invHgtCorr=GV_Hoehengrid_V1.gtx)" geodeticToProjected

The transformation pipeline starts with calculating the geodetic coordinates (lon/lat) from the projected UTM33 coordinates (x/y) followed by the height transformation from ellipsoidal to orthometric heights. At this stage, the geodetic datum transition from ETRS89 to MGI is performed via grid shift transformation, inversely applying the lon/lat corrections stored in the ntV2 grid shift file AT_GIS_GRID.gsb published by the Austrian national mapping agency BEV. Orthometric heights are invariant w.r.t. datum transitions, thus the remaining steps are to convert the orthometric heights to the custom MGI height system (i.e. small corrections stored in GV_Hoehengrid_V1.gtx, again published by BEV) and the final transformation of lon/lat to Gauss-Krueger system. The resulting planimetric coordinates (x,y) constitute a better approximation of the national Austrian MGI coordinate reference system (accuracy: better 15cm) compared to using the 7P-transformation demonstrated in the above example.

-20087.214 5300137.514 836.812
-18790.413 5300160.098 521.755
-19582.592 5300455.459 835.051

Current restrictions:

Currently only point datasets are supported, when running Module Translate with splitByAttribute and with CRS transformations.

Author
G. Mandlburger, J. Otepka
Date
28.06.2018
@ outParamFile
final parameter export
opalsTranslate is the executable file of Module Translate
Definition: ModuleExecutables.hpp:233
@ Y
sessions.adjustment.leverArm group(opalsStripAdjust)
@ paramMapping
mapping of parameters from file to own parameters
@ movingPlanes
moving (tilted) plane interpolation
opalsImport is the executable file of Module Import
Definition: ModuleExecutables.hpp:113
@ Z
sessions.adjustment.leverArm group(opalsStripAdjust)
@ strip
strip ID of an image (opalsStripAdjust)
@ outCRS
output coordinate reference system as WKT/proj4 description string or EPSG code (opalsTranslate)
Contains the public interface of OPALS.
Definition: ApplyTrafo.hpp:5
@ inParamFiles
parameters to import from file
opalsGrid is the executable file of Module Grid
Definition: ModuleExecutables.hpp:93
@ X
sessions.adjustment.leverArm group(opalsStripAdjust)
@ filter
string to be parsed in construction of DM::IFilter (various modules)
opalsLSM is the executable file of Module LSM
Definition: ModuleExecutables.hpp:128
@ inCRS
input coordinate reference system as WKT/proj4 description string or EPSG code (opalsTranslate)
@ EchoNumber
This is the k-th return/echo for a certain pulse, where for the first return: k==1 (see LAS spec....
@ strips
strip group (opalsStripAdjust)
@ operation
coordinate operation for crs transformations (opalsTranslate)