Module Export
See also
opals::IExport

Aim of module

Exports vector data stored in an OPALS data manager (ODM) in various data formats.

General description

This manual starts with a short description of the OPALS Data Manager and data formats that are available for export. Furthermore, the parameters to control data export are explained including the filtering features (echo filter, geometric filters ...). Finally, some illustrative examples for the usage of Module Export are given.

OPALS uses the OPALS Data Manager (ODM) for the administration of ALS point clouds, as well as line related geometries (eg. breaklines, formlines, borderlines). OPALS modules dealing with primary topographic data use data manager files (*.odm) for input (e.g. Module Grid, Module Cell, etc.) and output (e.g. Module EchoRatio, Module Normals, etc.).

The ODM can be seen as a database system supporting efficient spatial queries, but also the storage of arbitrary attributes (referred to as additional info in OPALS) for each geometry object. These additional infos are added during import or by different OPALS modules (e.g. Module Normals which estimates surface normals in each point). Depending on the selected data format the geometry objects including the corresponding attribute columns are exported.

Module Export supports the following export formats and export strategies, respectively:

Format Description
original Restore the original file(s) in the imported data format (not yet implemented)
wnp, bwnp Winput and binary Winput files
xyz, bxyz xyz and binary xyz files
las LAS
sdw Riegl SDW (Sample Data World) files
fwf IPF internal full waveform text format (currently only FWF 1.0 can be exported)
shape Shapefiles
<opals format definition xml file> Generic export of text and binary file

Parameter description

-inFileinput odm files
Type: opals::Vector<opals::Path>
Remarks: mandatory
The specified input odm files can be exported into a 'new' output file (arbitrary output format) or the original file(s) as imported are reproduced. Note that wildcard characters (*,?) can be used for the selection of multiple ODM files at once (e.g. strip*.odm).
-outFileoutput file/directory
Type: opals::Path
Remarks: mandatory
The specified input odm file can be exported into a 'new' output file (arbitrary output format) or the original file(s) as imported are reproduced.
-oFormatoutput format [original, xyz, bxyz, wnp, bwnp, las, sdw, fwf, odm, shape, <opals format def. xml file>]
Type: opals::String
Remarks: optional
Defines the output format or in case of 'original' the original file(s) as imported are reproduced.
-filtermodify the export using filter(s)
Type: opals::Vector<opals::FilterWithPlaceHolders<DM::IFilter::ReadAccess::coordinatesAttributes, DM::IFilter::WriteAccess::coordinates | attributes | delayedResults>>
Remarks: optional
Reduce and/or transform the output data set: export last echoes only, transform coordinates, modify attributes, etc. When specifying one filter it will be applied to all input files. However, it is also possible to use different filter for different input files. In this case, the order and the number of input files and filters have to match.
FilterWithPlaceHolders<coordinatesAttributes, coordinates | attributes | delayedResults>: has read access to: coordinatesAttributes; has write access to: coordinates | attributes | delayedResults;
See Filters
-trafogeometrically transform the data during export
Type: opals::TrafPars3dAffine
Remarks: optional
An eventually passed affine 3-d transformation is applied during data export. Internally, the passed value is incorporated into 'filter'. If both are to be used, then mark with {trafo} the position where 'trafo' shall be introduced into 'filter'.
TrafPars3dAffine: Describes a relative or absolute 3D-affine transformation. This class contains the transformation parameters of an affine 3D (i.e. 12-parameter) transformation, given as 12-array
-restoreOrderrestore natural order of data
Type: bool
Remarks: optional
In case the odm was created with the 'storeOrder' parameter, it is possible to export the data in natural order. Otherwise the internal (=spatially sorted) order is used.
-limitdata limit window: left lower right upper
Type: opals::Array<double, 4>
Remarks: optional
If specified, only data within the given window area (left lower right upper) are considered for export.Since a spatial range query is performed to access the data, this parameter cannot be combined with the 'restoreOrder' flag.
-tileSizetile size length for ODM files
Type: double
Remarks: optional
This parameter is only relevant if exported into an ODM and the automatic tile size estimation should be overruled. Inappropriate tile sizes can lead to memory problems or slow performance in later processing steps. On average, 100,000 to 200,000 points should be stored within one tile (check index statistics after export).

Examples

The examples shown in this section rely on the full waveform ALS data set fullwave.fwf located in the $OPALS_ROOT/demo/ directory and can be imported into the ODM by the following command:

opalsImport -inFile fullwave.fwf

XYZ

To export the point cloud in a simple text XYZ format, type:

opalsExport -inFile fullwave.odm -outFile odmExport.xyz

The resulting text file contains the 3D-coordinates only and starts with the following lines:

24820.774 311160.141 322.452
24820.035 311161.159 319.200
24820.599 311160.576 322.863
24820.348 311160.921 321.763
...

SDW

However, the imported file contains additional attributes from full waveform data processing (GPS time, amplitude, echo width, ...) which is well preserved in the respective ODM file. To maintain this information, data formats capable of storing full waveform information like Riegl SDW should be used. The following example also illustrates how filters can be used to export a limited set of data from the ODM (i.e. all last echoes in this case):

opalsExport -inFile fullwave.odm -outFile odmExport.sdw -filter Echo[last]

LAS

To demonstrate further possibilities offered by Module Export, a few more preparation steps are carried out. First, the surface normals are calculated and, subsequently, the echo ratio is determined for each point.

opalsNormals -inFile fullwave.odm -neighbours 8 -normalsAlg robustPlane -searchRadius 2 -storeMetaInfo medium
opalsEchoRatio -inFile fullwave.odm -searchRadius 2 -ratioMode slopeAdaptive

After executing the above commands the ODM file additionally contains information about the surface normals (additional info: NormalX, NormalY, NormalZ, NormalSigma0, NormalEigenValue1, ...) and the echo ratio (additional info: EchoRatio).

Module Export also supports the ASPRS LAS file format, which is widespread in ALS data processing. While any version of the LAS file format standard allows a user-defined number of bytes to be appended to each point record, only LAS 1.4 has introduced means to document these Extra Bytes. OPALS supports the import (see Module Import) and export of ODM attributes from/to LAS 1.4 files by usage of a LAS Format Definition . To export the attributes calculated above, type

opalsExport -inFile fullwave.odm -outFile odmExport.las -oFormat $OPALS_ROOT/addons/formatdef/LAS_1.4_extra_bytes.xml

at the command prompt, using the OPALS Format Definition mentioned here. In general, it is also possible to provide the format definition as inline XML string to the oFormat parameter (also see OFD as Parameter). In case of exporting attributes a xml-file might be more preferable but for simple situation where only the LAS version (and point data record format) is relevant the string option is well suitable. As shown below, abbreviation are allowed in the same manner as for module parameter in the command line (all commands produce the same output file).

opalsExport -inFile fullwave.odm -outFile export1_4.laz -oFormat "<las versionMinor='4' pointDataRecordFormat='6' />"
opalsExport -inFile fullwave.odm -outFile export1_4.laz -oFormat "<las ver='4' point='6' />"
opalsExport -inFile fullwave.odm -outFile export1_4.laz -oFormat "<l v='4' p='6'/>"

For completeness, an example of exporting the attribute NormalSigma0 as extra bytes using an inline XML string is also given:

opalsExport -inFile fullwave.odm -outFile export_sig0.laz -oFormat "<l v='4' p='6'><extraByte name='NormalSigma0'/></l>"

Generic Format

The extra attributes per point can also be output in a user-defined text or binary OPALS Generic Format, which is supplied to Module Export as the path to an OPALS Format Definition file, e.g.

<opalsFormatDefinition>
<description>XYZ Normals EchoRatio Text Format</description>
<!-- Example for an OPALS Format Definition (OFD) that stores coordinates, normals, and EchoRatio
Fit for both Import and Export -->
<text>
<header text='# x y z nx ny nz er
#-------------------------------------------------------------------------' />
<column name='x' format='12.3' />
<column name='y' format='12.3' />
<column name='z' format='9.3' />
<column name='NormalX' format='8.4' invalidValue='-99.9999' />
<column name='NormalY' format='8.4' invalidValue='-99.9999' />
<column name='NormalZ' format='8.4' invalidValue='-99.9999' />
<column name='EchoRatio' format='6.2' />
</text>
</opalsFormatDefinition>

The above format definition is an example for storing normal vector (NormalX, NormalY, NormalZ) and echo ratio (EchoRatio) information together with the point coordinates (x,y,z). It is provided as normalsEchoRatioAscii.xml in the directory $OPALS_ROOT/addons/formatdef of the OPALS distribution. Executing the command ...

opalsExport -inFile fullwave.odm -outFile odmExport.dat -oFormat $OPALS_ROOT/addons/formatdef/normalsEchoRatioAscii.xml

... will produce a file starting with:

# x y z nx ny ev1 er
#-------------------------------------------------------------------------
24820.774 311160.141 322.452 -0.1102 -0.9939 0.9450 52.63
24820.035 311161.159 319.200 -0.9520 0.2787 0.4300 100.00
24820.599 311160.576 322.863 -0.4301 -0.9028 0.9378 45.45
24820.348 311160.921 321.763 -0.9504 0.2013 0.6022 47.83
...

As OFD files are searched for in the current working directory as well as in the $OPALS_ROOT/addons/formatdef directory, the above command can be written in a more compact way omitting the full OFD file path:

opalsExport -inFile fullwave.odm -outFile odmExport.dat -oFormat normalsEchoRatioAscii.xml

The resulting file contains a file header consisting of 2 lines, as indicated by the respective header section of the format definition file. Furthermore, if an attribute to be exported is not available for a certain point, a default value is output, which is zero or an empty string, depending on the attribute type. This default value is overruled by specification of the XML-attribute invalidValue of element entry. Note that in the example shown above all point attributes are valid, however.

Cut out a window from a list of ODM files

opalsExport is an efficient tool to cut out rectangular regions from a large list of ODM files using the limit parameter. First the bounding of the ODMs are intersected with the given limit windows. Only in case of intersection the corresponding ODM is opened and a subsequent spatial query on the actual data is performed. Hence, the processing time depends on the amount of data that is covered by the given limit but it's (nearly) independent of number of files that are provided. Note that wildcard characters (*,?) can be used for defining multiple ODM files at once:

opalsImport -infile G111.las
opalsImport -infile G112.las
opalsImport -infile G113.las
opalsExport -inFile G11?.odm -outFile limitTest.las -limit -2000 338950 -1950 339000

Shape

ESRI Shape files can be utilised for data exchange
transporting geometry objects including their attributes. Using OFD Shape files specific attributes can be exported into the corresponding dbf file. In the following example single points including there fullwave attributes are exported based on the OFD fwfShape.xml which can be found in $OPALS_ROOT/addons/formatdef/

opalsImport -infile strip11.laz
opalsExport -inFile strip11.odm -outFile strip11.shp -oFormat fwfShape.xml

The resulting shape file can be loaded into e.g. QGIS to query/visualize the fullwave attributes of the points.

Fig. 1: Inspecting point attributes in QGIS

References

Otepka, J., Briese, C. and Nothegger, C., 2006. First steps to a topographic information system of the next generation. In: Symposium of ISPRS Commission IV - Geo Spatial Databases for Sustainable Development, Goa, India.

Mandlburger, G., Otepka, J., Karel, W., Wagner. W. and Pfeifer, N., 2009. Orientation and Processing of Airborne Laser Scanning data (OPALS) - concept and first results of a comprehensive ALS software. ISPRS Workshop Laserscanning 2009, Paris, France.

Author
jo,wk
Date
07.11.2017
@ storeMetaInfo
level of meta info that are stored (opalsNormals)
opalsNormals is the executable file of Module Normals
Definition: ModuleExecutables.hpp:148
@ ratioMode
echo ratio calculation mode
@ last
last stage to be processed (opalsStripAdjust)
opalsImport is the executable file of Module Import
Definition: ModuleExecutables.hpp:113
opalsExport is the executable file of Module Export
Definition: ModuleExecutables.hpp:73
@ slopeAdaptive
slope adaptive echo ratio (slope based extension of 3D radius) if slope available,...
@ filter
string to be parsed in construction of DM::IFilter (various modules)
@ robustPlane
Robust Plane.
@ medium
Storage of 'minimum' infos, eigenvalues, nr. of given points and nr. of used points.
opalsEchoRatio is the executable file of Module EchoRatio
Definition: ModuleExecutables.hpp:63