Module Import
See also
opals::IImport

Aim of module

Loads vector and raster data into an ODM for subsequent use in modules dealing with point cloud and line related data.

General description

Module Import tries to preserve the full information of imported file formats. Therefore, the ODM contains an additional administration level, storing header and layer information for each file. Hence, it is possible to reproduce the original files from an ODM file (even in case of a multiple file import). Currently, Module Import supports 6 different vector file formats. Additionally, any of the supported raster formats may be imported. However, please notice that this is a rather rare use case: raster data imported into an ODM will be treated like any other data in an ODM i.e. the topological information defined implicitly by rasters is lost.

See also
Module Import

OPALS uses the OPALS Data Manager (ODM) for the administration of ALS point clouds as well as line related geometries (eg. breaklines, formlines, borderlines). All subsequent OPALS modules dealing with primary topographic data are based on a data manager file (*.odm) as data input. Thus, running Module Import is a pre-requisite step for many OPALS application modules like Module Grid, Module Cell, Module Bounds ...

The ODM can be seen as a database system with focus on spatial data administration. Since the data are organized in a spatial order, the ODM provides efficient spatial access. The second key feature of the ODM is the storage of an arbitrary number of attributes (in OPALS they are referred to as additional info) for each geometry entry. Moreover, additional infos (addInfo) can be dynamically added at any time, allowing OPALS modules a transparent propagation of computational results across module boundaries (eg. Module Normals which estimates surface normals in each point).

For more details on the ODM and its features please refer to OPALS Data Manager manual.

In general it is not necessary to specify the file format of the input file since OPALS automatically detects the format based on the file content (and NOT based on the extension). In case the recognition fails and for formats described by a generic opals format definition, the file format can be specified by the iFormat parameter. opalsImport supports the listed format labels as shown below. For GDAL file formats it is either possible to use the generic format label gdal or the specific GDAL driver name as listed here .

Format Description
wnp, bwnp Winput and binary Winput files
xyz, bxyz xyz and binary xyz files
las LAS files
shape ESRI shape file format
sdw Riegl SDW files
fwf IPF internal full waveform text format (version: 0.8, 0.9, 1.0, and 1.1)
<opals format definition xml file> Generic import of text and binary file
trj, btrj Text and binary trajectory files. For details see Trajectory file
scop SCOP DTM file format (Random Digital Height, RDH)
gdal GDAL readable file format (see Supported grid/raster formats)
<GDAL driver name> Use a specific GDAL driver to read the file. Driver names are listed in Supported grid/raster formats

To import multiple files one can set all input files at once or make multiple Module Import runs in appending mode. To append data to an ODM file define the corresponding ODM file as first input file or define the ODM file as input and output file.

For some LiDAR processing steps it is necessary to know the beam vector for each point. Therefore opalsImport provides the possibility to consider trajectory files during import. It is possible to specify a single trajectory file that is used for all input files or multiple trajectory files where each trajectory file corresponds to an input file. In case the gps time range of the trajectory file doesn't fully cover the gps time range of the corresponding input file an exceptions will be thrown. For situations where the trajectory is given in multiple files which do not match the time range of input files, it is necessary to concatenate all trajectory data into a single trajectory file. Beside the beam vector in the world coordinate system (default option) it is possible to append the range, the scan angle and the beam vector in the sensor coordinate system (SCS) (see parameter storeBeamInfo). The latter two options require the computation of the beam vector in the sensor coordinate system which is the inverse operation of opalsDirectGeoref (a mounting calibration is not supported yet). Please note that the correct transformation can only be applied if the world coordinate system (see global parameter coord_ref_sys) is specified correctly (For details refer to the section World coordinate system). The beam vector in SCS can be used e.g. to improve the mounting calibration for data sets where only the final word coordinate point data and the trajectory is available by creating artificial strip files in the sensor coordinate system.

The link between point data and trajectory is given by the gps time, which limits this import parameter to file formats containing gps time information. The flight position at shot time is interpolated in a piecewise linear manner within the trajectory data. OPALS supports different trajectory file formats including the generic import. The format has to provide the data columns x, y, z, gps time, roll, pitch and yaw. Please refer to section trajectory file for further details.

The ODM tiles point data for efficient handling of huge data sets. Since tiles are always completely loaded into the memory, appropriate tile size estimation is important. This is why a spatial index statistics is reported after import. Too big tiles can lead to memory problems, too small tiles increase the administration overhead. The point density of LiDAR data is usually quite homogeneous making the tile size estimation non-crucial. However, in rare situations it can be necessary to influence the estimation process by setting the tileSize or the tilePointCount parameter. Please refer to section Analysing the index statistics of an ODM for further details.

Finally, it should be mentioned that it is possible to filter data during import in a very flexible way. This can be relevant if only a subset of the data should be passed to a processing chain. For details on filters and how they are defined please refer to the Filter manual.

Parameter description

-inFileinput files
Type: opals::Vector<opals::VectorOrRasterFile>
Remarks: mandatory
The specified input files are imported in the same order as defined.
To append data to an ODM file, specify the ODM file as input and outFile. Note that wildcard characters (*,?) can be used for the selection of multiple input files at once (e.g. strip*.las)
VectorOrRasterFile: File path of a vector or raster dataset.
See Supported vector formats, OPALS Datamanager, and Supported grid/raster formats
-iFormatfile format [auto, xyz, bxyz, wnp, bwnp, las, shape, sdw, fwf, odm, scop, <opals format def. xml file>, gdal, <GDAL driver name>]
Type: opals::Vector<opals::String>
Remarks: estimable
In case of multiple input files with present iFormat parameter, the number of iFormat and inFiles entries have to match or only a single iFormat entry is specified (=all input files have the same format).
This also applies if an additional data file is to be appended to an existing ODM in which case the ODM has to be specified twice, as input and output.
Estimation rule: auto. The file content is used to recognise the file format.
-trjFiletrajectory files
Type: opals::Vector<opals::Path>
Remarks: optional
If trajectory file(s) are provided, trajectory information (e.g. beam vector, see parameter storeBeamInfo) for each point is computed and stored as attributes within the manager. Prerequisite are inFile formats that contain gps time information (las, sdw, fwf). Trajectory files have to contain the following information: x, y, z, gps time, roll, pitch and yaw
-tFormattrajectory file format [auto, trj, btrj, odm, <opals format def. xml file>]
Type: opals::Vector<opals::String>
Remarks: estimable
In case of multiple trajectory files with present tFormat parameter, the number of tFormat and trjFiles entries have to match or only a single tFormat entry is specified (=all trajectory files have the same format).
Estimation rule: auto. The file content is used to recognise the file format.
-outFileoutput manager file name
Type: opals::Path
Remarks: estimable
Estimation rule: The current path and the name (body) of the first input file are used as file name basis
-filtermodify the input using a (tree of) filter(s)
Type: opals::FilterWithPlaceHolders<DM::IFilter::ReadAccess::coordinatesAttributes, DM::IFilter::WriteAccess::coordinates | attributes | delayedResults>
Remarks: optional
Reduce and/or transform the input data set: import last echoes only, transform coordinates, modify attributes, etc.
FilterWithPlaceHolders<coordinatesAttributes, coordinates | attributes | delayedResults>: has read access to: coordinatesAttributes; has write access to: coordinates | attributes | delayedResults;
See Filters
-trafogeometrically transform the data during import
Type: opals::TrafPars3dAffine
Remarks: optional
An eventually passed affine 3-d transformation is applied during data import. Internally, the passed value is incorporated into 'filter'. If both are to be used, then mark with {trafo} the position where 'trafo' shall be introduced into 'filter'.
TrafPars3dAffine: Describes a relative or absolute 3D-affine transformation. This class contains the transformation parameters of an affine 3D (i.e. 12-parameter) transformation, given as 12-array
-storeOrderstore natural order of data
Type: bool
Remarks: optional
If set, the import order of the data is preserved, which allows processing and exporting data in the original order (Not implemented yet).
-tilePointCountaverage tile point count
Type: int32
Remarks: default=200000
After importing a specific number of points the odm estimates the point density of the data. Based on the estimated density and the specified tilePointCount a tile size is computed. After that, the ODM administers the current and all subsequent point data in a tiling strategy.
-tileSizetile size length of spatial index
Type: double
Remarks: optional
The tile size length (the ODM uses squared tiles) can only be set for new ODMs or ODMs that are not in tiling mode yet. This parameter should be use with caution, since inappropriate tile sizes can lead to memory problems or slow performance in later processing steps. On average, 100,000 to 200,000 points should be stored within one tile (check index statistics after import).
-storeBeamInfobeam information to attach while import with trajectory
Type: opals::Vector<opals::BeamInfo>
Remarks: default=BeamVector
Possible values:  
  BeamVector ...... Beam vector in the world coordinate system
  BeamVectorSCS ... Beam vector in the sensor coordinate system
  Range ........... Measurment range
  ScanAngle ....... Scan angle as defined by LAS standard
  PanAngle ........ The pan angle is the rotation of scanner head around primary axis of scanner. In case of a 1-D scanner, this is the varying angle around the only axis of the scanner
If trajectory files are provided, the module computes and attaches the specified beam information to each imported point.
-mountingmounting calibration parameters
Type: opals::MountingPars
Remarks: optional
In case beam information should be attached to each point, it is possible to optional consider mounting calibration parameters
-mergePolygonmerge polygon mode
Type: opals::MergePolygonMode
Remarks: default=safe
Possible values:  
  safe ... incremental merge of polygon parts (slow but robust)
  fast ... build polygon object at once (fast but fails in case of topological incorrect polygon parts)
Per default the datamanager imports nested polygons in a safe and incremental manner such that polygon parts that are topological incorrected, still result in valid polygon objects. However, this algorithm can be very slow in case of complex nested polygon parts. If secured that only topological correct polygons are imported, using 'fast' mode can result in a huge import performance gain.

Short examples

Import one or multiple point cloud files into an odm

opalsImport -inf strip11.laz
opalsImport -inf strip*.laz -outf strip_all.odm
from opals.Import import Import
Import(inFile='strip11.laz').run()
Import(inFile='strip*.laz', outFile='strip_all.odm').run()

Examples

All data used in the following examples are located in the $OPALS_ROOT/demo/ directory of the OPALS distribution. To import the LAS file strip31.laz into the ODM, simply type:

opalsImport -inFile strip31.laz

Since no output file is specified, the name of the corresponding ODM file is estimated from the input file name and the file ...

strip31.odm

... constituting the ODM is created.

To add more than one strip to a single ODM file, multiple input files can be specified in a single command:

opalsImport -inFile "strip31.laz" "strip21.laz" "strip11.laz" -outFile allStrips.odm

Please note that quotes are only necessary if the file names contain blanks, which is not the case in this example. For convenience multiple files can be specified using wildcard characters (*,?) as well:

opalsImport -inFile strip?1.laz -outFile allStrips.odm

The same result could have been achieved by calling Module Import multiple times and by appending the data to a single ODM.

opalsImport -inFile strip31.laz -outFile allStrips.odm
opalsImport -inFile allStrips.odm strip21.laz -outFile allStrips.odm
opalsImport -inFile allStrips.odm strip11.laz -outFile allStrips.odm

In case, the automatic recognition of the input files' data format fails, it is nonetheless possible to import the files, if the proper input data format is specified. If multiple input files are used, then the number of files must match the number of input formats. This also applies for appending additional data files to an existing ODM. Thus, the very same example as above can be written as:

opalsImport -inFile strip?1.laz -iformat las -outFile allStrips.odm

Data in generic text (or binary) format can be imported by providing a OPALS Generic Format file in XML format. The file fullwave.fwf, for instance, contains eight columns (x, y, z, time, amplitude, echo width, echo number, echo qualifier) per line (=data set).

24820.774 311160.141 322.452 314358.431470 16 2.606 2 0
24820.035 311161.159 319.200 314358.431470 69 1.736 3 1
24820.599 311160.576 322.863 314358.431485 18 3.541 1 2
...

With the following format definition file ...

<opalsFormatDefinition>
<description>XYZ Text Format</description>
<!-- Example for a simple XYZ Text Format Definition for Import and Export -->
<text commentInitiator='#' >
<column name='x' />
<column name='y' />
<column name='z' />
<column name='GPSTime' />
<column name='Amplitude' />
<column name='EchoWidth' />
<column name='EchoNumber' />
<column name='_echoQualifier' type='int8' />
</text>
</opalsFormatDefinition>

... stored as myFormatDef.xml, the file can be imported as follows:

opalsImport -inFile fullwave.fwf -iFormat myFormatDef.xml

In fact, the file satisfies the I.P.F. full waveform format specification, and thus there is no need for a user defined format definition file in this place. But instead, sometimes it is desired to read only specific parts of a data file. Filters can be used to read, e.g. only the first echoes or echoes with amplitudes above a certain threshold.

opalsImport -inFile fullwave.fwf -iFormat fwf -filter Echo[First]
opalsImport -inFile fullwave.fwf -iFormat fwf -filter "Echo[First] and Generic[Amplitude>75]"

Please mind that quoting the filter string is necessary in the latter case due to the embedded blanks and special characters. It is recommended always to quote complex filter strings. For more detailed information about selective data import please refer to the Filter manual.

Computing beam info while data import

For some processing tasks, such as radiometric calibration and rigorous strip adjustment, beam information is required. Whereas the beam information is directly given for data in sensors coordinate system, sometimes only the final coordinates in the world coordinate reference system are available. In such situation the inverse operation to Module DirectGeoref is required. So in case the trajectory file exists and the input data file contain the corresponding GPS time stamps, Module Import can compute different beam informations and attach the values as attributes to the imported points.

In the following we re-compute sensors coordinates from a LAS file (world coordinate reference system) that was processed by Module DirectGeoref as shown in DirectGeoref Example 1. In case 060622_173547_dirGeoref.las doesn't exist yet, run the following commands in the $OPALS_ROOT/demo/ directory:

opalsFullwave -infile 060622_173547.sdf -detectThrLow 10 -outfile 060622_173547.sdc
opalsDirectGeoref -inFile 060622_173547.sdc -mounting "SCANNERSYS(D-F-R), MOUNTROTATION=LOCAL(ANGLES(-0.060 0.072 -0.325)), MOUNTSHIFT(-0.052 0.151 0.083), TIMELAG(-0.0032)" -trjFile F002_20060622_UTM33_part.txt -oformat las -coord_ref_sys EPSG:32633

Now, process the following command which imports the LAS file (world coordinate reference system) and appends the sensors coordinates of each points as attributes (_BeamVectorSCSX, _BeamVectorSCSY, _BeamVectorSCSZ)

opalsImport -inFile 060622_173547_dirGeoref.las -storeBeamInfo BeamVectorSCS -mounting "SCANNERSYS(D-F-R), MOUNTROTATION=LOCAL(ANGLES(-0.060 0.072 -0.325)), MOUNTSHIFT(-0.052 0.151 0.083), TIMELAG(-0.0032)" -trjFile F002_20060622_UTM33_part.txt -coord_ref_sys EPSG:32633

Using Module Info the attributes and their range of the newly created ODM can be checked

Sensors coordinates attached as attributes

By converting the original SDC file into a simple xyz text file the range of the sensors coordinates will be output at end

opalsTranslate -inf 060622_173547.sdc -outf 060622_173547.xyz

allowing for a simple comparison

...
Output file 060622_173547.xyz
Nr. points 80305
Nr. polylines 0
Nr. polygons 0
Minimum X-Y-Z 29.133 -0.282 -258.424
Maximum X-Y-Z 885.514 0.110 362.576
...
Author
J. Otepka, G. Mandlburger
Date
29.05.2016

References

Otepka, J., Briese, C. and Nothegger, C., 2006. First steps to a topographic information system of the next generation. In: Symposium of ISPRS Commission IV - Geo Spatial Databases for Sustainable Development, Goa, India.

Mandlburger, G., Otepka, J., Karel, W., Wagner. W. and Pfeifer, N., 2009. Orientation and Processing of Airborne Laser Scanning data (OPALS) - concept and first results of a comprehensive ALS software. ISPRS Workshop Laserscanning 2009, Paris, France.

@ mounting
compute 3D-shifts and flight direction aligned rotations (roll-pitch-yaw)
@ BeamVectorSCS
Beam vector in the sensor coordinate system.
opalsTranslate is the executable file of Module Translate
Definition: ModuleExecutables.hpp:238
@ Y
sessions.adjustment.leverArm group(opalsStripAdjust)
@ coord_ref_sys
default coordinate reference system (EPSG Code, WKT string or PRJ-File)
opalsDirectGeoref is the executable file of Module DirectGeoref
Definition: ModuleExecutables.hpp:58
opalsImport is the executable file of Module Import
Definition: ModuleExecutables.hpp:113
@ Z
sessions.adjustment.leverArm group(opalsStripAdjust)
@ strip
strip ID of an image (opalsStripAdjust)
opalsFullwave is the executable file of Module Fullwave
Definition: ModuleExecutables.hpp:83
@ X
sessions.adjustment.leverArm group(opalsStripAdjust)
@ filter
string to be parsed in construction of DM::IFilter (various modules)
@ storeBeamInfo
defines beam information that is attached during import (opalsImport)