Table of Contents
- See also
- python.workflows.opalsQuality
Aim of module
Package script providing a complete processing chain for quality assessment of ALS flight strips.
General description
opalsQuality calculates DSMs and visualizations thereof, point density maps, strip difference maps, and estimates of strip difference vectors.
- See also
- Script documentation
The Python script opalsQuality is the main script for the corresponding opalsQuality package. It uses multiple sub-scripts to create different products for checking and documenting the quality of an ALS flight campaign. The following aspects are covered:
- Data import: This is a pre-requisite for all other steps. ALS point cloud data files are read into the ODM (one ODM per flight strip, script: python._import).
- Strip outline: Based on the ODMs, strip outlines (minimum bounding rectangle, convex hull, or alpha shape) are derived (script: python._bounds).
- Overlapping pairs: Based on the strip outlines the overlapping strip pairs are identified (script: python._overlap).
- Point density: Raster based analysis is performed based on the ODM and point density maps are created for each strip and for the overall project area (Python script qltDensity)
- DSM and visualizations: For each strip a digital surface model (DSM) is calculated and a color coded hypsometric map as well as a shaded relief map are derived (Python script qltDSM).
- Strip differences: Based on the DSMs and the list of overlapping strip pairs, difference models and color coded raster maps of the relative strip differences are derived. This is a basic product for assessing the quality of the flight strips orientation via visual inspection (Python script qltStripDiff).
- strip difference vectors: A more quantitative analysis is carried out for all overlapping strip pairs, by estimating a best fitting 3D-affine-transformation via an Least Squares Matching approach (Python script qltLSM).
- point- or patchwise deviations between ALS data and independently measured control points/patches. This checks the absoute accuracy of the captured ALS data.
- precision: Raster based analysis of the measurement noise, measured as standard deviation of the plane fit during DEM grid interpolation (Python script qltPrecision).
Masking out areas for quality assessment
For estimating the actual quality of the captured ALS data, it is recommended to devide the area into parts considered for the derivation of quality measures and those, which should be exluded or masked out, replectively. Reasons for masking areas are manifold; examples are: rugged surface areas, areas covered by vegetation, scan shadows, water bodies, etc.
For controlling the masked areas, Python script opalsQuality contains the following parameters mask, gridMask and recalcMaskFiles, which are explained in detail in the following:
The mask parameter enables the user to create masked files using custom formulas for attributes calculated by opalsGrid, namely the opalsGridFeature. In the documentation of this module you can find the complete list of GridFeatures that can be calculated and later used as masks. The formulas should be given as Generic filter formulas. The syntax for specifying n formulas for n GridFeatures in the mask parameter should look like this.
The formulas are then combined and used in the computation. The standard formula for masked computations is specified at the top cfg file, which looks like this.
Parameter gridMask allows to specify a formula, which whether a negative mask (gridMask:"!r") is being used, for example to mask out unwanted areas like lakes or rivers. A positive mask (gridMask:"r") can be used to only compute certain areas of interests and mask out all areas which are not within the specified rasterfile.
As shown above implicitly, the script parameter mask can either be set in the commandline with the known syntax or as a paramater in the [script.opalsQuality]-section in the cfg-File.
Within the subscript *_grid*, the mask is combined and the final formula string is reported to the user via the opalsLog. Figure 1 shows an example of how the formula looks like with the above provided components.
The example 6 and the example 7 aim to show two uses cases for masking.
Finally, the boolean parameter recalcMaskFiles can be used to improve the masking in case the results of a previous run of opalsQuality call for an adaption of the masked areas. When activated in the repeat call of opalsQuality, this forces the re-creation of the mask files even if skipIfExists is set and the mask files (from the previous run) exist on disk.
Storage structure of intermediate and final results
Python script opalsQuality produces a series of intermediate and final results, which are stored in separate folders. The default folder structure is documented in the following tables:
| Default Directory | Produced by | Comments |
|---|---|---|
import | qltDSM, qltStripDiff, qltDensity, qltLSM, qltPrecision | The folder contains the imported ODM files. All sub scripts start with importing the data into the OPALS Datamanager format. |
grid | qltDSM, qltStripDiff, qltLSM, qltPrecision | The folder contains the DEM raster files for each strip of the flight block as well as additional grid feature raster files (sigma0, excentricity...) used for (i) masking and (ii) precision calculation. In addition, the folder conains the calculated mask for each strip and the masked/unmasked versions of the primary DEM and the sigma0 feature raster file. |
bounds | qltStripDiff | The folder contains the outlines of each strip (vector files contiaining the strip boundary polygon) as basis for the derivation of strip overlaps. |
cell | qltDensity | The folder contains the point density raster maps for each strip and the mosaic composed of all strips. |
diff | qltStripDiff | The folder contains the strip height difference raster maps for each strip pair and the mosaic composed of all strip pairs. If masking is activated, a separate raster is created for the masked and the unmasked version. |
overlap | qltStripDiff | The folder contains the file stripPairs.txt containing the list of overlapping strip pairs. |
| Default Directory | Produced by | Comments |
|---|---|---|
opalsQuality\Density | qltDensity | The folder contains sub-folders for the color-coded density maps (visu) and statistical point density analysis (histo). |
opalsQuality\DSM | qltDSM | The folder contains sub-folders for the hill shadings (shading) and color coded height maps (zcoding). |
opalsQuality\LSM | qltLSM | The folder contains the estimated transformation parameters for overlapping strip pairs in tabular form (LSM_TrafPars.csv) and as a parameter XML file (lsmOutput.xml). |
opalsQuality\Precision | qltPrecision | The folder contains sub-folders for masked and unmasked versions of the precision estimates (masked / unmasked). Each of the two folders consists of two sub-folders containing the color-coded height precision maps (zcoding) and the statistical analysis (histo). Separate files are generated for each strip and the mosaic composed of all strips. |
opalsQuality\StripDiff | qltStripDiff | As above, the folder contains sub-folders for masked and unmasked versions of the strip height differences. Each of the two folders consists of two sub-folders containing the color-coded height difference maps (zcoding) and the statistical analysis (histo). Separate files are generated for each strip pair and the mosaic composed of all strip pairs. |
Parameter description
Remark : optional, default: True
| possible input | evaluates to |
|---|---|
| 1, true, yes, Boolean(True), True | Boolean(True) |
| 0, false, no, Boolean(False), False | Boolean(False) |
Remark : optional, default: ['sigma0:r<0.1', 'excentricity:r<0.8*1', 'gridMask:!r']
Remark : optional
Remark : optional, default: []
Description: optional slope limits (in degrees) for slope classes based analyses. e.g. 20 30 will result in three classes: <20, >=20 to <30, >=30
Remark : optional, default: False
| possible input | evaluates to |
|---|---|
| 1, true, yes, Boolean(True), True | Boolean(True) |
| 0, false, no, Boolean(False), False | Boolean(False) |
Remark : optional
Remark : optional, default: ('qltDSM', 'qltDensity', 'qltStripDiff', 'qltLSM', 'qltPrecision')
Remark : optional, default: False
Description: This parameter could be used if a new mask is applied and only the masked outputfilesshould be recalculated.
| possible input | evaluates to |
|---|---|
| 1, true, yes, Boolean(True), True | Boolean(True) |
| 0, false, no, Boolean(False), False | Boolean(False) |
Remark : mandatory
Description: Test
Remark : optional
Description: Specifies the directory where the final results are stored. If not specified, the results are created in the current working directory.
Remark : optional, default: current directory
Description: Specifies the directory where all intermediate results are stored. If not specified, a TEMP subdirectory is created in the current working directory. Additional subdirectories (one per involved module) are created.
Remark : optional, default: $OPALS_ROOT/cfg/opalsQuality.cfg
Description: The name of a configuration file containing all relevant calculation parameters is expected. If not specified, the default cfg files as provided by the OPALS distribution are used.
Remark : optional, default: current directory
Description: The default project directory is the current working directory, but can be easily changed by this parameter. All path parameters, if not specified as absolute paths, are interpreted relative to the project directory.
Remark : optional, default: True
Description: Skip processing if result already exists. In order to re-run current script it is useful to repeat the processing only if the respective output does not already exist. This allows for incremental processing of large projects.
| possible input | evaluates to |
|---|---|
| 1, true, yes, Boolean(True), True | Boolean(True) |
| 0, false, no, Boolean(False), False | Boolean(False) |
Remark : optional
Description: according to the number chosen as the value, independent processes are being started to parallelise (parts of) the program and therefore enhance the runtime.
Remark : optional, default: info
Remark : optional, default: info
Remark : optional
Description: Logger is usually provided by the opals framework.The user may provide their own logger object, but it has to function in the same way as the opals Logger.
Examples
The data used in the following examples can be found in the $OPALS_ROOT/demo/ directory.
Example 1
In the first example, only the strip data file list is specified as input parameter:
striplist.txt may, e.g., contain the following lines:
Please note, that in the above example, strip31 and strip32 are ignored, since the corresponding line starts with a hash (=comment) character. Thus, the quality assessment is only carried out for the rest of the strips (strip11, strip21, strip21 and strip 22). Since no further parameters are specified, the output files are created in the current working directory ($OPALS_ROOT/demo/), all intermediate files are stored in subfolders of $OPALS_ROOT/demo/ and the default configuration files stored in $OPALS_ROOT/packages/cfg are used.
Example 2
To use all strip*.laz files and store the resulting and temporary files each in a separate folders, type:
Example 3
For examples 1 and 2 to work, it is necessary that the current working directory is $OPALS_ROOT/demo/. To achieve the same results from any current working directory, using the -p parameter, pointing to the directory containing the strip files, is recommended.
Example 4
If only the strip differences should be calculated (disregarding already calculated intermediate outputs) for the overlapping strip pairs but not for the entire block, use the following command:
Please note, that parameter names can be abbreviated as long as they can be unique resolved. E.g. -m is too short since it matches -mosaic and -mask, but -mo is ok. Hence, the very same command as above can also be written as:
Example 5: Slope dependent analysis
Text to come...
Example 6: standard (negative) mask
This first example should demonstrate the use of a negative mask. In this computation, the quality evaluation of the streets are not of interest, hence they are masked out. The raster file containing the approximate shape of the street is a rasterized shapefile called street_mask_ras.tif (see Figure 2a), which can be found in $OPALS_ROOT/demo/. Since a negative mask is desired, we do not need to change the default values in the cfg file. opalsQuality can therefore be called using the following command,
resulting in masked computations as shown in Figure 2b. It is clear to see that the masked out areas where excluded from the computation.
Example 7: positive mask (aoi) and -recalcMaskFiles parameter
Following up on the hypothetical example of the street masks above: Let's say the streets are the only thing that is interesting for a certain application. In this case our mask file can be used as a positive mask to treat the street as an aoi. As only the masked files should be recaculated and all other files need not be computed again, the recalcMaskFiles parameter can be used to only compute files necessary for mask creation and reuse everything else. The opalsQuality call could therefore look like this:
Note the subtle change in the formula of GridMask within the mask parameter. Since "<" and ">" are reserved for redirections in the command shell, quotation signs are necessary to specify formulas that contain those symbols. The same can be achieved if the formula is changed within the cfg like so.
Then the call can look like this:
The result can be seen in Figure 3, now in the masked files only the street areas are taken into account.
1.8.17