Table of Contents
This part describes the different categories and types of parameters supported by the OPALS framework, how they are specified via the 3 bindings and / or configuration files, and how to transfer parameter values of preceding module invocations to the current one using parameter files.
Each parameter is associated with a certain name, type, semantic, category, and optionality.
Parameter Categories
OPALS parameters belong to either of 3 categories:
- Specific parameters are peculiar to each module, having a module-specific type and semantic. For a list of all specific parameters supported by a certain module, see the respective section in the documentation of one of the modules.
- Common parameters control functionality that is implemented for all modules by the OPALS framework. Hence, they are supported by all modules, always having the same type and semantic. For a list of all common parameters, see the table of OPALS common parameters
- Global parameters also form a fixed set of parameters supported by all modules. However, the values of global parameters are thought to be constant for a certain OPALS installation, user, or project, and are thus expected to be specified in configuration files only. For a list of all global parameters, see the table of OPALS global parameters
Parameter Types
Every parameter is associated with a certain data type. All parameter types exposed by OPALS modules are either built-in data types defined by the C++ standard (int, double, etc.), container types defined in the C++ Standard Template Library (opals::Vector, opals::List, etc.), or custom data types declared in OPALS header files (e.g. enumerators like opals::LogLevel::Type). Thus, when using the C++ - interface and dynamic linkage, values of these data types may be instantiated and passed directly. However, when using OPALS modules as executables or Python modules, values that are convertible to the respective internal type need to be passed, where the set of convertible types / values depends on both the data type and the binding in use: parameter values for
- executables are always passed as strings, while values for
- Python modules are passed as Python types that are 'similar' to the respective C++ type
Parameter types can be classified into
- scalars, holding a single value only:
- numeric scalars
- (character) strings
- enumerators
- custom scalar types other than enumerators, e.g. file paths (opals::Path)
- containers, which hold one or more scalars of the container's element type
- containers with variable size: opals::Vector, opals::List
- fixed-size container: opals::Array, opals::Matrix
Referencing the installation root directory
Parameters that specify file paths are of type opals::Path. When encountering parameter values of type opals::Path that begin with the (case-sensitive) string $OPALS_ROOT, the OPALS framework internally replaces this sub-string with the installation's root directory. Thus, "$OPALS_ROOT/addons/pal/standardPalette.xml" is translated internally to the absolute path of the standard palette file.
Parameter Optionalities
A parameter's optionality defines if users must, may, or must not set its value:
- mandatory parameters must be specified;
- estimable parameters and parameters with a default may be given a custom value;
- optional parameters may be given a value; if not, respective functionality is skipped;
- output parameters hold results only. Hence, they must not be set, but only queried after the program run;
The optionality of each parameter is given in the respective parameter description.
Specification of Parameters on the Command-Line
OPALS executables may be invoked from the operating system's shell / command prompt by calling the respective file path. The characters following this path are split into a list of character strings (tokens) according to the system's rules. Under Windows, blanks are interpreted as token separators. This separation may be circumvented by enclosure in double quotes (for further information, see the respective page in MSDN).
The list of tokens is then interpreted by the OPALS framework:
- tokens beginning with one or two hyphens ('-') are interpreted as parameter names (excluding the hyphens)
- all tokens up to the next parameter name are considered as representing associated parameter values. These tokens are then converted to their internal representation. Thus, users may be confronted with 2 difficulties resulting from the syntax in use:
- (character) strings including blanks that should be converted to a single parameter value must be prevented from being split into different tokens using the operating system's rules (e.g. enclosure in double quotes). Above all, this may be necessary for file paths.
- tokens that start with a hyphen and represent parameter values: these tokens may need to be prevented from being recognized as parameter names by prefixing the hyphen with a backslash ('\').
Values extracted from the passed tokens are converted and set in alphabetic order of parameter names, except for the following common parameters, which are processed first, and in the following order:
Example
The following code submitted on a command-line on Windows calls Module Import, sets the minimum importance of log messages to be exported to file to opals::LogLevel::verbose, and imports 2 files:
Specification of Parameters in the Python Shell
In the Python shell, specific parameters are accessible as Python attributes of OPALS Python module instances. Of course, numeric types expect numeric Python types to be passed - the same holds for character strings. Other C++ types expect the following counterparts from the Python shell:
- opals::Path expects a Python string
- C++ enumerators and some custom OPALS types are provided as custom Python types located in the sub-package opals.Types, e.g. opals.Types.LogLevel.info, and opals.Types.HistoStats
- Other custom scalar types must be passed as strings
- Container types must be passed as Python lists. Instead of a single element list, a scalar may be passed as well.
Unlike specific parameters, common and global parameters are not accessible as direct attributes of an OPALS Python module instance. Instead, every module instance has the attributes "commons" and "globals", which in turn have these parameters as attributes.
Example
The following Python code snippet instantiates Module Import, sets the minimum importance of log messages to be exported to file to opals::LogLevel::verbose, and imports 2 files:
Note: by default, Python interprets backslashes contained in strings as escape characters. As shown above, this can be circumvented by prefixing the string definition with 'r'.
Specification of Parameters in Configuration Files
Parameter values may not only be specified on the command-line or via the Python / C++ interface, but all bindings of OPALS modules additionally parse configuration files at certain locations and set parameters from these, if present:
$OPALS_ROOT/cfg/opals.cfg
where$OPALS_ROOTdenotes the root directory of the OPALS installation- <home>/opals/opals.cfg
where <home> denotes the user's home path, e.g. "C:\Documents and Settings\Willi\" under Windows - <cwd>/opals.cfg
where <cwd> denotes the current working directory - <user-configuration-file>
only parsed if set on the command-line / via the interface, or in one of the configuration files above, using the common parameter 'cfgFile'
As the same parameter may be present in more than 1 configuration file, a precedence has been defined: parameter values specified in configuration files mentioned later in the list above override those specified in files listed earlier. Kindly note that this also holds true for container types: values specified in different configuration files do not add (while multiple definitions in the same file do, see below). Furthermore, parameter values specified via the command-line / interface override any corresonding values found in (non-user-) configuration files.
Note that user-configuration-files are processed (and respective values are set) immediately. Thus, if a parameter value is specified both in a supplied user-configuration-file and via the interface, the order of respective member function calls is of importance.
Configuration File Syntax
Configuration files are INI-like ASCII text files that are interpreted line by line. Lines containing white-space only are ignored, as all characters including and following a hash mark ('#') are. Parameter values are generally specified using the syntax
i.e. the character string up to the first equals sign ('=') (excluding leading and trailing white-space) is interpreted as the name of the parameter that the value(s) represented by the character string following the equals sign shall be assigned to. The string representing the value(s) to be assigned is split into a list of tokens in the same way as is done on the command-line (see Specification of Parameters on the Command-Line). Multiple specifications of container type parameters in the same configuration file add up. Thus, several elements may be assigned to a container type parameter in either of two ways:
Using this syntax, the value of the parameter associated with the respective name will be set for any module that supports this parameter. However, setting the value can be restricted to a certain module only using the following sytax:
The same effect can be achieved by placement of the module name in squared brackets on a preceding line:
Example
Variable Support in Configuration Files
OPALS supports variable substitution within configuration files. It is possible to refer to other parameters in the value string as shown below (The concept is also known as (extended) interpolation in the configparser class in python). The feature can be applied e.g. within package script configuration files for defining parameter values only once or defining all adaptable parameters in a central section called [script.variables]. Parameters from the current section or the [script.variables] section can be referenced without section prefix. Reference from other sections require the full section prefix. E.g.
An example using the central variable section is in the following
For completeness it is mentioned that the substitution mechanism is working recursively up to a depth of 10. Hence, a variable that refers to an other variable is fully supported.
The substitute implementation searches for the ${...} pattern in all parameter values. Currently, there is no way of disabling the substitution mechanism. Since the distinctive pattern may not occur in OPALS filter syntax and is unlikely to be used within file names, this shortcoming should be of minor concern.
Parameter Mapping
The invocation of each module's central function (run()) returns a list of all final module parameters, possibly including unchanged input-parameters, but more interestingly also (estimated) paths of exported files and computed results not stored to file. The list of final parameters may be
- stored to a parameter file (in XML format) using the common parameter 'outParamFile' (must be set before run() when using Python / C++)
- assigned to a variable on the calling side when using a Python or C++ binding
This list of final parameters may then be used to set the parameters of a later module call (probably of another module) by:
- reading the previously written file using the common parameter 'inParamFiles', or
- passing the list in memory using the variable from the calling side (Python and C++ only) and the function 'setParams(...)' present in all modules
When passing a list of parameters as input to the current module, the common parameter 'paramMapping' comes into play, as it specifies which parameters in the list shall be assigned to which parameters of a module. The syntax used is somehow similar to the one used in configuration files.
Takes the last parameter with name 'ParamNameToUse' found in the (ordered) list of parameters, and assigns it to the parameter named 'ParamNameToAssign' of the current module. This mapping of output to input parameters may be restricted to output parameters from certain modules only:
Assigns the parameter named 'ParamNameToUse' of the last call of only the module named 'OutputModule' to the parameter named 'ParamNameToAssign' of the current module. Likewise, parameters may be mapped to parameters of certain modules only:
Assigns the parameter named 'ParamNameToUse' to the parameter named 'ParamNameToAssign' only, if the current module is named 'InputModule'. These two types of restrictions may be combined to result in a most restrictive mapping:
Assigns the parameter named 'ParamNameToUse' from the last call of 'OutputModule' to the parameter named 'ParamNameToAssign' of the current module, if it is named 'InputModule'.
Multiple mappings to the same parameter must be separated by commas:
Assigns the last ocurrence of any parameter either named 'ParamNameToUse' or 'AnotherParamNameToUse' to the current module's parameter named 'ParamNameToAssign'.
Mappings to different parameters must be separated by semicolons:
Concerning container types to be assigned, parameter mapping works in a greedy manner: while scalar parameters and the fixed-size container opals::Array take exactly as many values from the list as needed (1 or N, resp., beginning at the end of the list), all values found in the list that satisfy the mapping rules will be assigned to vector-type parameters.
As the OPALS framework does not check the types of parameters to be mapped beforehand, it is the user's responsibility to assure that values to be used are convertible to the types of parameters to be assigned.
Note: as the character string specifying the value of the common parameter 'paramMapping' is split into tokens like any other parameter value on the command-line or configuration file, it is necessary to enclose it in double quotes under Windows, if it contains white-space.
Data-Dependent Conditions
In addition to restrictions based on module and parameter names, the parameter values of the current module and the parameter in question may optionally be combined to a logical expression. If for a certain mapping, the restrictions on module and parameter names are fulfilled, then the value from list/file is only assigned to the corresponding value of the current module, if this expression evaluates to true. If given, expressions must follow the mapping, separated by white space, and be formulated in Python syntax. The current module's parameters are exposed through the variable named 'me', having attributes only for the currently set parameters. The parameter under question may be accessed via the variable 'you'. 'you' and the exposed attributes of 'me' are fully typed, why respective methods/member functions may be invoked. Depending on the complexity of the involved parameter types, elaborate expressions may be composed, eventually using functions built into Python. As the parameter mapping string itself must be double quoted, it is advisable to enclose string constants in single quotes.
Example
In a configuration file, set the parameter mapping for all modules in a way that the final parameter 'outFile' of Module Import will be assigned to the parameter 'inFile' of Module Grid :
The same effect can be achieved by setting the parameter 'paramMapping' for Module Grid only:
The following code submitted to the command-line writes the final parameters of Module Import to a parameter file. The subsequent call to Module Grid uses this file in combination with the parameter mapping defined as above in a (default-) configuration file to set the ODM file path as input:
The code snippet below shows a parameter mapping being restricted by a data-dependent expression: a transformation of type opals::TrafPars3dAffine produced by opalsGeorefApprox shall be assigned to the parameter 'trafo' of opalsExport (having the same type). Separated by white space, the condition in Python syntax follows the actual mapping, restricting the set of candidate parameters to those whose member 'IdGridMov' is a prefix of the current module's inFile - strictly speaking, the respective file stems are compared:
- The grid ID of the moving grid which the candidate transformation parameter represented by 'you' refers to, is extracted using opals::TrafPars3dAffine's member function getIdGridMov() .
- Using the resulting string, an opals::Path is constructed, whose file stem (file name without extension) is returned from opals::Path's member function stem() .
- The value of parameter 'inFile' of type opals::Path of the current module is assessed via the respective attribute of 'me'.
- Again, a string results from stem() .
- As a Python string results, we may directly apply its method startswith, which is used to check whether the string originating from 'you' is a prefix. paramMapping = "trafo = opalsGeorefApprox.outTrafPars me.inFile.stem().startswith(Path(you.getIdGridMov()).stem())"
The following Python script passes the list of final parameters of Module Import as input to Module Grid, using the variable 'results' for storage:
As in this simple example, only one scalar, numeric parameter is passed via the parameter file, of course the following approach is equally practical:
Parameters Supported by all Modules
OPALS Common Parameter
Remarks: default=<number of logical CPUs>
The number of threads to spawn in modules with parallel processing support
Remarks: default=info
Possible values: none ...... Suppress all logging output. error ..... Some failure that cannot be handled. Program execution is aborted. warning ... Some weird program state which can still be handled (e.g. 'poor matrix condition', 'poor data distribution') info ...... Some progress that may be interesting in everyday-use verbose ... Something that may help in understanding normal program behaviour debug ..... Anything that may help 'debugging' by means of a release-build
The minimum level of log message importance to print on the screen
Remarks: default=verbose
Possible values: none ...... Suppress all logging output. error ..... Some failure that cannot be handled. Program execution is aborted. warning ... Some weird program state which can still be handled (e.g. 'poor matrix condition', 'poor data distribution') info ...... Some progress that may be interesting in everyday-use verbose ... Something that may help in understanding normal program behaviour debug ..... Anything that may help 'debugging' by means of a release-build
The minimum level of log message importance to export to the xml-log
Remarks: default=0
Flush the screen log stream (stdout) each time a log message has been emitted. Useful for watching progress when stdout has been redirected to a file. Otherwise, it is flushed only when its buffer is full.
Remarks: default=opalsLog.xml
The path of the xml log file. Write permission needed.
Remarks: default=
specify the mapping of parameters read from file to those of the current module
Remarks: optional
Restrict the assignment of parameters in user-cfg-files to scope.moduleName.paramName
Remarks: default=0
Delete output file after processing if it does not contain a single valid data value
OPALS Global Parameter
Remarks: default=0
Remarks: optional
A default coordinate reference system is assigned for all data sets lacking a specific geo-referencing information.
The system can either be defined as an integer EPSG Code (EPSG:nnnn), as string in either OGC WKT or PROJ.4 syntax (WKT:...), or as a path to a (.prj) text file that contains such a string.
Remarks: default=4000000
As long as the ODM has less points in memory than the given limit, an additional tile is loaded in to memory. Once the limit is exceed and a new tile is requested, the ODM swaps unneeded tils onto disk based on a recent used list.
Remarks: default=10
If the log file exceeds this size, then it is archived by renaming it using the current timestamp.
Remarks: default=las
This format is used as default LIDAR file format in case no format was specified (e.g. opalsEpxort)
Remarks: default=float32
Possible values: byte ....... 8 bit unsigned integer uint16 ..... 16 bit unsigned integer int16 ...... 16 bit signed integer uint32 ..... 32 bit unsigned integer int32 ...... 32 bit signed integer float32 .... 32 bit floating point float64 .... 64 bit floating point cint16 ..... Complex 16 bit signed integer cint32 ..... Complex 32 bit signed integer cfloat32 ... Complex 32 bit floating point cfloat64 ... Complex 64 bit floating point
Modules that create output grid/raster files may create these with this cell/pixel data type.
Remarks: optional
The creation of datasets (grid, vector, tin...) may be controlled by one or multiple create options (e.g., coordinate precision, tile size, compression ...). For grid/raster datasets, the create option strings must correspond to the defintion of the respective GDAL driver. Please not, that only well formatted options are considered.
Remarks: default=_nx
normal-x grids contain the x-component of the surface normal unit vector
Remarks: default=_ny
normal-y grids contain the y-component of the surface normal unit vector
Remarks: default=_open
openness grids contain the local viewshed (i.e. maximum opening angle of a cone)
Remarks: default=_kmin
kmin grids contain the minimum curvature of a specific surface point
Remarks: default=_kmax
kmmax grids contain the maximum curvature of a specific surface point
Remarks: default=_kmean
kmean grids contain the mean curvature of a specific surface point: kmin = (kmin+kmax)/2
Remarks: default=_kgauss
kgauss grids contain the gaussian curvature of a specific surface point: kgauss = kmin*kmax
Remarks: default=_kminDir
kminDir grids contain the direction (azimuth) of the the minimum curvature in a specific surface point
Remarks: default=_kmaxDir
kmaxDir grids contain the direction (azimuth) of the the maximum curvature in a specific surface point
Remarks: default=_absKmaxDir
absKmaxDir grids contain the direction (azimuth) of the the maximum absolute curvature in a specific surface point
- Date
- 09.11.2011
