The C++ OPALS Datamanager Library

The C++ OPALS Datamanager Library has been developed for efficient access to huge spatial data sets and for handling arbitrary attributes capsulated to geometry objects. It is a key component of OPALS, since all point/geometry processing modules access the data through this library. Although the library was mainly developed for OPALS, it does not depend on OPALS and therefore, can be seen as generic geometry managing framework. This is why the library is referred as DMlib (or DM) in the following.

The core features of the library are

  • management and storage of huge geometry data
  • flexible and dynamic extendable attribute schema for each geometry object
  • different spatial indices supporting different spatial queries
  • powerful filters for sub-selecting or modifying the geometry objects
  • multi-thread processing strategies for efficient data manipulation

This documentation mainly covers the technical aspects of the library interface. For details on the data organisation and handling concepts please refer to OPALS Datamanager section.

The C++ library favours the loose coupling principle. This means that the actual implementation details are hidden behind public interface. Using static creation functions (e.g. DM::IDatamanager::New) or factories in case of more complex creation situations (e.g. DM::IAddInfoLayoutFactory), new object can be constructed. Those function always return a pointer to the object. Hence it's the users responsibility to destroy the object to avoid memory leaks. Never delete an DMLib object using the standard C++ delete keyword. Always use the objects Delete member function (see DM::ObjectBase::Delete) (Since the object was created on the heap of the DLL, only the DLL itself has the ability to delete object correctly). The library provides a smart pointer class (see DM::Handle) that automatically destroys the referenced object if the last corresponding handle goes out of scope. So it highly recommended to use this template handle class to avoid memory leaks. For each interface class a corresponding handle typedef exists (DM::IDatamanager - DM::DatamanagerHandle, DM::IPoint - DM::PointHandle, etc.).

Internal Structure of the Datamanager

A DM::IDatamanager object can manage all a variety of different geometry object:

As it is the case for many applications (Laserscanning, photogrammetry, etc.) there will be a huge number of points to a comparatively lower number of other geometry object. This is why the datamanager provides two different indices. The first one is optimised for huge point clouds, but it is limited to points only (DM::IPointIndex). The second spatial index is designed for flexibility. It supports all geometry types, however, it cannot compete in terms of efficiency and spatial performance with the first spatial index. For simplicity this second spatial index is called polyline index (DM::IPolylineIndex) although, it can handle all geometry types (hence, it can manage points as well, but it's not recommended because of the previously mentioned reason)

Figure 1 shows the internal object structure of an datamanager object revealing the two separated spatial indices. The indices can be accessed separately or by the datamanager in a wrapped manner.

Fig. 1: Internal structure of the datamanager

As it can be seen the point index exposes the index leafs as well which contain the point objects. The leafs are represented as DM::IPointIndexLeaf objects, providing a second level spatial index securing high spatial performance. Within the datamanager the leafs are always written/read as single binary chunk to/from disk. Hence, if a leaf is changed/updated (even if only one point was changed) the full leaf has to be re-written to disk. However, this IO overhead is usually of minor concern since standard processing strategies changed all (or at least most) points. Additionally, bigger but sequential IO is usually faster than less non-sequential operations on hard disks. Further details on the different spatial indices can be seen in Figure 2.

Fig. 2: Internal structure of spatial indices

TO COME: Adding and removing points Updating attributes of points with Processor/Kernel concept

Managing Attributes

In the DMlib terminology a set of attributes is called AddInfo object (acronym for additional information).

Examples

Before starting our own development have a close look at the examples section. It contains a collections of various C++ and Python examples. Maybe your problem is already (partly) covered there.