Manual Data Processing

In addition to processing data from connectd devices and/or loaded data from disk, fluxEngine also allows the user to process data they have in memory. This is more advanced, as fluxEngine requires the user to specify precisely how the data is layed out in memory.

Creating a processing context

In order to process data with fluxEngine, a processing context must be created. A processing context knows about:

  • The model that is to be processed

  • How processing will be parallelized (it takes that information from the current handle) – changing parallelization settings will invalidate a given context

  • What kind of data is to be processed each time (full HSI cubes or individual PushBroom frames)

  • The size of data that is to be processed. fluxTrainer models are designed to be camera-independent (to an extent), and thus do not know about the actual spatial dimensions of the data that is to be processed. But once processing is to occur, the spatial dimensions have to be known

  • The input wavelengths of the data being processed. While a model build in fluxTrainer specifies the wavelengths that will be used during processing, cameras of the same model don’t map the exact same wavelengths onto the same pixels, due to production tolerances. For this reason cameras come with calibration information that tells the user what the precise wavelengths of the camera are. The user must specify the actual wavelengths of the input data, so that fluxEngine can interpolate those onto the wavelength range given in the model

  • Any white (and dark) reference data that is applicable to processing

There are two types of processing contexts that can be created for manual data processing: one for HSI cubes, one for PushBroom frames.

HSI Cube Processing Contexts

To process entire HSI cubes the user must use the constructor of ProcessingContext that takes a ProcessingContext::HSICube argument. It has the following parameters:

  • The model that is to be processed

  • The storage order of the cube (BSQ, BIL, or BIP)

  • The scalar data type of the cube (e.g. 8 bit unsigned integer)

  • The spatial dimensions of the cube that is to be processed

  • The wavelengths of the cube

  • Whether the input data is in intensities or reflectances

  • An optional set of white reference measurements

  • An optional set of dark reference measurements

There are two ways to specify the spatial dimensions of a given cube. The first is to fix them at this point, only allowing the user to process cubes that have exactly this size with the processing context. The alternative is to leave them variable, but specify a maximum size. This has the advantage that the user can process differently sized cubes with the same context, but has the major disadvantage that if a white reference is used, it will be averaged along all variable axes, that means that any spatial information of the reference data will be averaged out. (It is also possible to only fix one of the spatial dimensions.)

For referencing it is typically useful to average multiple measurements to reduce the effect of noise. For this reason, any references that are provided have to be tensors of 4th order, with an additional initial dimension at the beginning for the averages. For example, a cube in BSQ storage order has the dimension structure (λ, y, x), so the references must have the dimension structure (N, λ, y, x), where N may be any positive number, indicating the amount of measurements that is to be averaged. A cube in BIP storage order would have a dimension structure of (y, x, λ), leading to a reference tensor structure of (N, y, x, λ).

Note

It is possible to supply only a single cube as a reference measurement, in that case N would be 1. In that case the structure of the data is effectively only a tensor of third order – but the additional dimension still has to be specified.

Note

References supplied here must always be contiguous in memory, and may never have any non-trivial strides.

Reference cubes must always have the same storage order as the cubes that are to be processed.

The first example here shows how to create a processing context without any references, assuming that the input data is already in reflectances, with a 32bit floating point data type, and fixed spatial dimensions:

 1 using namespace fluxEngine;
 2
 3 try {
 4     std::int64_t width = 1024, height = 2150;
 5     std::vector<double> wavelengths{{ 900, 901.5, 903, ... }};
 6     ReferenceInfo referenceInfo;
 7     referenceInfo.valueType = ValueType::Reflectance;
 8     ProcessingContext context(model, ProcessingContext::HSICube,
 9         HSICube_StorageOrder::BSQ, DataType::Float32,
10         height, height, width, width, wavelengths,
11         referenceInfo);
12 } catch (std::exception& e) {
13     std::cerr << "An error occurred: " << e.what() << std::endl;
14     exit(1);
15 }

Alternatively, to create a processing context that uses a white reference cube, and where the y dimension has a variable size, the following code could be used:

 1 using namespace fluxEngine;
 2
 3 try {
 4     std::int64_t width = 1024, height = 2150;
 5     std::vector<double> wavelengths{{ 900, 901.5, 903, ... }};
 6     auto wavelengthCount = static_cast<std::int64_t>(wavelengths.size());
 7     ReferenceInfo referenceInfo;
 8     referenceInfo.valueType = ValueType::Intensity;
 9     /* get cube data from somewhere
10     * (this pointer only has to be valid until the call to
11     * the constructor has completed, then the user may
12     * discard the datea)
13     */
14     uint8_t* whiteReferenceData = ...;
15     referenceInfo.whiteReference = whiteReferenceData;
16     /* Just a single cube, with a height of just 40 in y direction,
17      * but the width being the same as the fixed width that is set
18      * here.
19      */
20     referenceInfo.whiteReferenceDimensions = { 1, wavelengthCount, 40, width };
21     ProcessingContext context(model, ProcessingContext::HSICube,
22         HSICube_StorageOrder::BSQ, DataType::UInt8,
23         height, -1, width, width, wavelengths,
24         referenceInfo);
25 } catch (std::exception& e) {
26     std::cerr << "An error occurred: " << e.what() << std::endl;
27     exit(1);
28 }

Note

The maximum size specified here also determines how much RAM is allocated in fluxEngine internally. Specifying an absurdly large number will cause fluxEngine to exhaust system memory.

Note

The white and dark references may have different spatial dimensions if those dimensions are specified as variable. In the above example, if a dark reference were to be specified, it would have to have the same width and number of bands (because those are both fixed), but it could have a different height.

PushBroom Frame Processing Contexts

To process one PushBroom frame after another the user must use the constructor of ProcessingContext that takes a ProcessingContext::PushBroomFrame argument. It has the following parameters:

  • The model that is to be processed

  • The storage order of the PushBroom frame

  • The scalar data type of each PushBroom frame

  • The spatial width of each PushBroom frame (which will be the actual with of each image if LambdaY storage order is used, or the height of each image if LambdaX storage order is used)

  • The wavelengths of each PushBroom frame

  • Whether the input data is in intensities or reflectances

  • An optional set of white reference measurements

  • An optional set of dark reference measurements

As PushBroom processing can be thought of as a means to incrementally build up an entire cube (but process data on each line individually), the spatial width must be fixed and cannot be variable. (The number of frames processed, i.e. the number of calls to ProcessingContext::processNext() is variable though.)

As it is often useful to average multiple reference measurements to reduce noise, the white and dark references must be supplied as tensors of third order, with a dimension structure of (N, x, λ) or (N, λ, x), depending on the storage order.

The following example shows how to set up a processing context without any references, assuming the input data is already in reflectances, stored as 32bit floating point numbers:

 1 using namespace fluxEngine;
 2
 3 try {
 4     int64_t width = 320;
 5     std::vector<double> wavelengths{{ 900, 901.5, 903, ... }};
 6     ReferenceInfo referenceInfo;
 7     referenceInfo.valueType = ValueType::Reflectance;
 8     ProcessingContext context(model, ProcessingContext::PushBroomFrame,
 9         PushBroomFrame_StorageOrder::LambdaY, DataType::Float32,
10         width, wavelengths, referenceInfo);
11 } catch (std::exception& e) {
12     std::cerr << "An error occurred: " << e.what() << std::endl;
13     exit(1);
14 }

Alternatively, if both white and dark reference measurements are to be supplied, and unsigned 8bit integer numbers, one could use the following code:

 1 using namespace fluxEngine;
 2
 3 try {
 4     int64_t width = 640;
 5     std::vector<double> wavelengths{{ 900, 901.5, 903, ... }};
 6     auto wavelengthCount = static_cast<std::int64_t>(wavelengths.size());
 7     ReferenceInfo referenceInfo;
 8     referenceInfo.valueType = ValueType::Intensity;
 9     /* Get the reference data from somewhere; in this case the white
10      * reference contains 5 measurements that are to be averaged,
11      * and the dark reference contains 10. Since this is in LambdaY
12      * storage order, the references are effectively cubes of BIL
13      * storage order themselves. (For PushBroom frames with LambdaX
14      * storage order the references would be cubes in BIP storage
15      * order.)
16      */
17     std::uint8_t* whiteReferenceCubeData = ...;
18     std::uint8_t* darkReferenceCubeData = ...;
19     referenceInfo.whiteReference = whiteReferenceCubeData;
20     referenceInfo.whiteReferenceDimensions = { 5, wavelengthCount, width };
21     referenceInfo.darkReference = darkReferenceCubeData;
22     referenceInfo.darkReferenceDimensions = { 10, wavelengthCount, width };
23     ProcessingContext context(model, ProcessingContext::PushBroomFrame,
24         PushBroomFrame_StorageOrder::LambdaY, DataType::UInt8,
25         width, wavelengths, referenceInfo);
26 } catch (std::exception& e) {
27     std::cerr << "An error occurred: " << e.what() << std::endl;
28     exit(1);
29 }

Processing Data

Once a processing context has been set up, the user may use it to process data. This happens in two steps:

  • Set the data pointer for the source data that is to be processed

  • Process the data

The first step has to be called at least once before processing happens. If the user always writes the data to the same memory location, it can be skipped for subsequent processing steps.

HSI Cubes Source Data

In order to process entire HSI cubes, a processing context for HSI cubes has to have been set up. (See the previous section.)

To set the source data for a HSI cube, use the method ProcessingContext::setSourceData() with a ProcessingContext::HSICube parameter. The following example shows how to use it:

 1 using namespace fluxEngine;
 2
 3 try {
 4     std::int64_t width = 1024, height = 2150;
 5     void const* cubeData = ...;
 6     context.setSourceData(ProcessingContext::HSICube, height, width, cubeData);
 7 } catch (std::exception& e) {
 8     std::cerr << "An error occurred: " << e.what() << std::endl;
 9     exit(1);
10 }

The width and height parameters must correspond to the spatial dimensions of the cube. If the spatial dimensions were fixed during creation of the processing context, the dimensions specified here must match.

This method will assume that the cube is contiguous in memory. For non-trivial stride structures please use the overload of the setSourceData() method that takes additional stride information. Refer to the refereence documentation for further details.

PushBroom Source Data

PushBroom source data is essentially a 2D image that must be provided to fluxEngine. As all of the dimensions of the image have been fixed during the creation of the processing context, the user must only specify the data pointer via the overload of the ProcessingContext::setSourceData() method that takes a ProcessingContext::PushBroomFrame parameter. The following example shows its usage:

1 using namespace fluxEngine;
2
3 try {
4     void const* frameData = ...;
5     context.setSourceData(ProcessingContext::PushBroomFrame, frameData);
6 } catch (std::exception& e) {
7     std::cerr << "An error occurred: " << e.what() << std::endl;
8     exit(1);
9 }

This method will assume that the frame is contiguous in memory. For non-trivial stride structures please use the overload of the setSourceData() method that takes an additional stride argument. Refer to the refereence documentation for further details.

Performing Processing

Once the data pointer has been set, the user may process the data with the processNext() method. This may be called in the following manner:

1 try {
2     context.processNext();
3 } catch (std::exception& e) {
4     std::cerr << "An error occurred: " << e.what() << std::endl;
5     exit(1);
6 }

After processing has completed, the user may obtain the results (see the next section).

To process another set of data, the user may do one of the following:

  • The user may simply overwrite the data in the memory area they specified when they set the source data, and call ProcessingContext::processNext() again.

    The data in the specified source memory area must only remain unchanged during processing, but between calls to ProcessingContext::processNext() the user is completely free to alter it.

  • The user may update the source data pointer and the call ProcessingContext::processNext() again, if the new data that is to be processed is found in a different memory region.

Obtaining Results

Results are obtained in the same manner as described in the section Obtaining Results for processing device data.