Data Processing

Creating a processing context

In order to process data with fluxEngine, a processing context must be created. A processing context knows about:

  • The model that is to be processed

  • How processing will be parallelized (it takes that information from the current handle) – changing parallelization settings will invalidate a given context

  • What kind of data is to be processed each time (full HSI cubes or individual PushBroom frames)

  • The size of data that is to be processed. fluxTrainer models are designed to be camera-independent (to an extent), and thus do not know about the actual spatial dimensions of the data that is to be processed. But once processing is to occur, the spatial dimensions have to be known

  • The input wavelengths of the data being processed. While a model build in fluxTrainer specifies the wavelengths that will be used during processing, cameras of the same model don’t map the exact same wavelengths onto the same pixels, due to production tolerances. For this reason cameras come with calibration information that tells the user what the precise wavelengths of the camera are. The user must specify the actual wavelengths of the input data, so that fluxEngine can interpolate those onto the wavelength range given in the model

  • Any white (and dark) reference data that is applicable to processing

There are two types of processing contexts that can be created: one for HSI cubes, one for PushBroom frames.

HSI Cube Processing Contexts

To process entire HSI cubes the user must use the function fluxEngine_C_v1_ProcessingContext_create_hsi_cube() to create said processing context. It has the following parameters:

  • The model that is to be processed

  • The storage order of the cube (BSQ, BIL, or BIP)

  • The scalar data type of the cube (e.g. 8 bit unsigned integer)

  • The spatial dimensions of the cube that is to be processed

  • The wavelengths of the cube

  • Whether the input data is in intensities or reflectances

  • An optional set of white reference measurements

  • An optional set of dark reference measurements

There are two ways to specify the spatial dimensions of a given cube. The first is to fix them at this point, only allowing the user to process cubes that have exactly this size with the processing context. The alternative is to leave them variable, but specify a maximum size. This has the advantage that the user can process differently sized cubes with the same context, but has the major disadvantage that if a white reference is used, it will be averaged along all variable axes, that means that any spatial information of the reference data will be averaged out. (It is also possible to only fix one of the spatial dimensions.)

For referencing it is typically useful to average multiple measurements to reduce the effect of noise. For this reason, any references that are provided have to be tensors of 4th order, with an additional initial dimension at the beginning for the averages. For example, a cube in BSQ storage order has the dimension structure (λ, y, x), so the references must have the dimension structure (N, λ, y, x), where N may be any positive number, indicating the amount of measurements that is to be averaged. A cube in BIP storage order would have a dimension structure of (y, x, λ), leading to a reference tensor structure of (N, y, x, λ).

Note

It is possible to supply only a single cube as a reference measurement, in that case N would be 1. In that case the structure of the data is effectively only a tensor of third order – but the additional dimension still has to be specified.

Note

References supplied here must always be contiguous in memory, and may never have any non-trivial strides.

Reference cubes must always have the same storage order as the cubes that are to be processed.

The first example here shows how to create a processing context without any references, assuming that the input data is already in reflectances, with a 32bit floating point data type, and fixed spatial dimensions:

 1 fluxEngine_C_v1_Error* error = NULL;
 2 fluxEngine_C_v1_ProcessingContext* context = NULL;
 3 fluxEngine_C_v1_ReferenceInfo reference_info;
 4 double wavelengths[] = { 900, 901.5, 903, ... };
 5 size_t wavelength_count = sizeof(wavelengths) / sizeof(wavelengths[0]);
 6 int64_t width = 1024, height = 2150;
 7 int rc = 0;
 8 memset(&reference_info, 0, sizeof(reference_info));
 9 reference_info.valueType = fluxEngine_C_v1_ValueType_Reflectance;
10 rc = fluxEngine_C_v1_ProcessingContext_create_hsi_cube(model, fluxEngine_C_v1_HSICube_StorageOrder_BSQ,
11                                                        fluxEngine_C_v1_DataType_Float32,
12                                                        height, height, width, width,
13                                                        wavelengths, wavelength_count,
14                                                        &reference_info, &context, &error);
15 if (rc != 0) {
16     fprintf(stderr, "Could not create processing context: %s\n",
17             fluxEngine_C_v1_Error_get_message(error));
18     fluxEngine_C_v1_Error_free(error);
19     return;
20 }

Alternatively, to create a processing context that uses a white reference cube, and where the y dimension has a variable size, the following code could be used:

 1 fluxEngine_C_v1_Error* error = NULL;
 2 fluxEngine_C_v1_ProcessingContext* context = NULL;
 3 fluxEngine_C_v1_ReferenceInfo reference_info;
 4 double wavelengths[] = { 900, 901.5, 903, ... };
 5 size_t wavelength_count = sizeof(wavelengths) / sizeof(wavelengths[0]);
 6 int64_t width = 1024, max_height = 5000;
 7 /* get cube data from somewhere
 8  * (this pointer only has to be valid until the call to
 9  * fluxEngine_C_v1_ProcessingContext_create_hsi_cube has completed,
10  * then the user may discard the datea)
11  */
12 uint8_t* white_reference_cube_data = ...;
13 int rc = 0;
14 memset(&reference_info, 0, sizeof(reference_info));
15 reference_info.valueType = fluxEngine_C_v1_ValueType_Intensity;
16 reference_info.whiteReference = white_reference_cube_data;
17 /* Just a single cube, with a height of just 40 in y direction,
18  * but the width being the same as the fixed width that is set
19  * here.
20  */
21 reference_info.whiteReferenceDimensions[0] = 1;
22 reference_info.whiteReferenceDimensions[1] = (int64_t) wavelength_count;
23 reference_info.whiteReferenceDimensions[2] = 40;
24 reference_info.whiteReferenceDimensions[3] = width;
25 rc = fluxEngine_C_v1_ProcessingContext_create_hsi_cube(model, fluxEngine_C_v1_HSICube_StorageOrder_BSQ,
26                                                        fluxEngine_C_v1_DataType_UInt8,
27                                                        max_height, -1, width, width,
28                                                        wavelengths, wavelength_count,
29                                                        &reference_info, &context, &error);
30 if (rc != 0) {
31     fprintf(stderr, "Could not create processing context: %s\n",
32             fluxEngine_C_v1_Error_get_message(error));
33     fluxEngine_C_v1_Error_free(error);
34     return;
35 }

Note

The maximum size specified here also determines how much RAM is allocated in fluxEngine internally. Specifying an absurdly large number will cause fluxEngine to exhaust system memory.

Note

The white and dark references may have different spatial dimensions if those dimensions are specified as variable. In the above example, if a dark reference were to be specified, it would have to have the same width and number of bands (because those are both fixed), but it could have a different height.

PushBroom Frame Processing Contexts

To process PushBroom frames one must use the function fluxEngine_C_v1_ProcessingContext_create_pushbroom_frame() to create the processing context. It has the following parameters:

  • The model that is to be processed

  • The storage order of the PushBroom frame

  • The scalar data type of each PushBroom frame

  • The spatial width of each PushBroom frame (which will be the actual with of each image if LambdaY storage order is used, or the height of each image if LambdaX storage order is used)

  • The wavelengths of each PushBroom frame

  • Whether the input data is in intensities or reflectances

  • An optional set of white reference measurements

  • An optional set of dark reference measurements

As PushBroom processing can be thought of as a means to incrementally build up an entire cube (but process data on each line individually), the spatial width must be fixed and cannot be variable. (The number of frames processed, i.e. the number of calls to fluxEngine_C_v1_ProcessingContext_process_next() is variable though.)

As it is often useful to average multiple reference measurements to reduce noise, the white and dark references must be supplied as tensors of third order, with a dimension structure of (N, x, λ) or (N, λ, x), depending on the storage order.

The following example shows how to set up a processing context without any references, assuming the input data is already in reflectances, stored as 32bit floating point numbers:

 1 fluxEngine_C_v1_Error* error = NULL;
 2 fluxEngine_C_v1_ProcessingContext* context = NULL;
 3 fluxEngine_C_v1_ReferenceInfo reference_info;
 4 double wavelengths[] = { 900, 901.5, 903, ... };
 5 size_t wavelength_count = sizeof(wavelengths) / sizeof(wavelengths[0]);
 6 int64_t width = 320;
 7 int rc = 0;
 8 memset(&reference_info, 0, sizeof(reference_info));
 9 reference_info.valueType = fluxEngine_C_v1_ValueType_Reflectance;
10 rc = fluxEngine_C_v1_ProcessingContext_create_pushbroom_frame(model, fluxEngine_C_v1_PushBroomFrame_StorageOrder_LambdaY,
11                                                               fluxEngine_C_v1_DataType_Float32,
12                                                               width,
13                                                               wavelengths, wavelength_count,
14                                                               &reference_info, &context, &error);
15 if (rc != 0) {
16     fprintf(stderr, "Could not create processing context: %s\n",
17             fluxEngine_C_v1_Error_get_message(error));
18     fluxEngine_C_v1_Error_free(error);
19     return;
20 }

Alternatively, if both white and dark reference measurements are to be supplied, and unsigned 8bit integer numbers, one could use the following code:

 1 fluxEngine_C_v1_Error* error = NULL;
 2 fluxEngine_C_v1_ProcessingContext* context = NULL;
 3 fluxEngine_C_v1_ReferenceInfo reference_info;
 4 double wavelengths[] = { 900, 901.5, 903, ... };
 5 size_t wavelength_count = sizeof(wavelengths) / sizeof(wavelengths[0]);
 6 /* Get the reference data from somewhere; in this case the white
 7  * reference contains 5 measurements that are to be averaged,
 8  * and the dark reference contains 10. Since this is in LambdaY
 9  * storage order, the references are effectively cubes of BIL
10  * storage order themselves. (For PushBroom frames with LambdaX
11  * storage order the references would be cubes in BIP storage
12  * order.)
13  */
14 uint8_t* white_reference_cube_data = ...;
15 uint8_t* dark_reference_cube_data = ...;
16 int64_t width = 640;
17 int rc = 0;
18 memset(&reference_info, 0, sizeof(reference_info));
19 reference_info.valueType = fluxEngine_C_v1_ValueType_Intensity;
20 reference_info.whiteReference = white_reference_cube_data;
21 reference_info.whiteReferenceDimensions[0] = 5;
22 reference_info.whiteReferenceDimensions[1] = (int64_t) wavelength_count;
23 reference_info.whiteReferenceDimensions[2] = width;
24 reference_info.darkReference = dark_reference_cube_data;
25 reference_info.darkReferenceDimensions[0] = 10;
26 reference_info.darkReferenceDimensions[1] = (int64_t) wavelength_count;
27 reference_info.darkReferenceDimensions[2] = width;
28 rc = fluxEngine_C_v1_ProcessingContext_create_pushbroom_frame(model, fluxEngine_C_v1_PushBroomFrame_StorageOrder_LambdaY,
29                                                               fluxEngine_C_v1_DataType_UInt8,
30                                                               width,
31                                                               wavelengths, wavelength_count,
32                                                               &reference_info, &context, &error);
33 if (rc != 0) {
34     fprintf(stderr, "Could not create processing context: %s\n",
35             fluxEngine_C_v1_Error_get_message(error));
36     fluxEngine_C_v1_Error_free(error);
37     return;
38 }

Processing Data

Once a processing context has been set up, the user may use it to process data. This happens in two steps:

  • Set the data pointer for the source data that is to be processed

  • Process the data

The first step has to be called at least once before processing happens. If the user always writes the data to the same memory location, it can be skipped for subsequent processing steps.

HSI Cubes Source Data

In order to process entire HSI cubes, a processing context for HSI cubes has to have been set up. (See the previous section.)

To set the source data for a HSI cube, use the method fluxEngine_C_v1_ProcessingContext_set_source_data_hsi_cube(). The following example shows how to use it:

 1 fluxEngine_C_v1_Error* error = NULL;
 2 int rc = 0;
 3 int64_t width = 1024, height = 2150;
 4 void const* cube_data = ...;
 5 rc = fluxEngine_C_v1_ProcessingContext_set_source_data_hsi_cube(context, height, width,
 6                                                                 cube_data, &error);
 7 if (rc != 0) {
 8     fprintf(stderr, "Could not set processing source data: %s\n",
 9             fluxEngine_C_v1_Error_get_message(error));
10     fluxEngine_C_v1_Error_free(error);
11     return;
12 }

The width and height parameters must correspond to the spatial dimensions of the cube. If the spatial dimensions were fixed during creation of the processing context, the dimensions specified here must match.

This method will assume that the cube is contiguous in memory. For non-trivial stride structures please use the fluxEngine_C_v1_ProcessingContext_set_source_data_hsi_cube() function instead, and refer to the documentation of that function for further details.

PushBroom Source Data

PushBroom source data is essentially a 2D image that must be provided to fluxEngine. As all of the dimensions of the image have been fixed during the creation of the processing context, the user must only specify the data pointer via fluxEngine_C_v1_ProcessingContext_set_source_data_pushbroom_frame().

The following examples shows its usage:

 1 fluxEngine_C_v1_Error* error = NULL;
 2 int rc = 0;
 3 void const* frame_data = ...;
 4 rc = fluxEngine_C_v1_ProcessingContext_set_source_data_pushbroom_frame(context, frame_data, &error);
 5 if (rc != 0) {
 6     fprintf(stderr, "Could not set processing source data: %s\n",
 7             fluxEngine_C_v1_Error_get_message(error));
 8     fluxEngine_C_v1_Error_free(error);
 9     return;
10 }

This method will assume that the frame is contiguous in memory. For non-trivial stride structures please use the fluxEngine_C_v1_ProcessingContext_set_source_data_pushbroom_frame_ex() function instead, and refer to the documentation of that function for further details.

Performing Processing

Once the data pointer has been set, the user may process the data with the fluxEngine_C_v1_ProcessingContext_process_next() function. This may be called in the following manner:

1 fluxEngine_C_v1_Error* error = NULL;
2 int rc = 0;
3 rc = fluxEngine_C_v1_ProcessingContext_process_next(context, &error);
4 if (rc != 0) {
5     fprintf(stderr, "Could not process data with fluxEngine: %s\n",
6             fluxEngine_C_v1_Error_get_message(error));
7     fluxEngine_C_v1_Error_free(error);
8     return;
9 }

After processing has completed, the user may obtain the results (see the next section).

To process another set of data, the user may do one of the following:

PushBroom Resets

As PushBroom cameras can be thought of as incrementally building up a cube line by line, at some point the user may want to indicate that the current cube is considered complete and a new cube starts. In that case the processing context has to be reset, so that all stateful operations are reset as well, such as object detection, but also kernel-based operations.

To achieve this the function fluxEngine_C_v1_ProcessingContext_reset_state() exists. Its usage is simple:

1 fluxEngine_C_v1_Error* error = NULL;
2 int rc = 0;
3 rc = fluxEngine_C_v1_ProcessingContext_reset_state(context, &error);
4 if (rc != 0) {
5     fprintf(stderr, "Could not reset fluxEngine processing context state: %s\n",
6             fluxEngine_C_v1_Error_get_message(error));
7     fluxEngine_C_v1_Error_free(error);
8     return;
9 }

Note

There is no requirement to actually perform such a reset. If fluxEngine is used to process PushBroom frames that are obtained from a camera above a conveyor belt in a continuous process, for example, it is possible to just simply process all incoming frames in a loop and never call the reset method. In that situation the reset method would be called though if the conveyor belt is stopped and has to be started up again. Though, depending on the specific application, that could also mean that a processing context would have to be created again, for example because references have to be measured again, and a simple state reset is not sufficient.

Obtaining Results

After processing has completed via the fluxEngine_C_v1_ProcessingContext_process_next() method, fluxEngine provides a means for the user to obtain the results of that operation.

When designing the model in fluxTrainer that is to be used here, Output Sinks should be added to the model wherever processing results are to be obtained later.

Note

If a model contains no output sinks, it can be processed with fluxEngine, but the user will have no possibility of extracting any kind of result from it.

fluxEngine provides a means to introspect a model to obtain information about the output sinks that it contains. The following two identifiers for output sinks have to be distinguished:

  • The output sink index, this ist just a number starting at 0 and ending at one below the number of output sinks that may be used to specify the output sink for the purposes of the fluxEngine API.

    The ordering of output sinks according to this index is non-obvious. Loading the same .fluxmdl file will lead to the same order, but saving models with the same configuration (but constructed separately) can lead to different orders of output sinks.

  • The output id, which is a user-assignable id in fluxTrainer that can be used to mark output sinks for a specific purpose. The output id may not be unique (but should be), and is purely there for informational purposes.

    For each output sink in the model the user will be able to obtain the output id of that sink. There is also a function fluxEngine_C_v1_ProcessingContext_find_output_sink() that can locate an output sink if the output id of that sink is unique. It will return the index of the sink with that output id.

To obtain the number of output sinks in a model, one may use the fluxEngine_C_v1_ProcessingContext_num_output_sinks() function (note the check count < 0 instead of rc != 0 here):

 1 fluxEngine_C_v1_Error* error = NULL;
 2 int count = 0;
 3 count = fluxEngine_C_v1_ProcessingContext_num_output_sinks(context, &error);
 4 if (count < 0) {
 5     fprintf(stderr, "Could not get the number of output sinks in model: %s\n",
 6             fluxEngine_C_v1_Error_get_message(error));
 7     fluxEngine_C_v1_Error_free(error);
 8     return;
 9 }
10 printf("Number of output sinks in the model: %d\n", count);

It is then possible to obtain information about a specific output sink. The main method for this is fluxEngine_C_v1_ProcessingContext_get_output_sink_meta_info(). It will return the following information:

  • The output id of the output sink

  • The name of the output sink as an UTF-8 string (this is the name the user specified when creating the model in fluxTraineer)

  • Storage type: what kind of data the output sink will return (the current options are either tensor data or detected objects)

  • The output delay of the output sink (only relevant in the case when PushBroom data is being processed, see Output Delay for PushBroom Cameras for a more detailed discussion of this.

The following example code shows how to use this method:

 1 /* Look at the first output sink */
 2 int sink_index = 0;
 3 int output_id = -1;
 4 char* name = NULL;
 5 fluxEngine_C_v1_OutputStorageType storage_type;
 6 int64_t delay = 0;
 7 int rc = 0;
 8 rc = fluxEngine_C_v1_ProcessingContext_get_output_sink_meta_info(context, sink_index, &output_id, &name, &storage_type, &delay, &error);
 9 if (rc != 0) {
10     printf("Could get output sink information from processing context: %s\n",
11            fluxEngine_C_v1_Error_get_message(error));
12     fluxEngine_C_v1_Error_free(error);
13     return;
14 }
15 /* The string for the name is now allocated after this call, so it
16  * has to be freed by the user.
17  */
18 fluxEngine_C_v1_string_free(name);

Output sinks store either tensor data or detected objects, depending on the configuration of the output sink, and where it sits in the processing chain.

Tensor Data

Tensor data within fluxEngine always has a well-defined storage order, as most algorithms that work on hyperspectral data are at their most efficient in this memory layout. While fluxEngine supports input data of arbitrary storage order, it will be converted to the internal storage order at the beginning of processing. The output data will always have the following structure:

  • When processing entire HSI cubes it will effectively return data in BIP storage order, that means that the dimension structure will be (y, x, λ) (for data that still has wavelength information) or (y, x, P), where P is a generic dimension, if the data has already passed through dimensionality reduction filters such as PCA.

  • When processing PushBroom frames it will effectively return data in LambdaX storage order, with an additional dimension of size 1 at the beginning. In that case it will either be (1, x, λ) or (1, x, P).

  • Pure per-object data always has a tensor structure of order 2, in the form of (N, λ) or (N, P), where N describes the number of objects detected in this processing iteration. Important: objects themselves are returned as a structure (see below), per-object data is data that is the output of filters such as the Per-Object Averaging or Per-Object Counting filter. Also note that output sinks can combine objects and per-object data, in which case the per-object data will be returned as part of the object structure.

    For PushBroom data it is recommended to always combine per-object data with objects before interpreting it, as the output delay of both nodes may be different, and when combining the data the user does not have to keep track of the relative delays themselves.

To obtain the tensor structure of a given output sink before processing has even begun, the function fluxEngine_C_v1_ProcessingContext_get_output_sink_tensor_structure() is available. It returns the following information:

  • The scalar data type of the tensor data (this is the same as the data type configured in the output sink)

  • The order of the tensor, which will be 2 or 3 (see above)

  • The maximum sizes of the tensor that can be returned here

  • The fixed sizes of the tensor that will be returned here. If the tensors returned here are always of the same size, the values here will be same as the maximum sizes. Any dimension that is not always the same will have a value of -1 instead. If all of the sizes of the tensor returned here are fixed, the tensor returned will always be of the same size. (There is one notable exception: if the output sink has a non-zero output delay of m, the first m processing iterations will produce a tensor that does not have any data.)

Please refere to the documentation of fluxEngine_C_v1_ProcessingContext_get_output_sink_tensor_structure() for information on how to call it.

Using fluxEngine_C_v1_ProcessingContext_get_output_sink_data_s() it is possible to obtain that tensor data after a successfull processing step. It is also possible to use fluxEngine_C_v1_ProcessingContext_get_output_sink_data() (no _s suffix), but that doesn’t return the stride structure, the tensor order and the scalar data type. While these can be reconstructed from the tensor structure that can be obtained using fluxEngine_C_v1_ProcessingContext_get_output_sink_tensor_structure(), the information required to create a tensor representation of the actual output result is nevertheless repeated here.

For example, if we know a given sink with index sink_index has signed 16bit integer data that spans the entire cube that is being processed (when processing HSI cubes), the following code could be used to obtain the results:

 1 /* obtained from previous introspection */
 2 int sink_index = ...;
 3 int rc = 0;
 4 int16_t const* classification_data = NULL;
 5 int64_t cube_width = ..., cube_height = ...;
 6 int order = -1;
 7 int data_type = -1;
 8 int64_t out_sizes[5] = { 0, 0, 0, 0, 0 };
 9 int64_t out_strides[5] = { 0, 0, 0, 0, 0 };
10 rc = fluxEngine_C_v1_ProcessingContext_get_output_sink_data_s(context, sink_index, &order, &data_type,
11                                                               out_sizes, out_strides,
12                                                               (void const**) &classification_data, &error);
13 if (rc != 0) {
14     printf("Could get classification results from fluxEngine: %s\n",
15            fluxEngine_C_v1_Error_get_message(error));
16     fluxEngine_C_v1_Error_free(error);
17     return;
18 }
19
20 /* Classification results have an inner dimension of 1, so the
21  * actual sizes should be (cube_height, cube_width, 1)
22  */
23 assert(out_sizes[0] == cube_height);
24 assert(out_sizes[1] == cube_width);
25 assert(out_sizes[2] == 1);
26
27 for (int64_t y = 0; y < cube_height; ++y) {
28     for (int64_t x = 0; x < cube_width; ++x) {
29         int64_t index = y * out_strides[0] + x * out_strides[1];
30         printf("Classification result for pixel (%lld, %lld) = %d\n",
31                (long long) x, (long long) y, (int) classification_data[index]);
32     }
33 }

Object Data

Objects will be returned as an array of C structures, either fluxEngine_C_v1_OutputObject or fluxEngine_C_v1_OutputExtendedObject, which contain information about objects that were detected in the model.

Which of these structures is returned depends on whether the user has selected to use extended objects via fluxEngine_C_v1_ProcessingContext_set_use_extended_objects(). By default fluxEngine_C_v1_OutputObject will be returned.

The function fluxEngine_C_v1_ProcessingContext_get_output_sink_object_list_structure() will return the following information about output sinks that return object results:

  • The maximum number of objects that can be returned in a single iteration.

  • Whether per-object data was output using the output sink in the model, and if so, how large it is (per-object data is always a vector, i.e. a tensor of order 1)

  • The scalar type of per-object data (if any)

Please refer to the documentation of fluxEngine_C_v1_ProcessingContext_get_output_sink_object_list_structure() for details on how to invoke that function.

Additionally, there are the functions fluxEngine_C_v1_ProcessingContext_get_output_sink_object_list_statistics_structure() and fluxEngine_C_v1_ProcessingContext_get_output_sink_object_list_quality_structure() that provide information about the structure of per-object statistics data and per-object quality data that is returned only when using the extended object structure.

Using fluxEngine_C_v1_ProcessingContext_get_output_sink_data() it is possible to obtain that object data after a successfull processing step. It is also possible to call fluxEngine_C_v1_ProcessingContext_get_output_sink_data_s() here, but the additional information provided by the _s suffixed method is not relevant for object data. The non-suffixed method may be called in the following manner, assuming sink_index is a sink that is known to return objects:

 1 /* obtained from previous introspection */
 2 int sink_index = ...;
 3 int rc = 0;
 4 fluxEngine_C_v1_OutputObject const* objects = NULL;
 5 int64_t out_sizes[5] = { 0, 0, 0, 0, 0 };
 6 int64_t object_count = 0;
 7 rc = fluxEngine_C_v1_ProcessingContext_get_output_sink_data(context, sink_index, out_sizes,
 8                                                             (void const**) &objects, &error);
 9 if (rc != 0) {
10     printf("Could not get object results from fluxEngine: %s\n",
11            fluxEngine_C_v1_Error_get_message(error));
12     fluxEngine_C_v1_Error_free(error);
13     return;
14 }
15 /* Only the first size is relevant here, which gives us the number
16  * of objects.
17  */
18 object_count = out_sizes[0];
19
20 for (int64_t i = 0; i < object_count; ++i) {
21     printf("Object: bbox topleft [%lld,%lld] -- bottomright [%lld,%lld], area %lld\n",
22            (long long) objects[i].bounding_box_x,
23            (long long) objects[i].bounding_box_y,
24            (long long) (objects[i].bounding_box_x + objects[i].bounding_box_width - 1),
25            (long long) (objects[i].bounding_box_y + objects[i].bounding_box_height - 1),
26            (long long) objects[i].area);
27 }

If extended objects are to be used, the following code will enable that. It should be called immediately after the creation of the processing context and will apply to all output sinks that return object data:

1 /* obtained from previous introspection */
2 int rc = 0;
3 rc = fluxEngine_C_v1_ProcessingContext_set_use_extended_objects(context, true, &error);
4 if (rc != 0) {
5     printf("Could not setup extended object use for fluxEngine: %s\n",
6            fluxEngine_C_v1_Error_get_message(error));
7     fluxEngine_C_v1_Error_free(error);
8     return;
9 }

Afterwards, objects may be retrieved similarly to the previous code, but using a different structure:

 1 /* obtained from previous introspection */
 2 int sink_index = ...;
 3 int rc = 0;
 4 fluxEngine_C_v1_OutputExtendedObject const* objects = NULL;
 5 int64_t out_sizes[5] = { 0, 0, 0, 0, 0 };
 6 int64_t object_count = 0;
 7 rc = fluxEngine_C_v1_ProcessingContext_get_output_sink_data(context, sink_index, out_sizes,
 8                                                             (void const**) &objects, &error);
 9 if (rc != 0) {
10     printf("Could not get object results from fluxEngine: %s\n",
11            fluxEngine_C_v1_Error_get_message(error));
12     fluxEngine_C_v1_Error_free(error);
13     return;
14 }
15 /* Only the first size is relevant here, which gives us the number
16  * of objects.
17  */
18 object_count = out_sizes[0];
19
20 for (int64_t i = 0; i < object_count; ++i) {
21     printf("Object: bbox topleft [%lld,%lld] -- bottomright [%lld,%lld], area %lld\n",
22            (long long) objects[i].bounding_box_x,
23            (long long) objects[i].bounding_box_y,
24            (long long) (objects[i].bounding_box_x + objects[i].bounding_box_width - 1),
25            (long long) (objects[i].bounding_box_y + objects[i].bounding_box_height - 1),
26            (long long) objects[i].area);
27     if (objects[i].quality.base_pointer && objects[i].quality.data_type == fluxEngine_C_v1_DataType_UInt8) {
28         uint8_t const* quality_values = (uint8_t const*) objects[i].quality.base_pointer;
29         printf("  - first quality entry: %d\n", (int) quality_values[0]);
30     }
31 }

Tensor Data in Fields of Extended Objects

Certain fields in the fluxEngine_C_v1_OutputExtendedObject structure are of type fluxEngine_C_v1_Tensor, which wraps tensor data. This structure contains a base pointer to the first element of the tensor data, as well as the order of the tensor, the scalar data type, the dimensions, and the strides of the tensor.

The mask field of fluxEngine_C_v1_OutputExtendedObject is a common example of where such a tensor may be stored. In that case the data type will always be fluxEngine_C_v1_DataType_Int8, and the order of the tensor will always be 2 – but for other fields this may vary. In the case of the mask this allows the user to view the shape of the object that was detected.

Accessing an element of such a tensor is done by first casting the base pointer to the correct scalar type, and then by indexing using the formula index0 * tensor.strides[0] + index1 * tensor.strides[1] + .... For example, to access the pixel with position y = 4 and x = 3 ((0, 0) being the first pixel in the top-left corner) of the mask, one would use:

 1 if (mask.data_type != fluxEngine_C_v1_DataType_Int8
 2     || mask.order != 2) {
 3     // Error handling, the mask does not have the anticipated
 4     // structure.
 5 }
 6 int8_t const* mask_data = (int8_t const*) mask.base_pointer;
 7 int64_t y = 4, x = 3;
 8 int64_t index = y * mask.strides[0] + x * mask.strides[1];
 9 if (y >= mask.dimensions[0] || x >= mask.dimensions[1]) {
10     // Error handling: the object is too small to contain the
11     // pixels.
12 }
13 bool is_pixel_present = (mask_data[index] == 0);

The same as with the object data itself, these tensors will only remain valid as long as the processing context is not freed, and no further data has been processed.