Clearing Vector of Sturct on next iteration.

[piyushmz]

Hey I've been working on IPC using iceoryx its a real beauty. The problem I'm facing here is I've created a vector of structure(having double & uint64_t). I'm doing emplace_back and a struct is placed in the vector of 10(i.e 10 sturcts are palced in vector), next this vector is published.
The thing is on the next iteration the vector must get cleared/erased and the values should be zero/NULL so that I can place my the new values in it the memory should get cleared over here. But in my case the values remain intact and doesn't change.

Now I've iterated the vector and tried assigning vector values to zero but I get the following error.
Condition: index < m_size in at is violated: "Out of bounds access". (/usr/local/include/iceoryx/v/iceoryx_hoofs/internal/cxx/vector.inl:271) 2022-10-13 12:04:46.386 [Error]: ICEORYX error! EXPECTS_ENSURES_FAILED

why is it accessing invalid memory? I just want to clear the vector on the next iteration. How do I erase/clear all the values of this vector?
Thanks.

[elfenpiff]

@piyushmz From your description and error message I assume you have a piece of code like this:

struct Structure {
    uint64_t a;
    uint64_t b;
};

iox::cxx::vector<Structure, 10> myVector;
// fill the vector with 10 elements of Structure here

myVector.clear();
for(uint64_t i = 0; i < 10; ++i) {
   myVector.at(i).a = 0;
   myVector.at().b = 0;
}

If this is the case then some remarks.
After you have called clear() on the vector your vector is empty and the vector calls the destructor on every element. So an access with .at() leads to an "Out of bounds access" as you experienced.
In the STL std::vector version you could still access the elements but would have undefined behavior in the end which is a worst case scenario for a safety critical application therefore we check it and fail with the "Out of bounds" error message.

If you want to zero the memory when an element is destroyed you have to explicitly implement a destructor for your structure since the default destructor of POD types (plain old data types like int, float, char, bool) leaves the memory unchanged since there is nothing to cleanup.
I think the best solution for you would be adding a destructor like:

struct Structure {
    ~Structure() {
        a = 0;
        b = 0;
    }
    uint64_t a;
    uint64_t b;
};

This would set your memory on destruction to zero. So the call

myVector.clear(); 

would call the destructor of the elements and zero them. But now the as-if rule maybe kicks in https://en.cppreference.com/w/cpp/language/as_if . In short, the compiler is allowed to optimize statements away when they cannot be observed by the program. This means the zeroing is maybe optimized away and you are back to square one.

Now I gave you the full overview of the problem - sorry it is maybe more then you asked for.

@piyushmz maybe we could solve this pickle also on a different front. Could you please send me the code snippet how you publish the vector? And why do you want to zero the vector in the first place?

[elBoberido]

@piyushmz I'm a bit confused. You say that you publish the vector and access it on the next iteration. How exactly do you access it? After the sample is published it should not be accessed by the publishing process anymore. In case you mean that you loan a new sample, then the sample should contain an empty vector. The unused memory of the vector contains random data which might be the data of the previous iteration in case it happens to loan the same sample as before. Calling emplace on the empty vector will construct a new element, independent of the random memory content. In case your struct does not initialize its member it will just use the content of this random memory which will be zero only for the very first usage of the sample. To use the example from @elfenpiff, this will ensure that emplaced structs are zeroed

struct Structure {
    uint64_t a{0};
    uint64_t b{0};
};

[piyushmz]

just a min I'm typing I'll just show you how thing are working.

[piyushmz]

I've the same code as the complexdata

topic_data.hpp will be imported in objecdetector.hpp

---------------------------topic_data.hpp----------------------------------

#ifndef IOX_EXAMPLES_COMPLEXDATA_TOPIC_DATA_HPP
#define IOX_EXAMPLES_COMPLEXDATA_TOPIC_DATA_HPP

#include "iceoryx_dust/cxx/forward_list.hpp"
#include "iceoryx_hoofs/cxx/list.hpp"
#include "iceoryx_hoofs/cxx/optional.hpp"
#include "iceoryx_hoofs/cxx/stack.hpp"
#include "iceoryx_hoofs/cxx/string.hpp"
#include "iceoryx_hoofs/cxx/variant.hpp"
#include "iceoryx_hoofs/cxx/vector.hpp"
#include <string>
#include <vector>
#include <utility>
#include <chrono>
using namespace std::chrono;
using namespace iox::cxx;

struct single_box
{
    double v_x;
    double v_y;
    double v_l;
    uint64_t tm_stamp;
};

struct box_s
{
	vector<single_box, 10> s_b;
};

--------------------------objectdetector.hpp-------------------------------

void function_something {

Something goes here

/*------------------IPC- Publisher Starts ----------------------- */

/* The main() calls the initiPublisher with app name hence not implemented here*/

iox::popo::Publisher<box_s> publisher({"Group", "Instance", "ComplexDataTopic"});
        
        /*----- This is a vector of single boxes -----*/
       

        box_s p;    /*comes from topic_data.hpp*/

        for (uint64_t i = 0u; i < p.s_b.capacity(); i++){
                single_box b = {};
             	uint64_t ms = duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();
             
                double m_x = m_detectedBoxList[i].x + (m_detectedBoxList[i].width / 2);           /* These values are obatained from function */
                double m_y = m_detectedBoxList[i].y + (m_detectedBoxList[i].height / 2);          /* Same as value */
                b.v_x = m_x;
                b.v_y = m_y;
                b.v_l = 1;
                b.tm_stamp = ms;
                std::cout << b.v_x << " " << b.v_y<< " "  << b.v_l << " "  << b.tm_stamp << std::endl;
                p.s_b.emplace_back(b);
        } 
        auto loanRes = publisher.loan();
        
        if(!loanRes.has_error()) {
            auto &sample = loanRes.value();
           
            sample->s_b = p.s_b;
           
            for (uint64_t i = 0u; i < sample->s_b.capacity(); i++) {
                std::cout << "x: " << sample->s_b[i].v_x
                        << " y: " << sample->s_b[i].v_y
                        << " l: " << sample->s_b[i].v_l
                        << " Time: " << sample->s_b[i].tm_stamp
                        << std::endl;
            }
            std::cout << std::endl; 
            sample.publish();

/* Now I want to clear p vector so that on next iteration I'll have new values */

        }
}

Here's the link for output asciinema output . In this you can see when object is not detected by the camera the values remain static in the vector I need them to be zero.

I tried p.clear() but it's not possible.
Thanks.

[elfenpiff]

@piyushmz Oh and I spot your mistake.

This here in the second loop

 for (uint64_t i = 0u; i < sample->s_b.capacity(); i++) {

Here you must use p.s_b.size() and not capacity.

• capacity() returns how much elements can be contained in the vector.
• size() returns how many elements are actually contained inside the vector

In the first loop it makes sense since you want to fill the vector with elements until it is full. In the second loop you would like to iterate over all elements of the vector and therefore you have to use size.
Even better is a range based loop like:

            for (auto & element : sample->s_b) {
                std::cout << "x: " <<element.v_x
                        << " y: " << element].v_y
                        << " l: " << element.v_l
                        << " Time: " << element.tm_stamp
                        << std::endl;
            }

[elBoberido]

@piyushmz try this

for (uint64_t i = 0u; i < p.s_b.capacity() && i < m_detectedBoxList.size(); i++)

You have an out of bounds access in m_detectedBoxList and just copy the old data from there.

cc @elfenpiff

[elBoberido]

@piyushmz ... in case you really want to always send 10 elements, you can also do the following after the first loop

while(p.s_b.size() < p.s_b.capacity) {
    p.s_b.emplace({0., 0., 0., 0});
}

[piyushmz]

After sample.publish() is there a way to clear the shared memory? after some time frame of milliseconds.

[elfenpiff]

@piyushmz No there is not and this for a good reason.

As soon as you published a sample you released the ownership of the memory to our IPC middleware and it delivers it to an arbitrary number of subscribers which can read the memory at any given time.
If you now modify the memory we would introduce race conditions on the subscriber side.

Why do you want to zero the memory at all?

[piyushmz]

ohhh... I got it very clear. I did one mistake that the m_detectedBoxList vector was not generating any values when the camera object wasn't detected hence the values stayed static over there. Now I handled it with an if else which works as desired.
Thanks very much for your help @elBoberido @elfenpiff

[elfenpiff]

@piyushmz we are always happy to help.

Happy Hacking 😄

[elBoberido]

@piyushmz you're welcome :)

[piyushmz]

Hey @elfenpiff @elBoberido sorry to bother you again. So below is my subscriber and the function onProcess() is called iteratively.
The thing happening here is as the function gets called the subscriber tr doesn't pick any values the program just keeps on printing the first line This is running.
But when I use a while loop it gets the values & is running fine. I've to break the while loop for each process.
Do you've any idea why isn't it taking the value on the process call. I've included the while loop in the comments. I need to do further
computation on the values received.
OR is this code fine that when process hits I enter a while loop and then break it on every process is it a good style to do.

This subscriber is included in class mapping:public DriveworksSample {public: void onProcess() {...} } above main().

If I declare subscriber globally I get following which is a bad idea I suppose.

 [Fatal]: Cannot initialize runtime. Application name has not been specified!
Segmentation fault (core dumped)

I've included the initRunitme(APP_NAME) in main

void onProcess() override 
    {
    
    /*----------- Subscriber here -----------*/
    std::cout << "This is running\n";
    box_s s;
    single_box s1;
    iox::popo::Subscriber<box_s> subscriber({"Group", "Instance", "ComplexDataTopic"});
    // uint64_t ms = duration_cast<milliseconds>(system_clock::now().time_since_epoch()).count();

    // while (!iox::posix::hasTerminationRequested())
    // { 
        auto tr = subscriber.take();

        if (!tr.has_error()) 
        {
            
            std::cout << "Entered the TAKE RESULT SECTION\n";
            
            for (uint64_t i = 0; i < s.s_b.capacity(); i++) 
            {
                    s1.v_x = tr.value()->s_b[i].v_x;
                    s1.v_y = tr.value()->s_b[i].v_y;
                    s1.v_l = tr.value()->s_b[i].v_l;
                    s1.tm_stamp = tr.value()->s_b[i].tm_stamp;
                    s.s_b.emplace_back(s1);
                // lable_v.push_back(s1.v_l);
            }
            // std::vector<cv::Point> mhl_pts;     /*My homography lane points*/
            // mho_pts.clear();
            // all_obs_pts.clear();
            for (uint64_t j = 0; j < s.s_b.capacity(); j++)
            {
                // mho_pts.push_back(obs_homography(s.s_b[j].v_x, s.s_b[j].v_y));
                std::cout << "entered this\n";
                
                std::cout << "x: " << s.s_b[j].v_x
                          << " y: " << s.s_b[j].v_y
                          << " l: " << s.s_b[j].v_l
                          << " Time: " << s.s_b[j].tm_stamp << std::endl;
            }
            // for(uint64_t i = 0u; i < mho_pts.size(); i++){
            //     all_obs_pts.push_back(gen_pts(mho_pts[i]));
            // }
            std::cout << std::endl;
            // break;
        }
    // }
}

Tried a lot of things on this just can't get where I'm going wrong?

[piyushmz]

@piyushmz

If I declare subscriber globally I get following which is a bad idea I suppose.

This is very bad. You should avoid singletons at all cost otherwise you run into the static initialization order fiasco: https://en.cppreference.com/w/cpp/language/siof But we have on our side also a singleton, the iox::runtime which is only initialized after you called initRunitme(APP_NAME) and your global subscriber requires this object which is at that time not yet present. The global runtime is a bad design on our side and we have to correct it the near future.

So below is my subscriber and the function onProcess() is called iteratively.

When is your onProcess called. In a busy while loop or are you attached to some listener/waitset? I have the suspicion that maybe the function is called 1000 times and only 1 of those 1000 times it actually receives a sample since you are calling it too quickly.

You are right here 1 of the 1000 data gets captured as the onProcess gets called really quickly but not on each call of onProcess I've to enter the while to capture every.

Otherwise your code looks good. When you enter the while (!iox::posix::hasTerminationRequested()) you do not have the problem with the output since the console is being quiet when no sample was received. Maybe you could remove the line td::cout << "This is running\n"; and see what this is doing to confirm or disprove my theory.

Now as I'm using while loop it's working good and excellent with further computation.

Without while loop
https://asciinema.org/a/GcqLvKONUCuxtgk1ftzRcjU4L

With while-break loop.
https://asciinema.org/a/LrLewPVmx0D46FdSVXejsGWeq

[elfenpiff]

I think the perfect tool for your use case is the WaitSet. Here you can attach subscribers and you are explicitly triggered when a new sample is received.
With this you avoid the busy loop which costs you a lot of CPU time.

See this example here: https://github.com/eclipse-iceoryx/iceoryx/tree/master/iceoryx_examples/waitset

[piyushmz]

Ok got it I'll try that. Just a further quick question @elfenpiff can I use the same subscriber as producer and pass the computed data further to another subscriber.

Thanks for your help.

[elfenpiff]

@piyushmz Not yet but we planned this as a feature, see #927

What you are describing in the end is similar to a unix pipeline with a source (publisher) and a sink (subscriber). This does not work at the moment since such a pipeline does not allow multiple subscribers.
Since the subscriber would modify the sample during the computation and this would lead to data races in the other subscribers.

[piyushmz]

Ok got it, Thank you very much.