Vxworks 32bit system, PPC, Diab Compiler, Embedded application problems

Yutao · June 3, 2023, 8:29am

Hi there,

I have successfully cross-compiled ACADOS for VxWorks 6.9, a 32-bit system running on PPC CPU. Here are the settings

jietu1
jietu3
jietu2

So

The target is set to GENERIC
EXT_DEP is turned off
LA uses REFERENCE
MF uses COLMAJ

The compilation is successful, despite a few warnings. Then I successfully compiled locally the c code of the bicycle model generated from the Python interface. For simplicity, I tested the integrator first.

When I try to run the executable, I got the following output
jietu4
So basically the code is not working. Ok, I start debug, and found that the error came from the struct mem in the integrator.
jietu5
When I try to give values to the mem->time_ad, mem->time_la and mem->time_sim, the error shows. Then I found that these three double variables are not created, or their memory is not assigned properly.
jietu6
Finally, I simply comment these three lines, and the integrator works!

My question is, why is the case? There seems memory alignment or assignment issues, but weirdly everything works fine on Ubuntu 64bit system, and on Speedgoat.

To further locate the problem, I test the OCP example as well, and the code stuck at model->scaling=1.0
The similarity of these two tests is that: both sim_irk_memory and model struct contains a few double pointers and a few double variables. The pointers work fine, but the double variables are crashed. I guess, this is the problem of align_char_to(8, &c_ptr), or any thing related to this?

Thank you for your advice.

Yutao

Yutao · June 3, 2023, 9:26am

A quick update:

By adding memory alignment before the introduction of the mem, the error disappears

Hence, it seems that an initial memory alignment is necessary in my case.

Although this solves my issue for the integrator test, further tests on OCP example show that I need to modify dozens of functions, which makes this workaround tedious. Is there better way to to it?

Yutao

FreyJo · June 5, 2023, 6:06am

Hi Yutao,

I tested for 32 bit compatibility over a year ago on a 32 bit Raspberry Pi, and wrote down this memory management guide:
https://docs.acados.org/developer_guide/index.html#memory-management-in-acados

It seems that on your architecture you get “less aligned” memory from the system and applying the guide more rigorously is required.

I hope you can fix some of the issues and make a PR.

For reference:

github.com/acados/acados

Support 32 Bit Architectures

opened 04:25PM - 24 Apr 20 UTC

FreyJo

I put a bit of work into this, see #597 but there are still some issues left. F…or now, acados is not fully supported on 32 Bit architectures, mainly because of memory alignment issues. In order to be able to make such a statement, the following should be done on a 32 Bit machine: - [ ] make C tests / examples work - [ ] make the Python tests work - [ ] ideally have some CI to check some form of 32 Bit compatibility. I hope this Guideline will be helpful for future developers/maintainers: https://github.com/acados/acados/blob/master/docs/memory_management.md

Best,
Jonathan

Yutao · June 6, 2023, 9:02am

Hi Jonathan,

Thank you for the reply. The guide you gave is very helpful.

However, after adapting the memory alignment guide to our architecture, we still got some errors. The errors indicate that HPIPM structs have memory problems, and this kind of struct is not explained in the guide you provided.

Specifically, we found that the first error lied in the function

ocp_qp_hpipm_memory_assign

and in which the sub-struct

d_ocp_qp_ipm_ws

is created by calling

d_ocp_qp_ipm_ws_create

We tried to align before/after this call, but still got errors. This means that

mem->hpipm_workspace

is not created succussfully.

However, this function is very complicate and given in external/hpipm, hence we don’t think it’s a good idea to change it.

What should/can I do? Any suggestions?

Yutao

FreyJo · June 7, 2023, 7:52am

Hi Yutao,

Can you try to add an align here?
i.e. before d_ocp_qp_ipm_ws, which has some doubles at the beginning.

github.com

acados/acados/blob/master/acados/ocp_qp/ocp_qp_hpipm.c#L204


      
          ocp_qp_dims *dims = dims_;
          ocp_qp_hpipm_opts *opts = opts_;
          ocp_qp_hpipm_memory *mem;
          
          
// char pointer
          char *c_ptr = (char *) raw_memory;
          
          
mem = (ocp_qp_hpipm_memory *) c_ptr;
          c_ptr += sizeof(ocp_qp_hpipm_memory);
          
          
mem->hpipm_workspace = (struct d_ocp_qp_ipm_ws *) c_ptr;
          c_ptr += sizeof(struct d_ocp_qp_ipm_ws);
          
          
struct d_ocp_qp_ipm_ws *ipm_workspace = mem->hpipm_workspace;
          
          
align_char_to(8, &c_ptr);
          assert((size_t) c_ptr % 8 == 0 && "memory not 8-byte aligned!");
          
          
// ipm workspace structure
          d_ocp_qp_ipm_ws_create(dims, opts->hpipm_opts, ipm_workspace, c_ptr);
          c_ptr += ipm_workspace->memsize;

Also, HPIPM has been successfully tested on a 32bit ARM, but it might be that there are issues left for PPC. It might be also related to the fact that PPC is big endian is big endian by default.

I hope this helps you to fix it and that you contribute the fix!

Best,
Jonathan

Yutao · June 11, 2023, 2:17pm

Hi Jonathan,

We tried your suggestion but that does not work.

However, we went through the code deeper and find out mem->hpipm_workspace->stat address can be printed out, and mem->hpipm_workspace->stat_m has a value of 17.

When we tried to print mem->hpipm_workspace->stat[0], the program stoped. Hence, it is obvious that the memory of this vector is not assigned properly.

We then checked this vector in x_ocp_qp_ipm.c (in function void OCP_QP_IPM_WS_CREATE), and found the codes below

// double/float stuff
	REAL *d_ptr = (REAL *) sv_ptr;
	workspace->stat = d_ptr;
	int stat_m = 17;
	d_ptr += stat_m*(1+arg->stat_max);

So, stat_m=17 is confirmed, and clearly the vector “stat” has been assigned memory here, but somehow not successfully when returned to function void *ocp_qp_hpipm_memory_assign

Finally, we tried a very direction solution, i.e. adding manually the memory for this vector in the function void *ocp_qp_hpipm_memory_assign as

assign_and_advance_double(17*(1+opts->stat_max), &ipm_workspace->stat, &c_ptr);

So the final function of void *ocp_qp_hpipm_memory_assign becomes

Magically, the program worked

Now I’m not sure if this can be a PR (does it work for other architectures?), and why the memory of a single vector is not assigned inside HPIPM create function correctly so that we have to do it manually in ACADOS create function.

Cheers,

Yutao

albert · June 16, 2023, 10:02am

I am also struggling with memory erros on a QNX with PPC machine (dSPACE) and tried this fix. However I get the following error, when applying your fix. Do we have different code versions?

‘ocp_qp_hpipm_opts {aka struct ocp_qp_hpipm_opts_}’ has no member named ‘stat_max’
assign_and_advance_double(17*(1+opts->stat_max), &ipm_workspace->stat, &c_ptr);

Maybe you have an idea?

Thanks,
Albert

Yutao · June 16, 2023, 10:29am

Hi albert,

I made a mistake. The correct fix should be

assign_and_advance_double(mem->hpipm_workspace->stat_m*(1+opts->hpipm_opts->stat_max), &mem->hpipm_workspace->stat, &c_ptr);

Yours,

Yutao

albert · June 16, 2023, 10:43am

Hi Yutao,

thanks for the quick answer!

I made the change and it compiles, at runtime the assertion in the following line does throw an error now.

In function ocp_qp_hpipm_memory_assign – C:/Users/praktika/acados/acados/ocp_qp/ocp_qp_hpipm.c:218 (char *) raw_memory + ocp_qp_hpipm_memory_calculate_size(config_, dims, opts_) >= c_ptr – assertion failed

Was this the case for you as well? It shouldnt be like this, right?

Albert

Yutao · June 17, 2023, 10:31am

Hi Albert,

You also need to fix accordingly in the function

ocp_qp_hpipm_memory_calculate_size

So, basically, you need to add this additional memory when calculating the size of the memory. Check the link Jonathan share above.

Best,

Yutao

giaf · June 20, 2023, 1:51pm

In case of 32-bit systems and 64-bit FP, it may be that in HPIPM the memory before assigning stat is not aligned to 8-bytes boundaries.
In this commit

I add an explicit alignment before it, can you check if this makes the job for you?

Yutao · June 28, 2023, 6:46am

Hi Gianluca,

I have tested your commit and it worked. The results are correct.

Thank you very much.

Just be curious, will these additional alignments affect 64-bit systems (Ubuntu 64 on a PC)?

Yutao

giaf · June 30, 2023, 9:50am

This extra alignment has no effect on 64-bit systems, other than using a tiny bit more memory.

I implemented the same fix also for the dense solver and the tree-structure ocp qp solver.

Thanks again for spotting and the precise hints to locate the issue.