Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add live graphic visualization of memory allocated/freed while nodeos is running #1041

Draft
wants to merge 21 commits into
base: main
Choose a base branch
from

Conversation

greg7mdp
Copy link
Contributor

@greg7mdp greg7mdp commented Nov 20, 2024

Please note that we do not intend to merge this PR, in only exists as documentation of the work done, and as a base for further developments if warranted.

Resolves #974.

This change is mostly done in two separate parts:

Boost interprocess memory occupancy tracking

The Boost interprocess library (which is used to manage memory within the shared memory segment allocated by chainbase) can use multiple memory allocation managers.

Chainbase, and therefore nodeos, use the rbtree_best_fit algorithm which keeps track of free areas using a rbtree (the nodes of the rbtree are intrusively stored in the free areas themselves, so there is no additional memory cost for this tree, besides the fact that it requires a minimum allocation size).

This PR adds a new member in the memory segment header managed by the interprocess library, of the type:

using occupancy_array_t = std::vector<uint8_t>; // one byte describes the occupancy percentage of each page of the segment

and updates this occupancy vector:

  • when a new function bool initialize_occupancy(); is called
  • when memory is allocated or freed

As we cannot add these changes in the original boost repo, this PR links to a private repo which includes the changes in the interprocess library. These changes are also included in the following file:

ip.txt

Use the occupancy vector maintained by Boost interprocess to update a texture and visualize it on-screen

The visualization is implemented using OpenGL. This required adding some dependencies to chainbase:

  1. some glad header files describing the OpenGL interface used. These were generated from https://gen.glad.sh/, specifying gl 4.6 and glx 1.4, compat profile, header only, GL_ARB_direct_state_access extension.
  2. the glm library, used as-is
  3. the glfw library, with this trivial change:
    glfw.txt

The visualization is implemented in the new files mem_visualizer[.hpp, .cpp], and takes place on a separate thread.

Notes

  • when using the rbtree_best_fit memory allocator (as chainbase does), the minimum size of a block is 48 bytes. When allocating 8 bytes, the overhead is 40 bytes. When allocating 32 bytes or more, the overhead is 16 bytes.
  • the minimum 16 bytes overhead is for two sizes (m_prev_size and m_size) with two 1 bit flags encoded in the lower two bits of m_size (m_prev_allocated and m_allocated).

@greg7mdp greg7mdp marked this pull request as draft November 20, 2024 19:59
@ericpassmore
Copy link
Contributor

Note:start
category: Tooling
component: Internal
summary: Added openGL tooling to visualize memory allocation in real time.
Note:end

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Investigate data locality of chainbase allocator
2 participants