Support for partition tables larger than 4MB with "linux" target (IDFGH-14080)

[higaski]

Is your feature request related to a problem?

I'm currently trying to get NVS running on a Linux target. Sadly this is more difficult than I've had anticipated. Just calling

nvs_flash_init() 

doesn't do it. It segfaults a couple of levels deep inside esp_partition_read.

The documentation is lacking a meaningful example so far, or at least I couldn't find one. I found the NVS host tests inside the nvs_flash components folder but that's quite a lot of code to go through for "just writing a blob to wherever". My best guess after maybe half an hour of reading through the test code is that I need some kind of mmap file first?

Describe the solution you'd like.

Could we maybe get a minimal example of "here is how to read and write NVS on Linux"?
I assume the functions for doing that in a couple of lines are already there... but I just can't find them.

Describe alternatives you've considered.

I've tried to extract what little I need from the test code that's already in ESP-IDF but that's a tedious task for what I assume is maybe 2 or 3 function calls.

Additional context.

No response

[igrr]

This should work with just nvs_flash_init, at least in recent IDF versions.

The fact that you get a segfault in esp_partition_read is probably a bug, although I can't reproduce it with the following simple example:

project CMakeLists.txt

cmake_minimum_required(VERSION 3.16)

include($ENV{IDF_PATH}/tools/cmake/project.cmake)
set(COMPONENTS main)
project(test)

main/CMakeLists.txt

idf_component_register(SRCS "test.c"
                    INCLUDE_DIRS "."
                    PRIV_REQUIRES "nvs_flash")

main/test.c

#include <stdio.h>
#include "esp_err.h"
#include "nvs_flash.h"
#include "nvs.h"

void app_main(void)
{
    ESP_ERROR_CHECK(nvs_flash_init());
    nvs_handle_t nvs;
    ESP_ERROR_CHECK(nvs_open("storage", NVS_READWRITE, &nvs));
    ESP_ERROR_CHECK(nvs_set_u32(nvs, "key", 42));
    ESP_ERROR_CHECK(nvs_commit(nvs));
    nvs_close(nvs);
    ESP_ERROR_CHECK(nvs_open("storage", NVS_READONLY, &nvs));
    uint32_t value;
    ESP_ERROR_CHECK(nvs_get_u32(nvs, "key", &value));
    nvs_close(nvs);
    printf("Value: %d\n", value);
}

results in

(output)

$ idf.py --preview set-target linux && idf.py build monitor
Adding "set-target"'s dependency "fullclean" to list of commands with default set of options.
Executing action: fullclean
Build directory '/private/tmp/test/build' not found. Nothing to clean.
Executing action: set-target
Set Target to: linux, new sdkconfig will be created.
Running cmake in directory /private/tmp/test/build
Executing "cmake -G Ninja -DPYTHON_DEPS_CHECKED=1 -DPYTHON=/Users/ivan/.espressif/python_env/idf5.5_py3.11_env/bin/python -DESP_PLATFORM=1 -DIDF_TARGET=linux -DCCACHE_ENABLE=1 /private/tmp/test"...
-- Existing sdkconfig '/tmp/test/sdkconfig' renamed to '/tmp/test/sdkconfig.old'.
-- Found Git: /opt/homebrew/bin/git (found version "2.39.2")
-- The C compiler identification is AppleClang 15.0.0.15000309
-- The CXX compiler identification is AppleClang 15.0.0.15000309
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test ver_gnu17_supported
-- Performing Test ver_gnu17_supported - Success
-- Performing Test ver_gnu++2b_supported
-- Performing Test ver_gnu++2b_supported - Success
-- ccache will be used for faster recompilation
-- The ASM compiler identification is AppleClang
-- Found assembler: /Library/Developer/CommandLineTools/usr/bin/cc
-- git rev-parse returned 'fatal: not a git repository (or any of the parent directories): .git'
-- Could not use 'git describe' to determine PROJECT_VER.
-- Building ESP-IDF components for target linux
-- Project sdkconfig file /tmp/test/sdkconfig
Loading defaults file /tmp/test/sdkconfig.defaults...
-- Linker Version: 1053.12
-- Components: esp_common esp_hw_support esp_partition esp_rom esp_system freertos hal heap linux log main nvs_flash partition_table soc spi_flash
-- Component paths: /Users/ivan/e/esp-idf/components/esp_common /Users/ivan/e/esp-idf/components/esp_hw_support /Users/ivan/e/esp-idf/components/esp_partition /Users/ivan/e/esp-idf/components/esp_rom /Users/ivan/e/esp-idf/components/esp_system /Users/ivan/e/esp-idf/components/freertos /Users/ivan/e/esp-idf/components/hal /Users/ivan/e/esp-idf/components/heap /Users/ivan/e/esp-idf/components/linux /Users/ivan/e/esp-idf/components/log /tmp/test/main /Users/ivan/e/esp-idf/components/nvs_flash /Users/ivan/e/esp-idf/components/partition_table /Users/ivan/e/esp-idf/components/soc /Users/ivan/e/esp-idf/components/spi_flash
-- Configuring done (5.4s)
-- Generating done (0.0s)
-- Build files have been written to: /tmp/test/build
Executing action: copy-compile-commands
Done
Executing action: all (aliases: build)
Running ninja in directory /private/tmp/test/build
Executing "ninja all"...
[66/82] Generating ../../partition_table/partition-table.bin
Partition table binary generated. Contents:
*******************************************************************************
# ESP-IDF Partition Table
# Name, Type, SubType, Offset, Size, Flags
nvs,data,nvs,0x9000,24K,
phy_init,data,phy,0xf000,4K,
factory,app,factory,0x10000,1M,
*******************************************************************************
[69/82] cd /tmp/test/build/esp-idf/partition_table && /opt/homebrew/...*******************************************************************"
Partition table binary generated. Contents:
*******************************************************************************
# ESP-IDF Partition Table
# Name, Type, SubType, Offset, Size, Flags
nvs,data,nvs,0x9000,24K,
phy_init,data,phy,0xf000,4K,
factory,app,factory,0x10000,1M,
*******************************************************************************
[82/82] Linking CXX executable test.elf
ld: warning: ignoring duplicate libraries: 'esp-idf/esp_common/libesp_common.a', 'esp-idf/esp_hw_support/libesp_hw_support.a', 'esp-idf/esp_partition/libesp_partition.a', 'esp-idf/esp_rom/libesp_rom.a', 'esp-idf/esp_system/libesp_system.a', 'esp-idf/freertos/libfreertos.a', 'esp-idf/hal/libhal.a', 'esp-idf/heap/libheap.a', 'esp-idf/linux/liblinux.a', 'esp-idf/log/liblog.a', 'esp-idf/nvs_flash/libnvs_flash.a', 'esp-idf/spi_flash/libspi_flash.a'
Executing action: monitor
Running idf_monitor in directory /private/tmp/test
Executing "/Users/ivan/.espressif/python_env/idf5.5_py3.11_env/bin/python /Users/ivan/e/esp-idf/tools/idf_monitor.py --toolchain-prefix  --target linux --revision 0 /private/tmp/test/build/test.elf -m '/Users/ivan/.espressif/python_env/idf5.5_py3.11_env/bin/python' '/Users/ivan/e/esp-idf/tools/idf.py'"...
--- esp-idf-monitor 1.5.0 on linux
--- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H
I (1309331414) port: Starting scheduler.
Value: 42

Executing action: copy-compile-commands
Done

Could you please add more details to help us reproduce or troubleshoot this segfault you are getting? (IDF version, code snippet if possible, whether you are mocking any of the components mentioned in the Components: list in the log above, any differences in your CMake files or build process, sdkconfig file, partition table in case you are using a custom one etc.)

[higaski]

This should work with just nvs_flash_init, at least in recent IDF versions.

Interesting. You're right, I just tested this on a new project and with default settings it really does work out of the box.

Could you please add more details to help us reproduce or troubleshoot this segfault you are getting? (IDF version, code snippet if possible, whether you are mocking any of the components mentioned in the Components: list in the log above, any differences in your CMake files or build process, sdkconfig file, partition table in case you are using a custom one etc.)

Yes sure. I'm using ESP-IDF v5.3, the project uses a custom directory layout with a "src" component as main and a "tests" folder as... well, test component. You have already listed the issue however! It seems to be related to a custom partition table. If I use the default one all is fine, no segfault occurs. However if I switch to my custom one

# ESP-IDF Partition Table
# Name,   Type, SubType, Offset,  Size, Flags
otadata,  data, ota,     ,        8K,
ota_0,    app,  ota_0,   ,        6M,
ota_1,    app,  ota_1,   ,        6M,
nvs,      data, nvs,     ,        6M,
data,     data, littlefs,,        14272K,

things explode

fish: Job 1, './build/ESP32BuildSystem.elf' terminated by signal SIGSEGV (Address boundary error)

[igrr]

I see, thanks for the reproducer. Indeed there is a leftover in spi_flash component emulation, total emulated flash size doesn't automatically follow the size occupied by all the partitions.

Please add this to your app as a temporary workaround:

// for esp_partition_get_file_mmap_ctrl_input:
#include "esp_private/partition_linux.h"


void app_main() {
    // before calling nvs_flash_init or any other storage-related functions
    esp_partition_file_mmap_ctrl_t *p_file_mmap_ctrl = esp_partition_get_file_mmap_ctrl_input();
    p_file_mmap_ctrl->flash_file_size = 32 * 1024 * 1024;
    strcpy(p_file_mmap_ctrl->partition_file_name, "build/partition_table/partition-table.bin");  // assuming the CWD is project root directory and the build directory is "build"
    ...
    
    

we also need to improve bounds checking in esp_partition_* functions for linux target, so that at least they don't segfault.