update NVS encryption docs to note that post flash encryption writing nvs_key with parttool.py must include --esptool-write-args encrypt flag

[bschick]

Environment

• Module or chip used: ESP32-C3-MINI-1-N4
• IDF version v4.4-beta1-275-g214d62b9ad
• Build System: idf.py
• Operating System: macO
• Using an IDE?: No
• Power Supply: USB

Problem Description

I am creating a separate encrypted NVS partition with pre-generated keys (in development flash encyption) following the directions here:
https://docs.espressif.com/projects/esp-idf/en/latest/esp32c3/api-reference/storage/nvs_flash.html#nvs-key-partition

I had uploaded the nvs_key partition with this template:
parttool.py --port /dev/ttyUSB0 --partition-table-offset "nvs_key partition offset" write_partition --partition-name="name of nvs_key partition" --input "nvs_key partition"

Also uploaded the encrypted custom NVS partition. But after enabling flash encryption and letting the bootloader encrypt appropriate partitions and initializing the custom partition with nvs_flash_secure_init_partition, opening the partition with a call like the following always returns ESP_ERR_NVS_NOT_FOUND.
nvs_open_from_partition("customer", "id", NVS_READONLY, &nvs_handle));

After much tinkering and reading esptool.py source, I found that you cannot use compression to upload encrypted partitions. This seems to matter whether you write the partition before or after flash encryption is applied. Before flash encryption is enabled, I assume you'd need to disable compression by adding the following to the parttool.py command (I have not tested this)
--esptool-write-args no-compress

What I did test and validate is that post flash encryption you must use the --encrypt flag for esptool.py. Adding the following to parttool.py does that and allows the pre-encrypted NVS partition to be opened and read:
--esptool-write-args encrypt

Proposed Change

Edit the docs so the parttool.py "template" command for pre-generated keys includes the esptool no-compress flag when run before flash encryption is applied (I have not validated this works, but I know it fails without):
parttool.py --esptool-write-args no-compress --port /dev/ttyUSB0 --partition-table-offset "nvs_key partition offset" write_partition --partition-name="name of nvs_key partition" --input "nvs_key partition"

Also add a description to the docs about writing nvs_key in developer mode after flash encryption is applied (I have validated this works, and I know corrupts the keys without):
parttool.py --esptool-write-args encrypt --port /dev/ttyUSB0 --partition-table-offset "nvs_key partition offset" write_partition --partition-name="name of nvs_key partition" --input "nvs_key partition"

[negativekelvin]

There should not be any problem using compression before flash encryption is enabled, the bootloader, partition table, and the app partitions get flashed with compression before they are encrypted.

[bschick]

I can try it again, but when I followed the steps in that link and wrote all those partitions in plaintext, the pre-encrypted NVS partition was not readable on the ESP32C3 device. And IIRC the Wi-Fi provisioning manager also could not read the default NVS partition and needed to be re-provisioned. Will try again today and reply.

[negativekelvin]

Note that Nvs partitions should not be marked as encrypted in your partition table, only nvs key partitions.

[bschick]

Thanks, I do not have the NVS partition itself marked encrypted. But I tried it again, and it worked as documented. Hard to say what went wrong the first few tries... Perhaps I had not rebuilt .bin files or something.

Either way, I'd still suggest that the docs are updated to note that in developer mode and post flash encyption, you can still update keys but must include --esptool-write-args encrypt in the parttool.py command.

If you don't include that flag the write appears to work (no errors or warnings about writing to an encrypted partition), but the data is corrupted, and the key won't decrypt a pre-generated NVS encrypted partition.

I updated the title to reflect that suggestion.

[0xjakob]

@bschick Sorry for the issue in the documentation, I'm taking a look right now and connecting a few things already. But somehow I can't reproduce the successful flash of the NVS key partition after encryption. What command and arguments did you use to re-flash the NVS key partition?

[0xjakob]

@bschick I'll add some note to the documentation. I could reproduce your issue almost entirely. The only difference is that I didn't see ESP_ERR_NVS_NOT_FOUND when trying to initialize or open an encrypted partition with wrongly flashed key partition. Instead, I always saw ESP_ERR_NVS_CORRUPT_KEY_PART. Furthermore, how did your initialization work? On my device, the NVS initialization (nvs_init, nvs_init_from_partition) fails already and even ignoring the error there would still give me an ESP_ERR_NVS_CORRUPT_KEY_PART error when trying to open a partition.

[AxelLin]

Hi @0xjakob
I got "A fatal error occurred: Active security features detected, erasing flash is disabled as a safety measure. Use --force to override, please use with caution, otherwise it may brick your device!" when I do below command:
(But parttool.py does not accept the --force setting)

$ parttool.py --esptool-write-args encrypt --port /dev/ttyUSB0 --partition-table-file=build/partition_table/partition-table.bin --partition-table-offset 0x8000 write_partition --partition-name="nvs_key" --input keys/my_nvs_key_partition_file.bin
Running /home/axel/.espressif/python_env/idf5.2_py3.10_env/bin/python /home/axel/esp/esp-idf/components/esptool_py/esptool/esptool.py --port /dev/ttyUSB0 erase_region 4190208 4096...
esptool.py v4.7.0
Serial port /dev/ttyUSB0
Connecting...
Detecting chip type... ESP32-S3
Chip is ESP32-S3 (QFN56) (revision v0.1)
Features: WiFi, BLE, Embedded PSRAM 8MB (AP_3v3)
Crystal is 40MHz
MAC: f4:12:fa:fa:49:40
Stub is already running. No upload is necessary.

A fatal error occurred: Active security features detected, erasing flash is disabled as a safety measure. Use --force to override, please use with caution, otherwise it may brick your device!
An exception: ** Command '['/home/axel/.espressif/python_env/idf5.2_py3.10_env/bin/python', '/home/axel/esp/esp-idf/components/esptool_py/esptool/esptool.py', '--port', '/dev/ttyUSB0', 'erase_region', '4190208', '4096']' returned non-zero exit status 2. ** occurred in _call_esptool.
Traceback (most recent call last):
File "/home/axel/esp/esp-idf/components/partition_table/parttool.py", line 363, in
main()
File "/home/axel/esp/esp-idf/components/partition_table/parttool.py", line 356, in main
op(**common_args)
File "/home/axel/esp/esp-idf/components/partition_table/parttool.py", line 179, in _write_partition
target.write_partition(partition_id, input)
File "/home/axel/esp/esp-idf/components/partition_table/parttool.py", line 167, in write_partition
self.erase_partition(partition_id)
File "/home/axel/esp/esp-idf/components/partition_table/parttool.py", line 155, in erase_partition
self._call_esptool(['erase_region', str(partition.offset), str(partition.size)] + self.esptool_erase_args)
File "/home/axel/esp/esp-idf/components/partition_table/parttool.py", line 130, in _call_esptool
raise e
File "/home/axel/esp/esp-idf/components/partition_table/parttool.py", line 127, in _call_esptool
subprocess.check_call(esptool_args, stdout=out, stderr=subprocess.STDOUT)
File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/axel/.espressif/python_env/idf5.2_py3.10_env/bin/python', '/home/axel/esp/esp-idf/components/esptool_py/esptool/esptool.py', '--port', '/dev/ttyUSB0', 'erase_region', '4190208', '4096']' returned non-zero exit status 2.

[KonstantinKondrashov]

Hi @AxelLin!
There is a way to pass the --force flag over this --esptool-erase-args', help='additional subcommand arguments when erasing regions of flash'. See the help parttool.py -h:

  --esptool-args ESPTOOL_ARGS [ESPTOOL_ARGS ...]
                        additional main arguments for esptool
  --esptool-write-args ESPTOOL_WRITE_ARGS [ESPTOOL_WRITE_ARGS ...]
                        additional subcommand arguments when writing to flash
  --esptool-read-args ESPTOOL_READ_ARGS [ESPTOOL_READ_ARGS ...]
                        additional subcommand arguments when reading flash
  --esptool-erase-args ESPTOOL_ERASE_ARGS [ESPTOOL_ERASE_ARGS ...]
...

parttool.py --esptool-erase-args=force \
--esptool-write-args encrypt --port /dev/ttyUSB0 \
--partition-table-file=build/partition_table/partition-table.bin --partition-table-offset 0x8000 write_partition --partition-name="nvs_key" --input keys/my_nvs_key_partition_file.bin

Could you try this way?

[AxelLin]

Hi @KonstantinKondrashov

Above command works, thanks.

It's a big surprise if copy-paste the command from document does not work.
Could you update the below document accordingly?
https://docs.espressif.com/projects/esp-idf/en/stable/esp32s3/api-reference/storage/nvs_encryption.html
(See the section: "Use a pre-generated NVS key partition":
If the device is encrypted in flash encryption development mode and you want to renew the NVS key partition, ...