SPIFFS可以成功挂载文件系统，POSSIX / C均可以打开文件，但是读文件失败，返SPIFFS_ERR_INDEX_REF_LU [-10016] (GIT8266O-328)

[EaonL]

环境描述：
ESP8266 + 4M Bytes SPI FLASH 【W25Q32】+ SDK release/v3.2

分区定义：
nvs, data, nvs, 0x9000, 0x6000,
phy_init, data, phy, 0xf000, 0x1000,
factory, app, factory, 0x10000, 512K,
storage, data, spiffs, , 448K,

操作步骤：

1. 使用esptools.py工具先整体擦除Flash；
2. 创建spiffs文件系统镜像目录：
   mkdir spiffs_root
   vim hello.txt 【随便写了一串字符串在文件里】
3. 使用如下命令制作spiffs文件系统镜像：
   python ./tools/spiffsgen.py 0x70000 spiffs_root/ spiffs.bin
4. 在storage分区烧写文件系统镜像：
   python ./tools/esptool.py --chip esp8266 --port /dev/ttyUSB0 --baud 115200 write_flash -z 0x90000 spiffs.bin
5. 改写SPIFFS_CHECK_RES宏函数，打印相关错误信息：
   #define SPIFFS_CHECK_RES(res)
   do {
   if ((res) < SPIFFS_OK)
   {
   printf("Err!! file: %s, func: %s, line: %d, ErrNo: %d\n\n", FILE, func, LINE, res);
   return (res);
   }
   } while (0);
6. make menuconfig将Log out等级设置为Debug模式，并打开SPIFFS的所有输出；
7. 改写example/storage/spiffs/main/spiffs_example_main.c :
   7.1 挂载spiffs；
   7.2 POSSIX 或 C 编写open("/spiffs/hello.txt"，...)，read(fd, ...)；
8. make flash && make monitor；
9. 输出如下日志：

D (701) SPIFFS: read: load objix page 0001:0000 for data spix:0000

D (711) SPIFFS: read: offset:0 rd:20 data spix:0000 is data_pix:ff00 addr:00ff0005

Err!! file: /root/esp/ESP8266_RTOS_SDK/components/spiffs/spiffs/src/spiffs_nucleus.c, func: spiffs_object_read, line: 2036, ErrNo: -10016

D (741) SPIFFS: SPIFFS_clearerr

Note:
spiffs的问题库上找到一条类似的问题描述： pellepl/spiffs#160
他的环境是MSP430 + IAR，由于地址对齐问题引发，他使能cache可以read；然而我在ESP8266上各种折腾也不能成功读出文件，我有尝试使用esptools.py工具读Flash，读出来的内容和写进去的内容一致。

[EaonL]

问题已解决：
问题原因：引发这个问题的根本原因应该是spiffsgen.py工具配置的SPIFFS_ALIGNED_OBJECT_INDEX_TABLES与SDK配置的不一致，SDK将SPIFFS_ALIGNED_OBJECT_INDEX_TABLES配置为4，不清楚spiffsgen.py工具配置的值是多少。
解决方法：使用mkspiffs制作文件系统镜像，编译mkspiffs时将SPIFFS宏配置与SDK中的值设置成一致。

1. git clone https://github.com/igrr/mkspiffs.git
2. cd mkspiffs
3. git submodule update --init
4. make dist CPPFLAGS=“-DSPIFFS_OBJ_META_LEN=4 -DSPIFFS_ALIGNED_OBJECT_INDEX_TABLES=4" BUILD_CONFIG_NAME=”-esp-8266”
5. ./tools/mkspiffs/mkspiffs -c spiffs_root_win/ -b 4096 -p 256 -s 0x70000 spiffs.bin
6. python ./tools/esptool.py --chip esp8266 --port /dev/ttyUSB0 --baud 115200 write_flash -z 0x90000 spiffs.bin
7. make flash && make monitor
8. 成功打开文件并读取内容。

[EaonL]

spiffsgen.py生成的spiffs.bin文件系统镜像和SDK配置的spiffs格式不一致，使用mkspiffs工具将spiffs文件系统镜像的格式与SDK配成一致，问题解决。

[tius2000]

Hi @EaonL,
thank you for posting your results! I have the same problem and still get an error SPIFFS_ERR_INDEX_REF_LU [-10016]:

$ grep SPIFFS <sdkconfig
CONFIG_USING_SPIFFS=y
CONFIG_SPIFFS_MAX_PARTITIONS=3
CONFIG_SPIFFS_CACHE=y
CONFIG_SPIFFS_CACHE_WR=y
# CONFIG_SPIFFS_CACHE_STATS is not set
CONFIG_SPIFFS_PAGE_CHECK=y
CONFIG_SPIFFS_GC_MAX_RUNS=10
# CONFIG_SPIFFS_GC_STATS is not set
CONFIG_SPIFFS_PAGE_SIZE=256
CONFIG_SPIFFS_OBJ_NAME_LEN=32
CONFIG_SPIFFS_USE_MAGIC=y
CONFIG_SPIFFS_USE_MAGIC_LENGTH=y
CONFIG_SPIFFS_META_LENGTH=4
CONFIG_SPIFFS_USE_MTIME=y
# CONFIG_SPIFFS_DBG is not set
# CONFIG_SPIFFS_API_DBG is not set
# CONFIG_SPIFFS_GC_DBG is not set
# CONFIG_SPIFFS_CACHE_DBG is not set
# CONFIG_SPIFFS_CHECK_DBG is not set
# CONFIG_SPIFFS_TEST_VISUALISATION is not set

$ mkspiffs --version
mkspiffs ver. 0.2.3-6-g983970e
Build configuration name: esp-8266
SPIFFS ver. 0.3.7-5-gf5e26c4
Extra build flags: -DSPIFFS_OBJ_META_LEN=4 -DSPIFFS_ALIGNED_OBJECT_INDEX_TABLES=4
SPIFFS configuration:
  SPIFFS_OBJ_NAME_LEN: 32
  SPIFFS_OBJ_META_LEN: 4
  SPIFFS_USE_MAGIC: 1
  SPIFFS_USE_MAGIC_LENGTH: 1
  SPIFFS_ALIGNED_OBJECT_INDEX_TABLES: 4

Is there a wrong setting? Could you please post the relevant settings of your sdkconfig?

[EaonL]

Hi @EaonL,
thank you for posting your results! I have the same problem and still get an error SPIFFS_ERR_INDEX_REF_LU [-10016]:

$ grep SPIFFS <sdkconfig
CONFIG_USING_SPIFFS=y
CONFIG_SPIFFS_MAX_PARTITIONS=3
CONFIG_SPIFFS_CACHE=y
CONFIG_SPIFFS_CACHE_WR=y
# CONFIG_SPIFFS_CACHE_STATS is not set
CONFIG_SPIFFS_PAGE_CHECK=y
CONFIG_SPIFFS_GC_MAX_RUNS=10
# CONFIG_SPIFFS_GC_STATS is not set
CONFIG_SPIFFS_PAGE_SIZE=256
CONFIG_SPIFFS_OBJ_NAME_LEN=32
CONFIG_SPIFFS_USE_MAGIC=y
CONFIG_SPIFFS_USE_MAGIC_LENGTH=y
CONFIG_SPIFFS_META_LENGTH=4
CONFIG_SPIFFS_USE_MTIME=y
# CONFIG_SPIFFS_DBG is not set
# CONFIG_SPIFFS_API_DBG is not set
# CONFIG_SPIFFS_GC_DBG is not set
# CONFIG_SPIFFS_CACHE_DBG is not set
# CONFIG_SPIFFS_CHECK_DBG is not set
# CONFIG_SPIFFS_TEST_VISUALISATION is not set

$ mkspiffs --version
mkspiffs ver. 0.2.3-6-g983970e
Build configuration name: esp-8266
SPIFFS ver. 0.3.7-5-gf5e26c4
Extra build flags: -DSPIFFS_OBJ_META_LEN=4 -DSPIFFS_ALIGNED_OBJECT_INDEX_TABLES=4
SPIFFS configuration:
  SPIFFS_OBJ_NAME_LEN: 32
  SPIFFS_OBJ_META_LEN: 4
  SPIFFS_USE_MAGIC: 1
  SPIFFS_USE_MAGIC_LENGTH: 1
  SPIFFS_ALIGNED_OBJECT_INDEX_TABLES: 4

Is there a wrong setting? Could you please post the relevant settings of your sdkconfig?

This is my config:

./mkspiffs --version
mkspiffs ver. 0.2.3-6-g983970e
Build configuration name: EaonConfig_ESP8266
SPIFFS ver. 0.3.7-5-gf5e26c4
Extra build flags: -DSPIFFS_OBJ_META_LEN=4 -DSPIFFS_ALIGNED_OBJECT_INDEX_TABLES=4
SPIFFS configuration:
SPIFFS_OBJ_NAME_LEN: 32
SPIFFS_OBJ_META_LEN: 4
SPIFFS_USE_MAGIC: 1
SPIFFS_USE_MAGIC_LENGTH: 1
SPIFFS_ALIGNED_OBJECT_INDEX_TABLES: 4

grep SPIFFS sdkconfig
CONFIG_USING_SPIFFS=y
CONFIG_SPIFFS_MAX_PARTITIONS=3
# SPIFFS Cache Configuration
CONFIG_SPIFFS_CACHE=y
CONFIG_SPIFFS_CACHE_WR=y
CONFIG_SPIFFS_CACHE_STATS=
CONFIG_SPIFFS_PAGE_CHECK=y
CONFIG_SPIFFS_GC_MAX_RUNS=10
CONFIG_SPIFFS_GC_STATS=
CONFIG_SPIFFS_PAGE_SIZE=256
CONFIG_SPIFFS_OBJ_NAME_LEN=32
CONFIG_SPIFFS_USE_MAGIC=y
CONFIG_SPIFFS_USE_MAGIC_LENGTH=y
CONFIG_SPIFFS_META_LENGTH=4
CONFIG_SPIFFS_USE_MTIME=y
CONFIG_SPIFFS_DBG=
CONFIG_SPIFFS_API_DBG=
CONFIG_SPIFFS_GC_DBG=
CONFIG_SPIFFS_CACHE_DBG=
CONFIG_SPIFFS_CHECK_DBG=
CONFIG_SPIFFS_TEST_VISUALISATION=

Which version of the SDK are you using？

[tius2000]

Dear @EaonL,

This is my config:
...

Thank you for our config. I can see no relevant differences.

Which version of the SDK are you using？

sdk version v3.2-261-g7ab9602e-dirty

I have no idea what is wrong :-(

[EaonL]

Dear @EaonL,

This is my config:
...

Thank you for our config. I can see no relevant differences.

Which version of the SDK are you using？

sdk version v3.2-261-g7ab9602e-dirty

I have no idea what is wrong :-(

This question is annoying，but it will definitely be resolved in the end.
I see your command line output <sdkconfig, what version of Linux do you use? maybe u should try to use Ubuntu 16.04 TLS.

Beside, the mkffs tool is generated with your system's gcc compiler，check the version of your system's gcc compiler. xtensa-lx106-elf-gcc is a cross-link provided by Espressif, if your system's gcc version is not consistent with the gcc version of the compiled xtensa-lx106-elf-gcc cross toolchain, it may also cause the file to be unreadable:

root@ubuntu:~# gcc --version
gcc (GCC) 5.2.0
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

root@ubuntu:~# xtensa-lx106-elf-gcc --version
xtensa-lx106-elf-gcc (crosstool-NG crosstool-ng-1.22.0-100-ge567ec7) 5.2.0
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

See, the gcc version of my system is the same as the gcc version that generated xtensa-lx106-elf-gcc, they are both Version-5.2.0.

At last, I provide you a file system image 《spiffs_img.bin》, you can burn this file system image and read the file 《eaon.txt》 to see if it can be read successfully:

root@ubuntu:/root/spiffs_img# pwd
/root/spiffs_img
root@ubuntu:/root/spiffs_img#

root@ubuntu:/root/spiffs_img# ls
eaon.txt
root@ubuntu:/root/spiffs_img#

root@ubuntu:/root/spiffs_img# cat eaon.txt
Hello! This is a file from eaon!!
[ Note, in file eaon.txt, here is a '\n' ]
root@ubuntu:/root/spiffs_img#

root@ubuntu:/root/spiffs_img# cd ..
root@ubuntu:/root#

root@ubuntu:/root# ./tools/mkspiffs/mkspiffs -c spiffs_img/ -b 4096 -p 256 -s 0x70000 spiffs_img.bin

root@ubuntu:/root# stat spiffs_img.bin
File: 'spiffs_img.bin'
Size: 458752 Blocks: 896 IO Block: 4096 regular file
Device: 801h/2049d Inode: 527112 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2019-11-26 05:47:31.723173135 -0800
Modify: 2019-11-26 05:47:31.723173135 -0800
Change: 2019-11-26 05:47:31.723173135 -0800
Birth: -
root@ubuntu:/root#

As you can see, the size of spiffs_img.bin was defined as 0x70000, which means you need to set the partition of spiffs to 448K.

root@ubuntu:/root# cd $IDF_PATH/examples/storage/spiffs/

root@ubuntu:/root/esp/ESP8266_RTOS_SDK/examples/storage/spiffs# cat partitions_example.csv
#Name, Type, SubType, Offset, Size, Flags
#Note: if you change the phy_init or app partition offset, make sure to change the offset in Kconfig.projbuild
nvs, data, nvs, 0x9000, 0x6000,
phy_init, data, phy, 0xf000, 0x1000,
factory, app, factory, 0x10000, 512K,
storage, data, spiffs, 0x90000, 448K,
root@ubuntu:/root/esp/ESP8266_RTOS_SDK/examples/storage/spiffs#

python /root/tools/esptool.py --chip esp8266 --port /dev/ttyUSB0 --baud 115200 write_flash -z 0x90000 spiffs_img.bin

modify main/spiffs_example_main.c to read /spiffs_img/eaon.txt:
//The default mount in the Espressif-Example is fine
......
// Open file for reading
ESP_LOGI(TAG, "Reading file");
FILE* f = fopen("/spiffs/eaon.txt", "r");
if (f == NULL) {
ESP_LOGE(TAG, "Failed to open file for reading");
return;
}
char line[64];
fgets(line, sizeof(line), f);
fclose(f);
// strip newline
char* pos = strchr(line, '\n');
if (pos) {
*pos = '\0';
}
ESP_LOGI(TAG, "Read from file /spiffs_img/eaon.txt: '%s'", line);

make flash && make monitor

Then, hope u can see, good luck:
I (497) reset_reason: RTC reset 2 wakeup 0 store 0, reason is 2
I (501) example: Initializing SPIFFS
I (533) example: Partition size: total: 414401, used: 502
I (536) example: Reading file
I (539) example: Read from file /spiffs_img/eaon.txt: 'Hello! This is a file from eaon!!'

You should unzip the 《spiffs_img.zip》, and then you get 《spiffs_img.bin》.
spiffs_img.zip

[tius2000]

I see your command line output <sdkconfig, what version of Linux do you use? maybe u should try to use Ubuntu 16.04 TLS.

I'm using Windows 10 with the msys32 enviroment provided by EspressIF.

Beside, the mkffs tool is generated with your system's gcc compiler，check the version of your system's gcc compiler. xtensa-lx106-elf-gcc is a cross-link provided by Espressif, if your system's gcc version is not consistent with the gcc version of the compiled xtensa-lx106-elf-gcc cross toolchain, it may also cause the file to be unreadable:

I use the gcc versions provided by EpressIF within the msys32 enviroment:

$ gcc --version
gcc (GCC) 7.3.0
...
$ which gcc.exe
/usr/bin/gcc.exe
$  xtensa-lx106-elf-gcc --version
xtensa-lx106-elf-gcc.exe (crosstool-NG crosstool-ng-1.22.0-100-ge567ec7b) 5.2.0
...

You are right, there is a mismatch!

At last, I provide you a file system image 《spiffs_img.bin》, you can burn this file system image and read the file 《eaon.txt》 to see if it can be read successfully:

Your file is working :-)

file /spiffs/eaon.txt:
Hello! This is a file from eaon!!

So neither spiffsgen.py nor mkspiffs produce a valid image on my windows system. Perhaps this is really caused by the different gcc versions.

To depend on specific compiler versions seems a bit difficult. Perhaps it is possible to remove this dependency by adding esp8266 support to spiffsgen.py?

[EaonL]

I see your command line output <sdkconfig, what version of Linux do you use? maybe u should try to use Ubuntu 16.04 TLS.

I'm using Windows 10 with the msys32 enviroment provided by EspressIF.

Beside, the mkffs tool is generated with your system's gcc compiler，check the version of your system's gcc compiler. xtensa-lx106-elf-gcc is a cross-link provided by Espressif, if your system's gcc version is not consistent with the gcc version of the compiled xtensa-lx106-elf-gcc cross toolchain, it may also cause the file to be unreadable:

I use the gcc versions provided by EpressIF within the msys32 enviroment:

$ gcc --version
gcc (GCC) 7.3.0
...
$ which gcc.exe
/usr/bin/gcc.exe
$  xtensa-lx106-elf-gcc --version
xtensa-lx106-elf-gcc.exe (crosstool-NG crosstool-ng-1.22.0-100-ge567ec7b) 5.2.0
...

You are right, there is a mismatch!

At last, I provide you a file system image 《spiffs_img.bin》, you can burn this file system image and read the file 《eaon.txt》 to see if it can be read successfully:

Your file is working :-)

file /spiffs/eaon.txt:
Hello! This is a file from eaon!!

So neither spiffsgen.py nor mkspiffs produce a valid image on my windows system. Perhaps this is really caused by the different gcc versions.

To depend on specific compiler versions seems a bit difficult. Perhaps it is possible to remove this dependency by adding esp8266 support to spiffsgen.py?

Maybe you should use your system's gcc to compile the toolchain yourself, rather than using the toolchain binary package compiled by Espressif. I don’t know how spiffsgen.py implements file system mirroring, and I don’t trust it. Here is the doc how to build the toolchain yourself:
Customized Setup of Toolchain

[tius2000]

Hi @EaonL,

I had problems setting up a different toolchain on windows. So I tried to investigate the problem with spiffsgen.py and found a problem with the alignment of the page indices. spiffsgen.py seems not to do any alignment here. According to spiffs_nucleus.h SPIFFS_ALIGNED_OBJECT_INDEX_TABLES is not an integer value, but a binary flag:

#if SPIFFS_ALIGNED_OBJECT_INDEX_TABLES
    __attribute(( aligned(sizeof(spiffs_page_ix)) ))
#endif

As a test I added the following code to spiffsgen.py at line 220 and got a valid image :-)

#   Add padding if required
while len(img) % SPIFFS_PAGE_IX_LEN: img += b"\x00"

Of course, this should made a command line option for spiffsgen.py and enabled from Makefile.projbuild. Someone with more knowledge than I might provide a complete patch.

Thank you very much your help to find a solution!

[GarMingLi]

不知道为什么，我的环境中没有spiffsgen.py这个文件

[yuanminglove]

export SPIFFS_ALIGNED_OBJECT_INDEX_TABLES=4
python spiffsgen.py 0x70000 data build/storage.bin --aligned-obj-ix-tables