External flash usage

[Avamander]

It would be good to map out where LittleFS would potentially be used in InfiniTime, right now and in the future. What would have to be migrated and what can't be migrated.

Current usages of flash contain:

• Bootloader scratch sections
• Settings storage
• Datetime backup (Littlefs (or similar) datetime backup #87)

Future usages might contain:

• Resource storage (for LVGL)
• Buffer for HR/step data (Bug/Feature: Persist Step Count Between Resets #493)
• ??? Please leave your comment below

[joaquimorg]

I managed to do a first performance test between RAWFS versus LITTLEFS.
The use of LITTLEFS adds another 20Kb in the firmware.
As we can see in the video the performance between the two is evident.

The LITTLEFS introduces some advantages in the management of the files, but I think that the loss of performance and increase of the firmware does not justify the gain of functionalities.

In my case, I will not invest more time in the littlefs solution, I will try to improve RAWFS and add some features to allow partial updates or subdivide between resources and watch faces, so that I can have customizable watch faces without having to always send everything.

[Avamander]

What's the LittleFS configuration you used?

[joaquimorg]

What's the LittleFS configuration you used?

https://github.com/joaquimorg/PinetimeLite/blob/1db8712ecd49ecb2ee37147eebe2e5b6dac521ac/src/components/fs/FS.cpp#L81

[Avamander]

I will try to optimize the settings, because I think better read performance is doable and worth the investment.

The advantages we'd get from LittleFS, such as reduced maintenance and development burden, full-flash wear leveling, better failure tolerance (CoW), thought-out and documented design, higher interoperability and alternative implementations are more than some.

[joaquimorg]

In my tests I had already identified that LVGL is always opening files to read, and did not keep the files open while refreshing the screen.
After all, it was my mistake, the code allow that there could be several files open at the same time, but LVGL does not take advantage of this if it is not indicated in LV_IMG_CACHE_DEF_SIZE how many images we want to keep open.
I will make this change and carry out further tests.

I would also prefer to use littlefs, but I don't want its use to translate into having something that makes a point of losing the little performance we have.

When I indicated that I would not invest more time in littlefs it was in the continuation of developing the mechanisms so that we can have a management of the files, delete files, create folders etc...

So whenever I discover something that can improve the performance I will test it.

[JF002]

Thanks @joaquimorg and @Avamander , great work!

From your video, the performance penality looks... surprisingly huge! Maybe @Avamander will be able to fine-tune/optimise the integration of littlefs with lvgl?

Now... Maybe we are trying to solve many problems with a single solution?
The use-case tested by @joaquimorg on his video is a "read-only" use case : we store files in the external memory once and only read them from the firmware. Those data are (over)written only when upgrading the graphical assets.
The use-case where a more advanced file system like littlefs would bring a lot of advantages are "read/write" use-cases like storing 24h or heart rate data, user settings, save the master piece the user's just drawn using InfiniPaint,...

Maybe we can explore the possibility to split the external flash space available in 2 parts : one read only to store fonts, icons, bitmaps, static assets. And the other one to store dynamic data that is often read/written.

What's your opinion?

[Avamander]

Thanks joaquimorg and Avamander , great work!
Maybe Avamander will be able to fine-tune/optimise the integration of littlefs with lvgl?

joaquimorg deserves more credit here and he also said that LV_IMG_CACHE_DEF_SIZE might provide a quick and easy way of regaining the lost performance, if that doesn't work out I'll investigate as well.

Maybe we can explore the possibility to split the external flash space available in 2 parts : one read only to store fonts, icons, bitmaps, static assets. And the other one to store dynamic data that is often read/written.

Yes, that's a theoretical possibility. But the returns are somewhat smaller. If it still turns out that LFS is not worth it, we just have to go with an alternative solution.

[joaquimorg]

Maybe we can explore the possibility to split the external flash space available in 2 parts : one read only to store fonts, icons, bitmaps, static assets. And the other one to store dynamic data that is often read/written.

I think it may be a viable solution, a few days ago we were discussing this on the discord.

Really the part of the resources for the firmware can be read-only and we don't have the need to have littlefs for that, and we can have an r/w zone with littlefs, in total we have about 3.4 Mb still free if we share between the two we have space for lots of applications to save data.

In the end we have to remember that we are talking about a device that has its limitations and we do not want to have a very complex system, my goal is always to look for a way to try to do things to get the maximum performance from the smartwatch which it often leads to doing things in a less generic way, more oriented towards the optimization of the few resources we have.

This weekend I will try to do some more tests to try to see if the changes have influence.

[ObiKeahloa]

Maybe we can explore the possibility to split the external flash space available in 2 parts : one read only to store fonts, icons, bitmaps, static assets. And the other one to store dynamic data that is often read/written.

We can also make a proper filesystem with folders (Directory's) , Example:
A fonts folder for the font files , Assets that have all assets for the firmware (Resources) , AppData (Contains appdata that is stored to the spi flash) , Userdata (User Settings , Time Settings etc,)

If possible we could also try to make the Assets r/w , so that the user can add or modify the assets without having to make an entirely new firmware....

This could be a vulnerability but a hierarchy system can be put in place: System > User > Apps.
This might need a lot of code so might be out of reach.

[joaquimorg]

I did some more tests changing the LVGL configuration (LV_IMG_CACHE_DEF_SIZE)
and it really made all the difference, of course there is some loss of performance, but I think it is possible to live with it.
Now the hardest work begins, turning everything into a PR ...

[Avamander]

I'm amazed, truly incredibly cool. Thanks @joaquimorg

[JF002]

This is really impressive! Thanks for your experimentation!

So, LVGL could be the unique answer for all our "external" storage use-cases?

For the PR, what would be the easiest way for you? A single PR of split the changes accross multiple PR to ease the review process?
Do you have a branch on github so we can have a look at how it works?

[joaquimorg]

Do you have a branch on github so we can have a look at how it works?

The branch with the test code is here
https://github.com/joaquimorg/PinetimeLite/tree/UsingLITTLEFS

[joaquimorg]

For the PR, what would be the easiest way for you? A single PR of split the changes accross multiple PR to ease the review process?

I will try to split as much as possible to be easier to include without causing breaks in what already exists, and thus also be easier to review, however it will only work in full when everything is together.