Add scoreboard 🏆

[mutatrum]

This PR adds a scoreboard screen that shows the top 20 shares:

And an api/system/scoreboard api call:

[
  {
    "difficulty": 956725.8468403114,
    "job_id": "684934fc00047bfd",
    "extranonce2": "0600000000000000",
    "ntime": 1758023248,
    "nonce": "F298047E",
    "version_bits": "040BC000"
  },
  {
    "difficulty": 942190.512488009,
    "job_id": "684934fc00047bc7",
    "extranonce2": "0e00000000000000",
    "ntime": 1758021685,
    "nonce": "5B7D04AA",
    "version_bits": "0122A000"
  },
  {
    "difficulty": 699177.2079349027,
    "job_id": "684934fc00047b90",
    "extranonce2": "1f00000000000000",
    "ntime": 1758020167,
    "nonce": "8B1F05EA",
    "version_bits": "04CA2000"
  },

Currently it captures all the fields from mining.submit, but others can be added. Not sure if adding the full header and resulting hash should be added.

[github-actions]

Test Results

19 tests  ±0   19 ✅ ±0   0s ⏱️ ±0s
 1 suites ±0    0 💤 ±0 
 1 files   ±0    0 ❌ ±0 

Results for commit e2653d0. ± Comparison against base commit 528f813.

♻️ This comment has been updated with latest results.

[mutatrum]

Thanks to @duckaxe for combing over my NG stuff 🤝

[skot]

Next step would be to show it somewhere on AxeOS and to persist this table in NVS. Adding the current all time best to the table would be a nice-to-have, even though we don't have the other fields.

As with all flash we need to be careful to not erase from NVS at high frequency as this can wear out the flash. I think keeping the high score list in NVS is a great idea. Just need to be aware that we only get something like 100,000 erase cycles on a ESP32 flash sector before it will wear out and be unreliable. An erase every minute would be bad (~ 2 months MTBF), an erase every hour (~ 11 years MTBF) would probably be fine. The esp-idf NVS library does some wear leveling afaik, but I wouldn't want to rely on that.

generally speaking, just changing a value in flash involves a erase cycle on the whole sector.

[mutatrum]

Next step would be to show it somewhere on AxeOS and to persist this table in NVS. Adding the current all time best to the table would be a nice-to-have, even though we don't have the other fields.

As with all flash we need to be careful to not erase from NVS at high frequency as this can wear out the flash. I think keeping the high score list in NVS is a great idea. Just need to be aware that we only get something like 100,000 erase cycles on a ESP32 flash sector before it will wear out and be unreliable. An erase every minute would be bad (~ 2 months MTBF), an erase every hour (~ 11 years MTBF) would probably be fine. The esp-idf NVS library does some wear leveling afaik, but I wouldn't want to rely on that.

generally speaking, just changing a value in flash involves a erase cycle on the whole sector.

Apologies for the long reply, it was a deeper rabbit hole than I expected.

I did some reading, and initially bad news: the current NVS partition is only 24kb, which means 6 pages of 4kb. However, the write pattern is more subtle, and that's good news: each page is an append log, and writes are append onto the same page without an erase. Updates are also appended, and the old value is marked as deleted. Only when a page is full, it starts to use a next page. If all but the last empty page is used, a full page is recycled, where still valid key/values are moved to a new page and it's actually flash erased. link

We are currently at ~33% usage of NVS, Added some statistics:

I (1104) nvs_device: Used entries: 249
I (1104) nvs_device: Free entries: 507
I (1104) nvs_device: Available entries: 381
I (1107) nvs_device: Total entries: 756

The difference between free and available is old values that are marked as deleted, and will at some time be garbage collected page. Each entry is 32 bytes: small values (8, 16 or 32 bit) are a single entry, blobs and strings can be more than one entry. Blobs have 2 entries overhead and string have 1 entry overhead.

I also looked at repartitioning, but that's a tricky avenue. It has been done, but makes f.e. downgrades very challenging. This PR doesn't warrant that.

So, not sure here. It's possible to add 20 NVS string entries with all values concatenated per line. This would be around 80 NVS entries. IMO this is a reasonable trade-off for write granularity. Having the scoreboard persistent is making it a lot more fun. More ways of celebrating high difficulties and to have an idea when they happened and how rare a high difficulty is gives a lot of insight in the process.

[mutatrum]

Besides the NVS storage aspect, are there other fields that should be added or replaced?

[skot]

This does feel a little bit outside of the practical use of the ESP32's limited NVS. I could image the high scoreboard being a good thing to punt to the client dashboard to keep track of.. Or maybe just ESP32 RAM?

[mutatrum]

lol I added it, as I wanted to test if it would work. With 20 entries as strings, I get the following NVS statistics:

I (1116) nvs_device: Used entries: 329
I (1116) nvs_device: Free entries: 427
I (1116) nvs_device: Available entries: 301
I (1119) nvs_device: Total entries: 756

Exactly 80 entries are added, the math works out. It uses about 10% of NVS storage, it's now at 43%.

The first few minutes have quite a few updates, but as the difficulty rises quickly, the number of updates will drop really fast. A minimum threshold before saving an entry would reduce that.

This does feel a little bit outside of the practical use of the ESP32's limited NVS. I could image the high scoreboard being a good thing to punt to the client dashboard to keep track of.. Or maybe just ESP32 RAM?

I agree, it's pushing it a bit. But as I mentioned above, at the same time it's also a really informative addition, IMO. And having the scoreboard survive a reboot is chefs kiss 😆

To not paint ourselves into the corner, I'll experiment a bit with adding a second NVS partition at the end of the partition table. I can't find a conclusive answer from the documentation if this is allowed/possible or not.

[duckaxe]

Some improvements mutatrum#4

[duckaxe]

A little fix for smaller screens mutatrum#6

[Travetown]

Wouldn't it be good if the highest difficulty level to date could be adopted? Of course, not all the information is available anymore. Perhaps it could be replaced with ‘*’ or something else. But then the list would be complete. At least in first place.

[mutatrum]

Wouldn't it be good if the highest difficulty level to date could be adopted? Of course, not all the information is available anymore. Perhaps it could be replaced with ‘*’ or something else. But then the list would be complete. At least in first place.

I though about that, but think that's more grief than necessary. I'm running my Gamma now for ~ 9 months, so to get a new higher best, it would take on average another 9 months. (I didn't calculate, but it has a 5G best difficulty, no idea what the odds are for that.) To get that score flushed out of the top 20, it would need 20 higher scores, which would probably take longer than the lifetime of the device. Then for the rest of the lifetime of the device, there will be a single line item that can't be sorted and shows empty fields.

[Travetown]

I though about that, but think that's more grief than necessary. I'm running my Gamma now for ~ 9 months, so to get a new higher best, it would take on average another 9 months. (I didn't calculate, but it has a 5G best difficulty, no idea what the odds are for that.) To get that score flushed out of the top 20, it would need 20 higher scores, which would probably take longer than the lifetime of the device. Then for the rest of the lifetime of the device, there will be a single line item that can't be sorted and shows empty fields.

I agree with you. However, as is easy to see, people are very focused on the interest/pursuit of high difficulty.
It will be interesting to see how the absence of the current highest difficulty level is received. I suspect there will be many questions/complaints.
Since most people are probably not very interested in the data displayed (except for difficulty /date), or are unable to interpret it, it would not be a great loss not to display it.

[seepv]

A Summary would be helpful:

The scoreboard difficulty distribution is:
0 T-Diff | 3 G-Diff and 17 M-Diff 

[Travetown]

A top 20 of all worker (swarm) would also be nice.

[0xf0xx0]

tested, my main concern is def the nvs though
could the scoreboard be updated in ram and written periodically?

[mutatrum]

tested, my main concern is def the nvs though could the scoreboard be updated in ram and written periodically?

After a week or two there's maybe one update a day or so, it's good for decades.