Migrate to new Session Structure

[tmack8001]

Instead of storing session files with metadata in the session filename and only allowing an array of data points in the JSON file itself (at this point should have just been CSV ).

Current File Structure (and would still need to be parsed correctly going forward for backwards compatibility)

example filename: 20210115_174247#30aea4c737ec#2c5a8b7ade5742c7bfd83330a78426b3#HopHeads_IPA_%231-3_With_First_Wart_&_Whirlpool#6.json

[
  { ... data t0 ... },
  { ... data t1 ... }
]

Proposed file structure:

filename: 2c5a8b7ade5742c7bfd83330a78426b3.json

{
  "id": "2c5a8b7ade5742c7bfd83330a78426b3",,
  "device": {
    "uid": "30aea4c737ec",
    "type": "ZSeries"
  },
  "startTime": "20210115_174247",
  "recipe": {
    "id": "1234",
    "name": "HopHeads_IPA_#1-3_With_First_Wart_&_Whirlpool"
  },
  "session": {
    "id": 6,
    "type": "Beer"
  },
  "dataLog": [
    { ... data t0 ... },
    { ... data t1 ... }
  ]
}

[tmack8001]

Having such a structure to our sessions will allow future use cases to allow more data to be added to the session after it is "archived" / "completed".

Say if we wanted to have the ability to link various machine sessions together we could do that by creating bi-directional links, or a new model for 'batch' or something where we link sessions to this new data type.

{
  "relatedSessions": [
    {
      "id": "20201207_200013",
      "startTime": "20201207_200013",
      "file": "/picobrew_pico/app/sessions/ferm/archive/20201227_200013#2c3ae834b45b.json",
      "device": {
        "type": "PicoFerm",
        "uid": "2c3ae834b45b"
      }
    },
    {
      "id": "20201207_200013",
      "startTime": "20201207_200013",
      "file": "/picobrew_pico/app/sessions/iSpindel/archive/20201227_200013#f12ae834b45b.json",
      "device": {
        "type": "iSpindel",
        "uid": "f12ae834b45b"
      }
    },
    {
      "id": "20201228_200013",
      "startTime": "20201228_200013",
      "file": "/picobrew_pico/app/sessions/still/active/20201228_200013#4b3ae834b45b.json",
      "device": {
        "type": "PicoStill",
        "uid": "4b3ae834b45b"
      }
    }
  ]
}

[Rudder2]

I really like the ability to link sessions! This has been wanted for a while. Looks like this update would add much wanted features in edition to fixing a file naming problem. Looks AWESOME!

[BuckoWA]

I’m in too! Like this approach.

[tmack8001]

I'm not looking at working on this right away, but as the thought has occurred to me how reformatting the session file structure could be leveraged for multiple features I started liking this approach more and more.

To do this right we should also change from just simply appending data to the file and instead upon each data point received serialize and deserialize the file contents and always write syntactically valid JSON we would have to in order to start adding meta data (links, session names, etc) during the active brew session vs afterwards.

[BuckoWA]

Started messing around with this is my spare time. Modified iSpindel to generate this type of session file. Dates are not stored properly, but wanted to see if I'm heading down the right path here. Here's a sample output file.

test.json.txt

[Intecpsp]

Question: why migrate to a different file structure instead of a local db?

[tmack8001]

DB is definitely not a recommended solution especially so for the RaspberryPi deployment environment unless you mount primary filesystem to say an external HD.

In theory the "most stable" deployment environment for a Pi setup would likely to mount the main sd filesystem as read only and store everything into S3 or some other "cloud based storage" and/or only local RAM storage.

[cgalpin]

I think perhaps he means something like RRD - the "database" behind rrdtool