Pull out path constants from imager into yml file

[aheschl1]

Paths to binaries were hardcoded in imager. They have been pulled out into a yml file. At build time, the file is parsed and imported as constants into the imager code.

Furthermore, the folder name for the output of the images is included in the config file, as well as the path to static files to be included.

Issue: #24

[aheschl1]

Approach taken is to expose the same constants as currently exist, though the build.rs file, which parses the yaml.

Introduces new dependency: rust-yaml

The other option is loading the file at runtime, performance isn't a concern here, but my opinion is that build time is a better option here because it is more aligned with the fact that the lists are constants, and should not be manipulated by logic.

[lasiotus]

Thanks a lot for the PR! I was actually imagining processing a config file in the main.rs, not in build.rs. Why do you think processing the config file in build.rs is better vs at runtime?

The benefits of having a "runtime" config (processed in main.rs) is that a single imager binary can build different images, based on different config files. So in our current setup, imager will be invoked (via Makefile) three times, once with full.yml, once with web.yml, and once with motor-fs.yml configs (so the config file describes input binaries and the file system to use). In the future I can imagine the same imager binary used to package a "test" image that will run tests "on boot".

[aheschl1]

Thanks for the comment - makes sense to me to do it at runtime since you are going for one build capable of generating multiple images. I had originally done it at build time as I thought of it more as easily configurable constants that won't change.

I've pulled it into runtime, and added more complex "syntax" for the config file. My goal here was to allow for defining
a) Name a set of configurations based on what static files they will contain
b) Include a list of binaries that will be used in these images
c) Allow for multiple different images, with configurable FS, using the same set of binaries/static files

Let me know if this is not what you had in mind:

image_sets: // root
  full:             // name in the static files folder
    binaries:
      - list of binary paths
    images: 
      - name: "motor-fs"  // image config 1, using the above binaries
        fs_partition_name: "fs_part.web.motor-fs"
        part3_fs: "motor-fs"
      - name: "motor.full.img" // image config 2, using the above binaries
        fs_partition_name: "fs_part.full"
        part3_fs: "srfs"
  web:  
    binaries:
      - "/bin/httpd"
      - "/sys/sys-init"
      - "/sys/sys-tty"
    images:
      - name: "motor.web.img"
        fs_partition_name: "fs_part.web"
        part3_fs: "flatfs"
paths:
  img_out_dir: "vm_images"
  static_files_dir: "img_files"

In this, an arbitrary number of images can be added and configured.

Some names should probably be changed so let me know what changes you'd have in mind. Maybe config structure as well if you don't like the format. For one, would you prefer part3_fs just to be 'fs_format' because partition 3 is the only one with a filesystem?

I have a couple questions. (I am very much a novice when it comes to OS topics)

1. How do srfs, flatfs, and motor-fs filesystems work, how do they differ?
2. What is on the part1 and part2 partitions?
3. Importantly: what steps would you recommend taking to assert the correctness of this PR? do you have any routines for validating build results?
4. Mostly importantly: What do you think of this setup?

Switching to a draft because I definetely have some cleanup to do. For example, I want to remove the types I defined, and think that it could be good to make the config have a bit more redundancy in exchange for clarity. For this, it could be good to make it such that every image requires relisting all binaries. Wanted to grab an opinion on this approach first though.

[lasiotus]

Thanks for the update!

I didn't look much into all the details - will do it later, when the changes suggested below are in place. For now I have two high-level comments:

• please make it so that one run of imager creates one image; so for three images three imager invocations with three different configuration files will be needed;
• please combine your two commits into a single one (I guess git rebase -i and then git push --force).

To answer your questions:

How do srfs, flatfs, and motor-fs filesystems work, how do they differ?

• flatfs is a simple read-only FS, similar to cpio used in Linux, just for serving static files over http(s);
• srfs is a fully functional FS; it is, however, rather slow; I implemented it quickly to let me work on other features of Motor OS;
• motor fs is a work-in-progress file system that will eventually replace srfs; I'm now doing it slowly and deliberately; it will definitely be better than srfs, and I hope it will be good enough for some time.

What is on the part1 and part2 partitions?

Partitions 1 and 2 are basically chained bootloaders, so that Qemu can load Motor OS from a bootable disk image. This is a technical detail related to booting an OS from a bootable disk, not really relevant to operating systems per se. You can read up on "chain loading", but this is somewhat tangential.

Importantly: what steps would you recommend taking to assert the correctness of this PR? do you have any routines for validating build results?

full-test.sh in vm_images dir will test the "full" image. If all tests pass, I'd say the full image works. To test the "web" image just boot it (replace "motor.full.img" with "motor.web.img" in the startup script). Motor FS is not yet ready, so don't bother with it; I'll test it manually.

Mostly importantly: What do you think of this setup?

As I mentioned above, get rid of ImageSet and have a config file describe a single image.

Thanks again for working on this!

[aheschl1]

Thanks @lasiotus for your answers!

I updated the configs to hold one image per file, and have modified the Makefile to generate all three existing configs.

In this PR I included a commit which adds an argument to run-qemu.sh so you can specify the image to run without modifying the run-qemu.sh file. For example ./run-qemu.sh --img=motor.web.img. The default img being motor.full.img. Let me know if you don't like this.

[lasiotus]

This looks close to what I had in mind, thank you! I'll test the PR later today.

I want to refactor it a bit, but it will be much easier/faster for me to do it myself rather than explain the refactor and go through another one or more rounds of reviews, so I think I'll prepare a refactor commit on top of this PR and then merge the PR and the commit immediately after. (Assuming the PR tests OK). So maybe another day or two.

Will this work for you? If so, maybe mark this PR as ready for review? It is currently a draft.

Thanks!