add - build script to generate improved documentation and config file for project - etcd, website

[webdawg]

Hello All!

I am currently implementing a new etcd stack in archlinux, and in these efforts I am using systemd config based environmental variables. In doing this the aur package in archlinux is using system.d style of configuration.

There is a /usr/lib/systemd/system/etcd.service.d/10-EnvironmentFile.conf that has a reference to the EnvironmentFile=-/etc/conf.d/etcd for the unit and a nice (but needs updated) ENV_* configuration file in the package that is /etc/conf.d/etcd

Currently I am undertaking a reformat and update of the /etc/conf.d/etcd for my work. I would like to write, eventually, a script to generate my configuration (/etc/conf.d/etcd) file based on the etcd --help output. I see there was a removal of ENV_ variables after etcd 3.4, and I can understand why. Who would want to maintain all that?

I rose this in #sig-etcd slack channel recently (https://kubernetes.slack.com/archives/C3HD8ARJ5/p1727313996375469?thread_ts=1727313558.679039&cid=C3HD8ARJ5), and I am submitting an issue here today to see if anyone thinks it may be wise to create a build script based on the output of etcd --help or similar. The build script would serve two purposes I think. The first to give folks a great .conf / ENV_ style configuration for their use, and then also generate pieces or parts of the actual https://etcd.io/docs/v3.5/op-guide/configuration/ page

How would I go about doing this? I just need the highlevel. Write script in bla (python?), submit to website repo, target X file, and Y file using these paths. Or even something like see website build script here and figure it out.

Please feel free to comment with guidance or advice.

Here is some sample work for the ENV_*/.conf file I am building right now. I am open for suggestions. For me it is a new take on the eggdrop config style work seen here: https://github.com/eggheads/eggdrop/blob/develop/eggdrop.conf

Here is a preview of what I am working on:

################################################################################
############################## Background ######################################
################################################################################


# Use this file to pull these env into a systemd service.  For instance:
# /usr/lib/systemd/system/etcd.service.d/10-EnvironmentFile.conf
#
# Contents:
# [Service]
# EnvironmentFile=-/etc/conf.d/etcd

# FYI
# Environment variables: every flag has a corresponding environment variable that has
# the same name but is prefixed withETCD_ and formatted in all caps and snake case. For
# example, --some-flag would be ETCD_SOME_FLAG.




################################################################################
############################## Member flags ####################################
################################################################################


#### --name
#
# ETCD_NAME="default"
#
# Human_readable name for this member.
#
# default: "default"
#
# This value is referenced as this node’s own entries listed in the
# --initial-cluster flag (e.g., default=http://localhost:2380). This needs to
# match the key used in the flag if using static bootstrapping. When using
# discovery, each member must have a unique name. Hostname or machine-id can
# be a good choice.

[jmhbnz]

Thanks for raising this @webdawg. As a maintainer of this repo in a broad sense I would be interested in some kind of automation to assist with the maintenance of these pages:

• https://github.com/etcd-io/website/blob/main/content/en/docs/v3.5/op-guide/configuration.md
• https://github.com/etcd-io/website/blob/main/content/en/docs/v3.6/op-guide/configuration.md

You're right - from 3.5 onwards we switched to a new layout for those pages which removed the explicit listing of the environment variables and instead just explained how the environment variables for each flag could be derived.

I think a challenge trying to implement fully automated generation of the pages will be that for some configuration flags I suspect (but haven't verified) that some configuration flags will have additional description flavour or notes added to them over and above what is currently output from etcd --help.

Perhaps a starting point for some partial automation we could begin with is a script we can run in CI that would:

1. compare the flags present in etcd --help for that minor version, and verify that for each flag is actually present in the website markdown file.
2. manipulate the file to append an environment variable value above/underneath the command line parameter for each flag.

That would address two problems we currently have where sometimes we are missing flags in the website docs which are actually present and available in the binary, and also bringing back environment variable values in a way that doesn't cause maintainer burden.

We could start with a simple script that could be run as a make recipe manually. Then if it's working nicely we could add a prow job in kubernetes/test-infra to run it as presubmit or on a schedule.

@ivanvc, @spzala what do you think about this?

[ivanvc]

I like the idea. 3.4's layout is more explicit/verbose, but it is also easier for end-users to reference what configuration option they are using.

We could start with a simple script that could be run as a make recipe manually. Then if it's working nicely we could add a prow job in kubernetes/test-infra to run it as presubmit or on a schedule.

Given our constraints with running automation jobs, I think the only option we have now is to run manually. We could run it along with the nightly job that updates the current version (as I would expect to run it with the released binaries).

[spzala]

Thanks @webdawg !! Agree with @jmhbnz and @ivanvc that some sort of automation, would be good. But, as it's mentioned in #491, "The Configuration pages for v3.5 and v.34 combined get ~2% traffic relative to the overall site: my guess is that most users simply run etcd -h." so considering the usage, I think we should consider if it's (automation/script) easier to produce and maintain. Also, for a reference, a sample config file is provided here, https://github.com/etcd-io/etcd/blob/main/etcd.conf.yml.sample which we try to keep up to date. Thanks! cc @ahrtr

[spzala]

I am not sure (but I will check unless someone beat me verifying it :)) if the Configuration doc pages provides a link to the https://github.com/etcd-io/etcd/blob/main/etcd.conf.yml.sample but if not, I think it's a good idea to do so.

[jmhbnz]

I don't think it's wise to use percentage traffic as a measure of a pages importance, because if we did that every page in the docs site would be irrelevant compared to the home page.

If you actually look at the page hits both our v3.4 and v3.5 configuration pages are actually in the top 10 most popular for the site.

Additionally they get significantly higher viewing times than any other pages in our top 10.

Below are the figures for last 12 months:

[spzala]

I was just referring to the comment from cncf doc staff (PR owner) for reducing some of the content in his PR for easier maintenance as he has mentioned there.
Traffic sure can vary and I am not sure about if it shows unique visitors. I guess we probably can safely ignore worry about it but if we need to know more, the cncf doc staff who help multiple projects can help get better pic.

[webdawg]

For me, for now it may be easier to write a bash / sh script. Would this be acceptable?

[ivanvc]

For me, for now it may be easier to write a bash / sh script. Would this be acceptable?

Writing a shell script and calling it from a Makefile target (like markdown-diff-lint) works (and is actually expected), as we can then call that make target from a prow job.

[webdawg]

Attached is what I have come up with so far. Please comment.

2024.10.20.output.txt
config_build.sh.txt

I think it would be easy to pipe etcd --help into this.

[webdawg]

I am continuing work on this - I have to continue to hone the script

[ivanvc]

Hi, @webdawg, I'd suggest opening a draft pull request so we can spearhead reviewing in place, rather than downloading the script. Thanks.