Add ability to generate starter compose/device/application services via CLI

[charles-knox-intel]

🚀 Feature Request

Description

As a developer trying to frequently create EdgeX-based solutions, I have found myself needing a capability that Angular CLI offers - the ability to generate a new project using the command line.

Describe the solution you'd like

For example:

# Angular's CLI generates a completely new project based off of a template using this command, and asks a few questions:
ng new
# ? What name would you like to use for the new workspace and initial project? test
# ? Would you like to add Angular routing? Yes
# ? Which stylesheet format would you like to use? CSS


# similarly, an EdgeX CLI generator capability would look like:
edgex new
# What would you like to name your project? [my-edgex-project]
# Which EdgeX version would you like to use for the compose file? [hanoi]
# Would you like a sample device service to be added? [y/n]
# Would you like a sample application service to be added? [y/n]

The edgex-cli program proceeds to generate a Hanoi compose with sample device/app services in place.

Additionally, the EdgeX CLI generator capability can continue to follow the Angular CLI by allowing for one-off service generation commands:

# angular: generates a "data" service in angular
ng g service data # shorthand of "ng generate service data"

# edgex: generates an application service
edgex g ds my-device-service-name # shorthand of "edgex generate device-service"
edgex generate app my-app-service-name # shorthand of "edgex generate application-service"

Acceptable permutations could be:

edgex g ds
edgex g dev
edgex g device-service
edgex generate ds
edgex generate dev
edgex generate device-service

edgex g as
edgex g app
edgex g app-service
edgex g application-service
edgex generate as
edgex generate app
edgex generate application-service
edgex generate app-service

I am indifferent on whether or not this feature should hook into any existing Git repositories in the users' local directory. It may be easier to just simply generate the files for the initial release of this capability, and have the user migrate them to their own repositories later. The time and confusion saved by having this feature makes a big difference - the Angular CLI adds an extremely valuable developer experience to their ecosystem, and I think that having this feature in EdgeX would make a difference too.

Describe alternatives you've considered

I've looked at the edgex-cli and it does not appear to have this capability. We can likely utilize the EdgeX examples repository and developer scripts repository to help bootstrap the development of this feature.

[difince]

@charles-knox-intel Interesting enhancement proposal. I will take a closer look at it in the beginning of the next year.

[charles-knox-intel]

Great! I look forward to hearing more. Thanks for reviewing my feature request.

[difince]

Happy New Year :)
@charles-knox-intel @lenny-intel
As I understand regarding device-service generation, I need to use as a base device-service repository - device-random-go, device-snmp-go ... depending on which we decide to use a basic example repository.

About Application Services - App-service-configurable looks perfectly fit the needs of this feature. (we will need to choose the default sample "-profile=< profile >" to be used, and update the Dockerfile accordingly )

And in addition I need to generate docker-compose file with the sample DS and APP-Service with all their dependent services

[lenny-goodell]

depending on which we decide to use a basic example repository.

IMHO, we should add template device and app services to examples. These would have better comments in the code directing the developer on what they need to add/change for their use case.

And in addition I need to generate docker-compose file with the sample DS and APP-Service with all their dependent services

IMO, the compose file is out of scope.

@charles-knox-intel , thoughts?

[charles-knox-intel]

As I understand regarding device-service generation, I need to use as a base device-service repository - device-random-go, device-snmp-go ... depending on which we decide to use a basic example repository.

@difince I think device-random-go is the most reasonable and simple starting point for developers trying to "just simply get a device service added to their project", which is the intent of my feature request. It sounds like we're on the same page, which is great!

About Application Services - App-service-configurable looks perfectly fit the needs of this feature. (we will need to choose the default sample "-profile=< profile >" to be used, and update the Dockerfile accordingly )

I see the appeal in using app service configurable. In the past, I have used the simple-filter-xml-http application service as a starting point. This is because app-service-configurable has extra "stuff" (I almost used the word "bloat" here, but it's definitely not) - all of its configuration.toml files probably doesn't simplify the developer experience for the new & confused or still-learning developers.

I often find that putting in the bare minimum to let developers (like me) code is the most effective way to increase adoption. Then, diligent developers will peruse the example repositories and bring in what they need.

But I do think that app service configurable is both minimal enough and extensible enough to be the default choice when running edgex generate as. Here's what I suggest:

• edgex generate as - adds app-service-configurable with a very simple profile selected by default, ideally one that processes data from the default device-random-go service
• edgex generate as-minimal and edgex generate app-service-minimal - adds a version of the simple-filter-xml-http application service that has extra comments indicating where to make your changes.

Something I want to point out that really highlights the spirit of this feature request:

https://github.com/edgexfoundry/app-service-configurable/blob/master/main.go#L40-L46

My opinion: This is the most important piece of code when it comes to DX. It's the decision point where developers will either use one of the preset pipelines, or they'll rip out this chunk of code and start adding their own code to do what they want. It's what I do every time. We should strive to add comments very clearly indicating that right here is where you get to decide what you want to do in your application service.

For the device-random-go device service it's a little different of course, maybe consider adding comments to e.g. https://github.com/edgexfoundry/device-random/blob/master/driver/randomdriver.go#L48 (assuming that's the right place to start).

And in addition I need to generate docker-compose file with the sample DS and APP-Service with all their dependent services

IMO, the compose file is out of scope.

@charles-knox-intel , thoughts?

I'm really hoping to get compose file functionality included as part of this feature request. There are two situations to consider when it comes to dealing with the compose file:

• Starting a brand new EdgeX project. In this case, it's a matter of copying the targeted compose file from the developer scripts repo directly to a local file. This is easy and I consider it very much needed as part of this feature request.
• Adding a new device/app service. This is a bit more involved, but all that needs to be done is:
  • Parse the existing compose file
  • Add the app/device service to the list of services in the compose file. Geneva/Fuji/Hanoi/etc all have slightly different service definitions in their compose files, so depending on the version the yaml object will vary. The user then should be advised to modify the service/compose file if they have customizations to make.

[lenny-goodell]

@difince , @charles-knox-intel , Note that app-service-configurable is NOT a good place to start for a custom app service. It uses the configurable pipeline that a custom App Service will not use. Again, best to create templates in edgex-examples that are specific for this, rather than attempt to use exist services not design as templates.

@charles-knox-intel , As for compose file, I see your point. The CLI could leverage off the work we did in the Compose Builder to generate a compose file.

[charles-knox-intel]

Again, best to create templates in edgex-examples that are specific for this, rather than attempt to use exist services not design as templates.

I agree completely. Curating a template is the best way to go forward. Hopefully it's not too much of a hassle to create a good template device + app service for this purpose :)

[tonyespy]

@charles-knox-intel @lenny-intel

While I do agree that the current Getting Started instructions for new Go device services is clunky and error prone, I'm not convinced the edgex-cli is the right place for initializing the files needed for a new device or application service. I think the current client command syntax is already a bit overloaded with too many top-level commands. Maybe if the client had implemented a per-service style command format (e.g. edgex-cli (m)etadata device list) as I'd previously suggested, I might be more inclined to agree with your suggestion. This approach would also lead to the client being dependent on the two SDKs. Finally, I also think that the primary purpose of the client is a replacement for the myriad of curl-based scripts many of us use to interact with EdgeX services.

I'll also point out usage of the term SDK is a misnomer that was carried over from the original Java Device SDK implementation (which is partially my fault). SDK stands for "software developer kit", whereas app-functions-sdk and device-sdk-go are both sets of Go packages which provide frameworks for creating new application and device services. One way to actually begin evolving them into true SDKs would be to add tools to help developers create new services. We could create a new SDK focused client(s) to handle what you suggest for both, although then we have the same problem we have with the client, namely how do we distribute the binaries? A simpler alternative would be to just leverage the existing Makefiles in both, and add new "init" targets. If we feel that user input (e.g. a user providing a name for the new service) is needed, then perhaps a shell-script approach would suffice.

Since this is a cross-cutting concern, I'd suggest that this get teed up for the architecture group to discuss. I think it also makes sense to discuss in both the application and device services working groups first.

[charles-knox-intel]

@tonyespy I appreciate your input on this feature request.

This approach would also lead to the client being dependent on the two SDKs.

In only as much as executing a git clone xyz (or similar) command is a dependency. Or at least that's how I envisioned it in my original request.

I don't want to over-simplify this though, if you know some other dependency between these EdgeX services here that I don't, I hope you can explain it for me.

I think the current client command syntax is already a bit overloaded with too many top-level commands.

When I came to this repository looking for this feature specifically, I didn't even notice all of the other capabilities, because I was looking only for it with tunnel vision. Other developers will likely do the same exact thing. It never hurts to re-evaluate or expand the purpose/usage/capabilities of this CLI if it enables faster adoption.

Finally, I also think that the primary purpose of the client is a replacement for the myriad of curl-based scripts many of us use to interact with EdgeX services.

Given that there is a growing trend of many other major software projects that have this capability that I've described... we should follow suit for the sake of smooth DX.

We want people to use EdgeX, right? The entrypoint into this framework should be the CLI, which is what other developers will be expecting. It's the way other major frameworks are trending, and it's a good pattern to follow.

We could create a new SDK focused client(s) to handle what you suggest for both, although then we have the same problem we have with the client, namely how do we distribute the binaries?

It sounds like you're suggesting we have two command line interfaces for EdgeX? I think that might just be more confusing. Let's keep it simple. People use a command line interface for its ability to get a range of things done in a lot of different ways, or just one or two frequent things done. It's a power tool for the user. Everyone will soon be able to use the CLI to bootstrap their projects, as well as use the curl commands you've described above. I don't see a problem with that.

I think I'm really just simply asking for EdgeX to have the ability to copy/paste some code from GitHub and update a compose file for me, because most developers probably don't have the time or desire to dig through all of the example repositories and different branches and release versions and permutations of compose services.

My feature request effectively adds a simple, singular source of truth for what it looks like to start a new EdgeX project and add a new blank device/app service for each version of EdgeX. Ideally all it takes is a few templates. No dependencies, no architecture overhauls. No over-complication... I hope! I hope that helps to communicate the essence and spirit of this feature request - I really think EdgeX will benefit from it.

[difince]

I like the enhancement suggestion and I see it as a handy users feature.
But agree to discuss it with a broader audience of edgex contributors.
I still think that CLI top level commands are not numerous enough to justify entering the service name (thus force the user to have a knowledge of the internal architecture of the project - knowing which microservice is responsible for which objects like - watchers, devices and etc. )
But no matter if we decide to rewrite the current structure of the CLI, I see this feature request as useful and nice to have.

[tonyespy]

@charles-knox-intel Thanks for your response. Again, my feedback wasn't that your idea isn't without merit, it's that I don't think edgex-cli is the right place for this feature. Now that's just my opinion, so as I suggested, we should first bring this discussion to the application and devices working groups as they are the owners of the SDKs (and the associated developer experiences). If there's consensus from the two groups that this is a good idea, then we can ask the client team to work on a design and prioritize its implementation.

[jpwhitemn]

Per Device Service WG meeting 1/18 - suggestion by Tony to add make init on the SDKs to create new service.
Under the covers - have sample code in SDK vs pull template files and create service from the templates
Proceed with discussion in app services group and then escalate discussion to monthly architect's meeting.