Swagger/OpenAPI to MST generator for REST-like APIs

[OnkelTem]

Let me describe a pretty common situation in a regular JS project with a REST-like API:

• Backend provides an API
• The API comes as a Swagger/OpenAPI spec and is the source of truth for the frontend
• This spec is processed to generate DTOs (data transfer objects) and methods in TypeScript.

One of simplest tools for that is swagger-typescript-api
It generates API types and methods. The only real JavaScript part is methods.

Other people use more sophisticated generators from OpenAPITools set, e.g.:

In addition to types these generators usually produce some JS for data format conversions.

Now back to types.

MST models cannot use TypeScript DTOs as generics.¹

So we cannot create a model for an existing type. We can only create an "unattended" model and then claim its type to be the right one (still - not without tricks, e.g.: #2238)

But here is another approach: we can make a step backwards at the generator level and generate not the TS defs, but instead - MST JS models and get typings with typeof <model>.infer.

Yeah, I know, I am a great dreamer haha. Seriously, did you folks run across something like this?

Footnotes

1. After some investigation I can add: as well as Zod, Yup and presumably many other validators, in fact I found only one exception - ArkType 1:1 validator and that part is still a matter of future changes ↩

[tm-3]

If you're just looking for client side validation to and from the API, zod and others are much better tools. If you're looking for a way to generate a state store from the swagger API, you could use the volatile state within a MST model, or just a regular mobx observable depending on your needs. MST is opinionated and likes to have its structure known from the start. Mobx observable/auto observables work well with DTO's and plain JS objects, maps, arrays, etc. and you can create a more generic set of actions to update the state.

Where I think MST shines is in the middleware so even if you wrap your state store inside MST models, you can still use MST actions to interact with that state. Then you have all kinds of options with the dependency injection examples, logging, validation, etc. So if you think through your needs a bit, you could create a library of generic actions that can be composed and added to any model. Middleware is basically a pipe function that receives not just the state on the node it was called, but of the entire tree. It's both fun and dangerous.

The parts you'll lose from MST are the JSON patches and some snapshotting capabilities and other events because the state for the app is not in the MST model itself. You could probably get tricky with frozen types and just have them hydrate your app state without necessarily making any API calls. The problem with frozen types is that the entire value updates, so if you have a client component watching that frozen type you will probably render more than you want to.

The biggest thing to keep in mind is that MST is an opinionated structure for storing observable state data. It works best when you define your data structures. While it is typed and you can do some transforms to and from snapshots, I lean towards zod for this type of functionality, or just write my own mappers/validators. You can get somewhat creative by creating functions to generate models at runtime, but you'll lose much of the developer experience this way and probably run into some funky typescript challenges as well.

I may have misunderstood your thoughts entirely and if I did let me know and maybe I have some other thoughts. Everything I just mentioned may be overkill for most things, but maybe you'll find some value and come up with basic patterns that are useful to you.

hth

Here's a very sloppy example of some of what I was talking about:

import {
  addMiddleware,
  flow,
  getEnv,
  type IAnyModelType,
  type IAnyStateTreeNode,
  t,
} from 'mobx-state-tree';
import { type IObservable, observable, type ObservableMap } from 'mobx';

type DtoGeneratedFromSwagger<A = any> = A;

export const SomeClientSideState = t.model('SomeClientSideState', {
  clientSideProps: t.string,
  currentState: t.map(t.frozen()),
}).extend(
  (s) => {
    const appState: Record<string, DtoGeneratedFromSwagger> = observable.map();
    const someFunctionToGetApiData = async () => await {};
    const someFunctionToUpdateApiData = async (params: any) => await {};

    return {
      actions: {
        setCurrentState(key: string, value: any) {
          s.currentState.set(key, value);
        },
        fromApi: flow(function* fromApi() {
          // do some async stuff
          const data = yield someFunctionToGetApiData();
          // or using dependency injection
          // const data = yield getEnv<ApiAdapter>(s).fromApi();
          appState.set('someKey', data);
        }),
        toApi: flow(function* toApi(key: string, value: any) {
          // do some async stuff
          yield someFunctionToUpdateApiData({ key, value }).then(() => {
            appState.set(key, value);
          }).catch((error) => {
            // handle error
          });
        }),
      },
      views: {
        get someViewImportantToYourClient() {
          return appState.get('data');
        },
      },
      state: {
        /**
         * You can return all or part of the state here. Be sure not to mutate it without using actions.
         * Maps get a little complicated as well since they return references. If you do something like const item = appState.get('item') you can mutate item directly like item['prop'] = 'value'. Be responsible!! :)
         * You could use autorun and other mobx utilities with this object as needed.
         */
        appState,
      },
    };
  },
);

const someMiddleware = (
  state: IAnyStateTreeNode,
  validators?: (args: any) => boolean,
) =>
  addMiddleware(state, (call, next) => {
    call.name === 'toApi' && console.log('toApi called');

    validators && validators(call.args)
      ? console.log('valid')
      : console.log('invalid');

    return next(call);
  });

// API Adapter

const someApiAdapter = () => {
  return {
    toApi: (key: string, value: any) => {},
    fromApi: () => {},
  };
};

type ApiAdapter = ReturnType<typeof someApiAdapter>;

// Setup
const createStateNode = (apiAdapter: ApiAdapter) =>
  SomeClientSideState.create({
    clientSideProps: 'some value',
  }, { apiAdapter });

// Usage

const clientState = createStateNode(someApiAdapter());

someMiddleware(clientState);
clientState.fromApi();
clientState.someViewImportantToYourClient;
clientState.toApi('key', { key: 'data', value: 'some value' });

[OnkelTem]

That's a little too much for me right now, Tom, as I'm still at the stage of getting acquainted with MST. Anyway many thanks for sharing!

What I was talking about is the first step in getting MST adopted on a project and particularly - registering DTOs as MST models.

Say we had this in our API:

type User = {
  id: number,
  name: string,
}

How we'd create a model from it? Someone might expect that it can be done like this:

const UserModel = t.model<User>({
  id: t.number,
  name: t.string,
})

but it's not how things work, as what model generics accept are not the DTO types. And without one, e.g. without this <User> generic:

• IntelliSense doesn't work
  • awful editor experience
• As model is not connected with the type
  • changes to API (which happens every other day) will be occasionally breaking project
    • and what is worse - not in design-time, but it run-time
      • so we'd have to introduce a sheepload of tests, just because of this.

So we basically lose the key feature of TypeScript - having things settled down in editor, in design-time.

All this means, that it's not how models should be used on a TypeScript project. We need to make models the source of truth, not the other way around:

const UserModel = t.model({
  id: t.number,
  name: t.string,
})

type User = Instance<typeof UserModel>

And UserModel should be generated from API spec.

[OnkelTem]

Speaking about validation (unrelated to the task of modeling with MST), with Zod we'd get the same problem, as this:

const UserSchema = z.object<User>({
  id: z.number(),
  name: z.string(),
});

won't work too. Instead, we'd also have to generate Zod schemas from the API.

[coolsoftwaretyler]

@OnkelTem - I want to really commend you for jumping in with this. I think it's an interesting idea, and I get where you're coming from.

I still need to digest the actual proposal here before I can respond to the technical parts of it. But I can provide some project-level guidance to you.

This is a pretty different use case than our typical users ask for, and we already have an informal initiative to reduce our API surface area. At the same time, we have some high priority items to take care of in the library, and limited time to do it in (this is how it goes in open source, haha).

That said, I always prioritize community contributions, either to the repo, or when people produce projects that are complementary to MST.

My recommendation for you, if you really want to see this in MST, is to write a separate tool to do this kind of Swagger/OpenAPI to MST codegen. It doesn't need to be production ready. Just a POC. Then drop a link to it in this discussion so we can dig in.

You certainly don't have to go it alone. I would love to help with something like that, although I can only commit to providing guidance, not actual hands-on-keyboard time. But if you wanted to get started, I could give you a tour of the MST codebase, a rundown on the design philosophy, and details about our roadmap, along with feedback on your code.

If you (and/or others) did something like that, I would do everything in my power to make sure you have what you need to complete it. I also can't guarantee we will merge it into MST proper, since that has far reaching implications for our users. But I can certainly at least make sure it ends up in the official documentation, and that we collaborate across the codebases to maintain compatibility. If you needed a small code change to hook into MST somewhere, we'd be quite amenable to that. And if things go really well with the separate project, it's very likely we could find a path forward to merge it.

This is the process I'm taking for my own project, to make MST compatible with the React compiler. My plan is to develop it separately from MST, and find natural integration points where they make sense.

I'll continue to think about this proposal, and I encourage people to discuss it further. But nothing beats a working prototype!

[tm-3]

I'll work on some more of the details and write something up a bit more formal. I'm working through it now on a project so I'll share what I find.

[coolsoftwaretyler]

Awesome thank you. A PR against the recipes docs would be ideal, but if you just drop me a markdown snippet I can add it on my end as well. Happy to edit anything you write as needed.

[OnkelTem]

I didn't think this would work and I've tried something like this before and failed. However, It looks like if you start with a zod schema you can create MST models from it.

Sorry, I got lost somewhere at the beginning of the explanation. Why build such a chain as Zod-to-Mst in the first place, what task is being solved?

[tm-3]

This approach described above shows how to generate MST models dynamically based on a schema. It's not a complete solution or pattern, and that's what I'm working out as I work through other projects. The problem that will become painfully apparent with runtime typing and JavaScript is that each library handles this a slightly different way. So if you're saying you want to start with an OpenAPI schema document you're first starting with a dialect of a JSON schema. Now you want to create models and functions from that schema? ok. there are tools for that, but then you want to use other tools that were designed for different purposes? ok...well here's one way.

This was just an example as a place to start. You're looking for a single source of truth from what I gather. Zod can do that. But if you start with an OpenAPI schema, you can generate different models using different libraries. You could start with a JSON schema as well. The point here is that in order to generate a model using MST, you typically need to explicitly define the shape of the object along with the MST runtime type that it will use:

const MstModel = t.model('MstModel').props({
  id: t.identifier,
  name: t.string,
})

By using the Zod schema, you can forgo explicitly defining those MST Model Types and MST will infer the property names along with the MST runtime types. You still get all of the developer experience if you start with the Zod schema. Additionally, that Zod schema is now a part of your dynamically created MST model which can be easily used to validate inputs and outputs, run transformations using Zod functionality, or even work as a factory function for creating instances of the model type if you get creative enough. I'm sure the same could be done using some OpenAPI parser that generates the shape of an object from the OpenAPI schema but I simply didn't have time to figure that out as I rarely work with OpenAPI. The benefit with zod is the ability to infer a typescript type from a zod schema. This is where you get the developer experience and intellisense from generics.

You can also do this from a JavaScript Object:

function createMstModelFromJsObject<T extends Record<string, any>>(
  obj: T,
) {
  return t.model<T>().props({});
}

const someObj = {
  id: 'idvalue1',
  name: 'name',
  distance: 334243,
}
const NewMstModel = createMstModelFromJsObect(someObj);
const instance = NewMstModel.create();

This all comes down to what your actual goals are for MST and state management in general. MST is by definition opinionated. Doing things like this dynamically goes against the spirit of MST. I'm pretty sure that all of the properties generated in these models this way are in volatile state as opposed to explicitly defined. Not a problem here as they are all still observable, but it helps to read up on the difference and their use cases.

I personally have never worked on anything that had a consistent data model from service to client to third party API to persistence to downstream consumer. Not for lack of trying, but after a while it seems wasteful. Each portion of a solution has a different problem to solve and should be able to do that regardless of the data structures it receives from APIs. I typically try to push all IO out to the edges so that even when that changes, all I need to do is change a reducer function, and often I don't even have to do that.

[OnkelTem]

Hey folks! I finally came up with a plan on how to do this.

Could you please take a look?
OpenAPITools/openapi-generator#20935

Thanks!