How to use glaze to create a polymorphic object parser?

[KROIA]

Hi there

Thank you for your library, I'm checkig if I can use this library in a project.
I have specific requirements it has to meet to be able to fit for my project.
The idea of parsing structs direct is very cool and I have nerver seen that before in c++.
In my case i can't utilize this feature because I have a different approach. (If there is any way I can do that with structs too, I'm open to learn)

My approach is the following:
I have a Serializable object interface that has two methodes:
"load" and "save".

Custom objects can be derived from that Serializable interface.
The derived objects have to implement the load and save funtions.

A object handler takes all pointers to Serializable objects and parses them to a file.
The object handler also can load the saved json back to the specific object instances.

What is the problem now?
[GREAT] I can save the objects correctly.
[BAD] I can't load them back.

How can I parse this structure back to my JsonArray and JsonObject types to be able to give the JsonObject's to my Serializable instances for loading?

Is there a better way to achieve the goal of saving and loading custom objects in a polymorphic structure?

Can I somhow create a custom data struct inside every derived class and parse that struct?
I don't want to return a string in the save function which would represent the parsed struct string because I first want to collect the
json data from every derived object instance and save them after I have collected all the data.

Optimum solution would be if I can give the save function a type in which i can store the custom struct from that derived object.
This is not possible because the Serializable interface does not know anything about the custom struct from the derived classes.
The same custom struct could be used to load back the data.

Thanks for any help and tips.

Here is my test code:

#include "glaze/glaze.hpp"

#include <iostream>
#include <unordered_map>
#include <variant>
#include <vector>
#include <string>


using std::vector;
class JsonArray;
class JsonObject;

// Class which holds a list of possible Json value types
class JsonArray: public 
    std::vector<
        std::variant<
            std::monostate, 
            std::string, 
            int, 
            double, 
            bool, 
            JsonObject, 
            JsonArray, 
            vector<JsonObject>, 
            vector<JsonArray>,
            vector<std::string>,
            vector<int>, 
            vector<double>, 
            vector<bool>
        >
    >
{
public:

};


class JsonObject : public 
    std::unordered_map<
        std::string,            // key of the value
        std::variant<           // value can be any type in the list
            std::monostate, 
            std::string, 
            int, 
            double, 
            bool, 
            JsonObject, 
            JsonArray, 
            vector<std::string>,
            vector<int>, 
            vector<double>, 
            vector<bool>, 
            vector<JsonObject>, 
            vector<JsonArray>
        >
    >
{
public:

};

// Interface for serializable objects
class Serializable
{
public:
    
    // Function gets called when the data of the object gets loaded back
    virtual bool load(const JsonObject& obj) = 0;

    // Function gets called when a object gets saved to json
    virtual bool save(JsonObject& obj) const = 0;
};

// Implement dummy Obj1 
class Obj1: public Serializable
{
public:
    Obj1(int val, const std::string &str)
        : value1(val), text1(str)
    {}

    bool load(const JsonObject& obj) override
    {
        // Get the values back from the loaded JsonObject 
        value1 = std::get<int>(obj.at("value"));
        text1 = std::get<std::string>(obj.at("text"));
        return true;
    }

    bool save(JsonObject& obj) const override
    {
        // Saves the data to the json structure
        obj["value"] = value1;
        obj["text"] = text1;

        std::vector<int> list{
            value1,
            value1 + 1,
            value1 + 2
        };
        obj["list"] = list;
        return true;
    }
    int value1;
    std::string text1;
};
class Obj2 : public Serializable
{
public:
    Obj2(int val, const std::string& str, int val2, const std::string &str2)
        : value1(val), text1(str), obj1(val2,str2)
    {}

    bool load(const JsonObject& obj) override
    {
        value1 = std::get<int>(obj.at("value"));
        text1 = std::get<std::string>(obj.at("text"));

        obj1.load(std::get<JsonObject>(obj.at("obj1")));
        return true;
    }
    bool save(JsonObject& obj) const override
    {
        obj["value"] = (value1);
        obj["text"] = (text1);

        // Save nasted object
        JsonObject buff; // Create a json container
        obj1.save(buff); // save the nasted object
        obj["obj1"] = buff; // Add the data from the nasted object to this json structure
        return true;
    }

    int value1;
    std::string text1;
    Obj1 obj1;
};

// Serializes all objects
std::string serialized(const std::vector< Serializable*>& objs);

// Loads all objects back
// For simplicity the objs list must contain the same amount and objects to match the 
// json data
bool deserialize(const std::vector< Serializable*>& objs, const std::string &buffer);

int main()
{
    // Create multiple test objects
    std::vector< Serializable*> objects1{
        new Obj1(1, "text 1"),
        new Obj1(2, "text 2"),
        new Obj2(3, "text 3", 10, "1000"),
        new Obj1(4, "text 4"),
    };

    // Save the objects
    std::string data = serialized(objects1);

    // Prints the correct parsed structure
    std::cout << "serialized 1:\n" << data;
    
    // Create the same objects as before but with different data to be able to 
    // verify that loading was successful
    std::vector< Serializable*> objects2{
        new Obj1(0, ""),
        new Obj1(0, ""),
        new Obj2(0, "", 0, ""),
        new Obj1(0, ""),
    };
    // Try to load the objects back from the created string
    if (deserialize(objects2, data)) // is true if loading was successful
    {
        // Save the loaded objects back to check for any difference
        // expected(data2 == data)
        std::string data2 = serialized(objects2);
        std::cout << "serialized 2:\n" << data2;
    }
    return 0;
}

std::string serialized(const std::vector< Serializable*>& objs)
{
    JsonArray jsonList;
    jsonList.reserve(objs.size());

    for (auto& obj : objs)
    {
        // collect all json object structures
        JsonObject data;
        obj->save(data);
        jsonList.push_back((data));
    }
    // parse the array to a string
    std::string buffer;
    glz::write< glz::opts{ .prettify = true }>(jsonList, buffer);
    return buffer;
}

bool deserialize(const std::vector<Serializable*>& objs, const std::string& buffer)
{
    JsonArray readed;
    // Try to load back to an array
    auto err = glz::read_json(readed, buffer);

    // Check the loaded object count 
    if (readed.size() != objs.size())
    {
        std::cout << "\nsize mismatch: "<<readed.size() << " != " << objs.size() << "\n";
        return false;
    }

    bool success = true;
    for (size_t i = 0; i < objs.size(); ++i)
    {
        // Load each object back
        success &= objs[i]->load(std::get<JsonObject>(readed[i]));
    }
    return success;
}

The output of the console looks like this:

serialized 1:
[
   {
      "value": 1,
      "text": "text 1",
      "list": [
         1,
         2,
         3
      ]
   },
   {
      "value": 2,
      "text": "text 2",
      "list": [
         2,
         3,
         4
      ]
   },
   {
      "value": 3,
      "text": "text 3",
      "obj1": {
         "value": 10,
         "text": "1000",
         "list": [
            10,
            11,
            12
         ]
      }
   },
   {
      "value": 4,
      "text": "text 4",
      "list": [
         4,
         5,
         6
      ]
   }
]
size mismatch: 1 != 4

[justend29]

looking into this

[justend29]

Okay, I'm not going to duplicate our efforts, then. I'll just add some notes:

1. the inherited load & store member functions of the Serializable interface are not what actually (de)serialize data, they merely transfer data to/from the JsonObject. It's JsonObject & JsonArray that are (de)serialized to/from JSON with glaze. This means that glaze is always interacting with the same two types, JsonObject and JsonArray, meaning children of the Serializable do not have to offer structs or anything for glaze - they don't touch glaze.

2. These two types have a fundamental flaw in that there's no unique mapping between a JSON type and an alternative in the
   variants. For example, the JSON number could be a double or an int. This non-unique mapping can be verified via:

   static_assert(!glz::detail::variant_is_auto_deducible<typename JsonArray::value_type>());

   Furthermore, there's no way for glaze to know which variant alternative to parse for. It will not "try" each one, it will instead
   parse for whatever the input variant holds. In this case, the parsed result is default constructed, thereby holding
   std::monostate, meaning glaze parses the input JSON for a single std::monostate.

The entire issue can be boiled down to the following, without any consideration of load/store.

    JsonArray parsed{};
    auto err = glz::read_json(parsed, input_json); 

To rectify this, use glaze's best feature: variant tagging. Essentially, you serialize an additional field that identifies the serialized type such that parsing can parse for a specific alternative; the one identified by the tag.

An alternative, which I've used in the past, is providing a custom from_json specialization for your types that will try each alternative. Note, however, that the first alternative that succeeds parsing will be what's read, which may not be what you intended. For example, 1 will be parsed as an int not a double since it comes first in the alternatives.

Onto you, @stephenberry!

[stephenberry]

We can actually deduce between the two objects because Obj2 has the unique key "obj1"

[justend29]

But, isn't the example serializing a JsonObject, not an Obj or Obj2?
In order to maintain the inherited structure in the example, doesn't that mean a custom deserialization would have to be implemented for the JsonObject with knowledge of that obj1 member in Obj2 in order to differentiate?

[stephenberry]

Correct, hence I suggest throwing out all the inheritance bloat and using the code I provided in the longer answer.

[KROIA]

@justend29
Yes you're right, glaze does only interact with the JsonObject and JsonArray.

[stephenberry]

Hi @KROIA,

If you use a std::variant, you'll eliminate most of your code.

Note that right now in your example you leak memory for both not calling delete and not having a virtual destructor.

Here is how you can write your objects:

struct Obj1
{
    int value;
    std::string text;
};

struct Obj2
{
    int value;
    std::string text;
    Obj1 obj1;
};

Now, we need to add some metadata to Obj1 because we need to customize the serialization to add the "list" element:

template <>
struct glz::meta<Obj1>
{
   using T = Obj1;
   static constexpr auto list_write = [](T& obj1) {
      const auto& value = obj1.value;
      return std::vector<int>{
                  value,
                  value + 1,
                  value + 2
              };
   };
   static constexpr auto value = object(&T::value, &T::text, "list", glz::custom<skip{}, list_write>);
};

Now we can serialize/deserialize as you like using glaze functions:

using Serializable = std::variant<Obj1, Obj2>;
std::vector<Serializable> objects{Obj1{1, "text 1"},
   Obj1{2, "text 2"},
   Obj2{3, "text 3", 10, "1000"},
   Obj1{4, "text 4"},};

constexpr auto prettify = glz::opts{.prettify = true};

std::string data = glz::write<prettify>(objects); // writing
const auto is_error = glz::read_json(objects, data); // reading

Here is my full test suite that I added to the unit tests:

suite custom_object_variant_test = [] {
   "custom_object_variant"_test = [] {
      using Serializable = std::variant<Obj1, Obj2>;
      std::vector<Serializable> objects{Obj1{1, "text 1"},
         Obj1{2, "text 2"},
         Obj2{3, "text 3", 10, "1000"},
         Obj1{4, "text 4"},};
      
      constexpr auto prettify = glz::opts{.prettify = true};
      
      std::string data = glz::write<prettify>(objects);
      
      expect(data == R"([
   {
      "value": 1,
      "text": "text 1",
      "list": [
         1,
         2,
         3
      ]
   },
   {
      "value": 2,
      "text": "text 2",
      "list": [
         2,
         3,
         4
      ]
   },
   {
      "value": 3,
      "text": "text 3",
      "obj1": {
         "value": 10,
         "text": "1000",
         "list": [
            10,
            11,
            12
         ]
      }
   },
   {
      "value": 4,
      "text": "text 4",
      "list": [
         4,
         5,
         6
      ]
   }
])");
      
      objects.clear();
      
      expect(!glz::read_json(objects, data));
      
      expect(data == glz::write<prettify>(objects));
   };
};

[KROIA]

correct.
I need the inheritance.
My code example is only a lightwaigth example on how i want to use glaze in my final project.
My final project is a library that manages multiple Serializable objects as a database.
save and load is the interface to the filesystem.
I already have this library and it works. The main flaw is the speed which is needed by QJson, which is used to serialize and deserialize my objects.

I try'd to write my own json object structure and serializer/deserializer. This also works and is faster as the QT library.
But QT is faster in reading a string in to a QJsonObject structure than my Json parser is.
My implementation is only faster because it does not take as long as QT to call the load function on all objects to grab the parsed data.

I can switch between QT and my own Json stuff in that library using preprocessor defines. This way I can mesure the performance of both pretty well.
I'm not an expert in parsing strings and don't know how to optimize my implementation further.
I know that QT is slow in many things, so I wan't to have a faster alternative. Therefore I thougth I will test glaze.

I hope this gives more understanding on how I wan't to use this library.

Thank you both for your fast support

[stephenberry]

@KROIA, If you don't want to use std::variant you could also use a std::any with an any visitor pattern.

[KROIA]

I'm fine with using std::variant.
In my json parser I also use std::variant.
The way I decide what type I'm reding (int or double problem) is, that i check the parsing value string if it has a decimal or exponent symbol in it. if it has it is intepreted as a double, otherwise it is a int.
solved that problem it shuld be possible to parse the whole file without saving an additional parameter in the file that says what type the reading parameter will be.

[stephenberry]

@KROIA, you can also look at the glz::json_t generic type, which acts like your JsonArray and JsonObject. See generic-json

This comes with significant overhead, but will probably still be faster than QT's JSON, which has been benchmarked and is even slower than nlohmann JSON.

[KROIA]

@stephenberry

Thank you. After trying around I managed to get it working now.
That is exactly what I'm looking for.

Now I will try to implement that in my main project to see how much performance gain it will bring.
Thank you both for your effort and have a nice day.

For those who need to solve a similar problem, here is my final code:

#include "glaze/glaze.hpp"

#include <iostream>
#include <unordered_map>
#include <variant>
#include <vector>
#include <string>


using JsonValue = glz::json_t;
using JsonObject = glz::json_t::object_t;
using JsonArray = glz::json_t::array_t;

// Interface for serializable objects
class Serializable
{
public:
    
    // Function gets called when the data of the object gets loaded back
    virtual bool load(const JsonObject& obj) = 0;

    // Function gets called when a object gets saved to json
    virtual bool save(JsonObject& obj) const = 0;
};

// Implement dummy Obj1 
class Obj1: public Serializable
{
public:
    Obj1(int val, const std::string &str)
        : value1(val), text1(str)
    {}

    bool load(const JsonObject& obj) override
    {
        // Get the values back from the loaded JsonObject 
        value1 = obj.at("value").get<double>();
        text1 = obj.at("text").get<std::string>();
        return true;
    }

    bool save(JsonObject& obj) const override
    {
        // Saves the data to the json structure
        obj["value"] = value1;
        obj["text"] = text1;

        JsonArray list{
            value1,
            value1 + 1,
            value1 + 2
        };
        obj["list"] = list;
        return true;
    }
    int value1;
    std::string text1;
};
class Obj2 : public Serializable
{
public:
    Obj2(int val, const std::string& str, int val2, const std::string &str2)
        : value1(val), text1(str), obj1(val2,str2)
    {}

    bool load(const JsonObject& obj) override
    {
        value1 = obj.at("value").get<double>();
        text1 = obj.at("text").get<std::string>();

        obj1.load(obj.at("obj1").get<JsonObject>());
        return true;
    }
    bool save(JsonObject& obj) const override
    {
        obj["value"] = (value1);
        obj["text"] = (text1);

        // Save nasted object
        JsonObject buff; // Create a json container
        obj1.save(buff); // save the nasted object
        obj["obj1"] = buff; // Add the data from the nasted object to this json structure
        return true;
    }

    int value1;
    std::string text1;
    Obj1 obj1;
};

// Serializes all objects
std::string serialized(const std::vector< Serializable*>& objs);

// Loads all objects back
// For simplicity the objs list must contain the same amount and objects to match the 
// json data
bool deserialize(const std::vector< Serializable*>& objs, const std::string &buffer);

int main()
{
    // Create multiple test objects
    std::vector< Serializable*> objects1{
        new Obj1(1, "text 1"),
        new Obj1(2, "text 2"),
        new Obj2(3, "text 3", 10, "1000"),
        new Obj1(4, "text 4"),
    };

    // Save the objects
    std::string data = serialized(objects1);

    // Prints the correct parsed structure
    std::cout << "serialized 1:\n" << data << "\n";
    
    // Create the same objects as before but with different data to be able to 
    // verify that loading was successful
    std::vector< Serializable*> objects2{
        new Obj1(0, ""),
        new Obj1(0, ""),
        new Obj2(0, "", 0, ""),
        new Obj1(0, ""),
    };
    // Try to load the objects back from the created string
    if (deserialize(objects2, data)) // is true if loading was successful
    {
        // Save the loaded objects back to check for any difference
        // expected(data2 == data)
        std::string data2 = serialized(objects2);
        std::cout << "serialized 2:\n" << data2 << "\n";

        if (data == data2)
            std::cout << "PASS\n";
        else
            std::cout << "FAIL\n";
    }
    return 0;
}

std::string serialized(const std::vector< Serializable*>& objs)
{
    JsonArray jsonList;
    jsonList.reserve(objs.size());

    for (auto& obj : objs)
    {
        // collect all json object structures
        JsonObject data;
        obj->save(data);
        jsonList.push_back((data));
    }
    // parse the array to a string
    std::string buffer;
    glz::write< glz::opts{ .prettify = true }>(jsonList, buffer);
    return buffer;
}

bool deserialize(const std::vector<Serializable*>& objs, const std::string& buffer)
{
   // JsonArray readed;
    JsonArray readed;
    // Try to load back to an array
    auto err = glz::read_json(readed, buffer);

    // Check the loaded object count 
    if (readed.size() != objs.size())
    {
        std::cout << "\nsize mismatch: "<<readed.size() << " != " << objs.size() << "\n";
        return false;
    }

    bool success = true;
    for (size_t i = 0; i < objs.size(); ++i)
    {
        // Load each object back
        auto el = readed[i].get<JsonObject>();
        success &= objs[i]->load(el);
    }
    return success;
}

[justend29]

reminder to free those heap-allocated objects. std::unique_ptr would remember for you.