Automatic creation of string hashes

[Tisten]

Being able to write real strings in json and just store the string hash in the binary file (and putting backwards lookup info in a separate structure) and then being able to get strings again when converting back is an awesome space saver and debug feature which I'm missing in dl.

[wc-duck]

Im a bit hesitant to adding to much machinery around hashes directly in the core library mostly as I don't feel that DL can decide what hash-function to use and how a reverse-lookup should be performed.

However, what could be done is adding 2 callbacks on the type lib for hashing and "unhashing" string-data and let the user implement them how they see fit.

This could be default to a hash-function using the internal dl_hash and not implementing a reverse-lookup.
When unpacking to text and there is no reverse-lookup the output value would be the hash as an int. As long as txt-pack accept int or string for a hash32/hash64 it should be an OK compromise!?

Any thoughts on that?

[wc-duck]

When thinking a bit möte about this one more thing to take into consideration popped up.

I know of another similar library that store the typelib after each packed instance for simpler decompression. I'm not really a fan of this as it eats lots of space that is mostly the same among packed days ( there can be up to 90% waste in many files!).
It is however convenient!

Where am I going with this you might ask! Would it be worth adding an api or similar for users to alloc user data sections aster the packed instance where they could store things such as hashed strings and/or typelibs if they wish to?

[Tisten]

I agree that it would be good if the core of DL didn't make such decisions. DL would not have to support a HashedString type out of the box, but it would be amazing if there were the concepts needed for a user to add it, and an example of how to do it in dl.

I am absolutely a fan of "user callbacks" to transform the data per type. It could be generalized, so an "Angle" type could transform between randians and degrees, a "Rotation" could transform between euler and quaternions etc. The tricky part is that it would be conversion between two different types for json and binary, at least for the hashes. I think it would benefit from some experimentation.

About "that other library", I know it got API to store the typelib and backwards lookup info for hashes as separate files from the data itself, though it wasn't propagated to the build rules and compilers which wrote the data. I agree that it is convenient to have the user data embedded in the file, but if I would prefer to outputing the user data to a separate file, and then add a utility which archive of all the outputs into one file as part of the tooling. For example by extending dlpack with m6y own user code. I know that other library provided compression in the tools as well, something which really saved disk space, reduced the time for file checksums, and reduced bandwidth/made "compression over the wire" superfluous, so the archiving step could be part of other such additional tooling. But I don't think DL need to decide on that neither.

For this github issue I think the only thing which is really needed is user callbacks to transform data, and by using these the users of DL can implement user storage in the tooling themselves.

[wc-duck]

Yes... user provided transforms is a good start. I however need to think about the api, especially if you want to switch types str -> uint32 for example.

[Tisten]

If these transforms are meant to be between json and binary, DL doesn't have to care about the json format. So this could be between `const char*` and the binary type. I.e the user needs to handle the float parsing and writing of the "Euler" format and and just need to store the quaterion as 4 floats.

…

On Fri, Oct 7, 2022 at 5:31 PM Fredrik Kihlander ***@***.***> wrote: Yes... user provided transforms is a good start. I however need to think about the api, especially if you want to switch types str -> uint32 for example. — Reply to this email directly, view it on GitHub <#148 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABTKDO2KLOLBZYCWKACYEILWCA655ANCNFSM6AAAAAAQ65CWX4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[wc-duck]

I see bo real problem with this as long as your storage in dl is pod... but if it starts to contain subdata such as strings and arrays it get way more complex! But maybe it is a valid restriction to only support conversion to pod-types?

[wc-duck]

Supporting conversion to a struct with arrays or pointers would add quite a bit of complexity... especially since then you could actually chain them as well!

[Tisten]

I agree. Starting out sith POD only will cover the cases mentioned so far. I might be optimistic when thinking about pointers and arrays, but I think they ought to work as well if need them. But leave them for later

[wc-duck]

An idea that popped up during a walk... this might be combined with "type-conversion".
Adding a function to convert from type-a to type-b could be used to upgrade data, but with a callback to that process you could also implement these kind of workshops. By specifying an "edit"-type and a "runtime" type and do a "conversion" in your buildpipe maybe 2 issues can be solved at once?

Something like that could possibly be used for "more complex" type-upgrades as well?

[Tisten]

Yes, I really like the concept of "data migration" which a "data upgrade" callback was called in the machinery which I worked with lately. They really made it possible to make advanced schema upgrades without breaking old data. They weren't bidirectional though, since a data upgrade isn't normally intended to solve the case of old code with data which is "too new".

I like the idea of having a single concept which a developer can utilize to solves both of these kinds of issues. But with that ambition it would need to support non-POD types. And it wouldn't be enough to do bidirectional conversion between a general "json string" and a specific type.

[wc-duck]

When I talk about data-conversion I don't think about json even being in the picture, I see something in the vain of

dl_convert(packed_instance, &out_instance, new_type_id);

That would never need to go via json at all... not really sure how it would work, but if you could get something in your callback as:
error_code my_callback(old_stored_type* old_instance, new_stored_type* new_instance)
{
}

you could just write plain c to do you conversion? Especially together with the proposed "versioning of type" in another issue we could even generate the c-structs in a header for all versions of a type if applicable?

There is the question of how to implement your "string" -> hash, but if you have a string_hash type that is only a string-member and your hash32 type that is both types, it would work out quite nicely? I.e. your conversion function wwould get stored_type as "string_type*" and your new_stored_type would be a "hash32*".

that would make your conversion function to be:
new_type->hash = hash_the_string(old_type->str);

[Tisten]

Yes, I agree with your thoughts here. My comment about json was just to be clear about that the "old type" and "new type" is not about "old and new" but about "txt and binary" in that case, so the question about when to run the migration callback becomes more complex than to check for the highest version number since it needs to be bidirectional to and from the txt version.

I think I know you well enough to say that your callback signature is simplified, and that it should include a void* user_data which is registered with the callback function :)

[Tisten]

I took this prototype a bit further in https://github.com/Tisten/datalibrary/commits/patch_impl/ but never never finished it. The reason is that we so far don't use datalibrary for source data. We always create the datalibrary data from another source format and thus have not needed data migration. I like the idea of supporting data migration, but I don't really need it.

But lately I've thought more about the "human readable" part of datalibrary, and one feature which I would like to have is better visualization of key-value containers. I already have an implementation which can be stored in the binary version of datalibrary, and of cause it can be converted to json. But by doing so it just shows the implementation internals and the json becomes very hard to read for a human.

One solution could be to add a map/dictionary container into datalibrary. But I don't like that. It would need to define the data layout of the container, and questions like "should the data be sorted" are also locked down. The only feature I really need from datalibrary is the same as I need for the string hashes, and that is two callback functions for a specific typeid, one "binary to json" and one "json to binary".