[LIVY-1018] Extract field comments from Spark schema metadata and set in Datatypeutils

[ArnavBalyan]

What changes were proposed in this pull request?

• During schema fetch, ensure we can extract and persist the comments.
• Column comments extracted from Spark metadata for complexToDataType and schemaFromSparkJson
• Closes: Extract comment metadata for complexToDataType and schemaFromSparkJson #483

How was this patch tested?

• CI

[ArnavBalyan]

cc @gyogal could you please take a look

[gyogal]

Hi @ArnavBalyan , thank you for your contribution, I think it looks good! Could you please open a LIVY ticket for it at https://issues.apache.org/ and add it to the commit message? If possible, could you please include unit tests as well?

[ArnavBalyan]

Hi @ArnavBalyan , thank you for your contribution, I think it looks good! Could you please open a LIVY ticket for it at https://issues.apache.org/ and add it to the commit message? If possible, could you please include unit tests as well?

Done thanks

[gyogal]

@ArnavBalyan Thanks for adding the JIRA ticket to the PR description, however is LIVY-483 related to this issue? (It seems to be about sparkR memory configuration.) If a new issue is needed and you don't have a JIRA user yet, could you please request one?

[ArnavBalyan]

@ArnavBalyan Thanks for adding the JIRA ticket to the PR description, however is LIVY-483 related to this issue? (It seems to be about sparkR memory configuration.) If a new issue is needed and you don't have a JIRA user yet, could you please request one?

Oh I was following the Github issue, let me file a JIRA ticket thanks

[ArnavBalyan]

@ArnavBalyan Thanks for adding the JIRA ticket to the PR description, however is LIVY-483 related to this issue? (It seems to be about sparkR memory configuration.) If a new issue is needed and you don't have a JIRA user yet, could you please request one?

@gyogal updated could you PTAL thanks

[ArnavBalyan]

cc @gyogal gentle reminder thanks!

[gyogal]

Hi @ArnavBalyan , I was able to confirm that with your change the comments are populated for struct member fields on the server side, however I was not able to find a way to read this information with a JDBC client. Could you please let me know which client or API would allow me to check this on the client side?

[ArnavBalyan]

Hi @gyogal I've added a UT which should be able to e2e test the change thanks

[gyogal]

Thank you for adding tests! Please let me know if this is the final version of your change.

Just a note, it would be even better if there was a way to extract and validate the "Person ID", "Person name", etc. comments from the struct members.

[ArnavBalyan]

Thank you for adding tests! Please let me know if this is the final version of your change.

Just a note, it would be even better if there was a way to extract and validate the "Person ID", "Person name", etc. comments from the struct members.

Thanks, this is the final version of the change, I'll check if we can add recursive extraction in a future change

[gyogal]

Sounds good! Thank you @ArnavBalyan once again for your contribution!