Include Field/SparkField examples in Field metadata

[akhsiM]

Hello @mitchelllisle,

This PR adds field's examples to the metadata: {}, alongside the existing description/comment.

I am also recycling the existing test module.

[mitchstockdale]

Nice one.

Would you be able to add some documentation on how to add metadata and what fields are supported (i.e. comment and examples), please?

On a related note @mitchelllisle, we probably need start a discussion and come to an agreement on a few things:

• whether metadata should be locked down or be customisable
• metadata structure and naming (depending on the above)
• document how metadata fields relate to other Spark or Spark-adjacent tools/technologies, such as Delta Lake

[akhsiM]

@mitchstockdale Thanks and good point about..

.. whether metadata should be locked down or be customisable

My motivation for adding examples here is just to support pydantic's Field examples.

I'm going to wait for @mitchelllisle to respond. If they are happy to go ahead with support field examples, I will add some documentation to this PR.

[mitchelllisle]

Nice one.

Would you be able to add some documentation on how to add metadata and what fields are supported (i.e. comment and examples), please?

On a related note @mitchelllisle, we probably need start a discussion and come to an agreement on a few things:

• whether metadata should be locked down or be customisable
• metadata structure and naming (depending on the above)
• document how metadata fields relate to other Spark or Spark-adjacent tools/technologies, such as Delta Lake

@akhsiM Thanks for the contribution and sorry its taken me a while to respond. I'm happy to approve this PR if you wanted to add some docs before we get it merged. I'll keep an eye out so it gets merged soon.

I think we should limit the use cases based off suggestions from the community at first and if we reach a point where there is a lot of metadata use cases then we can come up with a more standard approach. I like the idea of making it customisable and allowing it to interoperate with spark / delta in some way if we can. But I haven't sat down and thought about how we could implement this. Happy to take suggestions.