New vsg::txt ReaderWriter and new capabilities of reading meta-data files over the web

[robertosfield]

Hi All,

This week I've been working implementing support for streaming image data, I'll talk about this in more detail later in the week once something functional to share, but for now I'd like to introduce a few new capabilities that I've added that are useful for tasks like streaming data, but also many other tasks. The changes I'll talk about here are now checked into VulkanSceneGraph and vsgXchange master.

The elements of what I have checked in are:

1. New vsg::txt ReaderWriter that makes it easy to read ascii files, whether on local disk or pulled down over http.
2. New vsg::compatibleExtensions(..) functions that standardize how extensions are checked and handled in ReaderWriter.
3. vsgXchange::curl now uses the vsg::Options::extensionHint if set to help tell which ReaderWriter should handle a file.
4. vsgXchange ReaderWriter's all now use the vsg::compatibleExtensions(..) function to standardize behavior.

The changes checked in are: VulkanSceneGraph changes and vsgXchanges changes.

What makes all this functionality interesting is what it enables. First up the new vsg::txt ReaderWriter simply reads the contents of files and returns it as a vsg::stringValue object. One it's own it not that interesting, but if you want you can do stuff like:

if (auto config = vsg::read_cast<vsg::stringValue>("configuration.txt"))
{
    std::cout<<"config = "<<config->value()<<std::endl;

    // assign to the scene graph for later use:
    scenegraph->setObject("configuration",  config);
}

Or on the command line:

vsgconv configuration.txt my_config.vsgt
more my_config.vsgt

Things get a little more interest when you want to read files from the web...

[robertosfield]

By combining using of vsgXchange::curl plugin with the new vsg::txt ReaderWriter we can directly read files from the web:

auto options = vsg::Options;
options->add(vsgXchange::all::create());

if (auto config = vsg::read_cast<vsg::stringValue>("http://website/configuration.txt", options))
{
      std::string& str = config->value();
      // process the string
}

To complicate matters sometimes config files and metadata is hosted on servers via an API so the URL used isn't an actual file, so the file extension may not exist or be followed by lots of extra characters. For instance to go get metadata from a Bing Maps server we need to use a URL like:

https://dev.virtualearth.net/REST/V1/Imagery/Metadata/AerialWithLabels?output=xml&key=lotsofrandomcharactersformykey
Which presents us with a problem with how we map the file that will be read by vsgXchange::curl to a ReaderWriter that can map the contents to something usable in our application. In this case it's going to be an xml file, but the VSG doesn't have an xml reader, and there is no convenient form like "myfile.xml" so we have to tell the curl ReaderWriter what to explicitly.

The way to do this, or at least once you've updated to vsgXchange master with the latest changes, is to specify that the new vsg::txt ReaderWriter should be used via the vsg::Options::extensionHint member. The code now to read the data is:

auto options = vsg::Options;
options->add(vsgXchange::all::create());
options->extensionHint = ".txt"; // we want to use the vsg::txt ReaderWriter to store the contents of the read
vsg::Path url("https://dev.virtualearth.net/REST/V1/Imagery/Metadata/AerialWithLabels?output=xml&key=lotsofrandomcharactersformykey");

if (auto metadata = vsg::read_cast<vsg::stringValue>(url, options))
{
      std::string& str = metadata->value();
      // process the string
}

For testing purposes it can also be used to try out our url's via vsgconv,, but this still needs to now our Options::extensionHint setting, so enable this I've added a --extension-hint <.ext> command line option to vsg::Options, which vsgconv reads so the above is now possible:

vsgonv https://dev.virtualearth.net/REST/V1/Imagery/Metadata/AerialWithLabels?output=xml&key=lotsofrandomcharactersformykey" test.vsgt --extension-hint .txt more test.vsgt

This can also be used to help force a loader to handle a file that doesn't have the correct extension, or a recognized extension i.e.

vsgconv myjpgthathasnoextension test.vsgt --extension-hint .jpg
vsgviewer myjpgthathasnoextension --extension-hint .jpg

The new vsg::compatibleExtensions(..) functions are now used in various ReaderWriter implementations in the core VSG and in vsgXchange so that if you set an Options::extensionHint it'll overrided extension that the file has.

The vsg::txt ReaderWriter is very simple - it just creates a std::string the same size as the file and reads the contents into it and returns this a vsg::stringValue. It currently automatically reads the following .extensions as text:

        ".txt",
        ".text",
        ".md",
        ".json",
        ".xml",
        ".sh",
        ".bat"

I have just put this in as placeholders to get us started, we can add/remove ones as we get more experience with using this class for different tasks.

I have included .json and .xml in the list as a first step to giving applications the ability to read these file types. At present there is no functionality built into the VSG for parsing these automatically, so if you are impatient you'll be able to pass the std::string onto a 3rd party i.e. nlohmann json parser:

if (auto config = vsg::read_cast<vsg::stringValue>("https://mywebsite.org/config.json", options))
{
       auto json_config = nlohmann::parse(config->value());
      // use nlohmann to read the data.
}

If you a confident with parsing std::string yourself then you may not even need a 3rd party tool to do this. This week I was able parse the bits I wanted from a xml metadata returned from a website just with some straight std::string::find calls, so it's certainly possible to do useful work without full blown parsers.

However, I do want to make parsing xml and json files a bit more straight forward, and without need big 3rd party dependencies like nlohmann::json header file is 26,000 lines of code, that half the size of the whole of the VSG and it has a whole scene graph, windowing integrating, binary and ascii parsers etc. Something is broken in the land of programming when small facilities take so much code.

I might not be able to tackle it this week, but my plan is have a bash at writing a very simple set of parsers for CSV, json and xml files. The aim would be to read the files with vsg::txt as a vsg::stringValue then parse it's std::string in a lightweight and flexible way - the aim would be to do 90% of the functionality with 10% of the code.. If users really want powerful json and xml parsers then they will still be able to use 3rd party libs for it.

[rolandhill]

Hi Robert, I don't know if you already have csv access code in mind. I've been using https://github.com/vincentlaucsb/csv-parser and have been happy with it so far. It's very fast, multithreaded, MIT licensed and has the option to just use a single header (that's what I use) or you can build a library. I think this was the second one I tested, so maybe there is better code out there. Regards, Roland

…

On Wed, 25 Jan 2023 at 07:28, Robert Osfield ***@***.***> wrote: By combining using of vsgXchange::curl plugin with the new vsg::txt ReaderWriter we can directly read files from the web: auto options = vsg::Options; options->add(vsgXchange::all::create()); if (auto config = vsg::read_cast<vsg::stringValue>("http://website/configuration.txt", options)) { std::string& str = config->value(); // process the string } To complicate matters sometimes config files and metadata is hosted on servers via an API so the URL used isn't an actual file, so the file extension may not exist or be followed by lots of extra characters. For instance to go get metadata from a Bing Maps server we need to use a URL like: https://dev.virtualearth.net/REST/V1/Imagery/Metadata/AerialWithLabels?output=xml&key=lotsofrandomcharactersformykey Which presents us with a problem with how we map the file that will be read by vsgXchange::curl to a ReaderWriter that can map the contents to something usable in our application. In this case it's going to be an xml file, but the VSG doesn't have an xml reader, and there is no convenient form like "myfile.xml" so we have to tell the curl ReaderWriter what to explicitly. The way to do this, or at least once you've updated to vsgXchange master with the latest changes, is to specify that the new vsg::txt ReaderWriter should be used via the vsg::Options::extensionHint member. The code now to read the data is: auto options = vsg::Options; options->add(vsgXchange::all::create()); options->extensionHint = ".txt"; // we want to use the vsg::txt ReaderWriter to store the contents of the read vsg::Path url("https://dev.virtualearth.net/REST/V1/Imagery/Metadata/AerialWithLabels?output=xml&key=lotsofrandomcharactersformykey"); if (auto metadata = vsg::read_cast<vsg::stringValue>(url, options)) { std::string& str = metadata->value(); // process the string } For testing purposes it can also be used to try out our url's via vsgconv,, but this still needs to now our Options::extensionHint setting, so enable this I've added a --extension-hint <.ext> command line option to vsg::Options, which vsgconv reads so the above is now possible: vsgonv https://dev.virtualearth.net/REST/V1/Imagery/Metadata/AerialWithLabels?output=xml&key=lotsofrandomcharactersformykey" test.vsgt --extension-hint .txt more test.vsgt This can also be used to help force a loader to handle a file that doesn't have the correct extension, or a recognized extension i.e. vsgconv myjpgthathasnoextension test.vsgt --extension-hint .jpg vsgviewer myjpgthathasnoextension --extension-hint .jpg The new vsg::compatibleExtensions(..) <https://github.com/vsg-dev/VulkanSceneGraph/blob/master/include/vsg/io/Options.h#L97> functions are now used in various ReaderWriter implementations in the core VSG and in vsgXchange so that if you set an Options::extensionHint it'll overrided extension that the file has. The vsg::txt ReaderWriter <https://github.com/vsg-dev/VulkanSceneGraph/blob/master/include/vsg/io/txt.h#L33> is very simple - it just creates a std::string the same size as the file and reads the contents into it and returns this a vsg::stringValue. It currently automatically reads the following .extensions as text: ".txt", ".text", ".md", ".json", ".xml", ".sh", ".bat" I have just put this in as placeholders to get us started, we can add/remove ones as we get more experience with using this class for different tasks. I have included .json and .xml in the list as a first step to giving applications the ability to read these file types. At present there is no functionality built into the VSG for parsing these automatically, so if you are impatient you'll be able to pass the std::string onto a 3rd party i.e. nlohmann json <https://github.com/nlohmann/json> parser: if (auto config = vsg::read_cast<vsg::stringValue>("https://mywebsite.org/config.json", options)) { auto json_config = nlohmann::parse(config->value()); // use nlohmann to read the data. } If you a confident with parsing std::string yourself then you may not even need a 3rd party tool to do this. This week I was able parse the bits I wanted from a xml metadata returned from a website just with some straight std::string::find calls, so it's certainly possible to do useful work without full blown parsers. However, I do want to make parsing xml and json files a bit more straight forward, and without need big 3rd party dependencies like nlohmann::json header file is 26,000 lines of code, that half the size of the whole of the VSG and it has a whole scene graph, windowing integrating, binary and ascii parsers etc. Something is broken in the land of programming when small facilities take so much code. I might not be able to tackle it this week, but my plan is have a bash at writing a very simple set of parsers for CSV, json and xml files. The aim would be to read the files with vsg::txt as a vsg::stringValue then parse it's std::string in a lightweight and flexible way - the aim would be to do 90% of the functionality with 10% of the code.. If users really want powerful json and xml parsers then they will still be able to use 3rd party libs for it. — Reply to this email directly, view it on GitHub <#708 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAPOEQ6KQ7OCEFUUTMR5CA3WUA3OPANCNFSM6AAAAAAUFODFO4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** .com>

[robertosfield]

Thanks for the link.

At the end of last year I implemented a CSV loader for a client so am comfortable with the task, it doesn't take much code to implement. Since writing it I've had further ideas of how it could be improved and generalized. Curiously I'm thinking of using std::string_view as the suggested vsg-parser talks about :-)

One of the things I'ld like to make easy is handling VSG specific types like vsg::vec3 etc. The vsg::CommandLine class has a neat way of enabling this, and the vsg::mem_stream class should help further too as a lightweight of integrating stream support.

Multi-threading is interesting topic, it's not one I've attempted to think deeply about at this point for parsing. First task is to get it to wok, then if required think about optimization.

My expectation is for most purposes speed of reading an ascii data or configuration file will not be critical as the really big datasets tend to be binary. If performance is an issue with whatever I come up with we can discuss how to make it faster., or we just fallback to suggesting users use dedicated third party libs like csv-parser.

[gwaldron]

HTTP responses include the Content-Type header, so why not support mime-types to identify ReaderWriters instead of (or in addition to) extensions. For now I'm doing this with a small mapping table so I can push data from httplib to the proper ReaderWriter (below). But direct support for mime-types in the ReaderWriter might make more sense. Just a thought

    static const std::unordered_map<std::string, std::string> ext_for_mime_type = {
        { "image/bmp", ".bmp" },
        { "image/gif", ".gif" },
        { "image/jpg", ".jpg" },
        { "image/jpeg", ".jpg" },
        { "image/png", ".png" },
        { "image/tga", ".tga" },
        { "image/tif", ".tif" },
        { "image/tiff", ".tif" }
    };

[robertosfield]

I'm open to suggestions, I have attempted to work with mime-types so far. I presume your can get the mime-type directly from libcurl?

[gwaldron]

For libcurl:

char* content_type_cp;
curl_easy_getinfo( _curl_handle, CURLINFO_CONTENT_TYPE, &content_type_cp );