Dygraphs format, Graph extension and other chart/visualzation formats

[krabina]

Goal

I'd like to point out and discuss the future of data visualization formats. Some like Timeseries format or d3 chart format are using external components that still are updated, while others like Dygraphs format are using external components that have not been worked on for years.

Graph extension

The mediawiki extension Graph (not to be confused with the Graph result format) seams to be a very actively developed extension, also used in Wikimedia projects.

So one goal could be to provide a "Semantic Graph" extension that brings the ability to a the Graph extension to display data annotated in SMW. (Just like "Semantic Maps" did this for the "Maps" extension, or "Semantic Glossary" for the "Lingo" extension.

This way, the SMW ecosystem could profit from an actively maintained extension. Such an integration would also reduce the burden of upgrading several other graph/chart result formats.

Features

These are missing in most result formats:

1. the ability to display data from an uploaded file (alongside with SMW data). The concept is lined out in the Dygraphs format which can use data from an uploaded csv file.
2. Using an external csv file (or other resource, e. g. JSON file)
3. putting raw data in wiki pages (which the Graph extension can do as well as Maps with the GeoJSON namespace)

Discussion

What are your thoughts on that?

[akuckartz]

Can http://vega.github.io/ be used / integrated?

[mwjames]

The mediawiki extension Graph (not to be confused with the Graph result format) seams to be a very actively developed extension, also used in Wikimedia projects.

The person who originally developed that extension and maintained it has left the WMF and we have no good track record of bundling SMW and WMF related things which in most cases end-up creating more work than previously anticipated. Furthermore, the `Graph`extension only makes use of `vega`/`vega-lite` which seems like a better approach to be used as integration platform instead of relying on an extra WMF middleware.

Can http://vega.github.io/ be used / integrated?

I made a prototype [0] in the past, so, yes it is possible and it only relies on native Vega functionality. [0] SemanticMediaWiki/SemanticMediaWiki#3431 (comment)

…

On 3/31/20, Bernhard Krabina ***@***.***> wrote: ### Goal I'd like to point out and discuss the future of data visualization formats. Some like [Timeseries format](https://www.semantic-mediawiki.org/wiki/Help:Timeseries_format) or [d3 chart format](https://www.semantic-mediawiki.org/wiki/Help:D3_chart_format) are using external components that still are updated, while others like [Dygraphs format](https://www.semantic-mediawiki.org/wiki/Help:Dygraphs_format) are using external components that have not been worked on [for years](http://dygraphs.com/versions.html). ### Graph extension The mediawiki extension [Graph](https://www.mediawiki.org/wiki/Extension:Graph) (not to be confused with the Graph result format) seams to be a very actively developed extension, also used in Wikimedia projects. So one goal could be to provide a "Semantic Graph" extension that brings the ability to a the Graph extension to display data annotated in SMW. (Just like "Semantic Maps" did this for the "Maps" extension, or "Semantic Glossary" for the "Lingo" extension. This way, the SMW ecosystem could profit from an actively maintained extension. Such an integration would also reduce the burden of upgrading several other graph/chart result formats. ### Features These are missing in most result formats: 1. the ability to display data from an uploaded file (alongside with SMW data). The concept is lined out in the [Dygraphs format](https://www.semantic-mediawiki.org/wiki/Help:Dygraphs_format) which can use data from an uploaded csv file. 2. Using an external csv file (or other resource, e. g. JSON file) 3. putting raw data in wiki pages (which the Graph extension can do as well as Maps with the GeoJSON namespace) ### Discussion What are your thoughts on that? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #583

[mwjames]

1. the ability to display data from an uploaded file (alongside with SMW data). The concept is lined out in the [Dygraphs format](https://www.semantic-mediawiki.org/wiki/Help:Dygraphs_format) which can use data from an uploaded csv file. 2. Using an external csv file (or other resource, e. g. JSON file) 3. putting raw data in wiki pages (which the Graph extension can do as well as Maps with the GeoJSON namespace)

I understand the sentiment but please don't forget that the data we display should be available through SMW and as part of the data repository (providing a context), just because it would be convenient displaying some random graph via SMW/SRF isn't part of what SMW should be doing. The information has to come from the store otherwise using SMW as simple graph engine doesn't bring any insights into the data you display, the analysis, or its context.

…

On 3/31/20, Bernhard Krabina ***@***.***> wrote: ### Goal I'd like to point out and discuss the future of data visualization formats. Some like [Timeseries format](https://www.semantic-mediawiki.org/wiki/Help:Timeseries_format) or [d3 chart format](https://www.semantic-mediawiki.org/wiki/Help:D3_chart_format) are using external components that still are updated, while others like [Dygraphs format](https://www.semantic-mediawiki.org/wiki/Help:Dygraphs_format) are using external components that have not been worked on [for years](http://dygraphs.com/versions.html). ### Graph extension The mediawiki extension [Graph](https://www.mediawiki.org/wiki/Extension:Graph) (not to be confused with the Graph result format) seams to be a very actively developed extension, also used in Wikimedia projects. So one goal could be to provide a "Semantic Graph" extension that brings the ability to a the Graph extension to display data annotated in SMW. (Just like "Semantic Maps" did this for the "Maps" extension, or "Semantic Glossary" for the "Lingo" extension. This way, the SMW ecosystem could profit from an actively maintained extension. Such an integration would also reduce the burden of upgrading several other graph/chart result formats. ### Features These are missing in most result formats: 1. the ability to display data from an uploaded file (alongside with SMW data). The concept is lined out in the [Dygraphs format](https://www.semantic-mediawiki.org/wiki/Help:Dygraphs_format) which can use data from an uploaded csv file. 2. Using an external csv file (or other resource, e. g. JSON file) 3. putting raw data in wiki pages (which the Graph extension can do as well as Maps with the GeoJSON namespace) ### Discussion What are your thoughts on that? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #583

[krabina]

well, your Dygraphs extension brought me the idea. And @JeroenDeDauw shows in the maps extension how useful it can be to have both: to use a GeoJSON from an external source and display SMW data as points over it. Same can be true for many other data types...

[mwjames]

well, your Dygraphs extension brought me the idea. And @JeroenDeDauw shows in the maps extension how useful it can be to have both: to use a GeoJSON from an external source and display SMW data as points over it. Same can be true for many other data types...

I see those as an exception rather that being the rule, data is only useful in context and context is that what you describe via SMW annotations.

…

On 3/31/20, Bernhard Krabina ***@***.***> wrote: well, your Dygraphs extension brought me the idea. And @JeroenDeDauw shows in the maps extension how useful it can be to have both: to use a GeoJSON from an external source and display SMW data as points over it. Same can be true for many other data types... -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #583 (comment)

[mwjames]

Another problem with external sources is that you never know whether the source will be available in a month, a year or so. Relying on "dead" source material you don't have any control over is a common problem for data repositories that think that just because it has an URL it means it will be available forever. The same issue Wikipedia is facing with dead cite links where something like the web archive helps but demonstrates the problem of relying on external link material.

…

On 3/31/20, James HK ***@***.***> wrote: > well, your Dygraphs extension brought me the idea. And @JeroenDeDauw > shows > in the maps extension how useful it can be to have both: to use a GeoJSON > from an external source and display SMW data as points over it. Same can > be > true for many other data types... I see those as an exception rather that being the rule, data is only useful in context and context is that what you describe via SMW annotations. On 3/31/20, Bernhard Krabina ***@***.***> wrote: > well, your Dygraphs extension brought me the idea. And @JeroenDeDauw > shows > in the maps extension how useful it can be to have both: to use a GeoJSON > from an external source and display SMW data as points over it. Same can > be > true for many other data types... > > -- > You are receiving this because you commented. > Reply to this email directly or view it on GitHub: > #583 (comment)

[krabina]

btw. an interesting read regarding the downside of the graph extension: https://www.mediawiki.org/wiki/User:Bawolff/Reflections_on_graphs

I'm impressed with [0] SemanticMediaWiki/SemanticMediaWiki#3431 (comment)
what can be done to make it happen?

[krabina]

Another problem with external sources is that you never know whether the source will be available in a month, a year or so. Relying on "dead" source material you don't have any control over is a common problem for data repositories that think that just because it has an URL it means it will be available forever. The same issue Wikipedia is facing with dead cite links where something like the web archive helps but demonstrates the problem of relying on external link material.

while this is true in general, often you have governanc over SMW "external" files. They can be an uploaded CSV, or JSON-data in a MediaWiki Namespace or an external resource you control yourself...

[mwjames]

while this is true in general, often you have governanc over SMW "external" files. They can be an uploaded CSV, or JSON-data in a MediaWiki Namespace or an external resource you control yourself...

If you own them and can make them available as "raw" content in some MediaWiki namespace then yes, but beware just uploading them and producing a "nice" graph isn't the job of SMW or SRF.

…

On 3/31/20, Bernhard Krabina ***@***.***> wrote: > Another problem with external sources is that you never know whether the > source will be available in a month, a year or so. Relying on "dead" > source material you don't have any control over is a common problem for > data repositories that think that just because it has an URL it means it > will be available forever. The same issue Wikipedia is facing with dead > cite links where something like the web archive helps but demonstrates the > problem of relying on external link material. while this is true in general, often you have governanc over SMW "external" files. They can be an uploaded CSV, or JSON-data in a MediaWiki Namespace or an external resource you control yourself... -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #583 (comment)

[krabina]

Well I tend to disagree. For data visualizations there are a lot of usecases where you want to describe different data sources (with SMW) and visualize them, without every data item to be stored in SMW. Just as you mentioned in dygraphs.
Not every measuring point of something needs to be stored if the data is coming from an outside source anyway.
My usecase currently is handling and visualizing spending data from municipalities. These datasets very easily get very big. However, being available to visualize them (interactively) AND being able to describe the data on a more general level could be done very smoothly in SMW.

Of course, this is not the regular use-case, so in general I agree with you :-)

[mwjames]

Well I tend to disagree. For data visualizations there are a lot of usecases where you want to describe different data sources (with SMW) and visualize them, without every data item to be stored in SMW. Just as you mentioned in dygraphs.

Yes, that data scenario exists but again, displaying some random data (here random is made with a reference to some non SMW source data) isn't the job of SMW or its extensions. SMW's responsibilities lies in making those data accessible that it has stored through its interface. You have to tread carefully here of what SMW should be, can be, and should not be because you don't want to make SMW a master of all trades and I recall " ... main task and objective of Semantic MediaWiki is to extend the MediaWiki platform with an ability to store structured information and make them retrievable either as part of a query result or as data for external federation ...". So, external sources can be apart of it but to a limit extend and with a very narrow definition of how the integration should happen. I don't think that loading some CSV data from somewhere as source to be displayed via SMW/SRF has anything to do with SMW aside from defining an URL to hold the data definition that can be displayed and loaded via a SMW related extension. I see that scenario outside of the scope of SMW/SRF because you could build an extension X that uses the same approach of loading some URL referenced data into a page via a parser function with no connection to SMW or SRF.

…

On 4/1/20, Bernhard Krabina ***@***.***> wrote: Well I tend to disagree. For data visualizations there are a lot of usecases where you want to describe different data sources (with SMW) and visualize them, without every data item to be stored in SMW. Just as you mentioned in dygraphs. Not every measuring point of something needs to be stored if the data is coming from an outside source anyway. My usecase currently is handling and visualizing spending data from municipalities. These datasets very easily get very big. However, being available to visualize them (interactively) AND being able to describe the data on a more general level could be done very smoothly in SMW. Of course, this is not the regular use-case, so in general I agree with you :-) -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #583 (comment)

[IRA1777]

General thoughts on graph representation of semantic data:
I use it a lot, especially through graphviz. To me, SRF "Graph format" is unusable because customizing capabilities through query parameters are too limited, you can exploit only few of graphviz.

A better solution that I use is to query SMW data in array format, store results in an array (array extension), then print the array inside a "tagged" graphviz , with full control over data formatting. {{#tag:graphviz | <my script>}}. Same apply for Pchar4MW extension or graph extension (which I never get to work properly by the way).

One typical "use case" comes from page name, which is generally assumed to be representative of the content (in SRF as in most semantic extension). But I tend to quit this practice and to store content in specific namespaces, with auto-generated number as page name (through extension page forms): ie a "Person" namespace, page names are like Person:0357. Human readable name is a generic SMW property like Has display name, holding what should be displayed. Has display name::Wilhelm Klink .

This allows great flexibility in data management, solves homonyms problems and many other. But it disallow use of many result formats that assume page name as display name.

So as a general rule, I don't see the need of a specific semantic wrapping of an extension, provided that this extension has a scripting language that I can generate by grabbing raw data from SMW query.

[alex-mashin]

I have managed to revive the graph format with ExternalData extension, which is now able to emulate GraphViz using new and yet undocumented parser function {{#get_program_data:}} in tag emulation mode. This function has been tested to successfully embed GraphViz, mscgen, ploticus, gnuplot, PlantUML and LilyPond output, including processed results of {{#ask:}} queries.

It might be worth trying to use this functionality in SRF. Of course, there are obvious security concerns, so testing for possible vulnerabilities is always welcome.

P.S. Speaking specifically of vega, I was unable to install it (let alone the graphoid service) in reasonable time; I gave up while trying to manually install with npm the dependencies required to run yarn build. I am afraid, this solution is deliberately made overcomplicated and practically usable only by WikiMedia Foundation, other examples of such microservice approach being VisualEditor and Math extensions. That can reduce the number of users of a possible new family of formats based on Graph.