Towards a new Parallels JSON structure

[ayya-vimala]

At the request of Bhante @sujato I want to make a proposal for a new, easily readable JSON structure for th Parallels.json file.

Methodology

There are four types of parallels, which are specified here: https://suttacentral.net/methodology?lang=en

Next to these, there are parallels of part of a text. So, for instance, if the first part of a sutta (A) is a full parallel to the first part of another sutta (B), but the rest of these suttas do not match. Then these two suttas can be represented as a "resembling parallel" or these first parts on their own can be represented as a "full parallel" by using their line numbers or paragraph ids to represent a range.

A is a resembling parallel to B and B is a resembling parallel to A
or
A#sc1-#sc10 is a full parallels to B#sc1-#sc10

Currently, the choice between these is sometimes clear, but sometimes a bit arbitrary.

Current JSON structure

Currently, the JSON structure is made to be concise, but it makes it not very readable. There is a wiki: https://github.com/suttacentral/suttacentral/wiki/Parallels-information

As an example, I take MN10: https://suttacentral.net/mn10?view=normal&lang=en

The first part of the parallels in this list is currently represented by:

    {
        "parallels": [
            "dn22",
            "ea12.1",
            "ma98",
            "mn10",
            "sht-sutta11",
            "~ma31",
            "~mn119",
            "~t32",
            "~ma81"
        ]
    },

The ~ represents a "resembling parallel".

Each of the resembling parallels is then also mentioned in a new object because it might be that for instance MA31 is resembling parallels with MN10, but not with MN119, etc. This has to be determined at in every case.

    {
        "parallels": [
            "mn141",
            "ea27.1",
            "ma31",
            "t32",
            "~dn22#18.1",
            "~ea12.1",
            "~ma98",
            "~mn10",
            "~sht-sutta11"
        ]
    },
    {
        "parallels": [
            "mn119",
            "ma81",
            "~dn22",
            "~ea12.1",
            "~ma98",
            "~mn10",
            "~sht-sutta11"
        ]
    },
.... Etc. for each of the resembling parallels.

Then there are the parallels of parts of the text. For instance for MN 10, paragraph 10.1:

    {
        "parallels": [
            "dn22#5.1",
            "mn10#10.1"
        ]
    },
    {
        "mentions": [
            "dn22#5.1",
            "ne17#22.1",
            "vb7#2.1"
        ]
    },
    {
        "mentions": [
            "mn10#10.1",
            "ne17#22.1",
            "vb7#2.1"
        ]
    },

In this case, we have a full parallel with dn22#5.1 and two mentions of each of those in ne17#22.1 and vb7#2.1.

And the same for the other paragraphs that are mentioned.

New JSON structure proposal

Now I propose a structure for the JSON that is radically different, namely a structure based on each sutta separately in the form of:

[
  "suttanr": {
            "full": [],
            "resembling": [],
            "mentions": [],
            "retelling": [],
            "sections": [
                         "suttanr#id-#id: {
                                         "full": [],
                                         "resembling": [],
                                         "mentions": [],
                                         "retelling": []
                                         }
                         ]
          }
]

Where "partial" represents the part-sutta parallels.
So for MN10 (in full), this would become:

[
  "mn10": {
            "full": [
                      "dn22",
                      "ea12.1",
                      "ma98",
                      "sht-sutta11"
                    ],
            "resembling": [
                      "ma31",
                      "mn119",
                      "t32",
                      "ma81"
                    ],
            "mentions": [],
            "retelling": [],
            "sections": [ "mn10#10.1": {
                                        "full": ["dn22#5.1"],
                                        "resembling": [],
                                        "mentions": [
                                                      "ne17#22.1",
                                                      "vb7#2.1"
                                                    ],
                                        "retelling": []
                                        },
                          "mn10#44.1": {
                                        "full": [
                                                  "dn22#17.1",
                                                  "mn9#14-18.1"
                                                ],
                                        "resembling": [],
                                        "mentions": [],
                                        "retelling": []
                                        },
                          "mn10#47.1": {
                                        "full": [
                                                  "dn22#22.24",
                                                  "sn47.1#2.1"
                                                ],
                                        "resembling": [],
                                        "mentions": ["kv1.9#10.1"],
                                        "retelling": []
                                        }
                        ]
          }
]

We could of course remove the empty fields so it becomes:

[
  "mn10": {
            "full": [
                      "dn22",
                      "ea12.1",
                      "ma98",
                      "sht-sutta11"
                    ],
            "resembling": [
                      "ma31",
                      "mn119",
                      "t32",
                      "ma81"
                    ],
            "sections": [ "mn10#10.1": {
                                        "full": ["dn22#5.1"],
                                        "mentions": [
                                                      "ne17#22.1",
                                                      "vb7#2.1"
                                                    ]
                                        },
                          "mn10#44.1": {
                                        "full": [
                                                  "dn22#17.1",
                                                  "mn9#14-18.1"
                                                ]
                                        },
                          "mn10#47.1": {
                                        "full": [
                                                  "dn22#22.24",
                                                  "sn47.1#2.1"
                                                ],
                                        "mentions": ["kv1.9#10.1"]
                                        }
                        ]
          }
]

And this for each and every sutta. So in this case, "dn22" would get it's own mention in the same way as well.

Pros and Cons

The proposed structure is more readable and intuitive, as it is basically the same as it actually shows on the website. The old structure is much more concise.

But it is also easier for someone adding a parallel to make the mistake to add it only in one place. For instance, if a full parallel X is found for mn10, you would need to add it with "dn22", "ea12.1", "ma98", "mn10", "sht-sutta11" and check if it has to be added to "ma31", "mn119", "t32" and "ma81". And X needs to get it's own entry as well if it wasn't there before.

Now this is something that had to be taken into account in the old structure also, but if one adds X to the full parallels list "dn22", "ea12.1", "ma98", "mn10", "sht-sutta11", it is automatically added with all of those already, without the need of doing this 5 times and making a new extra entry for X.

Integration into SuttaCentral

A new JSON structure would require the loading code in Python to be updated in the SuttaCentral backend.

So please let me know your ideas and thoughts about this proposal.

[sujato]

Loving the proposed JSON, it's a lot clearer. Let's talk it through bit by bit to see how we can make it better.

• One minor suggestion: I'd prefer something like sectional rather than partial, which is apt to be confused with resembling.
• I'm not entirely convinced that a separate category needs to be determined for these. Ultimately, anything with a # on either side is sectional. The division of texts on SC (or anywhere) is somewhat arbitrary anyway. For example, the Anguttara Ones are combined on SC into vaggas for ease of presentation on the web. So in our files they are one "text" but obviously they are several suttas—does that make a parallel sectional or full? We can resolve that issue of course, my point is simply that the difference between "full" and "sectional" is as much dependent on editorial choices as anything meaningful in the text. Still, never mind, keep it as-is for now, let's see.
• Speaking of which, I can't recall why we're using # in these files. Normally the IDs are specified with : and we only convert to # for the URL. Unless there's a reason not to, maybe we should use :?

So it seems to me we want some steps forward, and should consider these before going ahead.

1. Can we agree on the new data specification?
2. How easy is it to transform the old data to the new format?
3. Can we solve the editing problem?
4. How can we mesh the new format with the existing SC code?

I'm going to assume that 1, 2, and 4 can be solved by clever programmers. Let's consider the editing question.

This should be regarded as an important step forward in any case. There's been a lot of scholarship in this area and I'm sure we can improve the parallels a lot. Obvious steps would be to review the work of Vens Analayo and Dhammadinna and see what can be added; looking at texts that have not so far been covered; improving the Vinaya-Sutta parallels. There's even things like, say, adding Jataka parallels.

The main reason we haven't progressed is because there's no easy editing option. What we really need is a webapp for editing the files, where a user can add new parallels, and the updates are either added automatically if they can be, or the user is presented with relevant options to add by hand.

We don't currently have anything like that, but here's an idea: let's build that functionality into the new Bilara. Even though it is yet to launch, and development has taken waaaaaaaaaaay long, it's pretty powerful, and already designed to do similar tasks.

• permissions for editors, with different levels
• direct editing of bilara-data files
• search on those files
• we can automatically update bilara-data JSON, eg. if we add or remove a segment from a root text file.

It wouldn't be trivial, but I'm pretty sure we could build a parallels editor into the new Bilara.

[thesunshade]

can be solved by clever programmers.

Just want to make sure @khemarato, @winterdharma @brahmali @ccronje knows about this issue. As a heads up as well as for possible contributions.

I'm pretty sure we could build a parallels editor into the new Bilara.

Of course I know zero about this side of things. But I'm wondering @ayya-vimala if you feel you need something sooner than that might possibly be ready. Or can you continue your work manually as you have been? It makes perfect sense to build this into Bilarav2, but that could be years away and I don't want to lose momentum on the parallels work. And of course nothing at all can happen until other decisions are made.

[ayya-vimala]

But I'm wondering @ayya-vimala if you feel you need something sooner than that might possibly be ready.

For me it is not necessary. I work well with the old structure. It is for other people to work with in the future.

Or can you continue your work manually as you have been?

This is all still manual in the JSON file. It is just that the JSON file is a bit more intuitive and mirrors the website.

One minor suggestion: I'd prefer something like sectional rather than partial

Agreed and changed

I'm not entirely convinced that a separate category needs to be determined for these

I don't understand exactly what you mean. The "sections" category is just to have a place to put parts of the text and parallels thereof. For instance, in the Mahapajapati parallels I do now, the story of Mahapajapati is just part of the whole KV20. And it has its own distinct parallels. Verses are another example, which have their own parallels. This has to be a separate "sections" category because it's important to have an anchor you can use to iterate over when converting the JSON file via Python into entries in the Arango Database. The output on the website is exactly the same as it is now.

I also recall that part of these "sections" comes from legacy: things that Rod Bucknell added at the time for paragraphs with parallels.

my point is simply that the difference between "full" and "sectional" is as much dependent on editorial choices as anything meaningful in the text.

It's more the difference between "resembling" and "sectional" can be an editorial choice. Sometimes it is very clear, like in the case of a verse. You don't want to say that long sutta A and long sutta B are resembling parallels because there happens to be one verse in it that is the same. But you can say that A#versenr1 is a full parallels to B#versenr2.
And like I said before, you need something like the "sections" category to have a handle for the iterations. Where else to put this verse parallel? The main point of suggesting this structure is to have everything from a single sutta together in one JSON object/dictionary (things between {}), rather than having multiple objects that are about the same sutta (as is the case in the old structure).

I can't recall why we're using # in these files.

It doesn't really matter what is used, as long as the backend knows how to parse it. So it has to be something unique that is clear. I feel a ":" would get confusing with the ":" that are used as part of the JSON structure itself.

I recall we used # because ultimately on the frontend it has to become a link in the form of https://suttacentral.net/mn10/pli/ms#44.1, which opens the sutta at that specific point. But it does not matter what the JSON file shows, as long as in the SC backend we convert whatever sign is used in the JSON back to #.

Another thing we have to keep in mind is ranges. Right now we have ranges of paragraphs within a sutta like this "mn37#1.1-#3.1". The doubling up of the # sign was done in order to make a difference with sutta-ranges, like mn37-mn38. So it was done for ease of parsing because we have so many different notations for the sutta numbers itself, that it became difficult to parse if we didn't do this.

How easy is it to transform the old data to the new format?

Very easy

How can we mesh the new format with the existing SC code?

Needs to be done in the SuttaCentral backend. I'm a bit hampered here because I won't be able to test the code because my computer is much too old for that. So it would have to be @ihongda who does that. I can help by writing the code but it then needs to be tested.

I don't want to lose momentum on the parallels work

Yes, I'd rather do what I can first before conversion. Note that I don't have the skills to do ALL the parallel issues. (I don't speak Tibetan and the Tibetan texts are not there, for many there are no references on where to find the parallels, etc.)

Obvious steps would be to review the work of Vens Analayo and Dhammadinna and see what can be added

Yes, agreed, but that will take a long time; they published a lot and many of their sources are not on SuttaCentral so have to be added. I've been working on the parallels from just the Mahapajapati story that he points out and it is complex, especially since suttas (at least their suttaplex cards) needing to be added to the structure, which needs changes in sc-data and in bilara-data. I've been working on just that that for a number of days now.

I would say the main work is the changes to the structure needed before parallels can be added, and the misunderstanding that you only add a parallel once. I've been working with @winterdharma on some of the Chinese parallels and that took some time to figure out which parallels should go where. And this was all work BEFORE adding it to the JSON file. Once it was clear WHAT needs to be added, it takes me 5 minutes with the current JSON.

where a user can add new parallels, and the updates are either added automatically if they can be, or the user is presented with relevant options to add by hand.

Of course a UI that can do that is ideal. But that would not necessarily need a new JSON.

The JSON is only something that is used to talk to the SC backend, which then converts it into the data in the Arango Database. Once there is a UI, the JSON editing would be redundant because the data goes straight from Bilara into the Database or Bilara generates a JSON that is then read by the SC backend. So what that JSON looks like is then pretty much unimportant.

All the work I do now is:

• Editing JSON files for structure in sc-data
• Creating new JSON files for segmented files in bilara-data
• Editing JSON files for root and english site management (header texts for suttaplex cards and blurbs) in bilara-data
• And after all that, I add the parallels in the parallels.json file, which is the least amount of work

As and example, please see my recent pull requests for the Mahapajapati parallels: https://github.com/suttacentral/sc-data/pull/370/files and https://github.com/suttacentral/bilara-data/pull/4903/files. It's a total change of 19 files, of which only one is parallels.json
And to show you how much work it is just to get through one of Bhante Analayo's books, here are the parallels from the same book that I haven't done yet! (let alone that you have to read the book entirely to be able to pick out the parallels): #371

Saying that, of course when people want to add parallels for already existing files (you only need a handle for the suttaplex card) in SuttaCentral, it's pretty easy but a better, more intuitive, JSON structure would be ideal. If there is a UI to do that, the JSON structure doesn't matter.