adapt macrosynteny blocks services to provide gene correspondences within blocks (as a separate service or differently formatted version of existing)

[adf-ncgr]

behind the scenes of the pairwise_macro_synteny_blocks microservice, the blocks are constructed from a pairing of individual genes belonging to the same families and a DAG-chainer inspired algorithm for tracing the pairs producing collinear blocks. The current implementation basically just wants the endpoints of the blocks (for graphical display purposes) but in some contexts actually knowing the gene pair correspondences behind the blocks would be useful.

An example is given by Jbrowse using output from MCScan which is similar to DAG-chainer:
https://jbrowse.org/code/jb2/latest/?config=test_data%2Fconfig_dotplot.json&session=share-nqAhsTDpDM&password=CKdgg

In that session, there are two linear synteny displays, the first is using the "simple anchors" format which is basically similar to our blocks, just giving the endpoints for the blocks, but no details for internal correspondences. The second uses the regular "anchors" which provides each pairwise correspondence between genes (or proteins) as an individual element. It's something along the lines of the latter that I'd like to see added.

@matthewwiese let me know if you want to discuss any of that further.

[matthewwiese]

@adf-ncgr I have a very rudimentary "working" version of this but I am not in any way confident in its biological accuracy - are there legume JBrowse examples with data readily available that I can use to mimic in our microservices?

[adf-ncgr]

Great, that was fast! You could use a GCV legume instance in microsynteny mode to verify correspondences, although there could be some minor discrepancies due to the fact that the algorithms behind macro and micro are different. We also have some files where the MCScanX algorithm has been run in a way that is similar to how the GCV macrosynteny works; if you prefer I could point you to those.

[matthewwiese]

@adf-ncgr Yeah the MCScanX data would be helpful - some bed files I could use with the existing JBrowse MCScanAnchorsAdapter would provide a good comparison to sanity check my results to start

[matthewwiese]

@adf-ncgr I looked over my changes for this some more and felt confident enough to push it to the macro-synteny-paf branch. You can play around by pulling the latest commit, as well as the one for the JBrowse plugin.

The core change that facilitated everything else is this diff to the pairwise service's request handler (plus the addition of another Redis pipeline):

             if end in pointers:  # note: singletons aren't in pointers
                 if scores[end] < matched:
                     break
-                begin = end
-                while begin in pointers:
-                    begin = pointers.pop(begin)
+                # Collect the full path during traceback
+                path = [end]
+                current = end
+                while current in pointers:
+                    current = pointers.pop(current)
+                    path.append(current)
+                path.reverse()  # Start to end order
+                begin = path[0]
                 length = scores[end] - scores[begin] + 1
                 if length >= matched:
-                    yield (begin, end)
+                    yield (begin, end, path)

We now append to a list all the intermediary nodes instead of just keeping the beginning and end. If you could send along some other data to compare against I'd appreciate it.