Clustering and cluster classification or node classification and link prediction

[maiiabocharova]

Thank you a lot for a wonderful library!

I am new to GNN, but I want to use graphs for document layout analysis problem

I want to consider each document as a graph, each line as a node. Nodes features are going to be properties of the lines. The edges features would be distances to the closest neighbouring lines.

Can you please advise me how to approach this problem? I am not even sure how to formulate the task - should this be first clustering and then classification of the clusters? Or should it be classification and link prediction? x

I am sorry if my questions are stupid.

[rusty1s]

What is the task that you are trying to solve? I think the graph representation you are aiming for sounds good. Here is some tutorial that hopefully gets you going :)

[maiiabocharova]

Thank you a lot! Actually the first part is already done - I have created a dataset with nodes as lines (words which are very close together horizontally), their edges and features of both nodes and edges.

The task I am trying to solve is layout analysis - detecting columns, segmenting the PDF page etc. Document understanding is also my PhD topic. I tried SOTA Computer vision and vision+text approaches, but they did not give the desired results on more challenging data, where columns are close together. So afterwards I created a script based on networkx library which iteratively merged words and lines to form separate blocks. The results were very promising, so I want to get started with GNN for Document layout analysis to do it with ML. So actually main problem is to decide for which architecture to go and how to produce grouping with ML and not based on consecutive merging and rules.

[rusty1s]

Sounds wonderful. I am happy to help if you have any questions regarding implementation.

[maiiabocharova]

Sounds wonderful. I am happy to help if you have any questions regarding implementation.

I would very much appreciate advice - I though a lot about most straight forward way for this approach:
The dataset will look as follows:

1. nodes.csv
   node_id, features (as features I want to try in 2 different settings - one of the settings if to take representation for each line produced by LayoutLM v3 (which looks at both text&image)

2. edges.csv
   node_scr, node_dst, x_distance, y_distance, label
   With label here I will identify the fact should the lines be merged to form a block or not

And first I want to train a model to predict the edge label (so just edge classification. Which is going to be binary classification)

And second model will solve the node classification task.

(I know it is very computationally expensive, but I am doing research so this is the first setting which I want to try as prove of concept that graphs could be successfully applied for solving document layout analysis problem.)

Can you please advise which network better to look at? And also from your impressive experience with graphs can you give some advice about edge features - should I normalise distances (say 0-100) and use one-hot encoding?

[rusty1s]

I think you should start with the SplineConv model. In my experience, this one works pretty well for any spatially embedded graphs. For this, you utilize edge_attr = [x_distance, y_distance] and need to normalize edge_attr to the interval [0, 1] for each graph.

[maiiabocharova]

Thank you a lot!