RFC on get_marginals

[bertsky]

Seeing problems (like this) in the results of get_marginals, I am trying to fully understand the code and the intents behind it. Ideally this will flow into a PR...

[bertsky]

eynollah/src/eynollah/utils/marginals.py

Lines 102 to 105 in a2359ea

	try:
	mask_marginals[:,point_left:point_right]=1
	except:
	mask_marginals[:,:]=1

This seems wrong: when neither point_left != first_nonzero nor point_right != last_nonzero, then no further marginalia detection should be attempted. (The above link shows exactly this state: the entire text mask is regarded as marginalia, but due to small differences caused by deskewing, only a narrow stripe survives, which in turn completely spoils textline segmentation...)

Also, why do you allow marginalia on both sides here? Shouldn't you try to limit the range to the "best" (in the sense of most prominent peak) side only?

[bertsky]

eynollah/src/eynollah/utils/marginals.py

Line 131 in a2359ea

polygon_mask_marginals_rotated = return_contours_of_interested_region(mask_marginals,1,min_area_text)

This seems wrong: mask_marginals is already deskewed (because it is based off text_with_lines, which was already deskewed before calling get_marginals), while text_regions is not. Your mask_margins_rotated therefore undeskews the image, so the result is in the same shape as text_regions. (Of course, it is not really, exactly in the same coordinate space, because rotating back does not undo the coordinate offset caused by enlarging the image – to the contrary, it increases the offset even further.)

So my point is: why not use mask_marginals_rotated here?

[bertsky]

(Of course, it is not really, exactly in the same coordinate space, because rotating back does not undo the coordinate offset caused by enlarging the image – to the contrary, it increases the offset even further.)

Actually, that side point was invalid: since your rotate_image does not enlarge the image (as it should, if the corners are not to be cropped off), the coordinate space always stays the same, and rotating back will end up in the same coordinates (just without the corners in the image).

[bertsky]

eynollah/src/eynollah/utils/marginals.py

Lines 38 to 43 in a2359ea

	if thickness_along_y_percent<30:
	min_textline_thickness=8
	elif thickness_along_y_percent>=30 and thickness_along_y_percent<50:
	min_textline_thickness=20
	else:
	min_textline_thickness=40

These min_textline_thickness constants seem to be calibrated for some fixed image resolution, but text_with_lines actually varies in size. Shouldn't this be relative to the image size?

[bertsky]

eynollah/src/eynollah/utils/marginals.py

Line 72 in a2359ea

peaks, _ = find_peaks(text_with_lines_y_rev, height=0)

For robustness, IMO the height / width / prominence / distance of peaks should also be constrained (or at least considered on the output side).

Moreover, IMO locally-minimising the vertical sum of the text mask is much less reliable than locally-maximising it. Take this example:

Here, Eynollah classified both the small text patches on the left of the book spine and the page number and catch word on the right edge of the page as marginalia. (Like I said above, get_marginals should first decide which side the marginalia is located at, if any.) Disregarding the fact that these are not actually marginalia in this case (because they are page number and catch word), the right side should clearly win here.

And this is a plot of the internals leading up to that decision:

The upper left shows the (deskewed) text mask, with red lines indicating point_left / point_right, and green lines depicting first_nonzero / last_nonzero. The upper right shows the resulting mask_marginals (i.e. everything between point_left and point_right get marked). The lower left shows the signal which find_peaks is searching minima in. Comparing the two local minima left vs right of the main text, you can see that it is difficult to favour one over the other solely on its height (exactly zero) or width (just a few pixels). In contrast, compare the local maxima: the bulge on the right is much more prominent / wider / higher than the one on the left. So it would be more robust to apply find_peaks in that way (accompanied by suitable constraints on prominence / width / height). Then it could favour the right side.

[bertsky]

In contrast, compare the local maxima: the bulge on the right is much more prominent / wider / higher than the one on the left. So it would be more robust to apply find_peaks in that way (accompanied by suitable constraints on prominence / width / height). Then it could favour the right side.

A little less robust, but still viable IMHO seems to be using the product of prominence and height – as score to be maximised.

[bertsky]

Another problem I see in the code (but I still don't know if it really is one):
Looking at the call chain of run_marginals → get_marginals, the array image_page gets promoted (renamed) to image_page_rotated – without actually rotating it:

eynollah/src/eynollah/eynollah.py

Lines 4431 to 4433 in a2359ea

	textline_mask_tot, text_regions_p, image_page_rotated = \
	self.run_marginals(image_page, textline_mask_tot_ea, mask_images, mask_lines,
	num_col_classifier, slope_deskew, text_regions_p_1, table_prediction)

eynollah/src/eynollah/eynollah.py

Line 2939 in a2359ea

image_page_rotated, textline_mask_tot = image_page[:, :], textline_mask_tot_ea[:, :]

[bertsky]

The reason for this is actually that crop_image_inside_box is used when only the bbox needs to be converted, hence image_page_rotated gets dragged through lots of functions, without actually doing anything.

[bertsky]

@vahidrezanezhad, can you please take a look at the above points? I have a related PR with some improvements already in the works, but would like to have your opinion before I finalise and submit it.

(We can discuss it in video conference if that's preferable to you.)

Many thanks in advance!