Proper grandstaff detection

[liebharc]

homr processes sheet music in four stages:

1. detect individual staffs
2. group staffs connected by a brace
3. merge paired staffs into grandstaffs
4. run symbol recognition on the resulting staff images

Step 3 is weak. Piano notation uses a grandstaff (upper + lower staff). Running recognition on each staff separately reduces accuracy because the model loses vertical context across both staffs.

The task: implement a reliable way to decide whether two adjacent staffs form a grandstaff or are independent voices.

Current state: grandstaff creation is done blindly.

Possible direction:

• Use brace detections from the segmentation model.
• Match brace bounding boxes to staff vertical ranges.
• Derive stable pairing rules from brace-staff alignment instead of fixed ordering.

Relevant code:

• _create_grandstaffs in brace_dot_detection.py
• create_grandstaffs in model.py

This issue has an example: #60

[fablau]

Thanks for posting this.

As a side note about my posting #60, I just tested my three-stave example, and it worked very well.

I am wondering what other cases involve the perfection of that weak step? Do you have any examples to work on?

[liebharc]

I good result would be a somewhat reliable detection if two staffs are a (piano) grandstaff or individual staffs for different types of sheet music: Grandstaff only, grandstaff before another voice, after another voice, or somewhere in between an orchestra piece with many voices.

I don't have more examples for this. You might find some in the OpenScore Lieder dataset. But more likely creating a dataset should be the first step for anyone working on this. The algorithm for this would need to work on pictures takes with a camera as this is the goal of the homr project. If camera images aren't available then perhaps data augmentation can be used.

All this work will be in the staff detection part. This part uses more traditional Computer Vision with operates on the results of a segmentation model

[fablau]

Got it. Thanks!

[fablau]

I good result would be a somewhat reliable detection if two staffs are a (piano) grandstaff or individual staffs for different types of sheet music: Grandstaff only, grandstaff before another voice, after another voice, or somewhere in between an orchestra piece with many voices.

I don't have more examples for this. You might find some in the OpenScore Lieder dataset. But more likely creating a dataset should be the first step for anyone working on this. The algorithm for this would need to work on pictures takes with a camera as this is the goal of the homr project. If camera images aren't available then perhaps data augmentation can be used.

I see the training package relies on the deepscores dataset as well. That dataset includes many full, orchestral scores. Hasn't the model already been trained on those scores?

[liebharc]

Deepscore is used for training of the segmentation model. If the model needs to be retrained for this task then I would definitely use it. It might however be that the task simply requires to use information which the current model already provides (the --debug flag writes the segmentation output to disk, so someone can check), if so then I would avoid using the deepscore dataset again. Since the dataset is used in training the segmentation model will of course perform much better on it than on other images.

The path I described as "possible direction" in the original description would not require a retraining, as the braces are already there.

[fablau]

Deepscore is used for training of the segmentation model. If the model needs to be retrained for this task then I would definitely use it. It might however be that the task simply requires to use information which the current model already provides (the --debug flag writes the segmentation output to disk, so someone can check), if so then I would avoid using the deepscore dataset again. Since the dataset is used in training the segmentation model will of course perform much better on it than on other images.

The path I described as "possible direction" in the original description would not require a restraining, as the braces are already there.

Good point! I'll review everything and discuss further if needed. Thanks!