Details how symbol detection is made

[mtalhabalci]

Hey,

I'm currently working on OMR with a focus on Turkish music notation, and I've been digging the oemer repo for a while now. I've also built a few proof-of-concept experiments based on it.

I noticed that apart from the common models like U-Net and SegNet, there are several custom ones in the repo—like clef.model, rests_above8.model, all_rests.model and sfn.model etc.. From what I gather, these are used to differentiate between various types of musical symbols.

Just wanted to ask—are you following the same logic for symbol differentiation in your implementation, or have you taken a different approach?
If it's the same, could you possibly share a bit more detail on how the training process works for those custom models—like how the datasets are extracted from ds2, structured, etc.?

Thanks indeed

[liebharc]

Hi, that sounds exciting. I'm happy to help.

As you likely know, oemer was the starting point of this repo as well. Some history, but I will be brief: I started homr because I was looking for an OMR algo which can work with pictures taken with a mobile camera. So images which are warped due to perspective or page geometry. Oemer performs well on this, and my idea was to further improve it by taking page warp into account at every stage of the detection (that's why homr uses angled rectangles pretty much everywhere, while oemer uses rectangles). At that time I created the models you found and I experimented with e.g. the clef model to tell clefs apart.

However rhthm is a big headache. It's very hard to detect the rhythm of a measure. There is a lot of variation and small details. I failed to update the algorithm of oemer to something which works well if symbols are warped and bent. At the time I found a transformer based model (TrOMR) which works really well even in difficult cases.

So today homr uses a staff detection which is based on the oemer models (unet, segnet). It then passes all the staffs in a transformer model which is based on TrOMR. It doesn't use classifiers for individual symbols anymore. In other words, the models you found are unused. The code base also still contains code which is unused and could be simplified.

OMR is quite a complex field and I don't have a great overview. In my experience the transformer based approach works much better than individual classifiers. But that's nothing more than my personal experience.

I'm not familiar with Turkish music notation. Assuming that is uses staffs then you could reuse the staff detection from homr.

If rhythm notation is as complex as for e.g. piano notation then you might want to consider the transformer approach. The training code of the transformer is part of this repo. You would likely need to define a new vocabulary and build a dataset to train the transformer.

[mtalhabalci]

Thank you for your quick reply,

I'll try to be clear. Turkish music has some microtones as we see in many eastern culture. For example, when you say give me "si flat" in european music therminology, we understand exactly which tone do you want. Because there's a certain note between la-si; si flat. But in microtonal music it's not that certain because there are more than one "si flats" (they have different names). You'll be replied which siflat do you want :)

In most of music scores we see 4 flats (for any note) mostly, and the symbols for these are different from each other. You can think sharps in the same way.

Let's say there are 10 more symbols compared to european music notation (but changes whole sensation).
I'm trying to include these symbols into the workflow.
This is a brief summary of what i am trying to do.

Note: Polyphony is not very much of my worry. Because in general there are scores with monophony.

So what understand from your answer is; training a new model with transformer approach would work for my case (with new dataset ).

By the way, I am willing to send you an e-mail for some technical details.

Thanks

[liebharc]

If I try to understand what this means for the OMR process itself, then it sounds like the detection must support additional accidentals (perhaps like the ones mentioned here https://en.wikipedia.org/wiki/Turkish_makam). Is that correct?

Assuming it's correct:

• The TrOMR model works with multiple decoders. The decoder for accidentals is one of the smaller ones, so I assume it wouldn't hurt the overall system performance too much if it needs to detect additional symbols.
• Rhythm can also be tough if it's monophonic, from the examples on the Wikipedia page I would assume that the transformer approach could be beneficial in your case. And that you can use the existing staff detection.

The steps to take might look like this:

• Assemble a data set, the existing data sets have around 100000 images of staffs. I don't know what the minimum number of staffs would be for the transformer to perform reasonable well on the new symbols.
• Update the training code and vocab (https://github.com/liebharc/homr/blob/main/homr/transformer/tokenizer_lift.json) to support the new symbols
• Retrain the model, possibly you could also try a fine-tuning approach which takes the existing model as basis and then requires less additional training time
• Update the musicxml generation code for the new symbols

This would still be a large project. Training the transformer takes about 3 days on my machine. After a training run you usually get disappointing results (at least I do) and then you have to think about possible reasons. Most of the time it's about improving the data set. And then you let it train for another 3 days. Repeat that a couple of times until you get good results.

Internally homr uses an own notation for music which is derived from TrOmr which again is derived from the PrIMuS dataset. You would likely need to learn that to extend it. I also try to move the internal notation to humdrum kern on one branch, but not sure if or when this will be completed.

And sure, you can always send me a mail. I think you find my address in the git commits.

[liebharc]

Hi, I haven't heard back from you. If you sent me a mail, then I fear that it was lost. I will close this issue for now. If you want to follow up, then just drop me a message.

[mtalhabalci]

Hey, hope you're doing well.
It's been a while. Sorry for the late reply. I finally managed to finish generating the dataset with the Turkish music symbols—barely.

I ran an algorithm that looks for appropriate empty spaces in the notes (images/segmentation/instance) and places the new symbols there, then writes the updated symbol coordinates into the related JSON file with the correct categoryId. I did this for the small part of the dataset (ds2_dense).
The full version takes quite a long time to generate, but I'll deal with that later.

I did a first training run and I think it went pretty well for an initial try :)
Now I'm going to dive into your code and try to understand how you handled everything—especially the transformer part. I might drop a few questions if I get stuck.

I'll leave the dataset link here in case you want to take a look. Any symbol you do not recognise is a turkish music notation symbol.

This is the simple result of the first try.

Greetings

[liebharc]

Hi, from what I understood you retrained the segmentation model? In homr the segmentation model is only used to detect the position of the staffs. And this part should work as well for Turkish music as it does for western music as the differences with e.g. more lifts don't matter for the staff detection. So before you spend more time on the segmentation, I would recommend you to look into the transformer datasets, as you will need to extend the transformer vocabulary and then add a new dataset for the additional symbols.

PS: I'm reopening the ticket for the moment as we are continuing this conversation