LightGlue is now available in 🤗 Transformers

[NielsRogge]

Hi there,

The LightGlue model is officially integrated in the Hugging Face Transformers library thanks to @sbucaille 🤗

It enables easy inference in a few lines of Python code, using pip install tranformers.

Note: you'll have to do pip install git+https://github.com/huggingface/transformers.git for now as the model is brand new at the time of writing.

Resources

• Model weights are here: https://huggingface.co/ETH-CVG/lightglue_superpoint
• Docs: https://huggingface.co/docs/transformers/main/en/model_doc/lightglue
• Demo for you to try: https://huggingface.co/spaces/ETH-CVG/LightGlue

[Phil26AT]

Great news, it's exciting to see LightGlue in transfomers! Thank you @sbucaille for making this possible.

I added some links and credit attributions to the README.

[sbucaille]

@Phil26AT you might not have noticed but batching is supported end to end in this version aswell, kinda proud of that 😁

[Mollylulu]

@Phil26AT you might not have noticed but batching is supported end to end in this version aswell, kinda proud of that 😁

hi~ I wonder what does the "batching" here mean?

[sbucaille]

@Phil26AT you might not have noticed but batching is supported end to end in this version aswell, kinda proud of that 😁

hi~ I wonder what does the "batching" here mean?

Hey @Mollylulu , thanks for asking !
Originally, in SuperPoint, images are dealt with one by one. Each image produces potentially different number of keypoints. So the output tensors are not "concatenable" as is.
SuperGlue (and LightGlue) were built on top of that the same way. Image pairs are dealt one by one, all layers only see one set of keypoints being matched with another.
In my implementation in transformers, I applied padding, similar to how LLM's treat inputs with different token lengths. So now if you use LightGlue with multiple pairs of images, all of them pass through the model at once. This way you can benefit speedups using GPU's.
Let me know if you have any questions !

[Mollylulu]

Thanks for the prompt and detailed explanation! Great work！