"Short" form of token shape

[kinghuang]

The Token shape attribute shows the orthographic features of the token's string. Sequences of the same characters are truncated after length 4.

Sometimes, I am more interested in something that generalizes the shape of the text even more. For example, say I have the following text.

The product name is My Great Product™.

Given the token for Product, I want to scan previous tokens that share the same basic shape. The current shapes for My Great Product are Xx Xxxxx Xxxxx. I can further truncate the shapes by doing something the following.

token_shape = "Xxxxx"
short_shape = "".join(dict.fromkeys(token_shape))

This truncates the shapes of those tokens to Xx Xx Xx, making it possible to correlate the shapes of the three words.

I would be interested in a Token attribute that acted like the current shape, but truncated sequences of the same character after length 1, instead of length 4. Something like a short_shape/short_shape_ attribute.

Text	Shape	Short Shape
Product	Xxxxx	Xx
Hi	Xx	Xx
2,000.02	d,ddd.dd	d,d.d
2000.02	dddd.dd	d.d
3.14159	d.dddd	d.d

Your Environment

• spaCy version: 2.3.2
• Platform: Linux-4.19.121-linuxkit-x86_64-with-glibc2.2.5
• Python version: 3.8.6

[svlandeg]

I see your point, but isn't this easily achievable with some of the code you've already written, and accessing the results from a custom token attribute?