Invalid links returned with some chinese characters as delimiters

[tommedema]

Steps to reproduce:

1. linkify the following text

【视频奇志大兵《发烧友》 在线观看 - 酷6视频】奇志大兵《发烧友》 在线观看，奇志 大兵 搞笑双簧 _ 发烧友 （追星族） http://t.cn/RZwjG7U（分享自 @酷6网）

1. the output link is

http://t.cn/RZwjG7U（分享自

whereas the output link should be

http://t.cn/RZwjG7U

The reason is that （ is not recognized as a separating delimiter, yet it is quite common in Chinese.

Out of 500 posts I gathered, about 20 to 30 of them had links like this, resulting in invalid links reported by linkify.

Note that I realize that these users are technically posting invalid URLs, but 20-30 out of 500 is very common and therefore there should be a way to deal with this. Any suggestion?

[puzrin]

Could you post a permalink to demo http://markdown-it.github.io/linkify-it/ ? You can type there all examples at once, with results.

[puzrin]

Example

Problem is in (, used in your example. There are NO space before it. I'm not familiar with asian punctuation. Are such ( always word terminators? Is is possible to have it in links, like in wiki?

[tommedema]

See http://markdown-it.github.io/linkify-it/#t1=http%3A%2F%2Ft.cn%2FRv4VRqQ%EF%BC%88%E5%88%86%E4%BA%AB%E8%87%AA%20%40%E7%88%B1%E5%A5%87%E8%89%BA%EF%BC%89%0A%0Ahttp%3A%2F%2Fhttp%3A%2F%2Ft.cn%2FR2t287H

I added another issue where a user mistakenly added http:// twice.

These are both (technically speaking) user behavior issues, but the first one especially is quite common.

Formally speaking ( is not a word terminator, however because it takes so much white space, many chinese people don't put a space in front of it (it's an act of being lazy, but it's quite common).

Perhaps I should do some string manipulation before I pass it to linkify-it?

[puzrin]

Example with double http:// is technically correct. That's like http://localhost - any local domain allowed. Imho such typo do not need fixes.

Example with ( is more serious. It uses chineese unicode scopes, and can be distinguished from english scopes. But i need recommendations about grammar to be sure.

[tommedema]

I'm sorry I'm not a Chinese speaker. Perhaps someone else can help out here. There are also other Chinese punctuation characters, like ，

[puzrin]

Let's leave it open until someone can formalize info for asian languages group (probably japan language has the same issues). I'm ready to fix as soon as possible, but would like to avoid kludges with inccomplete workarounds.

[tommedema]

More examples of how difficult these Chinese posts are to parse: link

What's your idea on this?

[puzrin]

Seems you given wrong link, it shows default text.

[tommedema]

Sorry, I've fixed the link now.

[puzrin]

Thanks for examples. Posted a question at commonmark forum http://talk.commonmark.org/t/linkifier-lets-discuss-and-test/1045/9?u=vitaly

As far as i see in last example, spaces are not used at all. It's possible to to track locale change (non-english -> english), but that's not safe.

[tommedema]

That's correct, spaces are not used. The Chinese use different punctuation marks or even don't use them because URLs are in English script and therefore the disctinction is easily visible to the eye, but more difficult for machines.

[puzrin]

What about links with chineese chars? I can find link start somehow by [ any non english ]http://, but can't use this rule to search link end.

PS. Anyway, it worth to start collecting test samples for china language separately. Something like this

[tommedema]

You're absolutely right. I think the first rule makes sense, but URLs can probably have chinese characters. Perhaps the first rule by itself would help though, and somekind of smart processing for certain delimiters that are unlikely to appear within an URL, e.g.:

str.replace(/[\uff08\uff09\uff0c\uff01\u3002\uff1f\u3010\u3011\uff3b\uff3d\u3001\u300a\u300b\u2605]/g, ' ') //（ ） ， ！ 。 ？ 【 】 ［ ］ 、 《 》 ★

[LeonLiuY]

Found this URL doesn't match:

http://markdown-it.github.io/linkify-it/#t1=http%3A%2F%2F172.26.142.48%2Fviewerjs%2F%23..%2F0529%2Fslides.pdf

BTW, I found a website comparing different regexps for URL:
https://mathiasbynens.be/demo/url-regex