Is it possible to use Swift for the IDNA support?

[MahdiBM]

This is mostly a question to see if there is such interest / need at all.

The whole thought was also triggered by this issue: #1560
(I'm not saying solving that issue will require to use this new library I'll be talking about).

I recently developed swift-idna, which is also very well tested with the help of the Unicode test suite, so I'm confident in the correctness of the code there and that the library could possibly be used in such a widely-used package as swift-foundation.

I'm not that familiar with swift-foundation, but I see there are some ICU being used for handling IDNA.
My question is, for those who are familiar with swift-foundation, that is there a possibility of moving to using swift-idna at all? What would be missing for such a move?

It's 100% fine by me if this is a no-go. I'm just asking and not trying to force using that new library 😅

The biggest issue I see is that swift-idna while having benchmarks, does not have benchmarks to benchmark itself against the current foundation implementation which uses ICU, so I'm not sure how the performance comparison will be.
Frankly, as things currently stands, I expect the current swift-foundation to be faster although I don't know for sure. But I'm confident I can make things fast enough.
For example see the swift inet_ntop/pton implementation in swift-endpoint, where I eventually noticed my implementation is actually faster than Darwin and mostly faster than Glibc C APIs (With no hidden downsides to the Swift implementation).

[MahdiBM]

@jrflat I can see you're the one who has recently worked with the IDN-related codes.
I'm curious to know what's your opinion whenever you have time 🙂.

[jrflat]

Hi @MahdiBM! I'm excited to see your Swift implementation of IDNA/punycode! Re: #1560, I think this will be simple fix in the current swift-foundation code that wraps the ICU functions.

Overall, I don't think we'll be looking to adopt swift-idna at this time since the current path is well-tested. However, I'm happy to help test some benchmarks of swift-idna vs. the current ICU functions we're using in URL. If swift-idna shows significant performance benefits over the current implementation, we could re-visit the question of adoption.

I'll play around with benchmarking this week and let you know what I find! 😄

[MahdiBM]

@jrflat happy to hear there could be a way.

To be clear I think right now we're just discussing to confirm if such a thing can come to happen at all. If we think it can, then I can make a forums post about it if that's needed to gather community feedback, although I think for the most part the additive changes go through the forums, not an implementation-detail change like what I imagine this would be.

About performance, I'd be really curious to have an idea how my implementation compares to the implementation currently used in swift-foundation.

I think there are 3 possible factors other than pure algorithmic performance here, 2 help swift-idna, the other one doesn't.
I have not confirmed any of these. It could be that all 3 of these are non-factors:

• 1. swift-idna is a purely Swift implementation which might mean it'll be able to skip some memory operations or such, since the data is already in Swift's String, and not some kind of C type. I saw such an advantage when comparing swift-endpoint's IPAddress encoding methods against inet methods. The allocation counts were still the same, but I highly suspect there was a big conversion cost (or who knows, maybe my implementation was just 10 times more efficient than Glibc, 20 times than Darwin, and there were no excessive conversion costs).
• 2. Unfortunately some APIs required to properly implement IDNA where only recently exposed publicly in Swift. Not that I can't go hand-write them, but I'm not excited to reinvent APIs just because they are/were not exposed. More specifically, the UTF8Span type really helps a lot (although doesn't solve all API shortcomings), but that API is only available on macOS 26+.
     This comes with some issues. First of all, well, the API is unavailable below macOS 26. Second of all, when I implemented some backward compatibility hacks, I noticed this issue with using #available attributes.
• 3. A lot of times a domain name doesn't really need to be touched, or only needs to simply be converted into lowercased, instead of going through the whole idna conversion. swift-idna detects those and skips conversions in those cases. From what I see in the current code, this is not really taken into account and the code just does the conversion. This could help a ton, specially for the 99% of the cases where a domain name is just simple ASCII. Though it's also worth noting that even if it does provide some advantages to the swift-idna implementation, it can still be implemented in swift-foundation without the need to pull in swift-idna. In that case, my major concern would be about testing. swift-idna tests those short-circuits against the massive Unicode test suite to ensure correctness. I remember some time in the past I messed up the implementation only for it to be later caught by the test suite.

The #1 and #3 points might be non-factors if swift-foundation is doing something that ends up not introducing much overhead when interacting with ICU, and if ICU is also applying those short-circuits as well.
The #2 point might be a non-factor if swift-foundation is simply fine with requiring macOS 26, considering it's a package that's included in the toolchain.

I also understand that regardless of correctness, we'll need to keep the behavior the same. For example IDNA allows for some configuration options, and we'll need to make sure both the swift-idna code and the current code are using the same configuration options if swift-idna is to make it to swift-foundation.

I would have offered a possibility of "merging" these types into swift-foundation, but we need swift-idna in swift-endpoint, and swift-endpoint might in year or so become some kind of official package (that is still in the very early days, might not come to happen), and then the ecosystem will have to maintain 2 different IDNA copies which would be suboptimal.

In the end I'd be happy if you could provide an example of encoding to and decoding from IDNA domains, as well as some normal domains that are not supposed to be affected, and then I can add those to the benchmark suite that swift-idna has, like what I did in swift-endpoint with the inet functions. That way I can keep track of the performance diff between the implementations and I can just run the benchmarks in the repo's Linux CI (uses a dedicated core VM. The results are reliable/reproducible).

Also for the record, this is what I had to say ~1 month ago when I last tried to optimized swift-idna: https://forums.swift.org/t/pitch-standard-network-address-types/82288/15
I know a handful of things that could be improved in swift-idna, and I didn't go for them because 1- they required even more backward compatibility hacks and 2- would take some time, and at that point I had already spent a good amount of time on swift-idna and was trying to wrap things up for the time being.

[MahdiBM]

I also understand that regardless of correctness, we'll need to keep the behavior the same. For example IDNA allows for some configuration options, and we'll need to make sure both the swift-idna code and the current code are using the same configuration options if swift-idna is to make it to swift-foundation.

For compatibility checking, I can run the whole Unicode test suite against both swift-foundation and my implementation and ensure they output the same result.
That should guarantee that swift-foundation's behavior won't change after using swift-idna.