Request for comment : Possiblly Removing length limit of array

[Xyncgas]

There's gotta be a way for array to be able to store an unlimited amount of data with one final exception that say it can not, so https://source.dot.net/#System.Private.CoreLib/src/libraries/System.Private.CoreLib/src/System/Convert.cs,e54b8e5a7f9d845a doesn't have to throw an exception here for example

If we make byte[] to have unlimited amount of bytes it means we can use more than 2^32 bytes in 32bit system, it's possible we just gotta implement array differently

what's other people's opinion here

[huoyaoyuan]

See: #12221

The size limit of string actually comes from GC.

doesn't have to throw an exception here for example

The ASCII string will always consume twice memory of byte array. Since there will always be a hard limit of object size (even if Int64.MaxValue), arrays larger than half of the limit can't be converted to hex string ever.

If we make byte[] to have unlimited amount of bytes

There's no unlimited in computer space. There can only be a very large limit.

we can use more than 2^32 bytes in 32bit system

I don't know this is possible with in a single process.

To summarize, the most important reasons that 64bit array isn't easy:

• GC can't move such giant object. Allocating in a dedicated pinned object heap will be similar to allocating unmanaged memory.
• foreach over array is lowered to use int index, and people are using int in for loop too.

[Clockwork-Muse]

we can use more than 2^32 bytes in 32bit system

I don't know this is possible with in a single process.

Getting that much contiguous memory can be a problem, even on a 64-bit system. Even outside of .NET/GC constraints.

Even on a 32-bit system, there's ways of virtualizing memory so that applications can access more than 4GB of memory. If the OS doesn't directly support it via PAE or similar, this might require some lower-level hacking, but it's there.

In almost all cases there's better ways to handle the situation, though.

[Xyncgas]

Everyone of you are right, I really want byte array to have unlimited length not physically unlimited but, I'm gonna figure out a way one day

[Clockwork-Muse]

.... Is there a reason you want to do this? Or just for bragging rights?

[Xyncgas]

Please don't say that to me, you don't know me like I do, let me explain bit more, imagine array is unlimited size, assumptions of 32 / 64 bit wouldn't exist, if in the future we get 128 bit addresses people automatically can use an array that big and if one day people implement a way to use a bunch memory rather than only 2^32 addresses you also automatically get that and if you don't no big deal throw exception when it runs out but we don't say array is 32 bit long we don't want to put that on the table it looks embarrassing if you tell me your array can only hold 2^32 elements I would be embarrassed for you

[huoyaoyuan]

people automatically

The practical way is to use nint to access lengths, like size_t in C++, but the whole ecosystem is using int.

I'm very against the usage of the word "automatic". There will always be unforeseen side-effects for solutions that looks fine when created.

array can only hold 2^32 elements

The most important thing I'm always pointing out in memory-length related issues is that, the way people accessing arrays can only access 2^31 elements.

[Clockwork-Muse]

The practical way is to use nint to access lengths, like size_t in C++, but the whole ecosystem is using int.

FWIW, there's a sizable portion of the C++ community that thinks the unsignedness of size_t was a mistake. And in any case, it's rare that just doubling the available size at that point is likely to be sufficient.

if in the future we get 128 bit addresses people automatically can use an array that big

From a practical standpoint, it's unlikely you'd ever want that much contiguous memory in the first place - the only reason to do so is if you were planning on iterating over the entire thing with one thread, but the amount of time it would take would be exorbitant. As opposed to using multiple threads, at which point you might as well have multiple segments/arrays. There are other practical problems with simply accessing that much memory in general, never mind getting a contiguous span.

Note that 128-bit addresses have been around for decades, at least in terms of language support (the venerable System/36), regardless of whether the underlying hardware actually supported addressing that many contiguous bytes (in the case of the System/36 and its followers, it did a lot of stuff over virtualized memory).