[C++] Move to C++17

[asfimport]

The upcoming abseil release has dropped support for C++11, so {}eventually{}, arrow will have to follow. More details here.

Relatedly, when I tried to switch abseil to a newer C++ version on windows, things apparently broke in arrow CI. This is because the ABI of abseil is sensitive to the C++ standard that's used to compile, and google only supports a homogeneous version to compile all artefacts in a stack. This creates some friction with conda-forge (where the compilers are generally much newer than what arrow might be willing to impose). For now, things seems to have worked out with arrow specifying C+\11 while conda-forge moved to C\+17 - at least on unix, but windows was not so lucky.

Perhaps people would therefore also be interested in collaborating (or at least commenting on) this issue, which should permit more flexibility by being able to opt into given standard versions also from conda-forge.

Update:

It was voted on the dev ML to move to C++17:

• vote announcement:
  https://lists.apache.org/thread/h9v83rwdl015z2j6s8zwdr1qp4svb5j8
• vote result:
  https://lists.apache.org/thread/dod96gbqtfz7pf096vhlczq6f5hv81z8

Reporter: H. Vetinari

Subtasks:

• [C++][CI] Mandate C++17 instead of C++11
• [C++] Remove pre-C++17 compatibility measures
• [C++] Adopt C++17 idioms and features for maintainability

Related issues:

• [C++] C++14 laundry list (supercedes)

_{Note: This issue was originally created as ARROW-17110. Please see the migration documentation for further details.}

[asfimport]

David Li / @lidavidm:
We still have to support down to GCC 4.8 (for some older R versions at least) so this will be a while coming. And even then I think C++14 will be the highest attainable version

[asfimport]

H. Vetinari:

We still have to support down to GCC 4.8 (for some older R versions at least) 

Where is the lower bound of R support defined (and how/why)? I tried looking but couldn't find anything. I think it'd be a good idea to define some sort of support policy (note, we did this in scipy over the last year or so, allowing us to move from 4.8 to 6.x and now to 8.x).

And even then I think C++14 will be the highest attainable version

Barring some progress on the lower bounds for compilers, you'll then be limited from upgrading abseil beyond 20220623.0 (and that's already working more or less by accident since 20211102 on unix is compiled with C+\17 in c-f. Unless we introduce multiple builds per CXX version in conda-forge, this problem will only get worse, because once vc142 becomes the minimum toolchain in the not too distant future, c-f can move to C\+17 also on windows globally).

[asfimport]

Neal Richardson / @nealrichardson:
Slight correction: GCC 4.8 is not an R requirement. It comes from CentOS 7, where it is the default compiler. We do happen to get a number of bug reports from R users on CentOS 7, though.

R 3.6 on Windows used an odd gcc 4.9 mingw compiler, and that's the main source of "R requires an old compiler". But we already disable many features on R < 4.0 on Windows, and conditionally disabling more is not a problem. (GCS filesystem support, which uses abseil, is one of those already.) We could drop support for R 3.6 now, but since we can just disable features on the build, we haven't been forced to do so yet.

CRAN checks are now all running gcc 8 or newer: https://cran.r-project.org/web/checks/check_flavors.html

We have CI that builds arrow on C+\17 (and maybe also 14?). I think Homebrew also bumped up building arrow with C\17 to match abseil (or maybe that's still in the copy of the formula we test in apache/arrow). We also have an open PR to add Azure Blob Storage, which will require C\14: https://github.com/apache/arrow/pull/12914/files#r899724290. So maybe the solution for the abseil issue is to require the newer C\+ standard if using abseil built with it?

[asfimport]

David Li / @lidavidm:
Thanks Neal - yeah I realize I read that backwards now. If we just need to build with 17 on conda-forge/when using newer Abseil that shouldn't be a problem (we'd have to update various pipelines/scripts), we just can't raise our minimum supported version.

[asfimport]

Antoine Pitrou / @pitrou:
See also ARROW-12816.

I agree that for now conda-forge can simply build using C+\17. Just before the minimum version for Arrow is C\+11 doesn't mean you are forbidden to use a newer one :-D

[asfimport]

H. Vetinari:

Slight correction: GCC 4.8 is not an R requirement. It comes from CentOS 7, where it is the default compiler. We do happen to get a number of bug reports from R users on CentOS 7, though.

Centos 7 has the devtoolset backports until GCC 11 (except aarch where it's GCC 10) though... These are obviously available & in use for the manylinux images, and I think they're a very much acceptable requirement for users on such old platforms.

I agree that for now conda-forge can simply build using C+\17. Just before [because?] the minimum version for Arrow is C\+11 doesn't mean you are forbidden to use a newer one :-D

Well, I would like to avoid breaking your CI if possible. :)
And as I tried to explain, if conda-forge switches to C+\17 (especially on windows) while you still try to compile with C\+11, breakage is all-but-guaranteed.

PS. I hate the JIRA text parser mangler with a burning passion :O

[asfimport]

David Li / @lidavidm:
Thanks. So let's make this ticket, "Make all conda-forge based CI pipelines specify C++17"?

[asfimport]

Weston Pace / @westonpace:
It probably seems good for this ticket to focus on conda-forge but are the devtoolset backports a workable solution? If so, it would be nice to update the minimum version as well.

[asfimport]

Antoine Pitrou / @pitrou:
The devtoolset backport won't do anything for the gcc 4.9 requirement on R Windows builds, I'm afraid.

[asfimport]

H. Vetinari:

The devtoolset backport won't do anything for the gcc 4.9 requirement on R Windows builds, I'm afraid.

What shape does this requirement take? The defining feature of the devtoolset backports is that they're fully ABI-compatible with the default compiler (i.e. 4.8), and I doubt R hard-depends on the presence of specific bugs in GCC 4.x that were fixed in later versions.

[asfimport]

David Li / @lidavidm:
The R Windows builds use a distribution of MinGW, not Centos: https://cran.r-project.org/bin/windows/Rtools/history.html

[asfimport]

Kouhei Sutou / @kou:
Can we avoid depending on Abseil ABI by removing Abseil use in cpp/src/arrow/filesystem/gcsfs_internal.cc and cpp/src/arrow/filesystem/gcsfs_test.cc?

[asfimport]

Antoine Pitrou / @pitrou:

Can we avoid depending on Abseil ABI by removing Abseil use in cpp/src/arrow/filesystem/gcsfs_internal.cc and {}cpp/src/arrow/filesystem/gcsfs_test.cc{}?

AFAIU, the problem is that the GCS C++ headers use Abseil, so we depend on the ABI no matter what. @coryan Is my understanding right?

[asfimport]

H. Vetinari:

The R Windows builds use a distribution of MinGW, not Centos: https://cran.r-project.org/bin/windows/Rtools/history.html

Sorry, I read too quickly; should have been obvious that CentOS has nothing to do with windows.

However, the good news is that as soon as you drop support for R<4, the lower bound should be able to move up to GCC 8 (in rtools40; for R 4.0 & 4.1) resp. GCC 10 (in rtools42; for R>=4.2).

[asfimport]

Carlos O'Ryan / @coryan:

AFAIU, the problem is that the GCS C++ headers use Abseil, so we depend on the ABI no matter what.  @coryan  Is my understanding right?

That is correct, google-cloud-cpp uses Abseil in headers.

FWIW, gRPC now requires C++ >= 14:

https://github.com/grpc/proposal/blob/master/L98-requiring-cpp14.md
https://github.com/grpc/grpc/releases/tag/v1.47.0

so does google-cloud-cpp:

googleapis/google-cloud-cpp#8740
https://github.com/googleapis/google-cloud-cpp/releases/tag/v2.0.0

I expect that Protobuf will follow suit sooner rather than later.

The following policy, while yet adopted by any of these projects, may be informational:

https://opensource.google/documentation/policies/cplusplus-support