MacOS Xcode-10 tokend fails to build

[mouse07410]

With update of Xcode to Xcode-10, OpenSC.tokend can no longer be built. The main cause seems to be that Xcode-10 finally dropped completely support for the deprecated libstdc++.

Attempts to build as-is lead to libstdc++ not found. Switch to libc++ by using -std=libc++ message.

Switching the project (via Tokend.xcodeproj) from "libstdc++" (which no longer exists on the system) to "Compiler default" results in a lot of errors like

. . . . .
In file included from /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/exception:82:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/cstdlib:98:28: error: expected ';' after top level declarator
_LIBCPP_BEGIN_NAMESPACE_STD
                           ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/cstdlib:162:1: error: unknown type name '_LIBCPP_END_NAMESPACE_STD'
_LIBCPP_END_NAMESPACE_STD
^
In file included from /Users/ur20980/src/OpenSC/OpenSC.tokend/OpenSC/OpenSCToken.cpp:25:
In file included from /Users/ur20980/src/OpenSC/OpenSC.tokend/OpenSC/OpenSCToken.h:32:
In file included from /Users/ur20980/src/OpenSC/OpenSC.tokend/Tokend/Token.h:33:
In file included from /Users/ur20980/src/OpenSC/OpenSC.tokend/build/security_utilities.framework/Headers/osxcode.h:25:
In file included from /Users/ur20980/src/OpenSC/OpenSC.tokend/build/security_utilities.framework/Headers/refcount.h:31:
In file included from /Users/ur20980/src/OpenSC/OpenSC.tokend/build/security_utilities.framework/Headers/threading.h:35:
In file included from /Users/ur20980/src/OpenSC/OpenSC.tokend/build/security_utilities.framework/Headers/utilities.h:32:
In file included from /Users/ur20980/src/OpenSC/OpenSC.tokend/build/security_cdsa_utilities.framework/Headers/cssmbridge.h:32:
In file included from /Users/ur20980/src/OpenSC/OpenSC.tokend/build/security_cdsa_utilities.framework/Headers/cssmerrors.h:31:
In file included from /Users/ur20980/src/OpenSC/OpenSC.tokend/build/security_utilities.framework/Headers/errors.h:33:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/exception:93:1: error: expected unqualified-id
namespace std  // purposefully not using versioning namespace
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/exception:97:29: error: variable has incomplete type 'class _LIBCPP_EXCEPTION_ABI'
class _LIBCPP_EXCEPTION_ABI exception
                            ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/exception:97:7: note: forward declaration of 'std::_LIBCPP_EXCEPTION_ABI'
class _LIBCPP_EXCEPTION_ABI exception
      ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/exception:97:38: error: expected ';' after top level declarator
class _LIBCPP_EXCEPTION_ABI exception
                                     ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/exception:105:29: error: variable has incomplete type 'class _LIBCPP_EXCEPTION_ABI'
class _LIBCPP_EXCEPTION_ABI bad_exception
                            ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/exception:97:7: note: forward declaration of 'std::_LIBCPP_EXCEPTION_ABI'
class _LIBCPP_EXCEPTION_ABI exception
      ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/exception:106:5: error: expected ';' after top level declarator
    : public exception
    ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/exception:106:7: error: expected unqualified-id
    : public exception
      ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]

** INSTALL FAILED **

Here's the complete log:
tokend-build-log.txt

Note, that under Xcode-9.4.1 it builds OK on High Sierra 10.13.6.

[mouse07410]

@frankmorgner I couldn't (so far) find a way to build OpenSC.tokend on Xcode-10 on High Sierra. As Mojave requires Xcode-10, this will be a critical/blocking issue.

In theory, one could say "just switch to the native pivtoken and don't bother with tokend". But in practice, there still are plenty of applications (Google Chrome, MS Office, to name a few) that people (like myself) need to run now, and that do not support CTK-based API. These people need a working tokend.

Could you help please? I have a few platforms running 10.13.6 with the latest Xcode-10.1, and tomorrow will have a "real" Mojave machine with Xcode-10.1. I can provide any logs that may be needed, and can test any modification.

Here's what's happening when I just use xcodebuild from Xcode-10, without making any changes to the Tokend.xcodeproj (that works fine with Xcode-9.4.1):
xcodebuild-out.txt

I know you're mostly interested in OpenSCToken as the next step after the dying tokend, but for apps that can speak CTK there already is a working pivtoken that addresses the majority of their needs, and for those apps that are still CDSA-based - OpenSCToken doesn't help.

[frankmorgner]

Switching to libc++ in OpenSC.tokend requires the use of unordered_map instead of hash_map:

@@ -33,14 +33,15 @@
 #include <security_utilities/globalizer.h>
 #include <security_cdsa_utilities/cssmerrors.h>
 
+#if __GNUC__ > 4
 #include <unordered_map>
-
-#if __GNUC__ > 2
+#elif __GNUC__ > 2
 #include <ext/hash_map>
 using __gnu_cxx::hash_map;
 #else
 #include <hash_map>
 #endif
+#define hash_map unordered_map
 
 namespace Security
 {
@@ -138,7 +139,7 @@ protected:
     virtual void lock();
     virtual bool tryLock();
     
-    typedef hash_map<_Handle, MappingHandle<_Handle> *> HandleMap;
+    typedef unordered_map<_Handle, MappingHandle<_Handle> *> HandleMap;
 
     MappingHandle();
 

However, this gives a linker error:

Undefined symbols for architecture x86_64:
  "Security::MappingHandle<long>::State::erase(std::__1::__hash_map_iterator<std::__1::__hash_iterator<std::__1::__hash_node<std::__1::__hash_value_type<long, Security::MappingHandle<long>*>, void*>*> >&)", referenced from:
      Tokend::Cursor& Security::MappingHandle<long>::findAndKill<Tokend::Cursor>(long, int) in tokend(Token.o)
      Tokend::RecordHandle& Security::MappingHandle<long>::findAndKill<Tokend::RecordHandle>(long, int) in tokend(Token.o)
      Tokend::KeyHandle& Security::MappingHandle<long>::findAndKill<Tokend::KeyHandle>(long, int) in tokend(Token.o)
  "Security::Context::dump(char const*, cssm_context_attribute const*) const", referenced from:
      Tokend::KeyHandle::wrapUsingKey(Security::Context const&, Security::AccessCredentials const*, Tokend::KeyHandle*, Security::CssmKey const*, Security::CssmData const*, Security::CssmKey&) in tokend(KeyHandle.o)
      Tokend::KeyHandle::unwrapKey(Security::Context const&, Security::AccessCredentials const*, Security::AclEntryPrototype const*, Security::CssmKey const&, unsigned int, unsigned int, Security::CssmData*, long&, Security::CssmKey&) in tokend(KeyHandle.o)
  "std::string::_Rep::_M_dispose(std::allocator<char> const&)", referenced from:
      Security::ModuleNexus<Security::CssmClient::Statics>::operator()() in security_cdsa_client(aclclient.o)
      Security::ModuleNexus<Security::MappingHandle<unsigned int>::State>::operator()() in security_cdsa_utilities(handletemplates.o)
      Security::ModuleNexus<Security::MappingHandle<long>::State>::operator()() in security_cdsa_utilities(handletemplates.o)
      Security::PCSC::Session::listReaders(std::vector<std::string, std::allocator<std::string> >&, char const*) in security_utilities(pcsc++.o)
      std::string* std::__uninitialized_copy_aux<std::string*, std::string*>(std::string*, std::string*, std::string*, std::__false_type) in security_utilities(pcsc++.o)
      std::vector<std::string, std::allocator<std::string> >::_M_insert_aux(__gnu_cxx::__normal_iterator<std::string*, std::vector<std::string, std::allocator<std::string> > >, std::string const&) in security_utilities(pcsc++.o)
      Security::ModuleNexus<DefaultAllocators>::operator()() in security_utilities(alloc.o)
      ...

There are some reports about clang's unordered_map, which may be worth exploring, but that didn't lead me anywhere useful

[metsma]

You cannot mix libc++ and libstdc++ libraries, they are not ABI compatible
you need compile then helper frameworks also with libc++
https://github.com/OpenSC/OpenSC.tokend/tree/master/build

otool -L SecurityTokend.framework/Versions/A/SecurityTokend
SecurityTokend.framework/Versions/A/SecurityTokend:
	/System/Library/PrivateFrameworks/SecurityTokend.framework/Versions/A/SecurityTokend (compatibility version 1.0.0, current version 36808.0.0)
	/System/Library/Frameworks/PCSC.framework/Versions/A/PCSC (compatibility version 1.0.0, current version 36160.0.0)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 36910.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 550.13.0)
	/usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.9.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.0.0)

[frankmorgner]

Apple has removed SecurityTokend.framework from MacOSX10.14.sdk entirely (we have an older copy in OpenSC.tokend). It seems it's time to switch to https://github.com/frankmorgner/OpenSCToken...

[metsma]

There is maybe possible compile them from source
https://opensource.apple.com/source/SecurityTokend/SecurityTokend-55111/
https://opensource.apple.com/source/Security/Security-58286.70.7/OSX/
libsecurity_cdsa_client/
libsecurity_cdsa_utilities/
libsecurity_utilities/

[mouse07410]

It seems it's time to switch to https://github.com/frankmorgner/OpenSCToken...

Given that pivtoken is provided by Apple and works reasonably fine, there are two reasons that I see to switch to OpenSCToken:

1. Excellent configurability. It's nice to have, but it does not help with my main problem: accessing PIV tokens from 3rd-party apps like MS Office and Google Chrome .
2. Ability to work with non-PIV cards. Probably useful for many, but my only non-PIV card is Latvian eID, and OpenSC does not support it at all yet.

OpenSCToken won't help me with S/MIME using MS Outlook, and such. Which is why I'm still bothering with OpenSC.tokend.

Apple has removed SecurityTokend.framework from MacOSX10.14.sdk entirely (we have an older copy in OpenSC.tokend)

Please excuse my ignorance - is that framework present in binary only, or in the source?

There is maybe possible compile them from source...

Do I need all of what you listed - including https://opensource.apple.com/source/SecurityTokend/SecurityTokend-55111/?

I guess I'll have to give it a try, as there is no alternative that I can see for now. Ideally, CDSA would be killed after the last "important" 3rd-party app is released with CTK support, but that's not the case...

[mouse07410]

@frankmorgner what file were you talking about here:

Switching to libc++ in OpenSC.tokend requires the use of unordered_map instead of hash_map

[mouse07410]

@metsma what's the best way to download those frameworks? The URLs point at Javascript-based web sites, and I wonder if there's a way and/or a tool to do it in a way more efficient that manually clicking on each and every file.

[metsma]

https://opensource.apple.com/release/macos-10136.html here you can download whole package
there is possible also use svn

[mouse07410]

@metsma thank you - it helped!

@martinpaljak I think it was you who built the original frameworks for OpenSC.tokend? Would you be able to help re-building them under Xcode-10?

[martinpaljak]

Sorry, I don't remember much. I remember changing them manually to work with newer compilers, but that was pretty much "do something sensible that makes the warning go away" kind of thing.

[frankmorgner]

@mouse07410, If you want to dig deeper, https://ludovicrousseau.blogspot.com/2018/12/macos-mojave-and-smart-card-source-code.html has a link to the Tokend sourcecode.

BTW, according to the docs, Google Chrome should have support for CryptoTokenKit now. Maybe other vendors also adopted this finally...

[mouse07410]

If you want to dig deeper...

I do. But it becomes more and more difficult, as less and less of the old MacOS-provided stuff, like
https://opensource.apple.com/source/SecurityTokend/SecurityTokend-55111/, compiles under the new Xcode.

... has a link to the Tokend sourcecode

Yes, but I don't think it compiles under Xcode-10, that's the problem.

BTW, according to the docs, Google Chrome should have support for CryptoTokenKit now. Maybe other vendors also adopted this finally...

I have no doubt that all the vendors will eventually move to CTK, especially since it's becoming more and more difficult to support CDSA.

My concern is how to "survive" the gap between OpenSC.tokend not compiling under Xcode-10 any more, and all the relevant apps moving to CTK and making OpenSC.tokend unnecessary. (And how long in time that gap is going to be.)