Hide symbols back

[Totktonada]

After #2971 we experience a bunch of problems (1, 2). The second one still have no good solution. You can find more on the topic here.

Another side of the problem is that I heard from time to time: hey, someone can check that a symbol is present and use it in a module despite that there is no statement that we'll keep it. Yet another side of the problem is that C API description on the website significantly behind the actual state (at least I see no actions around new APIs), but it is out of scope here.

My proposal:

• Set default visibility to 'hidden' back.
• Either expose the entire public API of a library or hide it and create wrappers / aliases for calls using LuaJIT's FFI or from a Lua/C module.
  • The former (expose) will allow to use the bundled library from a module. It'll disallow to bundle your own library version to the module (tarantool's one will anyway in use on Linux) — say, when you want to use a feature from a newer version of the library. It'll lead to more problems with bundling if the library does not provide stable ABI and intended to be statically linked into a project (like small).
  • The latter (hidding) will disallow to use the bundled library from a module, but will allow to bundle the library into a module or use a system one. We can expose wrappers for necessary APIs if there is a demand (like we did for the region related APIs).

The only advantage of the unhiding symbols: sometimes you can hack something for certain tarantool executable using LuaJIT's FFI, but you'll anyway unable to use this hack in a module, because the symbol may disappear in any build: even from the same tarantool version range, but with other OS or toolchain.

The situation for tarantool executable is the same as for any other library. A library should expose only its public API and should not increase probability of hard to debug problems due to clash of a user's code with an internal name from the library.

The discussion in PR #5720 motivates me to revisit our symbols visibility policy.

[Totktonada]

We discussed the problem with @Mons and @LeonidVas. The discussion results and my further thoughts are summarized below.

There are symbols from tarantool itself and from libraries we bundle into tarantool. The memcached problem was due to a clash of symbols from different version of the small library. We don't meet a problem with clashing of a symbol from a module (or a module library) with tarantool's own symbol, however it is possible.

I looked over tarantool symbols that're exported now (due to default visibility) and some of them looks quite clashable:

The list of private tarantool symbols with pretty simple names.

base64_encode
SHA1Update
sql_get
flopen
xlog_sync
uri_format
load_u8
store_u8
store_u16
store_u32
store_u64
memory_free
decNumberMinus
decimal_minus
sqlInsert
greeting_decode
func_delete
box_status
find_path
field_type_by_name
fiber_set_name
path_unlock
histogram_snprint
histogram_new
user_by_id
user_find
wal_dir
tuple_next
sql_exec
store_double
fdprintf
sql_limit
mkdirpath
random_bytes
title_free
request_str
func_new
func_by_id
trigger_run
mem_set_int
session_find
strbuf_free
log_set_fd

So it should be changed some day: we should hide tarantool's own symbols excepts ones that are part of our module API and Lua/C API. We should also create wrappers for built-in library symbols that are called with LuaJIT FFI (such as msgpuck ones) and eliminate direct exports.

Regarding libraries. We should hide symbols from our built-in libraries except several ones. The library should meet several criterias to be included in the list of the disclosed ones:

1. Have stable public API.
2. There is a demand to link with it in a module (if there is intention to use a library using FFI from a module -- don't know; maybe we should expose wrappers).

The libraries to export are the following:

• libcurl (Support linking against tarantool's built-in libcurl smtp#24)
• zstd (test: revise zstd symbols list in static-build #5720 (comment))

How exactly we should expose a library API:

• Expose only when a library is bundled from the repository (submodule) or when it is linked statically (see 47c19ee)
• Expose only public symbols from the library.
• Ship public headers in ${PREFIX}/include/tarantool/ with tarantool-devel/dev package when a library is built-in (bundled or linked statically).

Since @Mons want to keep ability to call hidden functions to have a chance to fix a known problem in a released version, we should propose a way to call a function using FFI by an address from a debug info. At least he want access to tarantool's own symbols and symbols from libev, libcoro, libeio.

We can eat the pie piece by piece: say, close small and msgpack symbols out of order to prevent problems that have higher change to meet.

[LeonidVas]

Issue #5932

[filonenko-mikhail]

Seems that discussion can be closed?

[Totktonada]

I summarized the status below. I hope it is what you want: actual status of the question.

There is no way to close a discussion (Q&A discussions can be marked as asnwered, others are not).

[Totktonada]

Status 2022-03-04

We resolved the main problem in #5932.

The symbols were open in 2.5.0-42-g03790ac55 and closed back in 2.9.0-244-g5ceabb378. The list of tarantool releases, affected by the problem:

• 2.5.1, 2.5.2, 2.5.3
• 2.6.1, 2.6.2, 2.6.3
• 2.7.1, 2.7.2, 2.7.3
• 2.8.1, 2.8.2, 2.8.3 (possibly also future 2.8.* if we'll not backport the change and will decide to release 2.8 branch one more time).

2.10.0-beta1 and following (pre)releases are NOT affected.

We will consider accurate opening (and freezing in the public C API) of useful symbols. See #6372 and #6369. This activity is stalled now.

I also found that some libsmall's symbols are still opened to use via LuaJIT FFI from inside tarantool itself (not as a public API). See #6873. We'll investigate, whether it may lead to problems for Lua/C modules, which use its own libsmall (such as memcached), and likely will remove them from exports.