Skip to content

Commit 279b18f

Browse files
KiterLucKiterLuc
authored
Merge branch 'dev' into smr/sc-52877/serialization-update-load-schema
2 parents 1829491 + 5b92634 commit 279b18f

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

45 files changed

+1801
-2560
lines changed

‎.github/workflows/build-windows.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -136,6 +136,8 @@ jobs:
136136
uses: seanmiddleditch/gha-setup-ninja@v4
137137
- name: Prevent vcpkg from building debug variants
138138
run: python $env:GITHUB_WORKSPACE/scripts/ci/patch_vcpkg_triplets.py
139+
- name: Install dependencies from pip
140+
run: python -m pip install pybind11[global]
139141

140142
- name: Configure TileDB
141143
shell: pwsh

‎HISTORY.md‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,67 @@
1+
# TileDB v2.25.0 Release Notes
2+
3+
## Announcements
4+
5+
* TileDB 2.25, includes the new current domain feature which allows to specify an area of the domain that is considered to be active for sparse arrays.
6+
7+
## Deprecation announcements
8+
9+
* The HDFS backend is no longer officially tested by TileDB. As announced before, it is scheduled to be removed in version 2.28, to be released in Q4 2024. [#5085](https://github.com/TileDB-Inc/TileDB/pull/5085)
10+
* Support for reading sparse fragments in dense arrays will be removed in version 2.27. Writting sparse fragments in dense arrays was removed in version 2.5. [#5116](https://github.com/TileDB-Inc/TileDB/pull/5116)
11+
* Support for returning the same results multiple times in sparse reads when ranges overlap will be removed in version 2.27. This was possible by setting `sm.merge_overlapping_ranges_experimental` to `false`, but the default `true` behavior has been there since version 2.17.
12+
13+
## New features
14+
15+
* REST support for current domain. [#5136](https://github.com/TileDB-Inc/TileDB/pull/5136)
16+
* Disallow writing outside of the current domain. [#5165](https://github.com/TileDB-Inc/TileDB/pull/5165)
17+
* Current domain: disallow reading outside of current domain. [#5168](https://github.com/TileDB-Inc/TileDB/pull/5168)
18+
19+
## Improvements
20+
21+
* Improve memory consumption for tile structures in dense reader. [#5046](https://github.com/TileDB-Inc/TileDB/pull/5046)
22+
23+
## Defects removed
24+
25+
* Fail early when trying to add members with relative URIs in remote groups. [#5025](https://github.com/TileDB-Inc/TileDB/pull/5025)
26+
* Correct defective return value in `Posix::ls_with_sizes`. [#5037](https://github.com/TileDB-Inc/TileDB/pull/5037)
27+
* Prevent constructing attribute with invalid cell_val_num. [#4952](https://github.com/TileDB-Inc/TileDB/pull/4952)
28+
* Do not mask failures when listing a directory fails on POSIX. [#5043](https://github.com/TileDB-Inc/TileDB/pull/5043)
29+
* Fix write queries using `sm.var_offsets.extra_element=true`. [#5033](https://github.com/TileDB-Inc/TileDB/pull/5033)
30+
* Fix segfaults in WebP queries ran in parallel. [#5065](https://github.com/TileDB-Inc/TileDB/pull/5065)
31+
* Fix exceptions with message: ```unknown exception type; no further information```. [#5080](https://github.com/TileDB-Inc/TileDB/pull/5080)
32+
* Fix check for out of bounds dimension in Dimension::dimension_ptr. [#5094](https://github.com/TileDB-Inc/TileDB/pull/5094)
33+
* Fix array latest schema selection for same MS timestamps schemas. [#5143](https://github.com/TileDB-Inc/TileDB/pull/5143)
34+
* Fix serialization issue with schema evolution for query v3. [#5154](https://github.com/TileDB-Inc/TileDB/pull/5154)
35+
36+
## Configuration changes
37+
38+
* Add `vfs.s3.storage_class` config option to set the storage class of newly uploaded S3 objects. [#5053](https://github.com/TileDB-Inc/TileDB/pull/5053)
39+
* Add `rest.custom_headers.*` config option to set custom headers on REST requests. [#5104](https://github.com/TileDB-Inc/TileDB/pull/5104)
40+
* Add `rest.payer_namespace` config option to set the namespace to be charged for REST requests. [#5105](https://github.com/TileDB-Inc/TileDB/pull/5105)
41+
42+
## API changes
43+
44+
### C API
45+
46+
* Add CurrentDomain API support. [#5041](https://github.com/TileDB-Inc/TileDB/pull/5041)
47+
48+
### C++ API
49+
50+
* Current Domain CPP API implementation. [#5056](https://github.com/TileDB-Inc/TileDB/pull/5056)
51+
52+
## Build System Changes
53+
54+
* Backwards compatibility with older CMake versions for libfaketime. [#5049](https://github.com/TileDB-Inc/TileDB/pull/5049)
55+
* Automatic downloading of vcpkg can be disabled by enabling the `TILEDB_DISABLE_AUTO_VCPKG` CMake option, in addition to setting the environment variable with trhe same name. [#5048](https://github.com/TileDB-Inc/TileDB/pull/5048)
56+
* Improve embedding of `magic.mgc` and allow compiling with any libmagic version. [#4989](https://github.com/TileDB-Inc/TileDB/pull/4989)
57+
58+
## Internal Improvements
59+
60+
* Implement actualize function that orders data underlying `alt_var_length_view` [#5087](https://github.com/TileDB-Inc/TileDB/pull/5087)
61+
* Implement a partitioning function to partition cells to fit into fixed size bins [#5092](https://github.com/TileDB-Inc/TileDB/pull/5092)
62+
* Implementation of a `chunk_view` class to provide a subset of C++23 chunk_view, suitable for supporting external sort. [#5035](https://github.com/TileDB-Inc/TileDB/pull/5035)
63+
* Tests that the chunks in a `chunk_view` can be separately sorted. [#5052](https://github.com/TileDB-Inc/TileDB/pull/5052)
64+
165
# TileDB v2.24.2 Release Notes
266

367
## Defects removed

‎cmake/Options/TileDBToolchain.cmake‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ if (NOT DEFINED CMAKE_TOOLCHAIN_FILE)
3636
# Inspired from https://github.com/Azure/azure-sdk-for-cpp/blob/azure-core_1.10.3/cmake-modules/AzureVcpkg.cmake
3737
message("TILEDB_DISABLE_AUTO_VCPKG is not defined. Fetch a local copy of vcpkg.")
3838
# To help with resolving conflicts, when you update the commit, also update its date.
39-
set(VCPKG_COMMIT_STRING 72010900b7cee36cea77aebb97695095c9358eaf) # 2023-12-05
39+
set(VCPKG_COMMIT_STRING 7aeffc91033ad35cc4e2c152f213a866ec6c11ac) # 2024-07-13
4040
message("Vcpkg commit string used: ${VCPKG_COMMIT_STRING}")
4141
include(FetchContent)
4242
FetchContent_Declare(

‎ports/README.md‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -26,13 +26,13 @@ $ git commit
2626

2727
After copying the port, add an entry to the table below. You should also contribute your changes to vcpkg and/or the upstream package repository.
2828

29+
For ease of review when patching existing ports, you are recommended to make one commit that copies the upstream port unchanged, and another commit that makes the changes you need.
30+
2931
## List of port overlays
3032

31-
| Port | Reason |
32-
|----------------------------|---------------------------------------------------------------------------------------------------------|
33-
| `libmagic` | Updating to the upstream port deferred due to failures. |
34-
| `pcre2` | To be removed alongside libmagic. |
35-
| `azure-storage-common-cpp` | Patching to disable default features on libxml2 (https://github.com/Azure/azure-sdk-for-cpp/pull/5221). |
36-
| `libfaketime` | Port does not yet exist upstream |
37-
| `vcpkg-cmake-config` | Patching to fix build issues with CMake 3.29.1. (https://github.com/microsoft/vcpkg/pull/38017) |
38-
| `google-cloud-cpp` | Patching to remove dependency on GMock. (https://github.com/microsoft/vcpkg/pull/39802) |
33+
| Port | Reason |
34+
|---------------|----------------------------------------------------------------------------------------------------------------------------|
35+
| `aws-c-io` | Patching to fix MinGW build failures. (https://github.com/awslabs/aws-c-io/pull/664) |
36+
| `aws-sdk-cpp` | Patching to fix MinGW build failures, and to avoid building test-only SDKs (https://github.com/aws/aws-sdk-cpp/pull/3061). |
37+
| `libmagic` | Updating to the upstream port deferred due to failures. |
38+
| `libfaketime` | Port does not yet exist upstream |

‎ports/aws-c-io/fix-pointer-mismatch.patch‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
diff --git a/source/windows/secure_channel_tls_handler.c b/source/windows/secure_channel_tls_handler.c
2+
index b62b1a0..48f4556 100644
3+
--- a/source/windows/secure_channel_tls_handler.c
4+
+++ b/source/windows/secure_channel_tls_handler.c
5+
@@ -53,7 +53,7 @@ struct secure_channel_ctx {
6+
struct aws_tls_ctx ctx;
7+
struct aws_string *alpn_list;
8+
SCHANNEL_CRED credentials;
9+
- PCERT_CONTEXT pcerts;
10+
+ PCCERT_CONTEXT pcerts;
11+
HCERTSTORE cert_store;
12+
HCERTSTORE custom_trust_store;
13+
HCRYPTPROV crypto_provider;
14+
@@ -188,7 +188,7 @@ static int s_manually_verify_peer_cert(struct aws_channel_handler *handler) {
15+
int result = AWS_OP_ERR;
16+
CERT_CONTEXT *peer_certificate = NULL;
17+
HCERTCHAINENGINE engine = NULL;
18+
- CERT_CHAIN_CONTEXT *cert_chain_ctx = NULL;
19+
+ PCCERT_CHAIN_CONTEXT cert_chain_ctx = NULL;
20+
21+
/* get the peer's certificate so we can validate it.*/
22+
SECURITY_STATUS status =

‎ports/aws-c-io/portfile.cmake‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
vcpkg_from_github(
2+
OUT_SOURCE_PATH SOURCE_PATH
3+
REPO awslabs/aws-c-io
4+
REF "v${VERSION}"
5+
SHA512 b5dc81635ce775c3783df2d4f6ccf0890b1e1669bddb1c10f6d200203684e84504ca87099cb88874010df28f98f658da5a4eacb9e4df0408d40d845f3d394f48
6+
HEAD_REF master
7+
PATCHES
8+
"fix-pointer-mismatch.patch"
9+
)
10+
11+
vcpkg_cmake_configure(
12+
SOURCE_PATH "${SOURCE_PATH}"
13+
OPTIONS
14+
"-DCMAKE_MODULE_PATH=${CURRENT_INSTALLED_DIR}/share/aws-c-common" # use extra cmake files
15+
-DBUILD_TESTING=FALSE
16+
)
17+
18+
vcpkg_cmake_install()
19+
20+
string(REPLACE "dynamic" "shared" subdir "${VCPKG_LIBRARY_LINKAGE}")
21+
vcpkg_cmake_config_fixup(CONFIG_PATH "lib/${PORT}/cmake/${subdir}" DO_NOT_DELETE_PARENT_CONFIG_PATH)
22+
vcpkg_cmake_config_fixup(CONFIG_PATH "lib/${PORT}/cmake")
23+
vcpkg_replace_string("${CURRENT_PACKAGES_DIR}/share/${PORT}/${PORT}-config.cmake" [[/${type}/]] "/")
24+
25+
file(REMOVE_RECURSE
26+
"${CURRENT_PACKAGES_DIR}/debug/include"
27+
"${CURRENT_PACKAGES_DIR}/debug/lib/${PORT}"
28+
"${CURRENT_PACKAGES_DIR}/debug/share"
29+
"${CURRENT_PACKAGES_DIR}/lib/${PORT}"
30+
)
31+
32+
vcpkg_copy_pdbs()
33+
34+
vcpkg_install_copyright(FILE_LIST "${SOURCE_PATH}/LICENSE")

‎ports/aws-c-io/vcpkg.json‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
{
2+
"name": "aws-c-io",
3+
"version": "0.14.9",
4+
"description": "Handles all IO and TLS work for application protocols.",
5+
"homepage": "https://github.com/awslabs/aws-c-io",
6+
"license": "Apache-2.0",
7+
"supports": "!(windows & arm) & !uwp",
8+
"dependencies": [
9+
"aws-c-cal",
10+
"aws-c-common",
11+
{
12+
"name": "s2n",
13+
"platform": "!uwp & !windows"
14+
},
15+
{
16+
"name": "vcpkg-cmake",
17+
"host": true
18+
},
19+
{
20+
"name": "vcpkg-cmake-config",
21+
"host": true
22+
}
23+
]
24+
}

‎ports/aws-sdk-cpp/find-dependency.patch‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
diff --git a/toolchains/core-config.cmake b/toolchains/core-config.cmake
2+
index d90bfaf747..ab14402577 100644
3+
--- a/toolchains/core-config.cmake
4+
+++ b/toolchains/core-config.cmake
5+
@@ -15,5 +15,9 @@ if (AWSSDK_CRYPTO_IN_SOURCE_BUILD)
6+
find_dependency(crypto)
7+
find_dependency(ssl)
8+
endif()
9+
+find_dependency(ZLIB)
10+
+if("@ENABLE_CURL_CLIENT@")
11+
+ find_dependency(CURL)
12+
+endif()
13+
set(BUILD_SHARED_LIBS ${BUILD_SHARED_LIBS_PREV})
14+
include("${CMAKE_CURRENT_LIST_DIR}/@[email protected]")

‎ports/aws-sdk-cpp/fix-aws-root.patch‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
diff --git a/cmake/AWSSDKConfig.cmake b/cmake/AWSSDKConfig.cmake
2+
--- a/cmake/AWSSDKConfig.cmake (revision 2f90f9fd6c56460bd382243aa215fcddcb5883c8)
3+
+++ b/cmake/AWSSDKConfig.cmake (date 1636913220527)
4+
@@ -54,18 +54,14 @@
5+
string(REPLACE ";" "${AWS_MODULE_DIR};" SYSTEM_MODULE_PATH "${CMAKE_SYSTEM_PREFIX_PATH}${AWS_MODULE_DIR}")
6+
list(APPEND CMAKE_MODULE_PATH ${AWS_MODULE_PATH} ${SYSTEM_MODULE_PATH})
7+
8+
-# On Windows, dlls are treated as runtime target and installed in bindir
9+
if (WIN32 AND AWSSDK_INSTALL_AS_SHARED_LIBS)
10+
- set(AWSSDK_INSTALL_LIBDIR "${AWSSDK_INSTALL_BINDIR}")
11+
# If installed CMake scripts are associated with dll library, define USE_IMPORT_EXPORT for customers
12+
add_definitions(-DUSE_IMPORT_EXPORT)
13+
endif()
14+
15+
16+
# Compute the default installation root relative to this file.
17+
-# from prefix/lib/cmake/AWSSDK/xx.cmake to prefix
18+
get_filename_component(AWSSDK_DEFAULT_ROOT_DIR "${CMAKE_CURRENT_LIST_FILE}" PATH)
19+
-get_filename_component(AWSSDK_DEFAULT_ROOT_DIR "${AWSSDK_DEFAULT_ROOT_DIR}" PATH)
20+
get_filename_component(AWSSDK_DEFAULT_ROOT_DIR "${AWSSDK_DEFAULT_ROOT_DIR}" PATH)
21+
get_filename_component(AWSSDK_DEFAULT_ROOT_DIR "${AWSSDK_DEFAULT_ROOT_DIR}" PATH)
22+
get_filename_component(AWS_NATIVE_SDK_ROOT "${CMAKE_CURRENT_SOURCE_DIR}" ABSOLUTE)

‎ports/aws-sdk-cpp/fix-winsock-headers.patch‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
diff --git a/src/aws-cpp-sdk-core/source/http/windows/WinHttpSyncHttpClient.cpp b/src/aws-cpp-sdk-core/source/http/windows/WinHttpSyncHttpClient.cpp
2+
index b8ab519..7cd91e9 100644
3+
--- a/src/aws-cpp-sdk-core/source/http/windows/WinHttpSyncHttpClient.cpp
4+
+++ b/src/aws-cpp-sdk-core/source/http/windows/WinHttpSyncHttpClient.cpp
5+
@@ -19,6 +19,9 @@
6+
7+
#include <Windows.h>
8+
#include <winhttp.h>
9+
+#ifdef __MINGW32__
10+
+#include <WinSock2.h>
11+
+#endif
12+
#include <mstcpip.h> // for tcp_keepalive
13+
#include <sstream>
14+
#include <iostream>

0 commit comments

Comments
 (0)