Skip to content

Commit 817235b

Browse files
balamuruganabalamurugana
authored
Add append_object() API (#1494)
Signed-off-by: Bala.FA <[email protected]>
1 parent 82c6c1a commit 817235b

File tree

3 files changed

+251
-24
lines changed

3 files changed

+251
-24
lines changed

‎docs/API.md‎

Lines changed: 75 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -63,30 +63,30 @@ client = Minio(
6363

6464
| Bucket operations | Object operations |
6565
|:------------------------------------------------------------|:----------------------------------------------------------------|
66-
| [`make_bucket`](#make_bucket) | [`get_object`](#get_object) |
67-
| [`list_buckets`](#list_buckets) | [`put_object`](#put_object) |
68-
| [`bucket_exists`](#bucket_exists) | [`copy_object`](#copy_object) |
69-
| [`remove_bucket`](#remove_bucket) | [`compose_object`](#compose_object) |
70-
| [`list_objects`](#list_objects) | [`stat_object`](#stat_object) |
71-
| [`get_bucket_versioning`](#get_bucket_versioning) | [`remove_object`](#remove_object) |
72-
| [`set_bucket_versioning`](#set_bucket_versioning) | [`remove_objects`](#remove_objects) |
73-
| [`delete_bucket_replication`](#delete_bucket_replication) | [`fput_object`](#fput_object) |
74-
| [`get_bucket_replication`](#get_bucket_replication) | [`fget_object`](#fget_object) |
75-
| [`set_bucket_replication`](#set_bucket_replication) | [`select_object_content`](#select_object_content) |
76-
| [`delete_bucket_lifecycle`](#delete_bucket_lifecycle) | [`delete_object_tags`](#delete_object_tags) |
77-
| [`get_bucket_lifecycle`](#get_bucket_lifecycle) | [`get_object_tags`](#get_object_tags) |
78-
| [`set_bucket_lifecycle`](#set_bucket_lifecycle) | [`set_object_tags`](#set_object_tags) |
79-
| [`delete_bucket_tags`](#delete_bucket_tags) | [`enable_object_legal_hold`](#enable_object_legal_hold) |
80-
| [`get_bucket_tags`](#get_bucket_tags) | [`disable_object_legal_hold`](#disable_object_legal_hold) |
81-
| [`set_bucket_tags`](#set_bucket_tags) | [`is_object_legal_hold_enabled`](#is_object_legal_hold_enabled) |
82-
| [`delete_bucket_policy`](#delete_bucket_policy) | [`get_object_retention`](#get_object_retention) |
83-
| [`get_bucket_policy`](#get_bucket_policy) | [`set_object_retention`](#set_object_retention) |
84-
| [`set_bucket_policy`](#set_bucket_policy) | [`presigned_get_object`](#presigned_get_object) |
85-
| [`delete_bucket_notification`](#delete_bucket_notification) | [`presigned_put_object`](#presigned_put_object) |
86-
| [`get_bucket_notification`](#get_bucket_notification) | [`presigned_post_policy`](#presigned_post_policy) |
87-
| [`set_bucket_notification`](#set_bucket_notification) | [`get_presigned_url`](#get_presigned_url) |
88-
| [`listen_bucket_notification`](#listen_bucket_notification) | [`upload_snowball_objects`](#upload_snowball_objects) |
89-
| [`delete_bucket_encryption`](#delete_bucket_encryption) | |
66+
| [`make_bucket`](#make_bucket) | [`append_object`](#append_object) |
67+
| [`list_buckets`](#list_buckets) | [`get_object`](#get_object) |
68+
| [`bucket_exists`](#bucket_exists) | [`put_object`](#put_object) |
69+
| [`remove_bucket`](#remove_bucket) | [`copy_object`](#copy_object) |
70+
| [`list_objects`](#list_objects) | [`compose_object`](#compose_object) |
71+
| [`get_bucket_versioning`](#get_bucket_versioning) | [`stat_object`](#stat_object) |
72+
| [`set_bucket_versioning`](#set_bucket_versioning) | [`remove_object`](#remove_object) |
73+
| [`delete_bucket_replication`](#delete_bucket_replication) | [`remove_objects`](#remove_objects) |
74+
| [`get_bucket_replication`](#get_bucket_replication) | [`fput_object`](#fput_object) |
75+
| [`set_bucket_replication`](#set_bucket_replication) | [`fget_object`](#fget_object) |
76+
| [`delete_bucket_lifecycle`](#delete_bucket_lifecycle) | [`select_object_content`](#select_object_content) |
77+
| [`get_bucket_lifecycle`](#get_bucket_lifecycle) | [`delete_object_tags`](#delete_object_tags) |
78+
| [`set_bucket_lifecycle`](#set_bucket_lifecycle) | [`get_object_tags`](#get_object_tags) |
79+
| [`delete_bucket_tags`](#delete_bucket_tags) | [`set_object_tags`](#set_object_tags) |
80+
| [`get_bucket_tags`](#get_bucket_tags) | [`enable_object_legal_hold`](#enable_object_legal_hold) |
81+
| [`set_bucket_tags`](#set_bucket_tags) | [`disable_object_legal_hold`](#disable_object_legal_hold) |
82+
| [`delete_bucket_policy`](#delete_bucket_policy) | [`is_object_legal_hold_enabled`](#is_object_legal_hold_enabled) |
83+
| [`get_bucket_policy`](#get_bucket_policy) | [`get_object_retention`](#get_object_retention) |
84+
| [`set_bucket_policy`](#set_bucket_policy) | [`set_object_retention`](#set_object_retention) |
85+
| [`delete_bucket_notification`](#delete_bucket_notification) | [`presigned_get_object`](#presigned_get_object) |
86+
| [`get_bucket_notification`](#get_bucket_notification) | [`presigned_put_object`](#presigned_put_object) |
87+
| [`set_bucket_notification`](#set_bucket_notification) | [`presigned_post_policy`](#presigned_post_policy) |
88+
| [`listen_bucket_notification`](#listen_bucket_notification) | [`get_presigned_url`](#get_presigned_url) |
89+
| [`delete_bucket_encryption`](#delete_bucket_encryption) | [`upload_snowball_objects`](#upload_snowball_objects) |
9090
| [`get_bucket_encryption`](#get_bucket_encryption) | |
9191
| [`set_bucket_encryption`](#set_bucket_encryption) | |
9292
| [`delete_object_lock_config`](#delete_object_lock_config) | |
@@ -836,6 +836,57 @@ client.set_object_lock_config("my-bucket", config)
836836

837837
## 3. Object operations
838838

839+
<a name="append_object"></a>
840+
841+
### append_object(bucket_name, object_name, data, length, content_type="application/octet-stream", metadata=None, sse=None, progress=None, part_size=0, num_parallel_uploads=3, tags=None, retention=None, legal_hold=False)
842+
843+
Appends from a stream to existing object in a bucket.
844+
845+
__Parameters__
846+
847+
| Param | Type | Description |
848+
|:----------------|:------------|:---------------------------------------------------------|
849+
| `bucket_name` | _str_ | Name of the bucket. |
850+
| `object_name` | _str_ | Object name in the bucket. |
851+
| `data` | _object_ | An object having callable read() returning bytes object. |
852+
| `length` | _int_ | Data size; -1 for unknown size and set valid part_size. |
853+
| `part_size` | _int_ | Chunk size. |
854+
| `progress` | _threading_ | A progress object. |
855+
| `extra_headers` | _dict_ | Extra headers. |
856+
857+
__Return Value__
858+
859+
| Return |
860+
|:----------------------------|
861+
| _ObjectWriteResult_ object. |
862+
863+
__Example__
864+
```py
865+
# Append data.
866+
result = client.append_object(
867+
"my-bucket", "my-object", io.BytesIO(b"world"), 5,
868+
)
869+
print(f"appended {result.object_name} object; etag: {result.etag}")
870+
871+
# Append data in chunks.
872+
data = urlopen(
873+
"https://www.kernel.org/pub/linux/kernel/v6.x/linux-6.13.12.tar.xz",
874+
)
875+
result = client.append_object(
876+
"my-bucket", "my-object", data, 148611164, 5*1024*1024,
877+
)
878+
print(f"appended {result.object_name} object; etag: {result.etag}")
879+
880+
# Append unknown sized data.
881+
data = urlopen(
882+
"https://www.kernel.org/pub/linux/kernel/v6.x/linux-6.14.3.tar.xz",
883+
)
884+
result = client.append_object(
885+
"my-bucket", "my-object", data, 149426584, 5*1024*1024,
886+
)
887+
print(f"appended {result.object_name} object; etag: {result.etag}")
888+
```
889+
839890
<a name="get_object"></a>
840891

841892
### get_object(bucket_name, object_name, offset=0, length=0, request_headers=None, ssec=None, version_id=None, extra_query_params=None)

‎examples/append_object.py‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
# -*- coding: utf-8 -*-
2+
# MinIO Python Library for Amazon S3 Compatible Cloud Storage,
3+
# (C) 2025 MinIO, Inc.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
import io
18+
from urllib.request import urlopen
19+
20+
from examples.progress import Progress
21+
from minio import Minio
22+
23+
client = Minio(
24+
"play.min.io",
25+
access_key="Q3AM3UQ867SPQQA43P2F",
26+
secret_key="zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG",
27+
)
28+
29+
# Upload data.
30+
result = client.put_object(
31+
"my-bucket", "my-object", io.BytesIO(b"hello, "), 7,
32+
)
33+
print(f"created {result.object_name} object; etag: {result.etag}")
34+
35+
# Append data.
36+
result = client.append_object(
37+
"my-bucket", "my-object", io.BytesIO(b"world"), 5,
38+
)
39+
print(f"appended {result.object_name} object; etag: {result.etag}")
40+
41+
# Append data in chunks.
42+
data = urlopen(
43+
"https://www.kernel.org/pub/linux/kernel/v6.x/linux-6.13.12.tar.xz",
44+
)
45+
result = client.append_object(
46+
"my-bucket", "my-object", data, 148611164, 5*1024*1024,
47+
)
48+
print(f"appended {result.object_name} object; etag: {result.etag}")
49+
50+
# Append unknown sized data.
51+
data = urlopen(
52+
"https://www.kernel.org/pub/linux/kernel/v6.x/linux-6.14.3.tar.xz",
53+
)
54+
result = client.append_object(
55+
"my-bucket", "my-object", data, 149426584, 5*1024*1024,
56+
)
57+
print(f"appended {result.object_name} object; etag: {result.etag}")

‎minio/api.py‎

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121
# pylint: disable=too-many-public-methods
2222
# pylint: disable=too-many-statements
2323
# pylint: disable=too-many-locals
24+
# pylint: disable=too-many-positional-arguments
2425

2526
"""
2627
Simple Storage Service (aka S3) client to perform bucket and object operations.
@@ -2006,6 +2007,124 @@ def put_object(
20062007
)
20072008
raise exc
20082009

2010+
def append_object(
2011+
self,
2012+
bucket_name: str,
2013+
object_name: str,
2014+
data: BinaryIO,
2015+
length: int,
2016+
chunk_size: int | None = None,
2017+
progress: ProgressType | None = None,
2018+
extra_headers: DictType | None = None,
2019+
) -> ObjectWriteResult:
2020+
"""
2021+
Appends from a stream to existing object in a bucket.
2022+
2023+
:param bucket_name: Name of the bucket.
2024+
:param object_name: Object name in the bucket.
2025+
:param data: An object having callable read() returning bytes object.
2026+
:param length: Data size; -1 for unknown size.
2027+
:param chunk_size: Chunk size to optimize uploads.
2028+
:return: :class:`ObjectWriteResult` object.
2029+
2030+
Example::
2031+
# Append data.
2032+
result = client.append_object(
2033+
"my-bucket", "my-object", io.BytesIO(b"world"), 5,
2034+
)
2035+
print(f"appended {result.object_name} object; etag: {result.etag}")
2036+
2037+
# Append data in chunks.
2038+
data = urlopen(
2039+
"https://www.kernel.org/pub/linux/kernel/v6.x/"
2040+
"linux-6.13.12.tar.xz",
2041+
)
2042+
result = client.append_object(
2043+
"my-bucket", "my-object", data, 148611164, 5*1024*1024,
2044+
)
2045+
print(f"appended {result.object_name} object; etag: {result.etag}")
2046+
2047+
# Append unknown sized data.
2048+
data = urlopen(
2049+
"https://www.kernel.org/pub/linux/kernel/v6.x/"
2050+
"linux-6.14.3.tar.xz",
2051+
)
2052+
result = client.append_object(
2053+
"my-bucket", "my-object", data, 149426584, 5*1024*1024,
2054+
)
2055+
print(f"appended {result.object_name} object; etag: {result.etag}")
2056+
"""
2057+
if length == 0:
2058+
raise ValueError("length should not be zero")
2059+
if chunk_size is not None:
2060+
if chunk_size < MIN_PART_SIZE:
2061+
raise ValueError("chunk size must be minimum of 5 MiB")
2062+
if chunk_size > MAX_PART_SIZE:
2063+
raise ValueError("chunk size must be less than 5 GiB")
2064+
else:
2065+
chunk_size = length if length > MIN_PART_SIZE else MIN_PART_SIZE
2066+
2067+
chunk_count = -1
2068+
if length > 0:
2069+
chunk_count = int(length / chunk_size)
2070+
if (chunk_count * chunk_size) < length:
2071+
chunk_count += 1
2072+
chunk_count = chunk_count or 1
2073+
2074+
object_size = length
2075+
uploaded_size = 0
2076+
chunk_number = 0
2077+
one_byte = b""
2078+
stop = False
2079+
2080+
stat = self.stat_object(bucket_name, object_name)
2081+
write_offset = cast(int, stat.size)
2082+
2083+
while not stop:
2084+
chunk_number += 1
2085+
if chunk_count > 0:
2086+
if chunk_number == chunk_count:
2087+
chunk_size = object_size - uploaded_size
2088+
stop = True
2089+
chunk_data = read_part_data(
2090+
data, chunk_size, progress=progress,
2091+
)
2092+
if len(chunk_data) != chunk_size:
2093+
raise IOError(
2094+
f"stream having not enough data;"
2095+
f"expected: {chunk_size}, "
2096+
f"got: {len(chunk_data)} bytes"
2097+
)
2098+
else:
2099+
chunk_data = read_part_data(
2100+
data, chunk_size + 1, one_byte, progress=progress,
2101+
)
2102+
# If chunk_data_size is less or equal to chunk_size,
2103+
# then we have reached last chunk.
2104+
if len(chunk_data) <= chunk_size:
2105+
chunk_count = chunk_number
2106+
stop = True
2107+
else:
2108+
one_byte = chunk_data[-1:]
2109+
chunk_data = chunk_data[:-1]
2110+
2111+
uploaded_size += len(chunk_data)
2112+
2113+
headers = extra_headers or {}
2114+
headers["x-amz-write-offset-bytes"] = str(write_offset)
2115+
upload_result = self._put_object(
2116+
bucket_name, object_name, chunk_data, headers=headers,
2117+
)
2118+
write_offset += len(chunk_data)
2119+
return ObjectWriteResult(
2120+
cast(str, upload_result.bucket_name),
2121+
cast(str, upload_result.object_name),
2122+
upload_result.version_id,
2123+
upload_result.etag,
2124+
upload_result.http_headers,
2125+
location=upload_result.location,
2126+
)
2127+
20092128
def list_objects(
20102129
self,
20112130
bucket_name: str,

0 commit comments

Comments
 (0)