Bump up API to support lifecycle and property (#78)

Author: hanahmily

.mvn/wrapper/maven-wrapper.properties

@@ -1,3 +1,2 @@
 distributionUrl=https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.6.1/apache-maven-3.6.1-bin.zip
 wrapperUrl=https://repo.maven.apache.org/maven2/io/takari/maven-wrapper/0.5.5/maven-wrapper-0.5.5.jar
-

CHANGES.md

@@ -11,6 +11,7 @@ Release Notes.
 * Bump up the API to support the new property.
 * Bump up the API to adopt the status field which is changed to the string type due to the compatibility issue.
 * Bump up the API to support getting the API version.
+* Bump up the API to support the lifecycle management.
 
 0.7.0
 ------------------

README.md

@@ -71,6 +71,62 @@ client.define(g);
 
 Then we may define a stream with customized configurations.
 
+#### Define a how-warm-cold Group
+
+Here illustrates how to use the lifecycle stages feature for hot-warm-cold data architecture:
+
+```java
+// build a group sw_record for Stream with hot-warm-cold lifecycle stages
+Group g = Group.newBuilder().setMetadata(Metadata.newBuilder().setName("sw_record"))
+            .setCatalog(Catalog.CATALOG_STREAM)
+            .setResourceOpts(ResourceOpts.newBuilder()
+                // Hot configuration
+                .setShardNum(3)
+                // Default segment interval (will be overridden by stages if defined)
+                .setSegmentInterval(
+                    IntervalRule.newBuilder()
+                        .setUnit(IntervalRule.Unit.UNIT_DAY)
+                        .setNum(1))
+                // Default TTL (will be overridden by stages if defined)
+                .setTtl(
+                    IntervalRule.newBuilder()
+                        .setUnit(IntervalRule.Unit.UNIT_DAY)
+                        .setNum(3))
+                // Define lifecycle stages (hot → warm → cold)
+                .addStages(LifecycleStage.newBuilder()
+                    .setName("warm")
+                    .setShardNum(2) // Fewer shards
+                    .setSegmentInterval(IntervalRule.newBuilder()
+                        .setUnit(IntervalRule.Unit.UNIT_DAY)
+                        .setNum(1))
+                    .setTtl(IntervalRule.newBuilder()
+                        .setUnit(IntervalRule.Unit.UNIT_DAY)
+                        .setNum(7)) // Keep in warm for 7 days
+                    .setNodeSelector("hdd-nodes") // Store on cheaper HDD nodes
+                    .build())
+                .addStages(LifecycleStage.newBuilder()
+                    .setName("cold")
+                    .setShardNum(1) // Minimal shards for archived data
+                    .setSegmentInterval(IntervalRule.newBuilder()
+                        .setUnit(IntervalRule.Unit.UNIT_DAY)
+                        .setNum(7)) // Larger segments for cold data
+                    .setTtl(IntervalRule.newBuilder()
+                        .setUnit(IntervalRule.Unit.UNIT_DAY)
+                        .setNum(30)) // Keep in cold for 30 more days
+                    .setNodeSelector("archive-nodes") // Store on archive nodes
+                    .setClose(true) // Close segments that are no longer live
+                    .build()))
+            .build();
+client.define(g);
+```
+
+This configuration creates a hot-warm-cold architecture where:
+- Hot stage: Data is stored on fast SSD nodes with many shards for 1 day, optimized for high query performance
+- Warm stage: Data moves to HDD nodes with fewer shards for 7 days, balanced between performance and cost
+- Cold stage: Data finally moves to archive nodes with minimal shards for 30 days, optimized for storage efficiency
+
+Data automatically flows through these stages according to the defined TTLs. The total retention of data is 38 days (1+7+30).
+
 #### Define a Stream
 ```java
 // build a stream trace with above group
@@ -469,68 +525,64 @@ MeasureWrite measureWrite = client.createMeasureWrite("sw_metric", "service_cpm_
 CompletableFuture<Void> f = measureBulkWriteProcessor.add(measureWrite);
 f.get(10, TimeUnit.SECONDS);
 ```
+# Property APIs
 
-## Property APIs
-
-Property APIs are used to store key-value pairs.
-
-### Apply(Create/Update)
-
-`apply` will always succeed whenever the property exists or not.
-The old value will be overwritten if already existed, otherwise a new value will be set.
+Before using properties, you need to define a property schema:
 
 ```java
-Property property = Property.create("default", "sw", "ui_template")
-    .addTag(TagAndValue.newStringTag("name", "hello"))
-    .addTag(TagAndValue.newStringTag("state", "successd"))
-    .build();
-this.client.apply(property); //created:true tagsNum:2
+// Define property schema
+BanyandbDatabase.Property propertyDef = 
+   BanyandbDatabase.Property.newBuilder()
+        .setMetadata(Metadata.newBuilder()
+            .setGroup("default")
+            .setName("ui_template"))
+        .setTagType(TagType.TAG_TYPE_STRING)
+        .build();
+
+client.define(propertyDef);
 ```
 
-The operation supports updating partial tags.
+After defining the schema, you can apply (create/update) properties:
 
 ```java
-Property property = Property.create("default", "sw", "ui_template")
-    .addTag(TagAndValue.newStringTag("state", "failed"))
+// Apply a property (create or update)
+Property property = Property.newBuilder()
+    .setMetadata(Metadata.newBuilder()
+        .setGroup("default")
+        .setName("ui_template"))
+    .setId("dashboard-1")
+    .setTagValue(BanyandbModel.TagValue.newBuilder()
+        .setStr("template-data-json"))
     .build();
-this.client.apply(property); //created:false tagsNum:1
-```
-
-### Query
 
-Property can be queried via `Client.findProperty`,
-
-```java
- BanyandbProperty.QueryResponse resp = client.query(BanyandbProperty.QueryRequest.newBuilder()
-        .addGroups("default")
-        .setContainer("sw")
-        .addIds("ui_template")
-        .build());
+ApplyResponse response = client.apply(property);
 ```
 
-The query operation could filter tags,
+You can also apply with a specific strategy:
 
 ```java
-BanyandbProperty.QueryResponse resp = client.query(BanyandbProperty.QueryRequest.newBuilder()
-        .addGroups("default")
-        .setContainer("sw")
-        .addIds("ui_template")
-        .addTagProjection("state")
-        .build());
+// Apply with merge strategy
+ApplyResponse response = client.apply(property, Strategy.STRATEGY_MERGE);
 ```
 
-### Delete
-
-Property can be deleted by calling `Client.deleteProperty`,
+Query properties:
 
 ```java
-this.client.deleteProperty("default", "sw", "ui_template"); //deleted:true tagsNum:2
+// Query properties
+BanyandbProperty.QueryRequest queryRequest = BanyandbProperty.QueryRequest.newBuilder()
+    .setMetadata(Metadata.newBuilder()
+        .setGroup("default")
+        .setName("ui_template"))
+    .build();
+
+BanyandbProperty.QueryResponse queryResponse = client.query(queryRequest);
 ```
 
-The delete operation could remove specific tags instead of the whole property.
+Delete a property:
 
 ```java
-this.client.deleteProperty("default", "sw", "ui_template", "state"); //deleted:true tagsNum:1
+// Delete a property
+DeleteResponse deleteResponse = client.deleteProperty("default", "ui_template", "dashboard-1");
 ```
 
 # Compiling project

src/main/java/org/apache/skywalking/banyandb/v1/client/BanyanDBClient.java

@@ -62,6 +62,7 @@
 import org.apache.skywalking.banyandb.v1.client.metadata.IndexRuleMetadataRegistry;
 import org.apache.skywalking.banyandb.v1.client.metadata.MeasureMetadataRegistry;
 import org.apache.skywalking.banyandb.v1.client.metadata.MetadataCache;
+import org.apache.skywalking.banyandb.v1.client.metadata.PropertyMetadataRegistry;
 import org.apache.skywalking.banyandb.v1.client.metadata.ResourceExist;
 import org.apache.skywalking.banyandb.v1.client.metadata.StreamMetadataRegistry;
 import org.apache.skywalking.banyandb.v1.client.metadata.TopNAggregationMetadataRegistry;
@@ -812,6 +813,64 @@ public List<IndexRuleBinding> findIndexRuleBindings(String group) throws BanyanD
         return registry.list(group);
     }
 
+    /**
+     * Define a new property.
+     * 
+     * @param property the property to be stored in the BanyanBD
+     * @throws BanyanDBException if the property is invalid
+     */
+    public void define(org.apache.skywalking.banyandb.database.v1.BanyandbDatabase.Property property) throws BanyanDBException {
+        PropertyMetadataRegistry registry = new PropertyMetadataRegistry(checkNotNull(this.channel));
+        registry.create(property);
+    }
+
+    /**
+     * Update the property.
+     *
+     * @param property the property to be stored in the BanyanBD
+     * @throws BanyanDBException if the property is invalid
+     */
+    public void update(org.apache.skywalking.banyandb.database.v1.BanyandbDatabase.Property property) throws BanyanDBException {
+        PropertyMetadataRegistry registry = new PropertyMetadataRegistry(checkNotNull(this.channel));
+        registry.update(property);
+    }
+
+    /**
+     * Find the property with given group and name
+     *
+     * @param group group of the metadata
+     * @param name  name of the metadata
+     * @return the property found in BanyanDB. Otherwise, null is returned.
+     */
+    public org.apache.skywalking.banyandb.database.v1.BanyandbDatabase.Property findPropertyDefinition(String group, String name) throws BanyanDBException {
+        PropertyMetadataRegistry registry = new PropertyMetadataRegistry(checkNotNull(this.channel));
+        return registry.get(group, name);
+    }
+
+    /**
+     * Find the properties with given group
+     *
+     * @param group group of the metadata
+     * @return the properties found in BanyanDB
+     */
+    public List<org.apache.skywalking.banyandb.database.v1.BanyandbDatabase.Property> findPropertiesDefinition(String group) throws BanyanDBException {
+        PropertyMetadataRegistry registry = new PropertyMetadataRegistry(checkNotNull(this.channel));
+        return registry.list(group);
+    }
+
+    /**
+     * Delete the property
+     *
+     * @param group group of the metadata
+     * @param name  name of the metadata
+     * @param id    identity of the property
+     * @return if this property has been deleted
+     */
+    public boolean deletePropertyDefinition(String group, String name) throws BanyanDBException {
+        PropertyMetadataRegistry registry = new PropertyMetadataRegistry(checkNotNull(this.channel));
+        return registry.delete(group, name);
+    }
+
     /**
      * Apply(Create or update) the property with {@link BanyandbProperty.ApplyRequest.Strategy#STRATEGY_MERGE}
      *
@@ -1046,6 +1105,20 @@ public ResourceExist existTopNAggregation(String group, String name) throws Bany
         return new TopNAggregationMetadataRegistry(checkNotNull(this.channel)).exist(group, name);
     }
 
+    /**
+     * Check if the given property exists.
+     *
+     * @param group group of the property
+     * @param name name of the property
+     * @return ResourceExist which indicates whether group and property exist
+     */
+    public ResourceExist existProperty(String group, String name) throws BanyanDBException {
+        Preconditions.checkArgument(!Strings.isNullOrEmpty(group));
+        Preconditions.checkArgument(!Strings.isNullOrEmpty(name));
+
+        return new PropertyMetadataRegistry(checkNotNull(this.channel)).exist(group, name);
+    }
+
     /**
      * Update the stream metadata cache from the server
      * @param group the group of the stream

src/main/java/org/apache/skywalking/banyandb/v1/client/PropertyStore.java

@@ -54,7 +54,7 @@ public DeleteResponse delete(String group, String name, String id) throws Banyan
         return HandleExceptionsWith.callAndTranslateApiException(() ->
                 this.stub.delete(BanyandbProperty.DeleteRequest.newBuilder()
                         .setGroup(group)
-                        .setContainer(name)
+                        .setName(name)
                         .setId(id)
                         .build()));
     }

src/main/java/org/apache/skywalking/banyandb/v1/client/metadata/PropertyMetadataRegistry.java

@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.banyandb.v1.client.metadata;
+
+import io.grpc.Channel;
+import org.apache.skywalking.banyandb.common.v1.BanyandbCommon;
+import org.apache.skywalking.banyandb.database.v1.BanyandbDatabase;
+import org.apache.skywalking.banyandb.database.v1.BanyandbDatabase.Property;
+import org.apache.skywalking.banyandb.database.v1.PropertyRegistryServiceGrpc;
+import org.apache.skywalking.banyandb.v1.client.grpc.MetadataClient;
+import org.apache.skywalking.banyandb.v1.client.grpc.exception.BanyanDBException;
+
+import java.util.List;
+
+public class PropertyMetadataRegistry extends MetadataClient<PropertyRegistryServiceGrpc.PropertyRegistryServiceBlockingStub,
+        BanyandbDatabase.Property> {
+
+    public PropertyMetadataRegistry(Channel channel) {
+        super(PropertyRegistryServiceGrpc.newBlockingStub(channel));
+    }
+
+    @Override
+    public long create(final Property payload) throws BanyanDBException {
+        BanyandbDatabase.PropertyRegistryServiceCreateResponse resp = execute(() ->
+                stub.create(BanyandbDatabase.PropertyRegistryServiceCreateRequest.newBuilder()
+                        .setProperty(payload)
+                        .build()));
+        return resp.getModRevision();
+    }
+
+    @Override
+    public void update(final Property payload) throws BanyanDBException {
+        execute(() ->
+                stub.update(BanyandbDatabase.PropertyRegistryServiceUpdateRequest.newBuilder()
+                        .setProperty(payload)
+                        .build()));
+    }
+
+    @Override
+    public boolean delete(final String group, final String name) throws BanyanDBException {
+        BanyandbDatabase.PropertyRegistryServiceDeleteResponse resp = execute(() ->
+                stub.delete(BanyandbDatabase.PropertyRegistryServiceDeleteRequest.newBuilder()
+                        .setMetadata(BanyandbCommon.Metadata.newBuilder().setGroup(group).setName(name).build())
+                        .build()));
+        return resp != null && resp.getDeleted();
+    }
+
+    @Override
+    public Property get(final String group, final String name) throws BanyanDBException {
+        BanyandbDatabase.PropertyRegistryServiceGetResponse resp = execute(() ->
+                stub.get(BanyandbDatabase.PropertyRegistryServiceGetRequest.newBuilder()
+                        .setMetadata(BanyandbCommon.Metadata.newBuilder().setGroup(group).setName(name).build())
+                        .build()));
+
+        return resp.getProperty();
+    }
+
+    @Override
+    public ResourceExist exist(String group, String name) throws BanyanDBException {
+        BanyandbDatabase.PropertyRegistryServiceExistResponse resp = execute(() ->
+                stub.exist(BanyandbDatabase.PropertyRegistryServiceExistRequest.newBuilder()
+                        .setMetadata(BanyandbCommon.Metadata.newBuilder().setGroup(group).setName(name).build())
+                        .build()));
+        return ResourceExist.create(resp.getHasGroup(), resp.getHasProperty());
+    }
+
+    @Override
+    public List<Property> list(final String group) throws BanyanDBException {
+        BanyandbDatabase.PropertyRegistryServiceListResponse resp = execute(() ->
+                stub.list(BanyandbDatabase.PropertyRegistryServiceListRequest.newBuilder()
+                        .setGroup(group)
+                        .build()));
+
+        return resp.getPropertiesList();
+    }
+}