-
Notifications
You must be signed in to change notification settings - Fork 224
Support Oracle db VECTOR type #3637
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
n0tl3ss
wants to merge
56
commits into
5.0.x
Choose a base branch
from
support-vector-type-oracledb
base: 5.0.x
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+9,361
−347
Draft
Changes from 1 commit
Commits
Show all changes
56 commits
Select commit
Hold shift + click to select a range
cf28195
Support Oracle db VECTOR type
n0tl3ss 70359a5
refactor Vector type
n0tl3ss efd704e
remove gradle properties
n0tl3ss cd72557
add support for multiple dialects and adds postgres vectors
n0tl3ss d4913f4
Merge branch '5.0.x' into support-vector-type-oracledb
n0tl3ss 0098291
fix requires
n0tl3ss 8196182
refactor for r2dbc
n0tl3ss 3996937
fix tests
n0tl3ss 683b1dd
fix tests
n0tl3ss f3268f0
fix impl
n0tl3ss 6180793
fix postgres
n0tl3ss ab18770
fix postgres
n0tl3ss e59b8f1
fix test
n0tl3ss 775a883
fix result reader
n0tl3ss ef095f8
remove extra annotation
n0tl3ss f700207
remove setup from tests
n0tl3ss d094e09
improve docs
n0tl3ss 1041e83
improve docs
n0tl3ss f4eee69
fix tests
n0tl3ss 2e2a76b
fix tests
n0tl3ss 5ab85e7
cleanup
n0tl3ss 15d3fcb
remove mapping
n0tl3ss c3dab64
fix bugs and duplicate code
n0tl3ss b68fbf1
fix test and duplication
n0tl3ss 7dc2cb1
add more tests
n0tl3ss f4e92d2
fix tests
n0tl3ss 210a290
add mysql support for vectors in jdbc
n0tl3ss 90155bf
add improvements
n0tl3ss f0632e1
Merge branch '5.0.x' into support-vector-type-oracledb
n0tl3ss 6a1dce2
use mysql connector-j instead of mysql-connector-java
n0tl3ss ce43e09
fix tests
n0tl3ss 36bbe0d
Merge branch '5.0.x' into support-vector-type-oracledb
n0tl3ss 583ce81
remove surplus
n0tl3ss a2d0174
fix bugs
n0tl3ss a9681f2
minor fixes
n0tl3ss 7a2567f
minor refactor
n0tl3ss 81b8f95
minor refactor
n0tl3ss 5c752fa
minor cosmetics
n0tl3ss 3eefef0
improve docs
n0tl3ss b2ed398
add kotlin and groovy examples
n0tl3ss 43983c3
Merge branch '5.0.x' into support-vector-type-oracledb
n0tl3ss e3b12aa
move to Jspecific
n0tl3ss 314fbc9
add vector indexes
n0tl3ss a42db6e
try to provide different image because of space on the runner
n0tl3ss 6322b91
add more tests
n0tl3ss 56db57e
try to remove /usr/local/lib/android on start to free more space
n0tl3ss acad5e0
try to remove /usr/share/dotnet on start to free more space
n0tl3ss 0f4d038
fix OracleVectorSqlIndexDefinitionProvider
n0tl3ss ee54edb
fix merge
n0tl3ss 2fba0bd
Merge branch '5.0.x' into support-vector-type-oracledb
n0tl3ss 1fc179c
fix r2dbc
n0tl3ss 70dada4
test build
n0tl3ss c2d7191
fix checkstyle
n0tl3ss b766655
test build
n0tl3ss 9078515
fix test
n0tl3ss 2bf5ccf
add suppress for S3776
n0tl3ss File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
215 changes: 215 additions & 0 deletions
215
.../test/groovy/io/micronaut/data/jdbc/oraclexe/vector/OracleJdbcByteVectorEntitySpec.groovy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,215 @@ | ||
| package io.micronaut.data.jdbc.oraclexe.vector | ||
|
|
||
| import io.micronaut.context.ApplicationContext | ||
| import io.micronaut.context.annotation.Parameter | ||
| import io.micronaut.data.annotation.GeneratedValue | ||
| import io.micronaut.data.annotation.Id | ||
| import io.micronaut.data.annotation.MappedEntity | ||
| import io.micronaut.data.annotation.Query | ||
| import io.micronaut.data.jdbc.annotation.JdbcRepository | ||
| import io.micronaut.data.jdbc.oraclexe.OracleTestPropertyProvider | ||
| import io.micronaut.data.model.Sort | ||
| import io.micronaut.data.model.Vector | ||
| import io.micronaut.data.model.query.builder.sql.Dialect | ||
| import io.micronaut.data.repository.PageableRepository | ||
| import io.micronaut.transaction.SynchronousTransactionManager | ||
| import spock.lang.AutoCleanup | ||
| import spock.lang.Shared | ||
| import spock.lang.Specification | ||
|
|
||
| import javax.sql.DataSource | ||
| import java.sql.Connection | ||
| import java.sql.Statement | ||
| import java.util.concurrent.Future | ||
| import java.util.concurrent.TimeUnit | ||
|
|
||
| class OracleJdbcByteVectorEntitySpec extends Specification implements OracleTestPropertyProvider { | ||
|
|
||
| @AutoCleanup | ||
| @Shared | ||
| ApplicationContext context = ApplicationContext.run(properties) | ||
|
|
||
| @Shared | ||
| VectorByteDocRepository vectorRepository = context.getBean(VectorByteDocRepository) | ||
|
|
||
| @Shared | ||
| DataSource dataSource = context.getBean(DataSource) | ||
|
|
||
| @Override | ||
| List<String> packages() { | ||
| // Ensure entity/repository in this package are scanned | ||
| return [getClass().package.name] | ||
| } | ||
|
|
||
| def setupSpec() { | ||
| // Create sequence and table if not exists (ignore errors if already present) | ||
| executeSilently "CREATE SEQUENCE VECTOR_DOC_SEQ" | ||
| // Oracle 23ai VECTOR: use 3 dims for tests (INT8) | ||
| executeSilently "CREATE TABLE vector_byte_doc (id NUMBER PRIMARY KEY, embedding VECTOR(3, BINARY))" | ||
| } | ||
|
|
||
| def cleanup() { | ||
| // Clean table between tests | ||
| executeSilently "DELETE FROM vector_byte_doc" | ||
| // no-op transaction boundary to flush | ||
| context.getBean(SynchronousTransactionManager).executeWrite { status -> null } | ||
| } | ||
|
|
||
| void "test save, find and update single entity (using custom queries with io.micronaut.data.model.Vector)"() { | ||
| given: | ||
| byte[] dv = [1, 2, -3] as byte[] | ||
| Vector.ByteVector v1 = Vector.of(dv) | ||
|
|
||
| when: "save via custom @Query using Vector parameter" | ||
| vectorRepository.saveCustom(v1) | ||
| def list = vectorRepository.findAll(Sort.of(Sort.Order.asc("id"))) | ||
| def e = list.find { it.embedding.toByteArray().toList() == dv.toList() } | ||
|
|
||
| then: "entity persisted and read conversion to Micronaut Vector works" | ||
| e != null | ||
| e.id != null | ||
| e.embedding != null | ||
| e.embedding.type == Byte.TYPE | ||
| e.embedding.toByteArray().toList() == dv.toList() | ||
|
|
||
| when: "update via custom @Query to a new vector" | ||
| byte [] dv2 = [3, 0, 7] as byte[] | ||
| Vector.ByteVector v2 = Vector.of(dv2) | ||
| vectorRepository.updateCustom(e.id, v2) | ||
| def updated = vectorRepository.findById(e.id).orElse(null) | ||
|
|
||
| then: | ||
| updated != null | ||
| updated.embedding.type == Byte.TYPE | ||
| updated.embedding.toByteArray().toList() == dv2.toList() | ||
| } | ||
|
|
||
|
|
||
| void "test save, find and update multiple entities"() { | ||
| given: | ||
| Vector.ByteVector vA = Vector.of([1, 2, 3] as byte[]) | ||
| Vector.ByteVector vB = Vector.of([4, 5, 6] as byte[]) | ||
|
|
||
| when: | ||
| vectorRepository.saveCustom(vA) | ||
| vectorRepository.saveCustom(vB) | ||
| def rows = vectorRepository.findAll(Sort.of(Sort.Order.asc("id"))) | ||
|
|
||
| then: | ||
| def idA = rows.find { it.embedding.toByteArray().toList() == [1, 2, 3] }?.id | ||
| def idB = rows.find { it.embedding.toByteArray().toList() == [4, 5, 6] }?.id | ||
| idA != null | ||
| idB != null | ||
| idA != idB | ||
|
|
||
| when: "update both" | ||
| Vector.ByteVector vA2 = Vector.of([7, 8, 9] as byte[]) | ||
| Vector.ByteVector vB2 = Vector.of([0, -1, -2] as byte[]) | ||
| vectorRepository.updateCustom(idA, vA2) | ||
| vectorRepository.updateCustom(idB, vB2) | ||
| def rows2 = vectorRepository.findAll(Sort.of(Sort.Order.asc("id"))) | ||
|
|
||
| then: | ||
| def updatedA = rows2.find { it.id == idA }?.embedding?.toByteArray()?.toList() | ||
| def updatedB = rows2.find { it.id == idB }?.embedding?.toByteArray()?.toList() | ||
| updatedA == [7, 8, 9] | ||
| updatedB == [0, -1, -2] | ||
| } | ||
|
|
||
| void "test custom and async queries"() { | ||
| given: | ||
| Vector.ByteVector vec = Vector.of([10, 11, 12] as byte[]) | ||
|
|
||
| when: | ||
| Future<Integer> saveFut = vectorRepository.saveAsync(vec) | ||
|
|
||
| then: | ||
| saveFut.get(10, TimeUnit.SECONDS) == 1 | ||
|
|
||
| when: | ||
| def all = vectorRepository.findAll(Sort.of(Sort.Order.asc("id"))) | ||
| def last = all.last() | ||
| Future<List<VectorByteDoc>> findFut = vectorRepository.findAsync(last.id) | ||
|
|
||
| then: | ||
| def found = findFut.get(10, TimeUnit.SECONDS) | ||
| found != null | ||
| found.size() == 1 | ||
| found.get(0).embedding.toByteArray().toList() == [10, 11, 12] | ||
|
|
||
| when: | ||
| Vector.ByteVector vec2 = Vector.of([13, 14, 15] as byte[]) | ||
| Future<Integer> updFut = vectorRepository.updateAsync(last.id, vec2) | ||
|
|
||
| then: | ||
| updFut.get(10, TimeUnit.SECONDS) != null | ||
| with(vectorRepository.findById(last.id)) { | ||
| it.isPresent() | ||
| it.get().embedding.toByteArray().toList() == [13, 14, 15] | ||
| } | ||
| } | ||
|
|
||
| private void executeSilently(String sql) { | ||
| Connection c = null | ||
| Statement st = null | ||
| try { | ||
| c = dataSource.getConnection() | ||
| st = c.createStatement() | ||
| st.execute(sql) | ||
| } catch (Throwable ignored) { | ||
| // ignore if already exists or unsupported in current XE version | ||
| } finally { | ||
| try { st?.close() } catch (ignored) {} | ||
| try { c?.close() } catch (ignored) {} | ||
| } | ||
| } | ||
| } | ||
|
|
||
| @MappedEntity("vector_byte_doc") | ||
| class VectorByteDoc { | ||
| @Id | ||
| @GeneratedValue(value = GeneratedValue.Type.SEQUENCE, ref = "VECTOR_DOC_SEQ") | ||
| Long id | ||
| Vector.ByteVector embedding | ||
|
|
||
| Long getId() { return id } | ||
| void setId(Long id) { this.id = id } | ||
|
|
||
| Vector.ByteVector getEmbedding() { return embedding } | ||
| void setEmbedding(Vector.ByteVector embedding) { this.embedding = embedding } | ||
| } | ||
|
|
||
| @JdbcRepository(dialect = Dialect.ORACLE) | ||
| interface VectorByteDocRepository extends PageableRepository<VectorByteDoc, Long> { | ||
|
|
||
| @Query("INSERT INTO vector_byte_doc(id, embedding) VALUES (VECTOR_DOC_SEQ.nextval, :vec)") | ||
| void saveCustom(@Parameter("vec") Vector vec) | ||
|
|
||
| @Query("INSERT INTO vector_byte_doc(id, embedding) VALUES (VECTOR_DOC_SEQ.nextval, :vec)") | ||
| void saveCustom(@Parameter("vec") Vector.ByteVector vec) | ||
|
|
||
| @Query("SELECT * FROM vector_byte_doc WHERE id = :id") | ||
| Optional<VectorByteDoc> findById(Long id) | ||
|
|
||
| @Query("UPDATE vector_byte_doc SET embedding = :vec WHERE id = :id") | ||
| void updateCustom(Long id, @Parameter("vec") Vector vec) | ||
|
|
||
| @Query("UPDATE vector_byte_doc SET embedding = :vec WHERE id = :id") | ||
| void updateCustom(Long id, @Parameter("vec") Vector.ByteVector vec) | ||
|
|
||
| @Query("INSERT INTO vector_byte_doc(id, embedding) VALUES (VECTOR_DOC_SEQ.nextval, :vec)") | ||
| Future<Integer> saveAsync(@Parameter("vec") Vector vec) | ||
|
|
||
| @Query("INSERT INTO vector_byte_doc(id, embedding) VALUES (VECTOR_DOC_SEQ.nextval, :vec)") | ||
| Future<Integer> saveAsync(@Parameter("vec") Vector.ByteVector vec) | ||
|
|
||
| @Query("SELECT * FROM vector_byte_doc WHERE id = :id") | ||
| Future<List<VectorByteDoc>> findAsync(Long id) | ||
|
|
||
| @Query("UPDATE vector_byte_doc SET embedding = :vec WHERE id = :id") | ||
| Future<Integer> updateAsync(Long id, @Parameter("vec") Vector vec) | ||
|
|
||
| @Query("UPDATE vector_byte_doc SET embedding = :vec WHERE id = :id") | ||
| Future<Integer> updateAsync(Long id, @Parameter("vec") Vector.ByteVector vec) | ||
|
|
||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.