Skip to content

kn: checksum bindings #121

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Dec 9, 2024
Merged

kn: checksum bindings #121

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
63ac5a4
Add package to checksum class files and implement MD5
lauzadis Nov 18, 2024
9938e2b
ktlint
lauzadis Nov 18, 2024
6044551
downgrade back to ios 15
lauzadis Nov 18, 2024
b668b29
Call CRT.initRuntime in checksum functions
lauzadis Nov 18, 2024
943ee32
Bump plugin version
lauzadis Nov 18, 2024
8fc034a
Fix hashing of empty inputs
lauzadis Nov 18, 2024
e4f58d6
ktlint
lauzadis Nov 18, 2024
5757c98
Add `WithCrt`
lauzadis Nov 18, 2024
8a5669a
Add KDocs
lauzadis Nov 18, 2024
c21bbe4
ktlint
lauzadis Nov 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions aws-crt-kotlin/native/src/aws/sdk/kotlin/crt/WithCrt.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
/*
* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
* SPDX-License-Identifier: Apache-2.0
*/
package aws.sdk.kotlin.crt

/**
* A mixin class used to ensure CRT is initialized before the class is invoked
*/
public open class WithCrt {
init {
CRT.initRuntime { }
}
}
Comment on lines +7 to +14
Copy link
Contributor

@ianbotsf ianbotsf Nov 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: I'm concerned with this approach for two reasons:

  1. Kotlin doesn't support true mixin classes so this pattern prevents inheritors from using any other superclass. Given that this functionality isn't truly polymorphic it seems prudent to leave the option open.
  2. This initializes the CRT runtime at object instantiation rather than when CRT functionality is actually necessary. If some code just instantiates a new Crc32 but never uses any methods on it, we'll've wasted cycles unnecessarily.

Could we move the initialization closer to the site of use?

Copy link
Member Author

@lauzadis lauzadis Nov 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we move the initialization closer to the site of use?

Sure, we'll just have to take care to initialize CRT everywhere we use it, which we currently don't do.

Copy link
Member Author

@lauzadis lauzadis Nov 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If some code just instantiates a new Crc32 but never uses any methods on it, we'll've wasted cycles unnecessarily.

This is true but in practice CRT is only initialized once, subsequent calls to CRT.initRuntime { } are no-ops. I'm assuming users of aws-sdk-kotlin Kotlin/Native will need to initialize CRT at some point

Copy link
Member Author

@lauzadis lauzadis Nov 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We spoke offline and agreed to continue this approach for now. Ideally we would create a singleton object CrtLib and use cinterop to generate extensions for it (like CrtLib.aws_checksums_crc32(...)). This CrtLib object would ensure CRT.initRuntime { ... } is called before invoking the actual C function.

3 changes: 1 addition & 2 deletions aws-crt-kotlin/native/src/aws/sdk/kotlin/crt/util/DigestNative.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,9 @@
* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
* SPDX-License-Identifier: Apache-2.0
*/

package aws.sdk.kotlin.crt.util

import Sha256
import aws.sdk.kotlin.crt.util.hashing.Sha256

/**
* Utility object for various hash functions
Expand Down
15 changes: 13 additions & 2 deletions aws-crt-kotlin/native/src/aws/sdk/kotlin/crt/util/hashing/CrcNative.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@
* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
* SPDX-License-Identifier: Apache-2.0
*/
package aws.sdk.kotlin.crt.util.hashing

import aws.sdk.kotlin.crt.WithCrt
import kotlinx.cinterop.*
import libcrt.aws_checksums_crc32
import libcrt.aws_checksums_crc32c
Expand All @@ -21,6 +24,10 @@ internal class Crc(val checksumFn: AwsChecksumsCrcFunction) : HashFunction {
private var crc = 0U

override fun update(input: ByteArray, offset: Int, length: Int) {
if (input.isEmpty() || length == 0) {
return
}

Comment on lines +27 to +30
Copy link
Contributor

@ianbotsf ianbotsf Nov 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: Why is this necessary? What breaks if we just pass an empty blob or a 0 length down to CRT?

Copy link
Member Author

@lauzadis lauzadis Nov 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The request would never make it to CRT, the following code snippet fails with an ArrayIndexOutOfBoundsException exception:

        val offsetInput = input.usePinned {
            it.addressOf(offset)
        }

val offsetInput = input.usePinned {
it.addressOf(offset)
}
Expand All @@ -42,7 +49,9 @@ internal class Crc(val checksumFn: AwsChecksumsCrcFunction) : HashFunction {
/**
* A CRC32 [HashFunction] implemented using bindings to CRT.
*/
public class Crc32 : HashFunction {
public class Crc32 :
WithCrt(),
HashFunction {
private val crc32 = Crc(::aws_checksums_crc32)
override fun update(input: ByteArray, offset: Int, length: Int) {
crc32.update(input, offset, length)
Expand All @@ -56,7 +65,9 @@ public class Crc32 : HashFunction {
/**
* A CRC32C [HashFunction] implemented using bindings to CRT.
*/
public class Crc32c : HashFunction {
public class Crc32c :
WithCrt(),
HashFunction {
private val crc32c = Crc(::aws_checksums_crc32c)
override fun update(input: ByteArray, offset: Int, length: Int) {
crc32c.update(input, offset, length)
Expand Down
2 changes: 2 additions & 0 deletions aws-crt-kotlin/native/src/aws/sdk/kotlin/crt/util/hashing/HashFunctionNative.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
* SPDX-License-Identifier: Apache-2.0
*/
package aws.sdk.kotlin.crt.util.hashing

/**
* A function which calculates the hash of given data
*/
Expand Down
65 changes: 65 additions & 0 deletions aws-crt-kotlin/native/src/aws/sdk/kotlin/crt/util/hashing/Md5Native.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
/*
* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
* SPDX-License-Identifier: Apache-2.0
*/
package aws.sdk.kotlin.crt.util.hashing

import aws.sdk.kotlin.crt.Allocator
import aws.sdk.kotlin.crt.WithCrt
import aws.sdk.kotlin.crt.awsAssertOpSuccess
import kotlinx.cinterop.addressOf
import kotlinx.cinterop.cValue
import kotlinx.cinterop.convert
import kotlinx.cinterop.pointed
import kotlinx.cinterop.reinterpret
import kotlinx.cinterop.usePinned
import libcrt.aws_byte_buf
import libcrt.aws_byte_cursor_from_array
import libcrt.aws_hash_destroy
import libcrt.aws_hash_finalize
import libcrt.aws_hash_update
import libcrt.aws_md5_new

/**
* MD5 hash function implemented using bindings to CRT
*/
public class Md5 :
WithCrt(),
HashFunction {
private var md5 = checkNotNull(aws_md5_new(Allocator.Default)) { "aws_md5_new" }

override fun update(input: ByteArray, offset: Int, length: Int) {
if (input.isEmpty() || length == 0) {
return
}

val inputCursor = input.usePinned {
aws_byte_cursor_from_array(it.addressOf(offset), length.convert())
}

awsAssertOpSuccess(aws_hash_update(md5, inputCursor)) {
"aws_hash_update"
}
}

override fun digest(): ByteArray {
val output = ByteArray(md5.pointed.digest_size.toInt())

val byteBuf = output.usePinned {
cValue<aws_byte_buf> {
capacity = output.size.convert()
len = 0U
buffer = it.addressOf(0).reinterpret()
}
}

awsAssertOpSuccess(aws_hash_finalize(md5, byteBuf, 0U)) { "aws_hash_finalize" }

return output.also { reset() }
}

override fun reset() {
aws_hash_destroy(md5)
md5 = checkNotNull(aws_md5_new(Allocator.Default)) { "aws_md5_new" }
}
}
15 changes: 13 additions & 2 deletions aws-crt-kotlin/native/src/aws/sdk/kotlin/crt/util/hashing/ShaNative.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,10 @@
* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
* SPDX-License-Identifier: Apache-2.0
*/
package aws.sdk.kotlin.crt.util.hashing

import aws.sdk.kotlin.crt.Allocator
import aws.sdk.kotlin.crt.WithCrt
import aws.sdk.kotlin.crt.awsAssertOpSuccess
import kotlinx.cinterop.*
import libcrt.*
Expand All @@ -17,7 +20,9 @@ internal typealias InitializeHashFn = (
/**
* SHA-1 hash function implemented using bindings to CRT
*/
public class Sha1 : HashFunction {
public class Sha1 :
WithCrt(),
HashFunction {
private val sha1 = Sha(::aws_sha1_new)
override fun update(input: ByteArray, offset: Int, length: Int) {
sha1.update(input, offset, length)
Expand All @@ -31,7 +36,9 @@ public class Sha1 : HashFunction {
/**
* SHA-256 hash function implemented using bindings to CRT
*/
public class Sha256 : HashFunction {
public class Sha256 :
WithCrt(),
HashFunction {
private val sha256 = Sha(::aws_sha256_new)
override fun update(input: ByteArray, offset: Int, length: Int) {
sha256.update(input, offset, length)
Expand All @@ -51,6 +58,10 @@ internal class Sha(val initializeFn: InitializeHashFn) : HashFunction {

// aws_hash_update
override fun update(input: ByteArray, offset: Int, length: Int) {
if (input.isEmpty() || length == 0) {
return
}

val inputCursor = input.usePinned {
aws_byte_cursor_from_array(it.addressOf(offset), length.convert())
}
Expand Down
19 changes: 19 additions & 0 deletions aws-crt-kotlin/native/test/aws/sdk/kotlin/crt/util/hashing/HashFunctionNativeTest.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
* SPDX-License-Identifier: Apache-2.0
*/
package aws.sdk.kotlin.crt.util.hashing

import aws.sdk.kotlin.crt.util.encodeToHex
import kotlin.test.Test
import kotlin.test.assertEquals
Expand Down Expand Up @@ -48,4 +50,21 @@ class HashFunctionTest {
assertEquals(expected, hash.digest().encodeToHex())
}
}

@Test
fun testEmptyUpdate() {
// algorithm -> hash("")
val tests = listOf(
(Sha1() to "da39a3ee5e6b4b0d3255bfef95601890afd80709"),
(Sha256() to "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"),
(Crc32() to "00000000"),
(Crc32c() to "00000000"),
)

tests.forEach { (hash, expected) ->
val data = "".encodeToByteArray()
hash.update(data, 0, 0)
assertEquals(expected, hash.digest().encodeToHex())
}
}
}
2 changes: 1 addition & 1 deletion gradle/libs.versions.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[versions]
kotlin-version = "2.0.21"

aws-kotlin-repo-tools-version = "0.4.14-kn"
aws-kotlin-repo-tools-version = "0.4.16-kn"

# libs
crt-java-version = "0.31.3"
Expand Down
Loading