Cannot create SageMaker Endpoint with Embed v3: poseidon exited with error

[ohbuchim]

I subscribed Cohere Embed Model v3 - Multilingual and tried to create a SageMaker endpoint using https://github.com/cohere-ai/cohere-aws/blob/main/notebooks/sagemaker/Deploy%20embed%20v3%20model.ipynb.

The endpoint creation failed and I found this error message in CloudWatch logs.

{
    "severity": "ERROR",
    "timestamp": "2024-11-13T00:04:31.072295282Z",
    "caller": "single-container/main.go:454",
    "message": "single container error",
    "logName": "single-container",
    "error": "poseidon exited with error: exit status 1",
    "logging.googleapis.com/labels": {},
    "logging.googleapis.com/sourceLocation": {
        "file": "github.com/cohere-ai/blobheart/go/cmd/single-container/main.go",
        "line": "454",
        "function": "main.startApps"
    },
    "serviceContext": {
        "service": "single-container"
    },
    "context": {
        "reportLocation": {
            "filePath": "github.com/cohere-ai/blobheart/go/cmd/single-container/main.go",
            "lineNumber": "454",
            "functionName": "main.startApps"
        }
    },
    "stack_trace": "single container error: poseidon exited with error: exit status 1\n\ngoroutine 1 [running]:\nmain.startApps(0xc000800000, 0xc0003e51f0)\n\tgithub.com/cohere-ai/blobheart/go/cmd/single-container/main.go:454 +0x425\nmain.run()\n\tgithub.com/cohere-ai/blobheart/go/cmd/single-container/main.go:251 +0x1534\nmain.main()\n\tgithub.com/cohere-ai/blobheart/go/cmd/single-container/main.go:139 +0x13\n"
}

[PhMueller-Haufe]

Hi,
the same error happens to me.
Is there any update on this?

Would appreciate any feedback,
Thanks!

[amarcin]

Confirmed behavior for this model, investigating on my end. Cohere team should do the same.

[chanhaosheng]

Hello, I am facing the same issue too for Embed Model v3 - English. Wonder if there is any solution. Thank you.

[ospiegel91]

very same issue ( large global american corp):
{
"severity": "ERROR",
"timestamp": "2025-01-27T09:27:37.492316139Z",
"caller": "single-container/main.go:458",
"message": "single container error",
"logName": "single-container",
"error": "poseidon exited with error: exit status 1",
"logging.googleapis.com/labels": {},
"logging.googleapis.com/sourceLocation": {
"file": "github.com/cohere-ai/blobheart/go/cmd/single-container/main.go",
"line": "458",
"function": "main.startApps"
},
"serviceContext": {
"service": "single-container"
},
"context": {
"reportLocation": {
"filePath": "github.com/cohere-ai/blobheart/go/cmd/single-container/main.go",
"lineNumber": "458",
"functionName": "main.startApps"
}
},
"stack_trace": "single container error: poseidon exited with error: exit status 1\n\ngoroutine 1 [running]:\nmain.startApps(0xc000792000, 0xc000417b20)\n\tgithub.com/cohere-ai/blobheart/go/cmd/single-container/main.go:458 +0x425\nmain.run()\n\tgithub.com/cohere-ai/blobheart/go/cmd/single-container/main.go:251 +0x1534\nmain.main()\n\tgithub.com/cohere-ai/blobheart/go/cmd/single-container/main.go:139 +0x13\n"