-
Notifications
You must be signed in to change notification settings - Fork 19
C2D Docker #705
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
C2D Docker #705
Conversation
FYI, i've created a PR on Ocean JS related with the breaking changes: other changes also included in this PR here: #735 #737 #736 #739 |
Hi @alexcos20 , what if the ' |
|
We should update the documentation to explain how this works and how to set it up. |
|
I tried to test it via the directCommand endpoint but I found it causes the node to crash. I'll give it another go when it's marked as ready for review, as changes are still being made. |
|
Update: it seems to work via the direct command now but I will do some more testing to see if there are any more bugs |
sure, but this is a draft in progress :-) expect it |
| @@ -0,0 +1 @@ | |||
| declare module 'docker-registry-client' | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This file seems to have a typo. Presumably it should have been src/@types/docker-registry-client.ts
| import { | ||
| ComputeGetEnvironmentsHandler, | ||
| ComputeStartHandler, | ||
| // ComputeStartHandler, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are a lot of imports and variables in this file that are commented out. Can we just remove them?
| @@ -0,0 +1,236 @@ | |||
| import { C2DDatabase } from '../../components/database/C2DDatabase.js' | |||
| // import { existsEnvironmentVariable, getConfiguration } from '../../utils/config.js' | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| // import { existsEnvironmentVariable, getConfiguration } from '../../utils/config.js' |
| C2DStatusText, | ||
| ComputeAlgorithm, | ||
| ComputeAsset, | ||
| // ComputeEnvironment, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| // ComputeEnvironment, |
| DBComputeJob, | ||
| RunningPlatform | ||
| } from '../../@types/C2D/C2D.js' | ||
| // import { computeAsset } from '../data/assets' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| // import { computeAsset } from '../data/assets' |
|
we need to fix the conflicts and eventually merge this |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks like there was a bug introduced in #860. Now the job gets stuck on publishing results status, which wasn't happening before:
Logs
Checking job status...
Job status: {owner: '0x4c495310EF3259830FEd12ea90DbB0B3D39103FD', did: null, jobId: '540f8543-ef1a-4623-9f5a-cef34a8c1cbb', dateCreated: '1742294240.728', dateFinished: null, …}
Status text: Job started
Checking job status...
Job status: {owner: '0x4c495310EF3259830FEd12ea90DbB0B3D39103FD', did: null, jobId: '540f8543-ef1a-4623-9f5a-cef34a8c1cbb', dateCreated: '1742294240.728', dateFinished: null, …}
Status text: Configuring volumes
Checking job status...
Job status: {owner: '0x4c495310EF3259830FEd12ea90DbB0B3D39103FD', did: null, jobId: '540f8543-ef1a-4623-9f5a-cef34a8c1cbb', dateCreated: '1742294240.728', dateFinished: null, …}
Status text: Running algorithm
Using nonce: 1742294255389
Generated result signature: 0xc183940019df1ed5b07c5fb24078ab61daf38b2d4708fe2528159eccb470ccb41bd3c689b27af3738cec07d40fff8a3f4c00439df418416c1ff66b99312debfd1c
Result signature valid: true
Response: Response {size: 0, timeout: 0, Symbol(Body internals): {…}, Symbol(Response internals): {…}}
Response body: PassThrough {_events: {…}, _readableState: ReadableState, _writableState: WritableState, allowHalfOpen: true, _maxListeners: undefined, …}
Checking job status...
Job status: {owner: '0x4c495310EF3259830FEd12ea90DbB0B3D39103FD', did: null, jobId: '540f8543-ef1a-4623-9f5a-cef34a8c1cbb', dateCreated: '1742294240.728', dateFinished: null, …}
Status text: Running algorithm
Stream complete
Checking job status...
Job status: {owner: '0x4c495310EF3259830FEd12ea90DbB0B3D39103FD', did: null, jobId: '540f8543-ef1a-4623-9f5a-cef34a8c1cbb', dateCreated: '1742294240.728', dateFinished: null, …}
Status text: Publishing results
Checking job status...
Job status: {owner: '0x4c495310EF3259830FEd12ea90DbB0B3D39103FD', did: null, jobId: '540f8543-ef1a-4623-9f5a-cef34a8c1cbb', dateCreated: '1742294240.728', dateFinished: null, …}
Status text: Publishing results
Checking job status...
Job status: {owner: '0x4c495310EF3259830FEd12ea90DbB0B3D39103FD', did: null, jobId: '540f8543-ef1a-4623-9f5a-cef34a8c1cbb', dateCreated: '1742294240.728', dateFinished: null, …}
Status text: Publishing results
Job status: {owner: '0x4c495310EF3259830FEd12ea90DbB0B3D39103FD', did: null, jobId: '540f8543-ef1a-4623-9f5a-cef34a8c1cbb', dateCreated: '1742294240.728', dateFinished: null, …}
Status text: Publishing results
Checking job status...
Job status: {owner: '0x4c495310EF3259830FEd12ea90DbB0B3D39103FD', did: null, jobId: '540f8543-ef1a-4623-9f5a-cef34a8c1cbb', dateCreated: '1742294240.728', dateFinished: null, …}
Status text: Publishing results
Checking job status...
Job status: {owner: '0x4c495310EF3259830FEd12ea90DbB0B3D39103FD', did: null, jobId: '540f8543-ef1a-4623-9f5a-cef34a8c1cbb', dateCreated: '1742294240.728', dateFinished: null, …}
Status text: Publishing results
Checking job status...
Job status: {owner: '0x4c495310EF3259830FEd12ea90DbB0B3D39103FD', did: null, jobId: '540f8543-ef1a-4623-9f5a-cef34a8c1cbb', dateCreated: '1742294240.728', dateFinished: null, …}
Status text: Publishing results
Checking job status...
Job status: {owner: '0x4c495310EF3259830FEd12ea90DbB0B3D39103FD', did: null, jobId: '540f8543-ef1a-4623-9f5a-cef34a8c1cbb', dateCreated: '1742294240.728', dateFinished: null, …}
Status text: Publishing results
Checking job status...
Job status: {owner: '0x4c495310EF3259830FEd12ea90DbB0B3D39103FD', did: null, jobId: '540f8543-ef1a-4623-9f5a-cef34a8c1cbb', dateCreated: '1742294240.728', dateFinished: null, …}
Status text: Publishing results
Checking job status...
Job status: {owner: '0x4c495310EF3259830FEd12ea90DbB0B3D39103FD', did: null, jobId: '540f8543-ef1a-4623-9f5a-cef34a8c1cbb', dateCreated: '1742294240.728', dateFinished: null, …}
Status text: Publishing results
Checking job status...
Job status: {owner: '0x4c495310EF3259830FEd12ea90DbB0B3D39103FD', did: null, jobId: '540f8543-ef1a-4623-9f5a-cef34a8c1cbb', dateCreated: '1742294240.728', dateFinished: null, …}
Status text: Publishing results
Checking job status...
Job status: {owner: '0x4c495310EF3259830FEd12ea90DbB0B3D39103FD', did: null, jobId: '540f8543-ef1a-4623-9f5a-cef34a8c1cbb', dateCreated: '1742294240.728', dateFinished: null, …}
Status text: Publishing results
Job status: {owner: '0x4c495310EF3259830FEd12ea90DbB0B3D39103FD', did: null, jobId: '540f8543-ef1a-4623-9f5a-cef34a8c1cbb', dateCreated: '1742294240.728', dateFinished: null, …}
Status text: Publishing results
Checking job status...
Job status: {owner: '0x4c495310EF3259830FEd12ea90DbB0B3D39103FD', did: null, jobId: '540f8543-ef1a-4623-9f5a-cef34a8c1cbb', dateCreated: '1742294240.728', dateFinished: null, …}
Status text: Publishing results
Checking job status...
Job status: {owner: '0x4c495310EF3259830FEd12ea90DbB0B3D39103FD', did: null, jobId: '540f8543-ef1a-4623-9f5a-cef34a8c1cbb', dateCreated: '1742294240.728', dateFinished: null, …}
Status text: Publishing results
Checking job status...
Job status: {owner: '0x4c495310EF3259830FEd12ea90DbB0B3D39103FD', did: null, jobId: '540f8543-ef1a-4623-9f5a-cef34a8c1cbb', dateCreated: '1742294240.728', dateFinished: null, …}
Status text: Publishing results
Checking job status...
Job status: {owner: '0x4c495310EF3259830FEd12ea90DbB0B3D39103FD', did: null, jobId: '540f8543-ef1a-4623-9f5a-cef34a8c1cbb', dateCreated: '1742294240.728', dateFinished: null, …}
Status text: Publishing results
Checking job status...
Job status: {owner: '0x4c495310EF3259830FEd12ea90DbB0B3D39103FD', did: null, jobId: '540f8543-ef1a-4623-9f5a-cef34a8c1cbb', dateCreated: '1742294240.728', dateFinished: null, …}
Status text: Publishing results
| DOCKER_SOCKET_PATH: { | ||
| name: 'DOCKER_SOCKET_PATH', | ||
| value: process.env.DOCKER_SOCKET_PATH, | ||
| required: false | ||
| }, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We're not using this anymore, can we remove it?
scripts/ocean-node-quickstart.sh
Outdated
| read -p "Enter the docker socket path: " DOCKER_SOCKET_PATH | ||
| DOCKER_SOCKET_PATH=${DOCKER_SOCKET_PATH:-''} | ||
| read -p "Enter the docker protocol: " DOCKER_PROTOCOL | ||
| DOCKER_PROTOCOL=${DOCKER_PROTOCOL:-''} | ||
| read -p "Enter the docker host: " DOCKER_HOST | ||
| DOCKER_HOST=${DOCKER_HOST:-''} | ||
| read -p "Enter the docker port: " DOCKER_PORT | ||
| DOCKER_PORT=${DOCKER_PORT:-0} | ||
| read -p "Enter the docker certificate authority path: " DOCKER_CA_PATH | ||
| DOCKER_CA_PATH=${DOCKER_CA_PATH:-''} | ||
| read -p "Enter the docker certificate path: " DOCKER_CERT_PATH | ||
| DOCKER_CERT_PATH=${DOCKER_CERT_PATH:-''} | ||
| read -p "Enter the docker key path: " DOCKER_KEY_PATH | ||
| DOCKER_KEY_PATH=${DOCKER_KEY_PATH:-''} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As far as I know, we are not using any of these anymore, can we remove them?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
make socker path hardcoded if you want and remove the rest
|
There seems to be something wrong with the deployed version we have. Sometimes, I get this error: Never seen this error before when I'm running the node on my vm. |
I can confirm this behaviour, tested with vscode-extension, now the Node URL points to the latest C2D modifications (including #860) and it gets stuck to the |
|
Let's also fix the conflicts here because it might cause the stucking problem when publishing results in vscode-extension. |
|
Testing with the updated branch still generates an infinite loop when publishing results in both scenarios (success and failure of the algo execution). algorithm: {
meta: {
rawcode: 'import time\n' +
'import asyncio\n' +
'import os\n' +
'\n' +
'# Constants for timing (in seconds)\n' +
'TOTAL_DURATION = 10 # 10 seconds\n' +
'LOG_INTERVAL = 1 # 1 second\n' +
'\n' +
'async def run_logging():\n' +
" print('RAW CODE: Starting logging process...')\n" +
' \n' +
' start_time = time.time()\n' +
' current_iteration = 1\n' +
' results = []\n' +
' \n' +
' while True:\n' +
' elapsed_time = time.time() - start_time\n' +
' \n' +
" log_entry = f'Log iteration {current_iteration}: {elapsed_time:.3f} seconds elapsed'\n" +
' print(log_entry)\n' +
' results.append(log_entry)\n' +
' current_iteration += 1\n' +
' \n' +
' if elapsed_time >= TOTAL_DURATION:\n' +
" print('Completed')\n" +
' \n' +
" # Create the output directory if it doesn't exist\n" +
" output_dir = '/data/outputs'\n" +
' os.makedirs(output_dir, exist_ok=True)\n' +
' \n' +
' # Save results to a text file\n' +
" txt_file = f'{output_dir}/results.txt'\n" +
' \n' +
" with open(txt_file, 'w') as f:\n" +
" f.write(f'PY Algorithm Results\\n')\n" +
" f.write(f'Total time: {elapsed_time:.3f} seconds\\n')\n" +
" f.write(f'Total iterations: {current_iteration - 1}\\n')\n" +
' \n' +
' print(f"Results saved as {txt_file}")\n' +
" return 'completed'\n" +
' \n' +
' await asyncio.sleep(LOG_INTERVAL)\n' +
'\n' +
'if __name__ == "__main__":\n' +
' asyncio.run(run_logging())',
container: [Object]
}
},
assets: [],
isRunning: true,
isStarted: false,
containerImage: 'oceanprotocol/algo_dockers:python-branin',
|
|
If I selected to run 2025-03-18T20:50:18.750Z info: CORE: CORE: consumer address and nonce signature mismatch
StoppedAnd still the job execution gets stuck at |
…/ocean-node into feature/c2d_docker
This is fixed now |
Yeah, I've tested it, and it works end to end now. |
|
I'm still occasionally getting a timeout error: Not sure what could be causing this; it doesn't happen every time. When it occurs, there are no errors in the node logs. I've seen it on both |
any logs in logs folder? |
* add some basic docs for c2d Co-authored-by: giurgiur99 <giurgiur99@gmail.com>

Breaking Changes proposed in this PR (requires other repos to be updated):
Changes: instead of having url, we have fileObject now, so we can support multiple types of non published assets
ocean-node/src/@types/commands.ts
Line 159 in d26a8c5
Changes: instead of having
datasetandadditionalDatasets, we merged them all underdatasetsChanges proposed in this PR:
All additional work should be based on PRs based against this one
Usage:
and then do a directCommand:
{ "command": "freeStartCompute", "consumerAddress": "0xC7EC1970B09224B317c52d92f37F5e1E4fF6B687", "nonce": 1, "signature": "0x123", "datasets": [ { "fileObject": { "type": "url", "url": "SOME_DATASET_URL", "method": "GET" } } ], "algorithm": { "fileObject": { "type": "url", "url": "SOME_ALGO_URL", "method": "GET" }, "meta": { "container": { "image": "SOME_CONTAINER", "tag": "latest", "entrypoint": "python $ALGO'" } } } }