Skip to content

Commit f53e93c

Browse files
authored
Merge pull request #214141 from elizabethhalper/elhalper/sdutil-changes
Address General Issues
2 parents b1f0c60 + e644ac9 commit f53e93c

File tree

1 file changed

+103
-163
lines changed

1 file changed

+103
-163
lines changed

articles/energy-data-services/tutorial-seismic-ddms-sdutil.md

Lines changed: 103 additions & 163 deletions
Original file line numberDiff line numberDiff line change
@@ -33,28 +33,24 @@ Linux
3333

3434
- [64-bit Python 3.8.3](https://www.python.org/ftp/python/3.8.3/Python-3.8.3.tgz)
3535

36-
Unix
36+
Unix/Mac
3737

3838
- [64-bit Python 3.8.3](https://www.python.org/ftp/python/3.8.3/Python-3.8.3.tgz)
3939
- Apple Xcode C++ Build Tools
4040

41-
Other requirements are addressed in the [installation](#installation) section below.
42-
43-
## Installation
44-
45-
Follow the directions in the sdutil documentation for [running sdutil in Azure environments](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-sdutil/-/tree/azure/stable#setup-and-usage-for-azure-env).
46-
47-
The utility requires other modules noted in [requirements.txt](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-sdutil/-/blob/azure/stable/requirements.txt). You could either install the modules as is or install them in virtualenv to keep your host clean from package conflicts. If you don't want to install them in a virtual environment, jump directly to step 3.
41+
The utility requires other modules noted in [requirements.txt](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-sdutil/-/blob/azure/stable/requirements.txt). You could either install the modules as is or install them in virtualenv to keep your host clean from package conflicts. If you don't want to install them in a virtual environment, skip the four virtual environment commands below. Additionally, if you are using Mac instead of Ubuntu or WSL - Ubuntu 20.04, either use `homebrew` instead of `apt-get` as your package manager, or manually install `apt-get`.
4842

4943
```bash
5044
# check if virtualenv is already installed
5145
virtualenv --version
5246

53-
# if not install it via pip
47+
# if not install it via pip or apt-get
5448
pip install virtualenv
49+
# or sudo apt-get install python3-venv for WSL
5550

5651
# create a virtual environment for sdutil
5752
virtualenv sdutilenv
53+
# or python3 -m venv sdutilenv for WSL
5854

5955
# activate the virtual environemnt
6056
Windows: sdutilenv/Scripts/activate
@@ -64,99 +60,99 @@ The utility requires other modules noted in [requirements.txt](https://community
6460
Install required dependencies:
6561

6662
```bash
67-
# run it from the extracted sdutil folder
63+
# run this from the extracted sdutil folder
6864
pip install -r requirements.txt
6965
```
7066

7167
## Usage
7268

7369
### Configuration
7470

75-
1. Clone the [sdutil repository](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-sdutil/-/tree/azure/stable) and open in your favorite editor.
71+
1. Clone the [sdutil repository](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-sdutil/-/tree/azure/stable) from the community Azure Stable branch and open in your favorite editor.
7672

7773
2. Replace the contents of `config.yaml` in the `sdlib` folder with the following yaml and fill in the three templatized values (two instances of `<meds-instance-url>` and one `<put refresh token here...>`):
7874

79-
```yaml
80-
seistore:
81-
service: '{"azure": {"azureGlabEnv":{"url": "https://<meds-instance-url>/seistore-svc/api/v3", "appkey": ""}}}'
82-
url: 'https://<meds-instance-url>/seistore-svc/api/v3'
83-
cloud_provider: 'azure'
84-
env: 'glab'
85-
auth-mode: 'JWT Token'
86-
ssl_verify: False
87-
auth_provider:
88-
azure: '{
89-
"provider": "azure",
90-
"authorize_url": "https://login.microsoftonline.com/",
91-
"oauth_token_host_end": "/oauth2/token",
92-
"scope_end":"/.default openid profile offline_access",
93-
"redirect_uri":"http://localhost:8080",
94-
"login_grant_type": "refresh_token",
95-
"refresh_token": "<put refresh token here from auth_token.http authorize request>"
96-
}'
97-
azure:
98-
empty: 'none'
99-
```
100-
101-
> [!NOTE]
102-
> Follow the directions in [How to Generate a Refresh Token](how-to-generate-refresh-token.md) to obtain a token if not already present.
75+
```yaml
76+
seistore:
77+
service: '{"azure": {"azureGlabEnv":{"url": "https://<meds-instance-url>/seistore-svc/api/v3", "appkey": ""}}}'
78+
url: 'https://<meds-instance-url>/seistore-svc/api/v3'
79+
cloud_provider: 'azure'
80+
env: 'glab'
81+
auth-mode: 'JWT Token'
82+
ssl_verify: False
83+
auth_provider:
84+
azure: '{
85+
"provider": "azure",
86+
"authorize_url": "https://login.microsoftonline.com/",
87+
"oauth_token_host_end": "/oauth2/token",
88+
"scope_end":"/.default openid profile offline_access",
89+
"redirect_uri":"http://localhost:8080",
90+
"login_grant_type": "refresh_token",
91+
"refresh_token": "<put refresh token here from auth_token.http authorize request>"
92+
}'
93+
azure:
94+
empty: 'none'
95+
```
96+
97+
> [!NOTE]
98+
> Follow the directions in [How to Generate a Refresh Token](how-to-generate-refresh-token.md) to obtain a token if not already present.
10399
104100
3. Export or set below environment variables
105101
106-
```bash
107-
export AZURE_TENANT_ID=check-env-provisioning-team-as-specific-to-cluster
108-
export AZURE_CLIENT_ID=check-env-provisioning-team-as-specific-to-cluster
109-
export AZURE_CLIENT_SECRET=check-env-provisioning-team-as-specific-to-cluster
110-
```
102+
```bash
103+
export AZURE_TENANT_ID=<your-tenant-id>
104+
export AZURE_CLIENT_ID=<your-client-id>
105+
export AZURE_CLIENT_SECRET=<your-client-secret>
106+
```
111107

112108
### Running the Tool
113109

114110
1. Run the utility from the extracted utility folder by typing:
115111

116-
```bash
117-
python sdutil
118-
```
119-
120-
If no arguments are specified, this menu will be displayed:
121-
122-
```code
123-
Seismic Store Utility
124-
125-
> python sdutil [command]
112+
```bash
113+
python sdutil
114+
```
126115

127-
available commands:
116+
If no arguments are specified, this menu will be displayed:
128117

129-
* auth : authentication utilities
130-
* unlock : remove a lock on a seismic store dataset
131-
* version : print the sdutil version
132-
* rm : delete a subproject or a space separated list of datasets
133-
* mv : move a dataset in seismic store
134-
* config : manage the utility configuration
135-
* mk : create a subproject resource
136-
* cp : copy data to(upload)/from(download)/in(copy) seismic store
137-
* stat : print information like size, creation date, legal tag(admin) for a space separated list of tenants, subprojects or datasets
138-
* patch : patch a seismic store subproject or dataset
139-
* app : application authorization utilities
140-
* ls : list subprojects and datasets
141-
* user : user authorization utilities
142-
```
118+
```code
119+
Seismic Store Utility
120+
121+
> python sdutil [command]
122+
123+
available commands:
124+
125+
* auth : authentication utilities
126+
* unlock : remove a lock on a seismic store dataset
127+
* version : print the sdutil version
128+
* rm : delete a subproject or a space separated list of datasets
129+
* mv : move a dataset in seismic store
130+
* config : manage the utility configuration
131+
* mk : create a subproject resource
132+
* cp : copy data to(upload)/from(download)/in(copy) seismic store
133+
* stat : print information like size, creation date, legal tag(admin) for a space separated list of tenants, subprojects or datasets
134+
* patch : patch a seismic store subproject or dataset
135+
* app : application authorization utilities
136+
* ls : list subprojects and datasets
137+
* user : user authorization utilities
138+
```
143139

144140
2. If this is your first time using the tool, you must run the sdutil config init command to initialize the configuration.
145141

146-
```bash
147-
python sdutil config init
148-
```
142+
```bash
143+
python sdutil config init
144+
```
149145

150146
3. Before you start using the utility and performing any operations, you must sign in the system. When you run the following sign in command, sdutil will open a sign in page in a web browser.
151147

152-
```bash
153-
python sdutil auth login
154-
```
148+
```bash
149+
python sdutil auth login
150+
```
155151

156-
Once you've successfully logged in, your credentials will be valid for a week. You don't need to sign in again unless the credentials expired (after one week), in this case the system will require you to sign in again.
152+
Once you've successfully logged in, your credentials will be valid for a week. You don't need to sign in again unless the credentials expired (after one week), in this case the system will require you to sign in again.
157153

158-
> [!NOTE]
159-
> If you aren't getting the "sign in Successful!" message, make sure your three environment variables are set and you've followed all steps in the "Configuration" section above.
154+
> [!NOTE]
155+
> If you aren't getting the "sign in Successful!" message, make sure your three environment variables are set and you've followed all steps in the "Configuration" section above.
160156

161157
## Seistore Resources
162158

@@ -305,111 +301,55 @@ Run the changelog script (`./changelog-generator.sh`) to automatically generate
305301
./scripts/changelog-generator.sh
306302
```
307303

308-
## Setup and usage for Microsoft Energy Data Services
304+
## Usage for Microsoft Energy Data Services
309305

310-
Below steps are for Windows Subsystem for Linux - Ubuntu 20.04
311-
Microsoft Energy Data Services instance is using OSDU&trade; M12 Version of sdutil
306+
Microsoft Energy Data Services instance is using OSDU&trade; M12 Version of sdutil. Follow the below steps if you would like to use SDUTIL to leverage the SDMS API of your MEDS instance.
312307

313-
1. Download the source code from community [sdutil](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-sdutil/-/tree/azure/stable/) Azure Stable branch.
308+
1. Ensure you have followed the [installation](#prerequisites) and [configuration](#configuration) steps from above. This includes downloading the SDUTIL source code, configuring your python virtual environment, editing the `config.yaml` file and setting your three environment variables.
314309

315-
2. In case python virtual env isn't installed, use below commands. Otherwise, skip to next section
310+
2. Run below commands to sign in, list, upload and download files in the seismic store.
316311

317-
```bash
318-
sudo apt-get update
319-
sudo apt-get install python3-venv
320-
```
321-
322-
3. Create a new virtual environment and install package
323-
324-
```bash
325-
#create new virtual env with name : sdutilenv
326-
python3 -m venv sdutilenv
327-
328-
#activate the virtual end
329-
source sdutilenv/bin/Activate
330-
331-
#install python package for sdutil
332-
pip install -r requirements.txt
333-
```
334-
335-
4. Replace the contents of `config.yaml` in the `sdlib` folder with the following yaml and fill in the three templatized values (tow `<meds-instance-url>` and `<put refresh token here...>`):
336-
337-
```yaml
338-
seistore:
339-
service: '{"azure": {"azureGlabEnv":{"url": "https://<meds-instance-url>/seistore-svc/api/v3", "appkey": ""}}}'
340-
url: 'https://<meds-instance-url>/seistore-svc/api/v3'
341-
cloud_provider: 'azure'
342-
env: 'glab'
343-
auth-mode: 'JWT Token'
344-
ssl_verify: False
345-
auth_provider:
346-
azure: '{
347-
"provider": "azure",
348-
"authorize_url": "https://login.microsoftonline.com/",
349-
"oauth_token_host_end": "/oauth2/token",
350-
"scope_end":"/.default openid profile offline_access",
351-
"redirect_uri":"http://localhost:8080",
352-
"login_grant_type": "refresh_token",
353-
"refresh_token": "<put refresh token here from auth_token.http authorize request>"
354-
}'
355-
azure:
356-
empty: 'none'
357-
```
358-
359-
> [!NOTE]
360-
> Follow the directions in [How to Generate a Refresh Token](how-to-generate-refresh-token.md) to obtain a token if not already present.
361-
362-
5. Export or set below environment variables
363-
364-
```bash
365-
export AZURE_TENANT_ID=check-env-provisioning-team-as-specific-to-cluster
366-
export AZURE_CLIENT_ID=check-env-provisioning-team-as-specific-to-cluster
367-
export AZURE_CLIENT_SECRET=check-env-provisioning-team-as-specific-to-cluster
368-
```
312+
1. Initialize
369313

370-
6. Run below commands to sign in, list, upload and download files in the seismic store.
314+
```code
315+
(sdutilenv) > python sdutil config init
316+
[one] Azure
317+
Select the cloud provider: **enter 1**
318+
Insert the Azure (azureGlabEnv) application key: **just press enter--no need to provide a key**
371319
372-
- Initialize
320+
sdutil successfully configured to use Azure (azureGlabEnv)
373321
374-
```code
375-
(sdutilenv) > python sdutil config init
376-
[one] Azure
377-
Select the cloud provider: **enter 1**
378-
Insert the Azure (azureGlabEnv) application key: **just press enter--no need to provide a key**
379-
380-
sdutil successfully configured to use Azure (azureGlabEnv)
381-
382-
Should display sign in success message. Credentials expiry set to 1 hour.
383-
```
322+
Should display sign in success message. Credentials expiry set to 1 hour.
323+
```
384324

385-
- Sign in
325+
2. Sign in
386326

387-
```bash
388-
python sdutil config init
389-
python sdutil auth login
390-
```
327+
```bash
328+
python sdutil config init
329+
python sdutil auth login
330+
```
391331

392-
- List files in your seismic store
332+
3. List files in your seismic store
393333

394-
```bash
395-
python sdutil ls sd://<tenant> # e.g. sd://<instance-name>-<datapartition>
396-
python sdutil ls sd://<tenant>/<subproject> # e.g. sd://<instance-name>-<datapartition>/test
397-
```
334+
```bash
335+
python sdutil ls sd://<tenant> # e.g. sd://<instance-name>-<datapartition>
336+
python sdutil ls sd://<tenant>/<subproject> # e.g. sd://<instance-name>-<datapartition>/test
337+
```
398338

399-
- Upload a file from your local machine to the seismic store
339+
4. Upload a file from your local machine to the seismic store
400340

401-
```bash
402-
python sdutil cp local-dir/file-name-at-source.txt sd://<datapartition>/test/file-name-at-destination.txt
403-
```
341+
```bash
342+
python sdutil cp local-dir/file-name-at-source.txt sd://<datapartition>/test/file-name-at-destination.txt
343+
```
404344

405-
- Download a file from the seismic store to your local machine
345+
5. Download a file from the seismic store to your local machine
406346

407-
```bash
408-
python sdutil cp sd://<datapartition>/test/file-name-at-ddms.txt local-dir/file-name-at-destination.txt
409-
```
347+
```bash
348+
python sdutil cp sd://<datapartition>/test/file-name-at-ddms.txt local-dir/file-name-at-destination.txt
349+
```
410350

411-
> [!NOTE]
412-
> Don't use `cp` command to download VDS files. The VDS conversion results in multiple files, therefore the `cp` command won't be able to download all of them in one command. Use either the [SEGYExport](https://osdu.pages.opengroup.org/platform/domain-data-mgmt-services/seismic/open-vds/tools/SEGYExport/README.html) or [VDSCopy](https://osdu.pages.opengroup.org/platform/domain-data-mgmt-services/seismic/open-vds/tools/VDSCopy/README.html) tool instead. These tools use a series of REST calls accessing a [naming scheme](https://osdu.pages.opengroup.org/platform/domain-data-mgmt-services/seismic/open-vds/connection.html) to retrieve information about all the resulting VDS files.
351+
> [!NOTE]
352+
> Don't use `cp` command to download VDS files. The VDS conversion results in multiple files, therefore the `cp` command won't be able to download all of them in one command. Use either the [SEGYExport](https://osdu.pages.opengroup.org/platform/domain-data-mgmt-services/seismic/open-vds/tools/SEGYExport/README.html) or [VDSCopy](https://osdu.pages.opengroup.org/platform/domain-data-mgmt-services/seismic/open-vds/tools/VDSCopy/README.html) tool instead. These tools use a series of REST calls accessing a [naming scheme](https://osdu.pages.opengroup.org/platform/domain-data-mgmt-services/seismic/open-vds/connection.html) to retrieve information about all the resulting VDS files.
413353

414354
OSDU&trade; is a trademark of The Open Group.
415355

0 commit comments

Comments
 (0)