Skip to content

Commit 2eddd6b

Browse files
docs: improve uri doc to clarify scheme and add performance tips (deephaven#7185)
Moving this PR over from the deephaven.io repo DOC-683
1 parent 47c5b7d commit 2eddd6b

File tree

2 files changed

+131
-13
lines changed

2 files changed

+131
-13
lines changed

docs/python/how-to-guides/use-uris.md

Lines changed: 130 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,29 @@
11
---
2+
id: use-uris
23
title: Use URIs to share tables
34
sidebar_label: URI
45
---
56

6-
This guide will show you to use Deephaven's [URIs](/core/pydoc/code/deephaven.uri.html#module-deephaven.uri) to share tables across instances and networks.
7+
This guide will show you how to use Deephaven's [URIs](https://docs.deephaven.io/core/pydoc/code/deephaven.uri.html#module-deephaven.uri) to share tables across server instances and networks.
78

8-
A URI, short for [Uniform Resource Identifier](https://en.wikipedia.org/wiki/Uniform_Resource_Identifier), is a sequence of characters that identifies a resource on the web. Think of a URI as a generalization of a URL. A Deephaven URI identifies a table. By linking to a URI, you share your results with others without them needing to replicate your setup or know the details of your queries.
9+
A URI, short for [Uniform Resource Identifier](https://en.wikipedia.org/wiki/Uniform_Resource_Identifier), is a sequence of characters that identifies a resource on the web. Think of a URI as a generalization of a URL. A Deephaven URI identifies a table on a server instance. By linking to a URI, you can access and work with tables from other Deephaven server instances without needing to replicate the data or queries that created them.
910

1011
> [!NOTE]
11-
> URIs can be used to share tables across Groovy and Python instances interchangably. For how to use URIs in Groovy, see [the equivalent guide](/core/groovy/docs/how-to-guides/use-uris).
12+
> URIs can be used to share tables across Groovy and Python instances interchangeably. For how to use URIs in Groovy, see [the equivalent guide](/core/groovy/docs/how-to-guides/use-uris).
13+
14+
## Why use URIs?
15+
16+
Deephaven URIs provide several key benefits:
17+
18+
- **Canonicalized resource identification**: Access resources through a standardized string format that works across server instances.
19+
- **Simplified data sharing**: Share tables between different Deephaven instances without duplicating data or queries.
20+
- **Distributed computing**: Build systems where processing is distributed across multiple Deephaven nodes.
21+
- **Real-time access**: Access live, updating tables from remote sources that reflect the latest data.
22+
- **Resource abstraction**: Reference remote tables and application fields using a consistent pattern regardless of location.
23+
- **Cross-language compatibility**: Access the same data from both Python and Groovy scripts.
24+
- **Environmental isolation**: Access data across different containers, servers, or networks.
25+
26+
By using URIs, you enable others to directly access your tables without needing to replicate your data pipeline, understand your query logic, or maintain duplicate datasets. This is particularly valuable in collaborative environments and distributed systems.
1227

1328
> [!NOTE]
1429
> URI and Shared Tickets are two different ways to pull tables. Both work on static or dynamic tables. URI pulls tables already on the server via a URL-like string. Shared Tickets let you pull tables you create or access via the Python Client. Learn more about using Shared Tickets with Deephaven in the [Shared Tickets guide](../how-to-guides/capture-tables.md).
@@ -28,15 +43,46 @@ The above URL can be broken down as follows:
2843
- Path
2944
- The path, in this case, is `/core/docs`. It is a path on the authority.
3045

46+
### Deephaven URI structure
47+
3148
Deephaven URIs use a similar syntax:
3249

33-
`dh+plain://<authority>/<path>`
50+
```
51+
dh://<authority>[:<port>]/<scope>/<resource_name>
52+
dh+plain://<authority>[:<port>]/<scope>/<resource_name>
53+
```
54+
55+
The components are:
56+
57+
- **`dh+plain`** or **`dh`** is the scheme.
58+
- `dh://` indicates a secure connection (TLS/SSL).
59+
- `dh+plain://` indicates an insecure connection (no encryption).
60+
- The scheme identifies the protocol for accessing Deephaven resources.
61+
- All Deephaven URIs use one of these schemes, regardless of the application type (script, static, dynamic, qst) configured in [Application Mode](./application-mode.md).
62+
- **`<authority>`** is the authority, which will be either:
63+
- A Docker container name (for local container-to-container communication).
64+
- A hostname/IP address (for network communication).
65+
- **`<port>`** is optional and only needed when:
66+
- The Deephaven instance is running on a non-default port (something other than 10000).
67+
- You're connecting across a network to a specific port.
68+
- **`<scope>`** identifies the namespace where the resource exists. This is typically `scope` for variables created in interactive console sessions, or `app/<app_name>/field` for resources exported from Application Mode applications.
69+
- **`<resource_name>`** is the exact name of the table or resource you want to access.
3470

35-
- `dh+plain` is the scheme.
36-
- `<authority>` is the authority, which will be either a Docker container name or hostname/IP.
37-
- `<path>` is the path to a table, which is generally `scope/<table_name>`.
71+
### Resolving URIs in your code
3872

39-
Let's explore this with a couple of examples.
73+
To access a table via its URI, use the [`resolve`](../reference/data-import-export/uri.md#parameters) function from the `deephaven.uri` module:
74+
75+
```python skip-test
76+
from deephaven.uri import resolve
77+
78+
# Basic usage
79+
table = resolve("dh+plain://hostname/scope/table_name")
80+
81+
# With explicit port
82+
table = resolve("dh+plain://hostname:9876/scope/table_name")
83+
```
84+
85+
The `resolve` function connects to the specified Deephaven instance, retrieves the table, and returns it as a local reference that you can use in your code.
4086

4187
## Share tables locally
4288

@@ -86,12 +132,66 @@ resolved_table = resolve("dh+plain://table-producer/scope/my_table")
86132

87133
By resolving the URI, we acquire `my_table` from the `table-producer` container using the syntax given above.
88134

135+
## Resource scopes and paths
136+
137+
A **scope** in a Deephaven URI is a namespace that identifies where a resource exists within a Deephaven server instance. Think of scopes as organizational containers that prevent naming conflicts and provide context for how resources were created.
138+
139+
Deephaven uses scopes to separate resources based on their origin and purpose:
140+
141+
### Query scope (`scope`)
142+
143+
The query scope contains variables created in interactive console sessions - when you create tables, variables, or other objects directly in the Deephaven IDE console or through client connections.
144+
145+
```python
146+
# This creates a table in the query scope
147+
from deephaven import empty_table
148+
149+
my_table = empty_table(100).update(["X = i", "Y = i * 2"])
150+
# Accessible via: dh://hostname/scope/my_table
151+
```
152+
153+
### Application scope (`app/<app_name>/field`)
154+
155+
The application scope contains fields exported from [Application Mode](./application-mode.md) applications. These are pre-configured resources that are available when the server starts, defined by application scripts.
156+
157+
```python syntax
158+
# In an Application Mode script, this exports a field
159+
# Accessible via: dh://hostname/app/trading_app/field/market_data
160+
```
161+
162+
Scopes ensure that:
163+
164+
- **No naming conflicts**: A table named `trades` in the query scope is completely separate from a field named `trades` in an application scope.
165+
- **Clear resource organization**: You know immediately whether a resource comes from interactive work or a pre-built application.
166+
- **Proper access control**: Different scopes can have different permission models.
167+
168+
### URI format by scope type
169+
170+
```syntax
171+
# Query scope variable (most common)
172+
dh+plain://hostname/scope/table_name
173+
dh://hostname/scope/table_name
174+
175+
# Application field
176+
dh+plain://hostname/app/my_application/field/my_field
177+
dh://hostname/app/my_application/field/my_field
178+
```
179+
180+
> [!NOTE]
181+
> When using URIs to access resources, you must have appropriate permissions to access the resources in those scopes.
182+
89183
## Share tables across a network
90184

91185
Tables can also be shared across networks, public or private. Just like the previous example of sharing across a machine, this works in the same way. Rather than the container name, you only need the hostname/IP and port of the instance producing the table.
92186

93187
> [!NOTE]
94-
> When sharing tables across a network, whether public or private, you do _not_ need the port if Deephaven is being run on the default Deephaven port `10000`. In all other cases, you _must_ provide the port on which the table can be found.
188+
>
189+
> - When sharing tables across a network, you do **not** need to specify the port if Deephaven is running on the default port `10000`.
190+
> - You **must** specify the port in the URI when:
191+
> - The remote Deephaven instance runs on a non-default port (not 10000).
192+
> - You're connecting to a custom port forwarding configuration.
193+
>
194+
> Example format with port: `dh+plain://hostname:9876/scope/table_name`
95195

96196
### Create a table
97197

@@ -119,16 +219,33 @@ If we have the hostname of our colleague's machine, that can be used in place of
119219

120220
If the machine on which a table exists is public, then consuming that table is done the same way as if it were a private network. All that's needed is the hostname/IP and table name.
121221

122-
<!-- TODO:
222+
## Performance considerations
223+
224+
When using URIs to share tables across instances, particularly over networks, there are several performance factors to consider:
225+
226+
### Network impact
227+
228+
- **Latency**: Table access over a network introduces latency that varies based on network conditions. For operations requiring low latency, consider co-locating instances when possible.
229+
- **Bandwidth**: The initial table snapshot and subsequent incremental updates consume bandwidth. Deephaven's Barrage protocol optimizes this by transmitting only changes rather than full table refreshes.
230+
- **Connection reliability**: Unstable network connections can affect the reliability of table access. Implement appropriate error handling for network disruptions.
231+
232+
### Table characteristics
123233

124-
## Paths
234+
- **Initial snapshot**: When first resolving a URI, Deephaven sends a snapshot of the current table state. Larger tables require more resources for this initial transfer.
235+
- **Update frequency**: Tables with high update frequencies generate more incremental updates over the network. Deephaven's Barrage protocol efficiently transmits only the changes (additions, removals, modifications).
236+
- **Column types**: Tables with complex data types like large strings or nested structures may have higher overhead during the initial snapshot and subsequent updates.
125237

126-
Tables can exist in different scopes, such as in app mode and others. When this is the case, the scope changes.
238+
### Optimization strategies
127239

128-
Update this section. I need to learn more about different scopes. -->
240+
- **Only share what's needed**: Filter, aggregate, and limit the amount of data you're sharing to only what a downstream consumer actually needs. This includes applying filters at the source, projecting only necessary columns, and pre-aggregating large datasets to reduce the volume of transferred data.
241+
- **Avoid repeated URI resolution**: Store resolved table references in variables rather than calling `resolve` multiple times for the same URI. Each call to `resolve` creates a new connection, so reuse the table reference when possible within your application.
242+
- **Use appropriate data consistency models**: For analysis requiring consistent data across multiple operations, use table snapshots instead of live updating tables. Point-in-time consistency ensures all your data represents the same moment in time, preventing issues where some data updates mid-analysis while other data remains static. Snapshots freeze the table state at a specific moment, guaranteeing consistent results and reducing network overhead from continuous updates.
129243

130244
## Related documentation
131245

132246
- [`empty_table`](../reference/table-operations/create/emptyTable.md)
133247
- [`time_table`](../reference/table-operations/create/timeTable.md)
134248
- [`update`](../reference/table-operations/select/update.md)
249+
- [Capture Python client tables](./capture-tables.md)
250+
- [Application Mode](./application-mode.md)
251+
- [Pydoc](https://deephaven.io/core/pydoc/code/deephaven.uri.html)
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"file":"how-to-guides/use-uris.md","objects":{"my_table":{"type":"Table","data":{"columns":[{"name":"X","type":"int"},{"name":"Y","type":"int"}],"rows":[[{"value":"0"},{"value":"0"}],[{"value":"1"},{"value":"2"}],[{"value":"2"},{"value":"4"}],[{"value":"3"},{"value":"6"}],[{"value":"4"},{"value":"8"}],[{"value":"5"},{"value":"10"}],[{"value":"6"},{"value":"12"}],[{"value":"7"},{"value":"14"}],[{"value":"8"},{"value":"16"}],[{"value":"9"},{"value":"18"}],[{"value":"10"},{"value":"20"}],[{"value":"11"},{"value":"22"}],[{"value":"12"},{"value":"24"}],[{"value":"13"},{"value":"26"}],[{"value":"14"},{"value":"28"}],[{"value":"15"},{"value":"30"}],[{"value":"16"},{"value":"32"}],[{"value":"17"},{"value":"34"}],[{"value":"18"},{"value":"36"}],[{"value":"19"},{"value":"38"}],[{"value":"20"},{"value":"40"}],[{"value":"21"},{"value":"42"}],[{"value":"22"},{"value":"44"}],[{"value":"23"},{"value":"46"}],[{"value":"24"},{"value":"48"}],[{"value":"25"},{"value":"50"}],[{"value":"26"},{"value":"52"}],[{"value":"27"},{"value":"54"}],[{"value":"28"},{"value":"56"}],[{"value":"29"},{"value":"58"}],[{"value":"30"},{"value":"60"}],[{"value":"31"},{"value":"62"}],[{"value":"32"},{"value":"64"}],[{"value":"33"},{"value":"66"}],[{"value":"34"},{"value":"68"}],[{"value":"35"},{"value":"70"}],[{"value":"36"},{"value":"72"}],[{"value":"37"},{"value":"74"}],[{"value":"38"},{"value":"76"}],[{"value":"39"},{"value":"78"}],[{"value":"40"},{"value":"80"}],[{"value":"41"},{"value":"82"}],[{"value":"42"},{"value":"84"}],[{"value":"43"},{"value":"86"}],[{"value":"44"},{"value":"88"}],[{"value":"45"},{"value":"90"}],[{"value":"46"},{"value":"92"}],[{"value":"47"},{"value":"94"}],[{"value":"48"},{"value":"96"}],[{"value":"49"},{"value":"98"}],[{"value":"50"},{"value":"100"}],[{"value":"51"},{"value":"102"}],[{"value":"52"},{"value":"104"}],[{"value":"53"},{"value":"106"}],[{"value":"54"},{"value":"108"}],[{"value":"55"},{"value":"110"}],[{"value":"56"},{"value":"112"}],[{"value":"57"},{"value":"114"}],[{"value":"58"},{"value":"116"}],[{"value":"59"},{"value":"118"}],[{"value":"60"},{"value":"120"}],[{"value":"61"},{"value":"122"}],[{"value":"62"},{"value":"124"}],[{"value":"63"},{"value":"126"}],[{"value":"64"},{"value":"128"}],[{"value":"65"},{"value":"130"}],[{"value":"66"},{"value":"132"}],[{"value":"67"},{"value":"134"}],[{"value":"68"},{"value":"136"}],[{"value":"69"},{"value":"138"}],[{"value":"70"},{"value":"140"}],[{"value":"71"},{"value":"142"}],[{"value":"72"},{"value":"144"}],[{"value":"73"},{"value":"146"}],[{"value":"74"},{"value":"148"}],[{"value":"75"},{"value":"150"}],[{"value":"76"},{"value":"152"}],[{"value":"77"},{"value":"154"}],[{"value":"78"},{"value":"156"}],[{"value":"79"},{"value":"158"}],[{"value":"80"},{"value":"160"}],[{"value":"81"},{"value":"162"}],[{"value":"82"},{"value":"164"}],[{"value":"83"},{"value":"166"}],[{"value":"84"},{"value":"168"}],[{"value":"85"},{"value":"170"}],[{"value":"86"},{"value":"172"}],[{"value":"87"},{"value":"174"}],[{"value":"88"},{"value":"176"}],[{"value":"89"},{"value":"178"}],[{"value":"90"},{"value":"180"}],[{"value":"91"},{"value":"182"}],[{"value":"92"},{"value":"184"}],[{"value":"93"},{"value":"186"}],[{"value":"94"},{"value":"188"}],[{"value":"95"},{"value":"190"}],[{"value":"96"},{"value":"192"}],[{"value":"97"},{"value":"194"}],[{"value":"98"},{"value":"196"}],[{"value":"99"},{"value":"198"}]]}}}}

0 commit comments

Comments
 (0)