Skip to content

SolarQuery Stream API

Jump to bottom Edit New page
Matt Magoffin edited this page May 4, 2022 · 19 revisions

The SolarQuery Stream API provides methods to query the data collected by SolarNodes, in an efficient datum stream format. The methods in this API do not limit the number of returned results by default, so your client must be prepared to handle potentially large results.

Response formats

The SolarQuery Stream API supports different response data formats, which are based on the Accept HTTP header used in the request. The supported Accept header values are:

  • application/cbor - CBOR
  • application/json - JSON

In addition the responses can be compressed with the gzip compressor by adding an Accept-Encoding: gzip HTTP header to the request.

Endpoints

Verb Endpoint Description
GET /datum/stream/datum Query raw or aggregate data by time, node, source, or stream.
GET /datum/stream/reading Query for data at specific dates or the difference between two dates.

Datum stream datum list

This method returns datum stream values, either raw data or aggregated if the aggregation parameter is provided.

GET /solarquery/api/v1/sec/datum/stream/datum
localStartDate An inclusive starting date to use in the query, in YYYY-MM-dd or YYYY-MM-ddTHH:mm patterns. The date will be interpreted in the time zone of the node associated with each datum.
localEndDate An exclusive ending date to use in the query, in YYYY-MM-dd or YYYY-MM-ddTHH:mm patterns. The date will be interpreted in the time zone of the node associated with each datum.
startDate An inclusive starting date to use in the query, in YYYY-MM-dd or YYYY-MM-ddTHH:mm patterns. The date must be provided in the UTC time zone.
endDate An exclusive ending date to use in the query, in YYYY-MM-dd or YYYY-MM-ddTHH:mm patterns. The date must be provided in the UTC time zone.
mostRecent If true then no date range is required, and the most recently posted datum for all matching nodes/sources will be returned. This can also be combined with the aggregation parameter to return the most recent calculated aggregate values instead of raw datum (only the Hour, Day, and Month types are supported). Date parameters may be included to restrict the results to the most recently available datum within the given date range.
aggregation An optional aggregation type. If not provided, then non-aggregated data will be returned. See the SolarNet aggregation guide for more info. ⚠️ Note the *Minute types enforce a maximum date range of 5 weeks.
partialAggregation An optional aggregation type for partial aggregation results to be included. See the partial aggregation guide for more info.
nodeId The ID of a single node.
nodeIds A comma-delimited list of node IDs. This parameter may be also provided multiple times.
sourceId The ID of a single source to restrict the results to. A wildcard pattern can be used to restrict the results to only source IDs that match the pattern.
sourceIds A comma-delimited list of source IDs to restrict the results to. This parameter may also be provided multiple times.
streamId The ID of a single stream to restrict the results to.
streamIds A comma-delimited list of stream IDs to restrict the results to. This parameter may also be provided multiple times.
combiningType An optional combining type to combine raw results into virtual ones. Specifying a type here requires some combintation of nodeIdMaps and sourceIdMaps parameters also be provided. See Virtual IDs for more information.
nodeIdMaps Return combined node IDs as virtual node IDs using the specified combiningType. The format of this parameter is VID:IDLIST where VID is a virtual node ID (integer) and IDLIST is a comma-delimited list of real node IDs. Multiple virtual IDs can be assigned by providing multiple nodeIdMaps query parameters. See Virtual IDs for more information.
sourceIdMaps Return combined source IDs as virtual source IDs using the specified combiningType. The format of this parameter is VID:IDLIST where VID is a virtual source ID and IDLIST is a comma-delimited list of real source IDs. Multiple virtual IDs can be assigned by providing multiple sourceIdMaps query parameters. This parameter requires the withoutTotalResultsCount also be set to true. See Virtual IDs for more information.

Datum list queries support the following sort query parameter keys:

  • created
  • node
  • source

Notes:

  • Use either the local*Date or *Date parameters; do not mix the two styles. For example, providing localStartDate and endDate parameters is not supported.

Datum stream datum list response

The response contains a general response object with the following properties:

Property Type Description
meta object An array of datum stream metadata, one element for every stream returned in the result data.
data array An array of datum stream data arrays for each datum matching the request criteria.

An example response object looks like this (see datum stream datum data for more detailed information):

{
  "success" : true,
  "meta": [
    {
      "streamId": "7714f762-2361-4ec2-98ab-7e96807b32a6",
      "zone": "Pacific/Auckland",
      "kind": "n",
      "objectId": 123,
      "sourceId": "/power/1",
      "i": [
        "watts",
        "current",
        "voltage",
        "frequency"
      ],
      "a": [
        "wattHours"
      ]
    }
  ],
  "data": [
    [
      0,
      1650667326308,
      12326,
      null,
      230.19719,
      50.19501,
      6472722
    ]
  ]
}

Datum stream datum data

Each datum that matches the request criteria is represented as an array with

  • A reference to the datum stream metadata associated with this datum. The metadata defines the property names and their order within the datum array, as well as the node/location/source/stream IDs associated with the datum stream
  • Datum timestamp
  • Instantaneous datum properties, as individual array elements
  • Accumulating datum properties, as individual array elements
  • Status datum properties, as individual array elements
  • Tags, as individual array elements

More formally, the datum data array structure looks like this:

Index Type Description
0 integer The 0-based index in the meta array of the datum stream metadata associated with this datum.
1 integer The datum timestamp, in millisecond epoch format.
2..x decimal The instantaneous datum property values, ordered by the meta.i array of the associated datum stream metadata.
(2+iLen)..x decimal The accumulating datum property values, ordered by the meta.a array of the associated datum stream metadata.
(2+iLen+aLen)..x string The status datum property values, ordered by the meta.s array of the associated datum stream metadata.
(2+iLen+aLen+sLen)..x string The tag values, if any tags available.

The iLen, aLen, and sLen variables represent the number of instantaneous, accumulating, and status datum properties (respectively) included in the metadata associated with the datum. null will be inserted for any property defined in the metadata that the datum does not have.

Take the example datum data array shown previously:

[
      0,
      1650667326308,
      12326,
      null,
      230.19719,
      50.19501,
      6472722
]

The elements of the array represent:

Index Value Description
0 0 the first datum stream metadata returned in the meta array, i.e. stream 7714f762-2361-4ec2-98ab-7e96807b32a6
1 1650667326308 the timestamp in millisecond epoch form, i.e. Saturday, April 23, 2022 10:42:06.308 AM in the Pacific/Auckland time zone given in the stream metadata.
2 12326 the watts instantaneous property
3 null the current instantaneous property (missing in datum)
4 230.19719 the voltage instantaneous property
5 50.19501 the frequency instantaneous property
6 6472722 the wattHours instantaneous property

Datum stream reading list

This method returns datum stream reading values, either raw data or aggregated if the aggregation parameter is provided.

GET /solarquery/api/v1/sec/datum/stream/reading
readingType The type of reading to perform. See Datum reading types for possible values.
localStartDate An inclusive starting date to use in the query, in YYYY-MM-dd or YYYY-MM-ddTHH:mm patterns. The date will be interpreted in the time zone of the node associated with each datum.
localEndDate An exclusive ending date to use in the query, in YYYY-MM-dd or YYYY-MM-ddTHH:mm patterns. The date will be interpreted in the time zone of the node associated with each datum.
startDate An inclusive starting date to use in the query, in YYYY-MM-dd or YYYY-MM-ddTHH:mm patterns. The date must be provided in the UTC time zone.
endDate An exclusive ending date to use in the query, in YYYY-MM-dd or YYYY-MM-ddTHH:mm patterns. The date must be provided in the UTC time zone.
aggregation An optional aggregation type. If provided, then individual aggregate results covering the time range between the start/end dates will be returned, instead of just one result for the overall range. Only Hour, Day, and Month values are supported. Only the Difference reading type is supported. See the SolarNet aggregation guide for more info.
nodeId The ID of a single node.
nodeIds A comma-delimited list of node IDs. This parameter may be also provided multiple times.
sourceId The ID of a single source to restrict the results to. A wildcard pattern can be used to restrict the results to only source IDs that match the pattern.
sourceIds A comma-delimited list of source IDs to restrict the results to. This parameter may also be provided multiple times.
streamId The ID of a single stream to restrict the results to.
streamIds A comma-delimited list of stream IDs to restrict the results to. This parameter may also be provided multiple times.
tolerance An optional time duration to restrict how far in time the query is allowed to look for matching raw datum. This duration is applied to both the start and end dates. The syntax is ISO 8601 and defaults to P1M (one month). Generally the shorter the duration, the faster the query can execute.

Notes:

  • Use either the local*Date or *Date parameters; do not mix the two styles. For example, providing localStartDate and endDate parameters is not supported.
  • When aggregation is provided, results for the specified aggregation are returned. For example, if the date range is specified as 2000-01-01 to 2001-01-01 and Month aggregation is used, 12 results will be returned per node and source combination, one for each month. Each month's results shows the reading values for just that month.
  • If no datum exists before a given start date, either because there simply isn't any data or there isn't any data within the tolerance range of the start date, the queries will include the first available datum within the given start/end dates. This can happen if datum are posted infrequently or at the very start of a datum stream, for example when querying for datum in the month the stream started and the stream starts in the middle of the month.

Datum stream reading list response

The response contains a general response object with the following properties:

Property Type Description
meta object An array of datum stream metadata, one element for every stream returned in the result data.
data array An array of either datum stream data or datum stream reading data arrays for each datum matching the request criteria.

The CalculatedAt reading type will return datum stream data data while all other types will return datum stream reading data.

An example response object looks like this (see datum stream reading data for more detailed information):

{
  "success": true,
  "meta": [
    {
      "streamId": "5514f762-2361-4ec2-98ab-7e96807b3255",
      "zone": "America/New_York",
      "kind": "n",
      "objectId": 123,
      "sourceId": "/pyrometer/1",
      "i": [
        "irradiance",
        "temperature"
      ],
      "a": [
        "irradianceHours"
      ]
    }
  ],
  "data": [
    [
      0,
      [
        1650945600000,
        1651032000000
      ],
      [
        3.6,
        2,
        0,
        7.2
      ],
      [
        19.1,
        2,
        18.1,
        20.1
      ],
      [
        1.422802,
        1138.446687,
        1139.869489
      ]
    ]
  ]
}

Datum stream reading data

Each datum that matches the request criteria is represented as an array with

  • A reference to the datum stream metadata associated with this datum. The metadata defines the property names and their order within the datum array, as well as the node/location/source/stream IDs associated with the datum stream
  • A 2-element array with the reading start and end timestamps, as millisecond epoch numbers
  • Instantaneous datum properties, each as 4-element arrays representing the average value, count of values, minimum value, and maximum value.
  • Accumulating datum properties, each as 3-element arrays representing the difference between the ending and starting values, the starting value, and the ending value

Note only the CalculatedAt reading type returns instantaneous values; all other types will return null values for the instantaneous properties.

More formally, the datum reading data array structure looks like this:

Index Type Description
0 integer The 0-based index in the meta array of the datum stream metadata associated with this datum.
1 array<integer> 2-element array representing the start and end reading timestamps, in millisecond epoch format. ⚠️ Note for aggregate results the end timestamp will be null, as the value is implied as the start timestamp plus the requested aggregation period.
2..x array<decimal> The instantaneous datum property values, each as a 4-element array representing the average value, count of values, minimum value, and maximum value, ordered by the meta.i array of the associated datum stream metadata. ⚠️ Note that not all queries returns instantaneous values; if they are not supported the results will have null instead of a 4-element array.
(2+iLen)..x array<decimal> The accumulating datum property values, each as a 3-element array representing the difference between the ending and starting values, the starting value, and the ending value, ordered by the meta.a array of the associated datum stream metadata.

The iLen variable represents the number of instantaneous datum properties included in the metadata associated with the datum. null will be inserted for any property defined in the metadata that the datum does not have, or for reading types that do not support instantaneous properties.

Take the example datum data array shown previously:

[
      0,
      [
        1650945600000,
        1651032000000
      ],
      [
        3.6,
        2,
        0,
        7.2
      ],
      [
        19.1,
        2,
        18.1,
        20.1
      ],
      [
        1.422802,
        1138.446687,
        1139.869489
      ]
]

The elements of the array represent:

Index Value Description
0 0 the first datum stream metadata returned in the meta array, i.e. stream 5514f762-2361-4ec2-98ab-7e96807b3255
1 [1650945600000,1651032000000] the start/end timestamps in millisecond epoch form, i.e. midnight Saturday, Tuesday April 26, 2022 - Wednesday April 27, 2022 in the America/New_York time zone given in the stream metadata.
2 [3.6,2,0,7.2] the irradiance instantaneous property: average of 3.6 from 2 datum that ranged in value from 0 to 7.2
3 [19.1,2,18.1,20.1] the temperature instantaneous property: average of 19.1 from 2 datum that ranged in value from 18.1 to 20.1
4 [1.422802,1138.446687,1139.869489] the irradianceHours accumulating property that changed by 1.422802 from 1138.446687 to 1139.869489
Add a custom footer

Clone this wiki locally