How can I retrieve useful fields via GraphQL API?

[niyue]

I am playing with vector's GraphQL endpoint and playground, and I tried to retrieve fields name and other fields like sources for sinks.
After enabling GraphQL and starting up vector, I found I can only retrieve pagination information about the objects like sinks.

However, in the public playground, I can retrieve fields like name and sources using the GraphQL API:
Query:

{
  sinks {
    name
    sources {
      name
    }
  }
}

Response:

{
  "data": {
    "sinks": [
      {
        "name": "prometheus",
        "sources": [
          {
            "name": "internal_metrics"
          }
        ]
      },
      {
        "name": "blackhole",
        "sources": []
      }
    ]
  }
}

I think even for the default vector configuration, it sets up 1 sink (sinks.print)/1 transform (transforms.parse_logs)/1 source (sources.dummy_logs) in vector.toml, so I think there should be some data I can retrieve. Is there anything I am missing here? Please forgive my ignorance since I am pretty new to both vector and GraphQL. Thanks.

[leebenson]

Vector's GraphQL API implements the GraphQL Cursor Connections Specification, to aid cursoring and pagination through potentially large datasets.

As an example, consider this Vector config:

[api]
  enabled = true

[sources.one]
  type = "generator"
  format = "json"

[sources.two]
  type = "generator"
  format = "json"

[sources.three]
  type = "generator"
  format = "json"

[sinks.blackhole]
  type = "blackhole"
  inputs = ["one", "two", "three"]

There are 3x sources, each feeding into a blackhole sink.

If we wanted to return all the sources in this config, along with the name of the sink(s) they feed into, we could issue this query:

{
  sources {
    pageInfo {
      startCursor
      endCursor
    	hasNextPage
      hasPreviousPage
    }
    totalCount
    edges {
      node {
        componentId
        sinks {
          componentId
        }
      }
    }
  }
}

Which would return:

{
  "data": {
    "sources": {
      "edges": [
        {
          "node": {
            "componentId": "one",
            "sinks": [
              {
                "componentId": "blackhole"
              }
            ]
          }
        },
        {
          "node": {
            "componentId": "two",
            "sinks": [
              {
                "componentId": "blackhole"
              }
            ]
          }
        },
        {
          "node": {
            "componentId": "three",
            "sinks": [
              {
                "componentId": "blackhole"
              }
            ]
          }
        }
      ],
      "pageInfo": {
        "endCursor": "Q3Vyc29yOjI",
        "hasNextPage": false,
        "hasPreviousPage": false,
        "startCursor": "Q3Vyc29yOjA"
      },
      "totalCount": 3
    }
  }
}

A few notes on some of these fields:

• Firstly, note that we didn't ask how many sources to return. By default, the Vector GraphQL API will return up to 10 components if a limit isn't specified.
• The source data is inside the node field. That's because the sibling fields - pageInfo and totalCount - return meta about the result set, which help a client know if there's further data to query.
• totalCount shows us how many total available records there are. In this case, we have 3 sources and we return them all. But as you'll see in a moment, the available dataset can be larger than what's actually returned.
• pageInfo contains 'cursors', which we can pass along to the next query to page forwards/backwards. We'll see that in action below. The hasNextPage and hasPreviousPage booleans tell us whether there's more data available.

Both edges.node and edges.pageInfo are part of the GraphQL Cursor Connections spec. Some GraphQL clients, like Relay, offer convenient tooling around this spec that allows for simpler querying (which was one of the motivating factors for designing the API this way.)

Now let's assume we only want the first source. We can add a first:1 to the sources field, like so:

{
  sources(first:1) {
    pageInfo {
      startCursor
      endCursor
    	hasNextPage
      hasPreviousPage
    }
    totalCount
    edges {
      node {
        componentId
        sinks {
          componentId
        }
      }
    }
  }
}

Which will now return:

{
  "data": {
    "sources": {
      "edges": [
        {
          "node": {
            "componentId": "one",
            "sinks": [
              {
                "componentId": "blackhole"
              }
            ]
          }
        }
      ],
      "pageInfo": {
        "endCursor": "Q3Vyc29yOjA",
        "hasNextPage": true,
        "hasPreviousPage": false,
        "startCursor": "Q3Vyc29yOjA"
      },
      "totalCount": 3
    }
  }
}

This is pretty much the same thing, but notice a few differences:

• This time, just one source is returned.
• The endCursor has changed. This marks the end of the result set, which in this case, is actually the first record and not the third like before. We can use this in a follow-up query to receive the next set of sources after this point.
• hasNextPage is now true, which tells us there are more results available if we want them.

This time, let's use that endCursor to mark the starting point for retrieving the next set of results:

{
  sources(after:"Q3Vyc29yOjA") {
    pageInfo {
      startCursor
      endCursor
    	hasNextPage
      hasPreviousPage
    }
    totalCount
    edges {
      node {
        componentId
        sinks {
          componentId
        }
      }
    }
  }
}

Returning:

{
  "data": {
    "sources": {
      "edges": [
        {
          "node": {
            "componentId": "two",
            "sinks": [
              {
                "componentId": "blackhole"
              }
            ]
          }
        },
        {
          "node": {
            "componentId": "three",
            "sinks": [
              {
                "componentId": "blackhole"
              }
            ]
          }
        }
      ],
      "pageInfo": {
        "endCursor": "Q3Vyc29yOjI",
        "hasNextPage": false,
        "hasPreviousPage": true,
        "startCursor": "Q3Vyc29yOjE"
      },
      "totalCount": 3
    }
  }
}

Because we dropped the first parameter, we now get up to 10 results back (by default). Since we provided the previous result's endCursor to after, we skip the first source and get back the remaining two.

These options can be combined -- for example, skip the first result, and only return the last result of what remains:

{
  sources(after:"Q3Vyc29yOjA", last:1) {
    pageInfo {
      startCursor
      endCursor
    	hasNextPage
      hasPreviousPage
    }
    totalCount
    edges {
      node {
        componentId
        sinks {
          componentId
        }
      }
    }
  }
}

Which now gives us just the third (and final) source:

{
  "data": {
    "sources": {
      "edges": [
        {
          "node": {
            "componentId": "three",
            "sinks": [
              {
                "componentId": "blackhole"
              }
            ]
          }
        }
      ],
      "pageInfo": {
        "endCursor": "Q3Vyc29yOjI",
        "hasNextPage": false,
        "hasPreviousPage": true,
        "startCursor": "Q3Vyc29yOjI"
      },
      "totalCount": 3
    }
  }
}

Now hasNextPage is false (since there are no more sources) and hasPreviousPage is true -- since there are 2 sources before it.

All components implement cursoring in the same way - including sources, transforms and sinks.

You may notice that component fields on node don't implement the cursor spec right now. That's why you can query sinks.componentId directly on a source (and likewise, you can query sources.componentId on a sink.)

Technically, there's nothing stopping us from implementing the cursor spec on all levels -- so you could, for example, retrieve the first 5 sources, and the first 2 sinks that each of them implement.

We chose not to do, to keep the surface area simpler. Generally, users that have narrowed down their 'root' component scope and are querying, say, sources that feed into a sink, will want to know all sources that are going in. But this is something we may revisit in the future if there's a need for it.

Finally, I'll leave you with one other scenario -- this allows you to query both sinks and sources, and filter against all of them:

{
  components(filter:{componentId:{contains:"o"}}) {
    edges {
      node {
        ...on Source {
          componentId
          sinks {
            componentId
          }
        }
        ...on Sink {
          componentId
        }
      }
    }
  }
}

Which returns:

{
  "data": {
    "components": {
      "edges": [
        {
          "node": {
            "componentId": "blackhole"
          }
        },
        {
          "node": {
            "componentId": "one",
            "sinks": [
              {
                "componentId": "blackhole"
              }
            ]
          }
        },
        {
          "node": {
            "componentId": "two",
            "sinks": [
              {
                "componentId": "blackhole"
              }
            ]
          }
        }
      ]
    }
  }
}

A few points to make here:

• Unlike sources or sinks, which return components of the same type, components field returns any component type.
• The node of what's returned is of type Component, which is a GraphQL Interface type. In order to 'cast' it to a definitive type, we use the spread operator (...) to effectively say "If this is a Source (or a Sink), return the fields I ask for"). These fields don't have to match one another. In the example above, if the type is a Source, we're additionally asking for all sinks it sends logs to, which is relevant for components "one" and "two".
• We can apply a global filter against components. In this case, we're matching all components that contain the letter "o".

Vector's GraphQL API is intended to be flexible for a range of use-cases.

It's also used to power vector top (for component metrics) and vector tap (for event lines) - two of our built-in CLI tools.

I'd encourage you to click "Docs" on the right side of the playground and explore the API further to get a better idea of what it's capable of.

Let me know if you have any further questions or a particular use-case in mind, and I'd be happy to dive in further!

[leebenson]

Regarding the difference in the publicly deployed playground vs. what you see locally:

Cursoring and pagination is one of the more recent features added to our GraphQL API, so I suspect you're seeing an older version where this wasn't yet implemented.

It's a little more boilerplate to issue an equivalent query, but it's also more flexible for limiting results and having the ability to search/filter components, so it's definitely the more useful of the two schemas.

[niyue]

@leebenson thanks so much for the detailed explanation. So far I only spent around 2 hours on learning GraphQL in order to use this API. I will look into GraphQL Cursor Connections Specification to see how I should use it.

BTW, I think we could include such information in the API documentation so that people can understand how this API can be used. I did some web searching previously, here are the two major sources of information I can find:

• https://vector.dev/docs/reference/api/
  • this doc doesn't mention this GraphQL Cursor Connections Specification, so I simply learn some GraphQL and can get something out of the public playground, but when I tried the local installation, I failed to get anything useful. Since vector's data models are not too complex, probably listing some specific examples in this page will help people when they start using it for the first time.
• https://vector.dev/blog/graphql-api/
  • this blog has some API example I can follow, for example, the hostMetrics query example, and it has some API example that stops to work, for example, the processedEventsTotal query example. And it may need some updates since when people googling vector graphql, this comes as the first search result

[leebenson]

I absolute agree. Docs are way lighter than we'd like, and we've got it on our roadmap to focus some real effort on fixing that.

#10027 tracks improving the API docs specifically.

In the meantime, I'll make it a priority to fix the examples in the blog post. Many users are exposed to our API for the first time from that post, so I can see how that's especially annoying when the queries don't run! Tracking in #10028.

Thanks so much for your feedback @niyue. If there's anything we can help you with, feel free to open up further discussions.

[kumarmah1981]

@leebenson thank you so much for the prompt replies and supporting this community. I have been looking at the vector graphql docs trying to find queries which can be called for vector tap. You mentioned that vector tap uses the graphql api. Can you please point me to the query which I can use to get event lines for a component before and after? I would like to use the graphql api in my code instead of using vector tap.

Thanks once again

[leebenson]

Sure thing. All of the GraphQL queries/subscriptions that power both top and tap are available in: https://github.com/vectordotdev/vector/tree/master/lib/vector-api-client/graphql

The one specifically for tap is referenced in this source, and can be found here: https://github.com/vectordotdev/vector/blob/master/lib/vector-api-client/graphql/subscriptions/output_events_by_component_id_patterns.graphql

Note that it's a subscription -- it sends data over a websocket as it's made available. Therefore whichever GraphQL client you're using would need to be capable of handling subscriptions over a websocket. Hope that helps!