Merge data from an external REST service with Hasura schema

[georgy-scifind]

Hi Husara team. Thank you so much for building such a fantastic tool. It helped our team immensely!

We have a question about how to merge data from an external REST service into our Hasura GraphQL schema. Let me explain our case in more detail.

We are building a social platform with a concept of Posts with some CRUD operations around them. Hasura covers that part perfectly. Now we need to add search to our platform, and for that, we will use the Algolia service. Upon search request from the front end, we want to seamlessly integrate data returned from Algolia search results into our schema and merge it with our Posts objects.

The first step we took was to build a Hasura Action around the Algolia REST API search endpoint. Here are the action types and an example response.
Action query definition and types:

type Query {
  search_posts(
    query: String!
    request_options: jsonb
  ): SearchPostsOutput
}

type SearchPostsOutput {
  hits: [PostHit]
  page: Int!
  nbHits: Int!
  nbPages: Int!
  hitsPerPage: Int!
}

type PostHit {
  post_id: uuid!
  title: String
  content: String!
  all_tags: [String]
  objectID: String!
  _highlightResult: PostHitHighlight
}

type PostHitHighlight {
  title: HighlightResult
  content: HighlightResult
  all_tags: [HighlightResult]
}

type HighlightResult {
  value: String!
  matchLevel: String!
  matchedWords: [String]
  fullyHighlighted: Boolean
}

An example response:

{
	"hits": [{
		"title": "sssfffs",
		"post_id": "c2fba996-2a11-42ad-a099-8d92a4d5ead0",
		"content": "fff",
		"all_tags": ["footag", "bartag"],
		"community_tags": ["footag"],
		"tooling_tags": ["bartag"],
		"objectID": "posts-c2fba996-2a11-42ad-a099-8d92a4d5ead0",
		"_highlightResult": {
			"title": {
				"value": "sssfffs",
				"matchLevel": "none",
				"matchedWords": []
			},
			"content": {
				"value": "fff",
				"matchLevel": "none",
				"matchedWords": []
			},
			"all_tags": [{
				"value": "<em>footag</em>",
				"matchLevel": "full",
				"fullyHighlighted": true,
				"matchedWords": ["footag"]
			}, {
				"value": "bartag",
				"matchLevel": "none",
				"matchedWords": []
			}]
		}
	}],
	"nbHits": 1,
	"page": 0,
	"nbPages": 1,
	"hitsPerPage": 20,
	"exhaustiveNbHits": true,
	"exhaustiveTypo": true,
	"exhaustive": {
		"nbHits": true,
		"typo": true
	},
	"query": "footag",
	"params": "query=footag",
	"renderingContent": {},
	"processingTimeMS": 1,
	"processingTimingsMS": {
		"total": 1
	}
}

The SearchPostsOutput.hits field contains an array of PostHit objects that represents a post entity in the Algolia index. We want to merge it with our schema via the SearchPostsOutput.hits.post_id field. But then we go to the Action relationships tab on Hasura Console, it shows us only the root level fields like hits, nbHits, etc. So it doesn't allow to select the two levels deep field hits.post_id to merge on.

As an alternative, we considered merging Algolia results with Posts from our schema during the action execution in the code. But since Hasura doesn't allow to reuse the auto-generated types (i.e. Posts) in a custom action, we can't simply nest the Post object under PostHit. Instead, we would need to duplicate everything from the Post object to the PostHit which we don't want to do because it contains many Hasura computed fields.

What can you recommend in our case? How to best achieve what we want?

[adas98012]

You should explore "configuring REST Connectors with Transform" (transform "response body" in your case). Here is the Hasura Document that you may find useful.

[georgy-scifind]

Thank you for the quick reply! We did explore REST Connectors as an alternative to writing the action code ourselves, but eventually, we leaned towards the latter. It was simply more convenient for us.

With either Hasura REST connectors or a custom action code, we can indeed adjust the response format. But the truth is, we don't want to do that. We don't want to extract Algolia Post entities out of the nested hits field to make the response look like a list of Post records, i.e. [PostHit]. In doing that, we would lose the supplementary search meta information that currently lives in response root fields like nbHits, page, nbPages, etc. Instead, we want to keep it in the response and just enrich the [PostHit] array nested under the hits field by merging it with our Hasura schema on the hits.post_id field.

Interestingly, I just tried to define the desired relationship on the nested PostHit type manually, bypassing the console. Here is how our metadata/actions.yaml file looks like:

actions:
  - name: search_posts
    definition:
      kind: ""
      handler: '{{WEBHOOK_URL_SEARCH_POSTS}}'
    permissions:
      - role: anonymous
      - role: user
custom_types:
  enums: []
  input_objects: []
  objects:
    - name: HighlightResult
    - name: SearchPostsOutput
    - name: PostHit
      relationships:
        - field_mapping:
            post_id: id
          name: post
          remote_table:
            name: posts
            schema: public
          source: default
          type: object
    - name: PostHitHighlight
  scalars: []

After applying the metadata to the server via hasura metadata apply, I saw the new relationship in the GraphQL API explorer. It all seems to work as we wanted. Here are examples of a GraphQL request and a response.
Request:

query MyQuery {
  search_posts(query: "") {
    nbPages
    page
    nbHits
    hitsPerPage
    hits {
      title
      post_id
      objectID
      content
      all_tags
      post {
        body
        author_id
        id
        shares_count
        slug
        title
      }
    }
  }
}

Response:

{
  "data": {
    "search_posts": {
      "nbPages": 1,
      "page": 0,
      "nbHits": 2,
      "hitsPerPage": 20,
      "hits": [
        {
          "title": "sssfffs",
          "post_id": "e967e0f6-aeaf-49cd-bf74-33927979d6d1",
          "objectID": "posts-e967e0f6-aeaf-49cd-bf74-33927979d6d1",
          "content": "fff",
          "all_tags": [],
          "post": {
            "body": "fff",
            "author_id": "674b1daf-f920-49fa-8917-887d32c7eae1",
            "id": "e967e0f6-aeaf-49cd-bf74-33927979d6d1",
            "shares_count": null,
            "slug": "sdsfsd88fffff888",
            "title": "sssfffs"
          }
        },
        {
          "title": "sssfffs",
          "post_id": "c2fba996-2a11-42ad-a099-8d92a4d5ead0",
          "objectID": "posts-c2fba996-2a11-42ad-a099-8d92a4d5ead0",
          "content": "fff",
          "all_tags": [
            "footag",
            "bartag"
          ],
          "post": {
            "body": "fff",
            "author_id": "674b1daf-f920-49fa-8917-887d32c7eae1",
            "id": "c2fba996-2a11-42ad-a099-8d92a4d5ead0",
            "shares_count": null,
            "slug": "sdsfsd88fffff222888",
            "title": "sssfffs"
          }
        }
      ]
    }
  }
}

Though in Hasura docs about custom types. You explicitly say that for nested objects, the relationship can be defined only on root fields. So I am a bit worried whether my approach bypassing the console is legit and we are safe to do it, or it's a hack.

I hope to hear back from you!

[adas98012]

@georgy-scifind
Thanks for your update. Looks like you have a working solution that meets your needs. Meanwhile, we also have developed a POC with Algolia API, that you may use as a reference.

Let us know if you need any additional info from us.

[georgy-scifind]

Thank you, @adas98012. Your POC project solves the exact problem we first encountered with Hasura Custom Actions. It's good to know that moving the action to a separate Hasura instance allows for the more sophisticated joining that we need.

Since we already found a way to define the joining within the single Hasura instance, we will stick to that approach. And if something won't work in the future, try your solution with the Remote Schema.

[dventimihasura]

You bet @georgy-scifind. Let us know if you if you need any other assistance. Cheers!

[adas98012]

@georgy-scifind
Just as a follow up, our engineering team validated that the Hasura GraphQL server does support Nested action joins via metadata config specs. Our Docs (and Console UI) have not been updated to reflect this feature enhancement. So, you should be good with your solution.

[georgy-scifind]

Thanks. That's good to hear!