Stress Yoga server

[trixobird]

I am evaluating yoga server for use in our company. I have created a minimum setup in server.ts with a mutation that once executed an event is fired through pub-sub. Then there is a subscription that is subscribing to this event and just outputs the payload.

import { createServer } from 'node:http'
import { createYoga } from 'graphql-yoga'
import { WebSocketServer } from 'ws'
import { useServer } from 'graphql-ws/lib/use/ws'
import {createSchema, createPubSub} from 'graphql-yoga';

interface BookMapper {
  id: string
  isbn: string
  read: boolean
}

const pubSub = createPubSub<{
  bookIsRead: [BookMapper]
}>()

const schema = createSchema({
    typeDefs: /* GraphQL */ `
        type Query {
            book(id: ID!): Book
        }
        type Mutation {
            markBookAsRead(id: ID!): Book!
        }
        type Subscription {
            bookReadEvent: Book!
        }
        type Book {
            id: ID!
            isbn: String!
            read: Boolean!
        }
    `,
  resolvers: {
    Query: {
      book: (parent, arg) => ({
        id: arg.id,
        isbn: `${arg.id}-isbn`,
        read: Math.random() < 0.5
      })
    },
    Mutation: {
      markBookAsRead: (parent, arg, _) => {
        const book = {
          id: arg.id,
          isbn: `${arg.id}-isbn`,
          read: Math.random() < 0.5
        };
        pubSub.publish('bookIsRead', book);
        return book
      }
    },
    Subscription: {
      bookReadEvent: {
        subscribe: () => pubSub.subscribe('bookIsRead'),
        resolve: (payload) => payload
      }
    }
  }
})

const yogaApp = createYoga({
  schema,
  graphiql: {
    // Use WebSockets in GraphiQL
    subscriptionsProtocol: 'WS'
  }
})

// Get NodeJS Server from Yoga
const httpServer = createServer(yogaApp)
// Create WebSocket server instance from our Node server
const wsServer = new WebSocketServer({
  server: httpServer,
  path: yogaApp.graphqlEndpoint
})

// Integrate Yoga's Envelop instance and NodeJS server with graphql-ws
useServer(
  {
    execute: (args: any) => args.rootValue.execute(args),
    subscribe: (args: any) => args.rootValue.subscribe(args),
    onSubscribe: async (ctx, msg) => {
      const { schema, execute, subscribe, contextFactory, parse, validate } =
        yogaApp.getEnveloped({
          ...ctx,
          req: ctx.extra.request,
          socket: ctx.extra.socket,
          params: msg.payload
        })

      const args = {
        schema,
        operationName: msg.payload.operationName,
        document: parse(msg.payload.query),
        variableValues: msg.payload.variables,
        contextValue: await contextFactory(),
        rootValue: {
          execute,
          subscribe
        }
      }

      const errors = validate(args.schema, args.document)
      if (errors.length) return errors
      return args
    }
  },
  wsServer
)

httpServer.listen(4000, () => {
  console.log('Server is running on port 4000')
})

Then we created a stress.ts that we subscribe X users to that event and then execute this subscription X times. We output the number of events (they shoud be X square) and then the time. For X = 100 this is less than a second. For X = 1000 this is taking 35seconds on my machine.

import {Client as WsClient, createClient as createWsClient} from 'graphql-ws';
import {Client as HttpClient, createClient as createHttpClient} from 'graphql-http';
import WebSocket from 'ws';
import {chunk} from "lodash";

type UserType = {
  id: number
  count: number
  httpClient: HttpClient
  wsClient: WsClient
}

const httpClient = () => createHttpClient({
  url: 'http://localhost:4000/graphql'
});
const wsClient = () => createWsClient({
  url: 'ws://localhost:4000/graphql',
  webSocketImpl: WebSocket
});

const stress = async () => {

  console.info('Start', new Date())
  const times = 1000;
  const users = createUsers(times);
  const usersMap = new Map(users.map(obj => [String(obj.id), obj]));

  const subscriptions: Promise<void>[] = [];
  const chunkedUsers = chunk(users,100);
  for (const us of chunkedUsers) {
    subscriptions.push(...us.map(u => wsSubscription(u.wsClient, usersMap.get(String(u.id)), times)))
    await new Promise(resolve => setTimeout(resolve, 100));
  }
  console.info('Start Stress', new Date())

  for (const us of chunkedUsers) {
    await Promise.all(us.map(async user => await readBook(user.httpClient, String(user.id))))
    await new Promise(resolve => setTimeout(resolve, 100));
  }

  await Promise.all(subscriptions);
  console.info(users.reduce((a, b) => a + b.count, 0))
  console.info('Done', new Date())
}

const readBook = async (httpClient: HttpClient, bookId: string) => {
  let cancel = () => {
    /* abort the request if it is in-flight */
  };

  const result = await new Promise((resolve, reject) => {
    let result;
    cancel = httpClient.subscribe(
      {
        query: 'mutation markBookAsRead($bookId: ID!) {\n  markBookAsRead(id: $bookId) {\n    id\n  }\n}',
        variables: {bookId}
      },
      {
        next: (data) => (result = data),
        error: reject,
        complete: () => resolve(result),
      },
    );
  });
};

const wsSubscription = async (wsClient, user: UserType, numOfUsers: number) => {
  const onNext = ({data}) => {
    user.count++;
    numOfUsers--
    if (numOfUsers === 0) {
      unsubscribe();
    }
    setTimeout(() => unsubscribe(), 40000);
  };

  let unsubscribe = () => {
    console.log('WS Complete Sub')
  };

  await new Promise((resolve, reject) => {
    unsubscribe = wsClient.subscribe(
      {
        query: 'subscription bookReadEvent{bookReadEvent{id isbn read}}',
      },
      {
        next: onNext,
        error: reject,
        complete: resolve as () => void,
      },
    );
  })
    .catch(e => console.error(e.message));
};

const createUsers = (times: number): UserType[] => {
  const users: UserType[] = [];
  for (let i = 0; i < times; i++) {
    users.push({
      id: i,
      count: 0,
      httpClient: httpClient(),
      wsClient: wsClient()
    })
  }
  return users;
}

stress().catch(e => console.error(e.message))

I know that this is less than optimal setup (events should be aggregated and sent every Y milliseconds), but is there anything that I am doing wrong? I was expecting a better performance.

Complete gist: https://gist.github.com/trixobird/b3953756e60af067f240a1781653a168

[n1ru4l]

@trixobird Can you please provide a benchmark without apollo-client, so we can know that yoga is actually the bottle-neck here?

[trixobird]

Thank you for the lightinig speed answer. I think I need some form of a client in order to send the mutation and check the subscription events. Is there another client that you might suggest? urql or relay maybe?

[n1ru4l]

Yoga follows GraphQL over HTTP specification, so you can use fetch or any other HTTP library for sending mutation and query operations to the server.

Apollo-Client/Urql and relay are suboptimal for benchmarking a server, as they have significant overhead with their cache.

For subscription operations, SSE is used. See https://the-guild.dev/graphql/yoga-server/docs/features/subscriptions#quick-start-with-simple-server-sent-events-sse
https://the-guild.dev/graphql/yoga-server/docs/features/subscriptions#handling-subscriptions-on-the-client

You should probably not have a single thread that sets up and handles all subscriptions, as in the real world all clients would be on different machines anyways.

[enisdenjo]

You can use graphql-http on the client for single result operations.

[trixobird]

@enisdenjo I managed to use graphql-http; this is only for query and mutation operation if I am not mistaken. For subscriptions I tried to use graphql-sse but I was getting Missing message event error. I also tried to use graphql-ws. This time I was getting UnhandledPromiseRejection if I don't await the subscription call, and Unexpected server response: 200 if awaited it.

I have updated the gist to reflect my latest tries, could you let me know where I am failing please?

[enisdenjo]

Yes, graphql-http is just for queries and mutations. If you're using graphql-sse on the client, you should use the accompanying plugin for Yoga: https://the-guild.dev/graphql/yoga-server/docs/features/subscriptions#graphql-over-server-sent-events-protocol-via-graphql-sse.

[trixobird]

hi @n1ru4l, @enisdenjo and others. I have droped apollo client, still with the similar results. I got 1M events from 1K users in 32sec. I will try to find time to spread the subscribers across different threads; is there any other pitfall in my test?

I have updated the code snipets and the gist: https://gist.github.com/trixobird/b3953756e60af067f240a1781653a168

[trixobird]

Hi again, just a kind reminder about the question above. As the company evaluates Yoga I need to make sure that my setup was correct therefore the results trustworthy.

[enisdenjo]

How do you measure your expectations? Is another server faster when compared or do you just personally expect Yoga to be faster?

To find a bottleneck, we need a reference point - we need something to compare to. Otherwise, you can just have an under-performing implementation JavaScript-wise (unrelated to the transport).

[trixobird]

hi Denis, thank you for your response,
I don't have any reference (in javascript world), I just expected Yoga to be faster. I have posted the whole gist, just to make sure that I have not made any errors in the configuration. Based solely on this experiment yoga is not sufficient fast for us. I'll try to find some time to play around with Apollo server as well, but I don' expect drastically different results, if my current configuration is correct.