[v5] Serialization and optional fields

[a-coruble]

Hi there ! It's the first time I'm using Adonis and so far I love it, thanks for this awesome framework !!

I'm building an API for a small project and I would like to have all the model fields returned when doing serialization even if the field is empty.

Here's my model :

export interface UserInterface {
  id: number
  email: string
  password: string
  firstName?: string
  lastName?: string
  dateOfBirth?: DateTime
  phoneNumber?: string
  createdAt: DateTime
  updatedAt: DateTime
}

export default class User extends BaseModel implements UserInterface {
  @column({ isPrimary: true })
  public id: number

  @column()
  public email: string

  @column({ serializeAs: null })
  public password: string

  @column({ serializeAs: 'firstName' })
  public firstName?: string

  @column({ serializeAs: 'lastName' })
  public lastName?: string

  @column.date({ serializeAs: 'dateOfBirth' })
  public dateOfBirth?: DateTime

  @column({ serializeAs: 'phoneNumber' })
  public phoneNumber?: string

  @column.dateTime({ autoCreate: true, serializeAs: 'createdAt' })
  public createdAt: DateTime

  @column.dateTime({ autoCreate: true, autoUpdate: true, serializeAs: 'updatedAt' })
  public updatedAt: DateTime

  @beforeSave()
  public static async hashPassword(user: User) {
    if (user.$dirty.password) {
      user.password = await Hash.make(user.password)
    }
  }
}

And here's my controller :

export default class AuthController {
  public async register({ auth, logger, request, response }: HttpContextContract) {
    const validationSchema = schema.create({
      email: schema.string({ trim: true }, [
        rules.email(),
        rules.maxLength(255),
        rules.unique({ table: 'users', column: 'email' }),
        rules.required(),
      ]),
      password: schema.string({ trim: true }, [rules.maxLength(180), rules.required()]),
      firstName: schema.string.optional({ trim: true }, [rules.maxLength(150)]),
      lastName: schema.string.optional({ trim: true }, [rules.maxLength(150)]),
      phoneNumber: schema.string.optional({ trim: true }, [rules.maxLength(50)]),
      dateOfBirth: schema.date.optional(),
    })

    const validationResult = await handleControllerError<{
      email: string
      password: string
      firstName?: string
      lastName?: string
      phoneNumber?: string
      dateOfBirth?: DateTime
    }>(() => request.validate({ schema: validationSchema }), PossibleErrors.VALIDATION_ERROR)
    if (isControllerError(validationResult)) {
      logger.error('register/validationResult', validationResult)
      return response.internalServerError(validationResult)
    }

    const userResult = await handleControllerError<User>(
      () => User.create(validationResult),
      PossibleErrors.DATABASE_ERROR
    )
    if (isControllerError(userResult)) {
      logger.error('register/userResult', userResult)
      return response.internalServerError(userResult)
    }

    const tokenResult = await handleControllerError<Promise<any>>(
      () => auth.login(userResult),
      PossibleErrors.AUTHENTICATION_ERROR
    )
    if (isControllerError(tokenResult)) {
      logger.error('register/tokenResult', tokenResult)
      return response.internalServerError(tokenResult)
    }

    return response.created({
      user: userResult.serialize(),
      // Tried including all the fields manually
      // user: userResult.serialize({
      //   fields: {
      //     pick: [
      //       'id',
      //       'email',
      //       'firstName',
      //       'lastName',
      //       'dateOfBirth',
      //       'phoneNumber',
      //       'createdAt',
      //       'updatedAt',
      //     ],
      //   },
      // }),
      token: tokenResult,
    })
  }
}

When I register a user by sending a POST request to the register endpoint with these informations : {"email": "[email protected]", "password": "testPassword"}, I get this response : { "user": { "email": "[email protected]", "createdAt": "2020-11-05T10:52:06.498+01:00", "updatedAt": "2020-11-05T10:52:06.512+01:00", "id": 15 }, "token": { "type": "bearer", "token": "MTQ.UPj5mpe0H5H9Q5HKTOabqMeRdm17gL9JOBdfDUOQ9IKZmxyqhvdRZjMHL6_A" } }

I would expect to have all the fields, but with undefined as value, whereas here, I don't even have the fields.

I have two questions concerning this code :

• how can I return all fields even if there's no value ?
• is my syntax to mark field as optional (using the ?) on the model right ?

Also, if I can suggest one thing, if returning empty fields is not feasible right now, what about implementing a parameter to serialize to return all serialazible fields like : user.serialize({fields: {all: true}}) or user.serializeAll() or any other syntax, but you get the idea. Of course, if a field has been marked as {serializeAs: null}, it should be ignored.

Thanks in advance for any future answer !!

[steschwa]

1. You can try the following:

return response.created({
    user: Object.entries(userResult.serialize()).reduce(
        (prev, [property, value]) => ({
             ...prev,
             [property]: value ?? null
        }),
        {}
    )
})

This will check every serializable property (from userResult.serialize()) and return it's value or null if the property value is undefined or null (using the ??) operator.

2. Marking fields as optional (eg. public firstName?: string) will omit the property from serialization if it's undefined

[a-coruble]

Hey, thanks for your answer @steschwa , unfortunately, I tried both your solution, and none of them worked.
I tried by replacing my register handler with this code :

public async register({ auth, logger, request, response }: HttpContextContract) {
    const validationSchema = schema.create({
      email: schema.string({ trim: true }, [
        rules.email(),
        rules.maxLength(255),
        rules.unique({ table: 'users', column: 'email' }),
        rules.required(),
      ]),
      password: schema.string({ trim: true }, [rules.maxLength(180), rules.required()]),
      firstName: schema.string.optional({ trim: true }, [rules.maxLength(150)]),
      lastName: schema.string.optional({ trim: true }, [rules.maxLength(150)]),
      phoneNumber: schema.string.optional({ trim: true }, [rules.maxLength(50)]),
      dateOfBirth: schema.date.optional(),
    })

    const validationResult = await handleControllerError<{
      email: string
      password: string
      firstName?: string
      lastName?: string
      phoneNumber?: string
      dateOfBirth?: DateTime
    }>(() => request.validate({ schema: validationSchema }), PossibleErrors.VALIDATION_ERROR)
    if (isControllerError(validationResult)) {
      logger.error('register/validationResult', validationResult)
      return response.internalServerError(validationResult)
    }

    const userResult = await handleControllerError<User>(
      () => User.create(validationResult),
      PossibleErrors.DATABASE_ERROR
    )
    if (isControllerError(userResult)) {
      logger.error('register/userResult', userResult)
      return response.internalServerError(userResult)
    }

    const tokenResult = await handleControllerError<Promise<any>>(
      () => auth.login(userResult, { expiresIn: '7 days' }),
      PossibleErrors.AUTHENTICATION_ERROR
    )
    if (isControllerError(tokenResult)) {
      logger.error('register/tokenResult', tokenResult)
      return response.internalServerError(tokenResult)
    }

    logger.info(`User ${userResult.dateOfBirth}`)
    return response.created({
      user: Object.entries(userResult.serialize()).reduce(
        (acc, [property, value]) => ({ ...acc, [property]: value ?? null }),
        {}
      ),
      token: tokenResult,
    })
  }

But I still get only the fields which have a value.

So I tried your second solution, and removed the optional on all my User model fields, and still no serialization of undefined properties.

Would you think that having an option to include undefined fields (except for the serializeAs: null) on the serialize method could be something that will be useful in the framework ? I would be glad to contribute and implement this.

[a-coruble]

When I go through the code of the serialize method in Lucid source code, I end up on the serializeAttributes function. When I follow the code, I see that Lucid checks for the column and looks if the serializeAs attribute of each column is different than null / undefined, which is the case for me as I set my serializeAs attribute in the decorators. Then we assign the value, but without doing a check of it, so I don't get why the fields would not be included as they are defined to be serialized.

[a-coruble]

Okay, I found out the reason of my issue. This is due to the JSON.stringify function that removes every undefined key of it's input. I tried to avoid having undefined as value by using this model :

import { DateTime } from 'luxon'
import Hash from '@ioc:Adonis/Core/Hash'
import { column, beforeSave, BaseModel } from '@ioc:Adonis/Lucid/Orm'

export interface UserInterface {
  id: number
  email: string
  password: string
  firstName?: string
  lastName?: string
  dateOfBirth?: DateTime
  phoneNumber?: string
  createdAt: DateTime
  updatedAt: DateTime
}

export default class User extends BaseModel implements UserInterface {
  @column({ isPrimary: true })
  public id: number

  @column()
  public email: string

  @column({ serializeAs: null })
  public password: string

  @column({
    serializeAs: 'firstName',
    serialize: (value?: string) => {
      return value ?? null
    },
  })
  public firstName?: string

  @column({
    serializeAs: 'lastName',
    serialize: (value?: string) => {
      return value ?? null
    },
  })
  public lastName?: string

  @column.date({
    serializeAs: 'dateOfBirth',
    serialize: (value?: DateTime) => {
      return value ?? null
    },
  })
  public dateOfBirth?: DateTime

  @column({
    serializeAs: 'phoneNumber',
    serialize: (value?: string) => {
      return value ?? null
    },
  })
  public phoneNumber?: string

  @column.dateTime({ autoCreate: true, serializeAs: 'createdAt' })
  public createdAt: DateTime

  @column.dateTime({ autoCreate: true, autoUpdate: true, serializeAs: 'updatedAt' })
  public updatedAt: DateTime

  @beforeSave()
  public static async hashPassword(user: User) {
    if (user.$dirty.password) {
      user.password = await Hash.make(user.password)
    }
  }
}

Even by doing so, my attributes are not getting serialized to null and I don't have the empty fields in my response.

[a-coruble]

Okay, I finally fixed my issue. Here's the full explanation of the issue.

Imagine that you have a User model like the following :

class User extends BaseModel {
 @column({ isPrimary: true })
  public id: number

  @column()
  public email: string

  @column({ serializeAs: null })
  public password: string

  @column({ serializeAs: 'firstName' })
  public firstName: string
}

Now, imagine that I allow my user to create an account without indicating the firstName. For this, I would have something along those lines in my controller :

export default class AuthController {
  public async register({ auth, logger, request, response }: HttpContextContract) {
    const validationSchema = schema.create({
      email: schema.string({ trim: true }, [
        rules.email(),
        rules.maxLength(255),
        rules.unique({ table: 'users', column: 'email' }),
        rules.required(),
      ]),
      password: schema.string({ trim: true }, [rules.maxLength(180), rules.required()]),
      firstName: schema.string.optional({ trim: true }, [rules.maxLength(150)]),
    })
    const validatedData = await request.validate(validationSchema)
    const user = await User.create(validatedData)
    return response.created(user.serialize())

Now, if I send a request to this controller, with this body :

{
    "email": "[email protected]",
    "password": "testPassword"
}

What's going to happen is that the validatedData in the controller will contain only the two fields, and not the firstName. Due to this, when doing the User.create(validatedData), the instance that will get returned will have firstName: undefined.
As it's undefined, serializing will ignore this field and not send it in the response, which is not the behaviour wanted.

To make the field appear in the serialization, we need to re-fetch from the database, as the value in the DB will be null for the firstName, and not undefined, which will allow the field to be included in the serialization. In order to avoid re-fetching the database manually, here's how you can modify the controller method to have the expected behaviour :

const user = await User.create(validatedData)
user.refresh()
return response.created(user.serialize())

And now, in the response, you will see the firstName field with null as value, as expected.

I think I will create a PR on the documentation of Adonis to explain that behaviour in the Model documentation.