React liveness/provide default device info

[riasatali42]

Description of changes

This PR introduces features to pass in and out the device information (device Id, device label, kind, group Id) for the camera of the liveness check. Also, provides support to add callback methods for camera changes or default device not found.

Key additions:

• Pass Default device id and label so that the device will be automatically selected.
• If device Id and device label both are present, device label will be prioritized.
• Pass out device info (device Id, device label, group Id, kind) once the liveness check is completed.
• Callback method added for default device not found if the device with the passed in device id/ label not found.
• Callback method added for camera changes if the camera is changed. So that we can track the updated camera device with device info.

Issue ##6587, Pass deviceId in and out of liveness check

Description of how you validated changes

1. Created unit tests to test the added features.

Checklist

• Have read the Pull Request Guidelines
• PR description included
• yarn test passes and tests are updated/added
• PR title and commit messages follow conventional commit syntax
• If this change should result in a version bump, changeset added (This can be done after creating the PR.) This does not apply to changes made to docs, e2e, examples, or other private packages.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[changeset-bot]

🦋 Changeset detected

Latest commit: 13ac976

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@aws-amplify/ui-react-liveness	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[Simone319]

Hello @riasatali42 , thanks for creating the PR! There's conflicts in the branch. Could you resolve the conflict and resubmit the PR?

[riasatali42]

Hello @Simone319 , I have updated the branch with the current main branch and resolved all the conflicts. I've updated this PR instead of creating a new branch. Please let me know if I should do otherwise.

[riasatali42]

Thanks @soberm for your reply. Should I merge this branch (#6673) with my branch to test the e2e testing? I've added the rekognition permissions with the IAM user (the one I'm using for the lambda).

[soberm]

Hi @riasatali42,
I already merged PR #6673 to main. If you update your branch with the main branch you should now be able to run the tests with your personal backend once you deploy the backend changes. Please let us know if you need any assistance.

[riasatali42]

Hello @soberm ,
Thank you so much, all of those issues are resolved now. I can run the test cases properly. But for some reason, once I run the e2e test case on liveness such as 'disable-start-screen', It's saying 'Camera is not accessible'. So, the test case is not passed. If I open the link on a browser, I can see the camera working properly.
Could you please help me here or give me any hint on why this could show this screen?

[soberm]

Hi @riasatali42,
I checked out your PR but unfortunately I don't see this issue on my side. Maybe there is a problem with your browser permissions.

[riasatali42]

Hello @soberm , I've updated the PR. Please let me know if we need anything else.

[soberm]

Hello @soberm , I've updated the PR. Please let me know if we need anything else.

Hey @riasatali42, I can't execute the E2E tests due to a missing step implementation.

Step implementation missing for "I see the "Device ID" selectfield".

[riasatali42]

Hello @soberm , I've added the step. Please take a look now.

[jasonfill]

Hello @soberm - circling back on this. We are eager to get this merged. Could the team please review and return any issues to us so we can bring this to a close?

[soberm]

Hello @soberm - circling back on this. We are eager to get this merged. Could the team please review and return any issues to us so we can bring this to a close?

We're currently reviewing this internally with our team and update you here once we have more information.

[soberm]

Hi @riasatali42, @jasonfill,

thank you for your work on implementing this feature for the FaceLivenessDetector component. We appreciate the implementation and thorough testing you've provided. Following our internal API review, here are the key changes needed to align your implementation with our design recommendations. Please let us know if you have any concerns or questions about these requested changes - we're happy to discuss the rationale behind any of these recommendations.

Requested Changes

1. Simplify Device Input Parameters

Your PR includes both deviceId and deviceLabel with prioritization logic. The device selection should be handled by looking up deviceId by label externally, keeping the API surface simple.

We suggest removing the deviceLabel parameter and only support deviceId in the FaceLivenessDetectorCoreConfig:

export interface FaceLivenessDetectorCoreConfig {
  deviceId?: string; // Only this parameter should be supported
  // ... other existing config options
}

2. Remove Additional Callback Methods

We suggest removing the callbacks that were added:

onDefaultDeviceNotFound
onCameraChange

The existing onError and onAnalysisComplete callbacks already provide the necessary mechanism to return device information when the liveness check succeeds or fails. Adding new callbacks complicates the API surface.

3. Update Existing Callback Signatures

We suggest using the existing callbacks to include optional DeviceInfo parameter to pass this information:

export interface DeviceInfo {
  deviceId: string;
  groupId: string;
  label: string;
}

export interface FaceLivenessDetectorCoreProps {
  onAnalysisComplete: (deviceInfo?: DeviceInfo) => Promise<void>;
  onError?: (livenessError: LivenessError, deviceInfo?: DeviceInfo) => void;
}

4. Ensure DeviceInfo Consistency

Ensure DeviceInfo properties match browser MediaDevices API standards (always strings, never null).

Thank you again for taking the initiative to implement this feature and for your patience as we work through the design.

[jasonfill]

@soberm I have a few items on these

1. Simplify Device Input Parameters
   The issue here is that deviceId can change. So if we are storing the last device used (which is what we are going to do) so we can pre-populate it, the deviceId will not be reliable. We provided both options and prioritized the label to offer choices. If the team wants to drop one, I would suggest we drop the deviceId since it is more ephemeral than the label. Please guide the team on what they would like in this regard.
2. Remove Additional Callback Methods
   We can see getting rid of onCameraChange, although there could be some use cases for this. However, if the team feels that it is too much extra, it can be removed. On the other hand, it makes a lot of sense and would be helpful to know if you're passing in a default device that device is not actually in the list of devices. onDefaultDeviceNotFound can be used for external cleanup or surfacing other messages.

Numbers 3 and 4 make sense, and we can incorporate them. Please provide the final direction on 1 & 2, and we will look to wrap this up.

cc @riasatali42

[soberm]

@jasonfill thank you for the feedback. Here are our thoughts on this:

1. Device Input Parameters
We understand your concern about deviceId being ephemeral. We want to avoid any ambiguity from using deviceLabel on the librarie's side since it's not unique compared to the deviceId according to the documentation. Then, consumers would have the responsibility to resolve this by mapping deviceLabel to the correct deviceId.

2. Callback Methods
Regarding onDefaultDeviceNotFound, we think this functionality can be better handled through the existing onError callback. Device not found scenarios are essentially error conditions, and we already return error types like CAMERA_ACCESS_ERROR through onError. This approach maintains consistency with our existing error handling patterns.

What are your thoughts on this approach?

[jasonfill]

@soberm since the deviceId is not reliable, it makes no sense for us to use that. Our goal is to preload the camera that was used last. The only level of reliability is to do that via label. If label is. Or permitted, this feature is rendered useless for us. Unless I am missing the concept of what you are outlining.

On point 2, that makes total sense. We can adjust to use the onError.

[soberm]

@soberm since the deviceId is not reliable, it makes no sense for us to use that. Our goal is to preload the camera that was used last. The only level of reliability is to do that via label. If label is. Or permitted, this feature is rendered useless for us. Unless I am missing the concept of what you are outlining.

On point 2, that makes total sense. We can adjust to use the onError.

@jasonfill Yes, the deviceId might change but if you already store the deviceLabel of the last used device it should be possible to get the DeviceInfo for the device with this label and then pass the deviceId to the liveness component. Maybe like this:

// Get device by label and extract deviceId
async function getDeviceIdByLabel(targetLabel) {
  const devices = await navigator.mediaDevices.enumerateDevices();
  const camera = devices.find(device => 
    device.kind === 'videoinput' && device.label === targetLabel
  );
  return camera?.deviceId;
}

const lastUsedLabel = getStoredCameraLabel(); // your stored device
const deviceId = await getDeviceIdByLabel(lastUsedLabel);

if (deviceId) {
  // Pass to liveness component
  <LivenessComponent deviceId={deviceId} />
}

Would this approach work for your use case ?

[jasonfill]

@soberm We can make that work. The team will roll these changes in ASAP.

Thanks for your time and collaboration.

[riasatali42]

Hello @soberm , I've updated the code according to your feedback. Please let me know if there's anything else.
cc. @jasonfill

[riasatali42]

Also, @soberm , please let me know if I should add a new changeset or update the old one with the changes.