I found that if you turn off srcset, the source element with AVIF is not added to the image.

AVIF - Generate images in this format
✓ Also generate fallback images in the original upload format
✓ Generate all fallback image sizes including custom sizes
✓ Use < picture > Element (experimental)

add_filter( 'wp_calculate_image_srcset', '__return_false' );

is this proper behavior?