All tests failing but Setup and Teardown

[aw-vm]

Hello,

I have tried running my tests on device farm but all the test are failing.
I have followed all the steps of the documentation and generated the files as the instructions said.
I have not encountered any issue uploading these files or creating a run with them but when the run is over I get these results:

I'm getting this from the automation output of one test:

User defaults from command line:
    IDEDerivedDataPathOverride = /tmp/scratchnZxqrd.scratch/xcode_screenshots7MfdjK/UITests15TwilioE4UITests/testExample
    IDETestRunOnlyIdentifiers = (
    "TwilioE4UITests/UITests15TwilioE4UITests/testExample"
)
    IDETestRunSpecificationPath = /Users/rwx/Library/Developer/Xcode/DerivedData/Build/Products/application.xctestrun

2018-07-30 09:42:09.687 xcodebuild[535:7368]  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/var/folders/tg/pk__v_v5785_6_qbhdg297gr0000gp/T/com.apple.dt.XCTest/IDETestRunSession-40A4F518-21AA-49FF-97D0-CDCD75DDF535/TwilioE4UITests-79D669D7-3634-4C57-8276-610333B02474/Session-TwilioE4UITests-2018-07-30_094209-hBcQkX.log
2018-07-30 09:42:09.689 xcodebuild[535:7354] [MT] IDETestOperationsObserverDebug: (032D8571-B44D-4931-A6DA-F7D1FA773B91) Beginning test session TwilioE4UITests-032D8571-B44D-4931-A6DA-F7D1FA773B91 at 2018-07-30 09:42:09.688 with Xcode 9F1027a on target <DVTiOSDevice: 0x7fc66fe14260> {
		deviceSerialNumber:         G6TVH3K9JCL8
		identifier:                 b2f96e8a3a53c1bd94d05fc1d0c06aa94954148a
		deviceClass:                iPhone
		deviceName:                 9902852853
		deviceIdentifier:           b2f96e8a3a53c1bd94d05fc1d0c06aa94954148a
		productVersion:             11.4
		buildVersion:               15F79
		deviceSoftwareVersion:      11.4 (15F79)
		deviceArchitecture:         arm64
		deviceTotalCapacity:        252371152896
		deviceAvailableCapacity:    249001328640
		deviceIsTransient:          NO
		ignored:                    NO
		deviceIsBusy:               NO
		deviceIsActivated:          YES
		deviceActivationState:      Activated
		isPasscodeLocked:           NO
		deviceType:                 <DVTDeviceType:0x7fc66fd80f20 Xcode.DeviceType.iPhone>
		supportedDeviceFamilies:    (
    1
)
		applications:              (null)
		provisioningProfiles:      (null)
		hasInternalSupport:        NO
		isSupportedOS:             YES
		developerDiskMountError:   (null)
(null)
	bootArgs:                  <unavailable>
		connected:                 yes
		isWirelessEnabled:         no
		connectionType:            direct
		} (11.4 (15F79))
MDMCreateDeltaDirectory:1920 calling MDMDirectoryDiff with:
state->old_bundle: /var/folders/tg/pk__v_v5785_6_qbhdg297gr0000gp/C/com.apple.DeveloperTools/All/Xcode/EmbeddedAppDeltas/977ae2eae09fab3f9a27a0f75731a39d/b2f96e8a3a53c1bd94d05fc1d0c06aa94954148a/TwilioE4UITests-Runner.app
state->new_bundle: /Users/rwx/Library/Developer/Xcode/DerivedData/Build/Products/Debug-iphoneos/TwilioE4UITests-Runner.app
state->dst_bundle: /var/folders/tg/pk__v_v5785_6_qbhdg297gr0000gp/C/com.apple.DeveloperTools/All/Xcode/EmbeddedAppDeltas/TwilioE4UITests-Runner.app.aYOj0V/TwilioE4UITests-Runner.app_sparse.ipa/Payload//TwilioE4UITests-Runner.app, binaryDiff flag: FALSE
	dst_ipa: /var/folders/tg/pk__v_v5785_6_qbhdg297gr0000gp/C/com.apple.DeveloperTools/All/Xcode/EmbeddedAppDeltas/TwilioE4UITests-Runner.app.aYOj0V/TwilioE4UITests-Runner.app_sparse.ipa
MDMDirectoryDiff_block_invoke:1473 calling writeDictToFile with: /var/folders/tg/pk__v_v5785_6_qbhdg297gr0000gp/C/com.apple.DeveloperTools/All/Xcode/EmbeddedAppDeltas/TwilioE4UITests-Runner.app.aYOj0V/TwilioE4UITests-Runner.app_sparse.ipa/ManifestCache.plist
writeDictToFile:1278 ==== Successfully wrote Manifest cache to /var/folders/tg/pk__v_v5785_6_qbhdg297gr0000gp/C/com.apple.DeveloperTools/All/Xcode/EmbeddedAppDeltas/TwilioE4UITests-Runner.app.aYOj0V/TwilioE4UITests-Runner.app_sparse.ipa/ManifestCache.plist
** TEST EXECUTE SUCCEEDED **

2018-07-30 11:42:16.421756-0500 TwilioE4UITests-Runner[588:26690] +[CATransaction synchronize] called within transaction
2018-07-30 11:42:16.442034-0500 TwilioE4UITests-Runner[588:26690] Running tests...
2018-07-30 11:42:17.072729-0500 TwilioE4UITests-Runner[588:26690] [AXValidations] UIAccessibility Error adding -[CAMLightingControl accessibilityLabel]
2018-07-30 11:42:17.072828-0500 TwilioE4UITests-Runner[588:26690] [AXValidations] UIAccessibility Error adding -[CAMLightingControl accessibilityValue]
2018-07-30 11:42:17.072880-0500 TwilioE4UITests-Runner[588:26690] [AXValidations] UIAccessibility Error adding -[CAMLightingControl accessibilityTraits]
2018-07-30 11:42:17.072926-0500 TwilioE4UITests-Runner[588:26690] [AXValidations] UIAccessibility Error adding -[CAMLightingControl isAccessibilityElement]
2018-07-30 11:42:17.072977-0500 TwilioE4UITests-Runner[588:26690] [AXValidations] UIAccessibility Error adding -[CAMLightingControl accessibilityIncrement]
2018-07-30 11:42:17.073017-0500 TwilioE4UITests-Runner[588:26690] [AXValidations] UIAccessibility Error adding -[CAMLightingControl accessibilityDecrement]
2018-07-30 11:42:17.073053-0500 TwilioE4UITests-Runner[588:26690] [AXValidations] UIAccessibility Error adding +[CAMLightingControl safeCategoryBaseClass]
2018-07-30 11:42:17.079126-0500 TwilioE4UITests-Runner[588:26690] [AXValidations] UIAccessibility Error adding -[PLManagedAsset accessibilityLabel]
2018-07-30 11:42:17.079200-0500 TwilioE4UITests-Runner[588:26690] [AXValidations] UIAccessibility Error adding -[PLManagedAsset accessibilityValue]
2018-07-30 11:42:17.079233-0500 TwilioE4UITests-Runner[588:26690] [AXValidations] UIAccessibility Error adding -[PLManagedAsset accessibilityTraits]
2018-07-30 11:42:17.079263-0500 TwilioE4UITests-Runner[588:26690] [AXValidations] UIAccessibility Error adding -[PLManagedAsset _accessibilityPhotoDescription]
2018-07-30 11:42:17.079292-0500 TwilioE4UITests-Runner[588:26690] [AXValidations] UIAccessibility Error adding +[PLManagedAsset safeCategoryBaseClass]
2018-07-30 11:42:17.096756-0500 TwilioE4UITests-Runner[588:26690] Continuing to run tests in the background with task ID 1
Test Suite 'Selected tests' started at 2018-07-30 11:42:17.717
Test Suite 'TwilioE4UITests.xctest' started at 2018-07-30 11:42:17.718
Test Suite 'TwilioE4UITests.xctest' passed at 2018-07-30 11:42:17.719.
	 Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds
Test Suite 'Selected tests' passed at 2018-07-30 11:42:17.720.
	 Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.003) seconds

Also checking the System logs I could find this log many times:

Jul 30 11:41:56 9902015857 amfid(Security)[195] <Notice>: could not enable test hierarchy: ApplePinningAllowTestCertsiPhoneApplicationSigning not true

Watching the test videos looks like the app is not opening and nothing is happening there.
Reading the logs I found this one:
Unable to generate launch image for com.apple.test.TwilioE4UITests-Runner because the app has no valid launch interfaces.

[nikhil-dabhade]

@valerymadiedo : Few questions:

1. Have you ran the same tests of a real physical device locally? The reason I ask is at times the test may have been built and run for simulator rather than a real device and error out when running on a real device.
2. Do you have entitlements in your iOS application? If yes, these will break on Device Farm as it re-signs the app. Using private devices gives you the ability to skip re-signing of your app.
3. I also see MDM in the logs. Do you require MDM client to be present on the device while the tests are running?

[aw-vm]

Hello @nikhil-dabhade Thanks a lot for your help.
Answers:

1. Yes I have run these tests in a physical device (iphone 5s - ios 11.2.1).
2. No. The ipa I uploaded to device farm was signed as developer, which is the right way to sign it?
3. Yes. I save some data locally using sqlite.swift.

Edit: here says I must create ad hoc profile and add the devices and I can find the uuid of the device on the device instance and add these to my provisioning profiles but these devices must be private and I dont have access to these yet.

[aw-vm]

Hi,
I have resigned the ipa with an adhoc profile and I'm getting the same behavior. The test opens but is not able to open the app.

[af-mke]

Hey @valerymadiedo , did you solve the problem? I also get these errors and can't figure out what the problem is

[croossin]

Any updates? Facing same issue