Rewrite images comparison tests using images input

[IvanNextToJunior]

This time from another branch

[dxCLA]

All committers have signed the CLA.

[schmidt9]

@rzakhar please check first test that uses images input (cd7d960), it works, but fails on pixels comparison inside assertComparison method (actualDifference.image == expectedDifference.image).

Meanwhile Difference values pass test with given accuracy 1e-6
expected 0.19183700000000001 vs actual 0.19183762087210363

Should it mean that we also should apply similar accuracy while comparing pixels arrays too?

[rzakhar]

1e-6 accuracy is equivalent to 3 bad pixels on the 1179 x 2556 screenshot. It looks like quite a reasonable threshold. By ISO 13406-2 standard, it is close to Pixel Fault Class II.

I would keep the accuracy as is

[schmidt9]

@rzakhar I mean pixels arrays are being compared always strict so test will always fail with any non-zero difference

[schmidt9]

XCTAssertTrue(actualDifference.image == expectedDifference.image, file: file, line: line) does not takes into account any accuracy

[rzakhar]

You can use method.compareImages twice

The first time to compare reference images and assert the difference value (delta +- 1e-6). Use the passed method for it.
The second time to compare different images and assert the difference value (0 +- 1e-6). Use the strict method for it.

[schmidt9]

Sorry I don't get you idea. In my understanding if I reuse images produced by SUITCase itself the strict test should pass out of the box.

I don't get why SUITCase produces inside compareImages method from the same Reference and Unexpected images a Difference image which is not equal to the same image loaded from assets. The only point I could imagine I save Difference image from test results log using Preview app, so this app could export Difference image with some specific png settings.

[rzakhar]

What is the difference between difference images? Would you please provide examples?

[schmidt9]

While testing difference images I noticed a strange thing - Difference image from assets turns green when converted to RGBAImage

let differenceImage = UIImage(named: "en_iPhone_X_difference_strict", in: .module, compatibleWith: nil)!
        
try! differenceImage.writePNG(filePath: "/Users/user/Desktop/assets_diff.png")
try! RGBAImage(uiImage: differenceImage).uiImage.writePNG(filePath: "/Users/user/Desktop/rgba_diff.png")

This gives me

[schmidt9]

What is the difference between difference images? Would you please provide examples?

Running in assertComparison with strict method:

            let actualDifference = try method.compareImages(actual: image1.uiImage,
                                                            reference: image2.uiImage)
            
            let differenceDifference = try method.compareImages(actual: actualDifference.image.uiImage,
                                                                reference: expectedDifference.image.uiImage)

differenceDifference.value is ≈ 0.129131, so difference images seem to be quite different, is it related to turning to green?

[rzakhar]

Hey! Thanks for the PR. Remember to include a proper explanation to the PR. Things that can help:

1. Clear title that shortly explains the changes
2. Proper PR description with a detailed explanation
3. Test data or setup required for a reviewer to verify changes.
4. An .xcresult file of the successfully passed tests

[rzakhar]

Looks like we reuse these images in all tests. Can you please

1. Reduce code repetition by moving these variables from the test cases to the test classes?
2. Double-check that assets only include the images we need.

[schmidt9]

Hey! Thanks for the PR. Remember to include a proper explanation to the PR. Things that can help:

1. Clear title that shortly explains the changes
2. Proper PR description with a detailed explanation
3. Test data or setup required for a reviewer to verify changes.
4. An .xcresult file of the successfully passed tests

@rzakhar
1 and 2 will be fixed
3 see please above my comments regarding difference images
4 Tests do not pass, that's the problem I do not know why, that's not clear about pixel arrays comparison and I still don't know why difference images are different, maybe you have a clue?

[IvanNextToJunior]

Rewrite images comparison tests using real images input from assets