🥇Launch instrumental tests in Firebase Test Lab. Part 1: iOS project

My name is Dmitry, I work as a tester in a company M.E.L. Science. Most recently, I finished dealing with a relatively fresh feature from firebase test lab - namely, with instrumental testing of iOS applications using the native testing framework XCUITest.

Before that, I already tried Firebase Test Lab for Android and I really liked everything, so I decided to try to put the project's iOS test infrastructure on the same rails. I had to google a lot and not everything worked out the first time, so I decided to write a tutorial article for those who still have to do it.

So, if you have UI tests on an iOS project, you can already try running them on real devices today, kindly provided by the Good Corporation. Interested - welcome under cat.

In the story, I decided to build on some source data - a private repository on GitHub and the CircleCI build system. Application name is AmazingApp, bundleID is com.company.amazingapp. I quote these data immediately, to reduce subsequent confusion.

If you implemented certain solutions in your project differently, share your experience in the comments.

1. The tests themselves

Create a new project branch for UI tests:

$ git checkout develop
$ git pull
$ git checkout -b “feature/add-ui-tests”

Let's open the project in XCode and create a new Target with UI tests [XCode -> File -> New -> Target -> iOS Testing Bundle], give it the telling name AmazingAppUITests.

Go to the Build Phases section of the created Target and check for Target Dependencies - AmazingApp, in Compile Sources - AmazingAppUITests.swift.

It is good practice to separate the various build options into separate Schemes. We create a scheme for our UI tests [XCode -> Product -> Scheme -> New Scheme] and give it the same name: AmazingAppUITests.

Build of the created scheme should include the Target of the main application - AmazingApp and Target UI tests - AmazingAppUITests - see screenshot

Next, we create a new build configuration for the UI tests. In Xcode, click on the project file, go to the Info section. Click on the “+” and create a new configuration, for example XCtest. We will need this in the future in order to avoid dancing with a tambourine when it comes to code signing.

There are at least three Targets in your project: the main application, unit tests (there are some, right?) and the Target UI of tests we created.

Go to Target AmazingApp, Build Settings tab, Code Signing Identity section. For the XCtest configuration, select iOS Developer. In the Code Signing Style section, select Manual. We have not yet generated a provisioning profile, but we will definitely return to it a little later.

For Target AmazingAppUITests, we do the same, but enter com.company.amazingappuitests in the Product Bundle Identifier column.

2. Setting up a project in the Apple Developer Program

We go to the Apple Developer Program page, go to the Certificates, Identifiers & Profiles section and then to the App IDs column of the Identifiers item. Create a new App ID named AmazingAppUITests and bundleID com.company.amazingappuitests.

Now we have the opportunity to sign our tests with a separate certificate, but ... The build procedure for testing involves building the application itself and building the test runner. Accordingly, we are faced with the problem of signing two bundle IDs with one provisioning profile. Fortunately, there is a simple and elegant solution - Wildcard App ID. We repeat the procedure for creating a new App ID, but instead of Explicit App ID, select Wildcard App ID as in the screenshot.

At this point, we're done with developer.apple.com, but we won't minimize the browser window. Let's go to site with documentation on Fastlane and read about the Match utility from cover to cover.

An attentive reader has noticed that in order to use this utility, we need a private repository and an account that has access to both the Apple Developer Program and Github. We create (if suddenly there is no such thing) an account of the form [email protected], come up with a strong password, register it at developer.apple.com, and appoint it as a project administrator. Next, give your account access to your company's github repository and create a new private repository with a name like AmazingAppMatch.

3. Setting up Fastlane and the match utility

Open the terminal, go to the folder with the project and initialize fastlane as indicated in official manual. After entering the command

$ fastlane init

you will be prompted to select the available usage configurations. We select the fourth item - manual configuration of the project.

A new fastlane directory has appeared in the project, in which there are two files - Appfile and Fastfile. In a nutshell - in Appfile we store service data, and in Fastfile we write jobs, in Fastlane terminology called lanes. I recommend reading the official documentation: time, two.

Open the Appfile in your favorite text editor and bring it to the following form:

app_identifier "com.company.amazingapp"       # Bundle ID
apple_dev_portal_id "[email protected]"  # Созданный инфраструктурный аккаунт, имеющий право на редактирование iOS проекта в Apple Developer Program.
team_id "LSDY3IFJAY9" # Your Developer Portal Team ID

We return to the terminal and, according to the official manual, start setting up match.

$ fastlane match init
$ fastlane match development

Next, enter the requested data - repository, account, password, etc.

Important: the first time you run the match utility, it will ask you to enter a password to decrypt the repository. It is very important to save this password, we will need it at the stage of setting up the CI server!

A new file has appeared in the fastlane folder - Matchfile. Open it in your favorite text editor and bring it to the form:

git_url("https://github.com/YourCompany/AmazingAppMatch") #Созданный приватный репозиторий для хранения сертификатов и профайлов.
type("development") # The default type, can be: appstore, adhoc, enterprise or development
app_identifier("com.company.amazingapp")
username("[email protected]") # Your Infrastructure account Apple Developer Portal username

We fill it in this way if we want to use match in the future to sign builds for uploading to Crashlytics and / or AppStore, i.e. to sign the bundle ID of your application.

But, as we remember, we created a special Wildcard ID to sign the test build. Therefore, open Fastfile and enter a new lane:

lane :testing_build_for_firebase do

    match(
      type: "development",
      readonly: true,
      app_identifier: "com.company.*",
      git_branch: "uitests"  # создаем отдельный бранч для development сертификата для подписи тестовой сборки.
    )

end

Save, enter into the terminal

fastlane testing_build_for_firebase

and see how fastlane created a new certificate and put it in the repository. Great!

Open XCode. Now we have the required provisioning profile of the Match Development com.company.* type, which must be specified in the Provisioning profile section for the AmazingApp and AmazingAppUITests targets.

It remains to add lane to build tests. Let's go to repository fastlane plugin project that makes it easy to set up export to Firebase Test Lab and follow the instructions.

Copy-paste from the original example so that our lane testing_build_for_firebase ends up looking like this:


 lane :testing_build_for_firebase do

    match(
      type: "development",
      readonly: true,
      app_identifier: "com.company.*",
      git_branch: "uitests"
    )

    scan(
      scheme: 'AmazingAppUITests',      # UI Test scheme
      clean: true,                        # Recommended: This would ensure the build would not include unnecessary files
      skip_detect_devices: true,          # Required
      build_for_testing: true,            # Required
      sdk: 'iphoneos',                    # Required
      should_zip_build_products: true,     # Must be true to set the correct format for Firebase Test Lab
    )

    firebase_test_lab_ios_xctest(
      gcp_project: 'AmazingAppUITests', # Your Google Cloud project name (к этой строчке вернемся позже)
      devices: [                          # Device(s) to run tests on
        {
          ios_model_id: 'iphonex',        # Device model ID, see gcloud command above
          ios_version_id: '12.0',         # iOS version ID, see gcloud command above
          locale: 'en_US',                # Optional: default to en_US if not set
          orientation: 'portrait'         # Optional: default to portrait if not set
        }
      ]
    )

  end

For complete information on setting up fastlane in CircleCI, I recommend reading the official documentation time, two.

Don't forget to add a new task to our config.yml:

build-for-firebase-test-lab:
   macos:
     xcode: "10.1.0"   
   working_directory: ~/project
   shell: /bin/bash --login -o pipefail
   steps:
     - checkout
     - attach_workspace:
         at: ~/project
     - run: sudo bundle install     # обновляем зависимости
     - run:
         name: install gcloud-sdk   # на mac машину необходимо установить gcloud
         command: |
           ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" < /dev/null 2> /dev/null ; brew install caskroom/cask/brew-cask 2> /dev/null
           brew cask install google-cloud-sdk
     - run:
         name: build app for testing
         command: fastlane testing_build_for_firebase  # запускаем lane сборки и отправки в firebase

4. What about our test stand? Setting up Firebase.

Let's proceed, in fact, to what the article was written for.

Your app may be using Firebase on a free plan, or it may not be using Firebase at all. There is absolutely no fundamental difference, because for the needs of testing we can create a separate project with a year of free use (cool, huh?)

We log in to our infrastructure account (or any other, it doesn’t matter), and go to firebase console page. Create a new project named AmazingAppUITests.

Important: In the previous step, in the Fastfile in lane firebase_test_lab_ios_xctest, the gcp_project parameter must match the name of the project.

The default settings suit us just fine.

We do not close the tab, we register under the same account in gcloud is a necessary measure, since communication with Firebase takes place using the gcloud console interface.

Google is giving away $300 for a year, which in the context of performing autotests is equivalent to a year of free use of the service. We enter payment data, wait for the test write-off of $1 and get $300 to the account. After a year, the project will be automatically transferred to a free tariff plan, so you should not worry about the possible loss of money.

Let's return to the tab with the Firebase project and transfer it to the Blaze tariff plan - now we have something to pay if the limit is exceeded.

In the gcloud interface, select our Firebase project, select the "Catalogue" main menu item and add the Cloud Testing API and Cloud Tools Result API.

Then go to the menu item "IAM and administration" -> Service accounts -> Create a service account. Grant permission to edit the project.

Create an API key in JSON format

We will need the downloaded JSON a little later, but for now we will consider the Test Lab setup complete.

5. Setting CircleCI

A reasonable question is brewing - what to do with passwords? To keep our passwords and other sensitive data securely, the mechanism of environment variables of our build machine will help us. In the CircleCI project settings, select Environment Variables

And set the following variables:

key: GOOGLE_APPLICATION_CREDENTIALS
value: contents of the gcloud service account key json file
key: MATCH_PASSWORD
value: password to decrypt github repository with certificates
key: FASTLANE_PASSWORD
value: Apple Developer Portal infrastructure account password

We save the changes, create a PR and send it to our team leader for review.

Results

As a result of these simple manipulations, we got a good, stable working stand with the ability to record video on the device screen at the time of testing. In the test case, I specified the iPhone X device model, but the farm provides a rich selection from a combination of different models and iOS versions.

The second part will be devoted to setting up Firebase Test Lab for an Android project step by step.

Source: habr.com

Run instrumental tests in Firebase Test Lab. Part 1: iOS project