My name is Dmitry, and I work as a tester at the company . Recently, I finished exploring a relatively new feature from — specifically, instrumental testing of iOS applications using the native testing framework XCUITest.
Prior to this, I had already experimented with Firebase Test Lab for Android and I really liked it, so I decided to try setting up the testing infrastructure for the iOS project in the same manner. I had to Google a lot, and it didn't all work out on the first try, so I decided to write a tutorial article for those who still have this ahead of them.
So, if you have UI tests on your iOS project, you can already try running them on real devices today, graciously provided by the Good Corporation. Interested parties are welcome under the cut.
In this narrative, I decided to base it on some initial data — a private repository on GitHub and the CircleCI build system. The application name is AmazingApp, bundleID — com.company.amazingapp. I provide this data upfront to reduce confusion later.
If you have implemented certain solutions differently in your project — share your experience in the comments.
1. The Tests Themselves
Create a new project branch for UI tests:
$ git checkout develop
$ git pull
$ git checkout -b “feature/add-ui-tests”
Open the project in XCode and create a new Target with UI tests [XCode -> File -> New -> Target -> iOS Testing Bundle], giving it a descriptive name AmazingAppUITests.
Go to the Build Phases section of the created Target and check for Target Dependencies — AmazingApp, and in Compile Sources — AmazingAppUITests.swift.
A good practice is to separate various build options into different Schemes. Create a scheme for our UI tests [XCode -> Product -> Scheme -> New Scheme] and name it the same: AmazingAppUITests.
The build of the created scheme should include the Target of the main application — AmazingApp and the Target for the UI tests — AmazingAppUITests — see screenshot.
Next, we create a new build configuration for UI tests. In XCode, click on the project file, go to the Info section. Click on “+” and create a new configuration, for example, XCtest. We will need this later to avoid complications when it comes to code signing.
Your project has at least three Targets: the main application, unit tests (they exist, right?), and the Target for the UI tests we created.
Go to the Target AmazingApp, the Build Settings tab, in the Code Signing Identity section. For the XCtest configuration, select iOS Developer. In the Code Signing Style section, choose Manual. We haven't generated a provisioning profile yet, but we'll come back to it a bit later.
For the Target AmazingAppUITests, do the same, but enter com.company.amazingappuitests in the Product Bundle Identifier field.
2. Setting Up the Project in the Apple Developer Program
Go to the Apple Developer Program page, navigate to Certificates, Identifiers & Profiles, and then to the App IDs section under Identifiers. Create a new App ID named AmazingAppUITests with the bundleID com.company.amazingappuitests.
Now we have the option to sign our tests with a separate certificate, but... The build procedure for testing involves building the application itself and the test runner builds. Consequently, we face the issue of signing two bundle IDs with one provisioning profile. Fortunately, there is a simple and elegant solution — the Wildcard App ID. Repeat the steps to create a new App ID, but instead of an Explicit App ID, choose Wildcard App ID as shown in the screenshot.
At this stage, work with developer.apple.com is finished, but we won't close the browser window. Let's go to and read about the Match utility from cover to cover.
The attentive reader may have noticed that to use this utility, we need a private repository and an account that has access to both the Apple Developer Program and Github. Create (if you don’t already have one) an account like InfrastructureAccount@your.company.domain, come up with a strong password, register it in developer.apple.com, and assign it as the project administrator. Next, grant the account access to your company’s GitHub repository and create a new private repository named something like AmazingAppMatch.
3. Setting Up Fastlane and the Match Utility
Open the terminal, navigate to the project folder, and initialize fastlane as specified in the . After entering the command
$ fastlane inityou will be prompted to choose from available configuration options. Select the fourth option — manual project setup.
A new directory called fastlane has appeared in the project, containing two files — Appfile and Fastfile. In brief, we store service data in the Appfile, and we define jobs, referred to as lanes in Fastlane, in the Fastfile. I recommend reading the official documentation: , .
Open the Appfile in your favorite text editor and set it to the following format:
app_identifier "com.company.amazingapp" # Bundle ID
apple_dev_portal_id "infrastructureaccount@your.company.domain" # The created infrastructure account with permission to edit the iOS project in the Apple Developer Program.
team_id "LSDY3IFJAY9" # Your Developer Portal Team ID
Return to the terminal and start configuring match according to the official manual.
$ fastlane match init
$ fastlane match development
Next, enter the requested data — repository, account, password, etc.
Important: On the first run, the match utility will ask for a password to decrypt the repository. It is very important to keep this password, as we will need it during the CI server setup!
A new file called Matchfile has appeared in the fastlane folder. Open it in your favorite text editor and set it to the following format:
git_url("https://github.com/YourCompany/AmazingAppMatch") # The created private repository for storing certificates and profiles.
type("development") # The default type, can be: appstore, adhoc, enterprise, or development
app_identifier("com.company.amazingapp")
username("infrastructureaccount@your.company.domain") # Your Infrastructure account Apple Developer Portal username
Fill it out exactly this way if you want to use match for signing builds for deployment to Crashlytics and/or App Store, i.e., for signing the bundle ID of your application.
But, as we remember, to sign the test build, we created a special Wildcard ID. Therefore, let's open the Fastfile and add a new lane:
lane :testing_build_for_firebase do
match(
type: "development",
readonly: true,
app_identifier: "com.company.*",
git_branch: "uitests" # create a separate branch for the development certificate to sign the test build.
)
end
Save it and enter in the terminal
fastlane testing_build_for_firebaseand see how fastlane created a new certificate and placed it in the repository. Great!
Open XCode. Now we have the required provisioning profile of type Match Development com.company.*, which needs to be specified in the Provisioning profile section for the AmazingApp and AmazingAppUITests targets.
It remains to add a lane for building tests. Let's go to the fastlane plugin project that simplifies the setup for exporting to Firebase Test Lab and follow the instructions.
Copy-paste from the original example so that our lane testing_build_for_firebase ultimately looks like this:
lane :testing_build_for_firebase do
match(
type: "development",
readonly: true,
app_identifier: "com.company.*",
git_branch: "uitests"
)
scan(
scheme: 'AmazingAppUITests', # UI Test scheme
clean: true, # Recommended: This would ensure the build would not include unnecessary files
skip_detect_devices: true, # Required
build_for_testing: true, # Required
sdk: 'iphoneos', # Required
should_zip_build_products: true, # Must be true to set the correct format for Firebase Test Lab
)
firebase_test_lab_ios_xctest(
gcp_project: 'AmazingAppUITests', # Your Google Cloud project name (we'll get back to this line later)
devices: [ # Device(s) to run tests on
{
ios_model_id: 'iphonex', # Device model ID, see gcloud command above
ios_version_id: '12.0', # iOS version ID, see gcloud command above
locale: 'en_US', # Optional: default to en_US if not set
orientation: 'portrait' # Optional: default to portrait if not set
}
]
)
end
For complete information on configuring fastlane in CircleCI, I recommend reading the official documentation .
Don't forget to add a new task to our config.yml:
build-for-firebase-test-lab:
macos:
xcode: "10.1.0"
working_directory: ~/project
shell: /bin/bash --login -o pipefail
steps:
- checkout
- attach_workspace:
at: ~/project
- run: sudo bundle install # update dependencies
- run:
name: install gcloud-sdk # need to install gcloud on mac machine
command: |
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" /dev/null ; brew install caskroom/cask/brew-cask 2> /dev/null
brew cask install google-cloud-sdk
- run:
name: build app for testing
command: fastlane testing_build_for_firebase # run the build lane and send to firebase
4. What about our test environment? Let's configure Firebase.
Now, let's get to what this article was actually written for.
Your application may use Firebase on a free plan, or it may not use it at all. There is absolutely no significant difference, as for testing purposes we can create a separate project with a year of free use (cool, right?)
Log in to our infrastructure account (or any other, it doesn't matter), and go to . Create a new project named AmazingAppUITests.
Important: In the previous step in Fastfile, the gcp_project parameter in the lane firebase_test_lab_ios_xctest should match the project name.
The default settings are perfectly fine for us.
Don't close the tab; register under the same account with — this is a necessary step, as communication with Firebase occurs through the gcloud console interface.
Google offers $300 for a year, which, in the context of running automated tests, is equivalent to a year of free service. We enter the payment details, wait for a test deduction of $1, and receive $300 in our account. After a year, the project will automatically switch to a free plan, so there’s no need to worry about potential loss of funds.
Let's return to the Firebase project tab and upgrade it to the Blaze plan — now we have a way to pay in case we exceed the limits.
In the gcloud interface, we select our Firebase project, choose the main menu item 'Catalog', and add the Cloud Testing API and Cloud Tools Result API.
Then we go to the menu item 'IAM & Admin' -> Service Accounts -> Create Service Account. We grant editing rights to the project.
We create an API key in JSON format.
The downloaded JSON will be needed later, but for now, we’ll consider the Test Lab setup complete.
5. Configuring CircleCI
A reasonable question arises — what about passwords? To securely store our passwords and other sensitive data, we can use the environment variable mechanism of our build machine. In the CircleCI project settings, we select Environment Variables.
And we create the following variables:
- key: GOOGLE_APPLICATION_CREDENTIALS
value: the contents of the JSON key file for the gcloud service account - key: MATCH_PASSWORD
value: the password to decrypt the GitHub repository with the certificates - key: FASTLANE_PASSWORD
value: the password for the infrastructure account on Apple Developer Portal
We save the changes, create a PR, and submit it for review to our team lead.
Summary
As a result of these simple manipulations, we have a good, stable environment with the ability to record video on the device screen while tests are being executed. In the test example, I specified the device model iPhone X, but the farm offers a rich selection of combinations of various models and iOS versions.
The second part will focus on the step-by-step setup of Firebase Test Lab for an Android project.
Source: habr.com
