Safari Automation on AWS - what we learned from the process

Intro

It all started with a realization within our automation team: a significant percentage of our product users were browsing our website using macOS and Safari. As we delved deeper into our analytics data, it became evident that we needed to provide our QA teams with the ability to run their cross-browser and cross-platform tests on Safari as well.

macOS and Safari bring with them a set of unique characteristics and quirks that can significantly impact the user experience and from a Test Automation standpoint a very challenging platform to support. 

What makes Safari so special? 

Safari, as the default browser for macOS users, introduces a distinct set of behaviors and rendering nuances that differentiate it from other browsers. From differences in font rendering and CSS handling to variations in JavaScript performance and browser behavior, Safari presents a dynamic testing landscape that requires meticulous attention to detail.

Font rendering and CSS handling, for example, may exhibit subtle discrepancies compared to other browsers, leading to layout inconsistencies and visual discrepancies that can affect the overall user experience. Similarly, variations in JavaScript performance and browser behavior may manifest as unexpected page rendering issues or functional inconsistencies.

Challenges from a Test Automation Standpoint

For Safari, the natural ecosystem would be running on macOS. However, it creates different additional challenges as opposed to Chromium-based browsers such as licensing, price, hardware, and other technical limitations. The traditional approach of acquiring a single physical device and connecting it as a Jenkins executor was impractical. Additionally, the idea of purchasing a license with a cloud provider for execution platforms was not feasible for various reasons.

Given that our automation runs on AWS EC2 slaves, we needed to explore available solutions within Amazon Web Services to address this challenge.

Soon enough we discovered that AWS offers Amazon Elastic Compute Cloud instances with OSX. After some research and a short meeting with a representative, we decided to go ahead and do a POC with the newly offered instances.

Spinning up an instance

As opposed to other virtual cloud instances, macOS EC2 instances can only run on physical Mac hardware due to licensing and technical limitations. Therefore, to provision a macOS EC2 instance, you need to allocate a dedicated host on AWS that is specifically equipped with Mac hardware. A few important things to consider is that a single Hardware (dedicated host) can only run a single EC2 instance. 

Also, it is worth noting that the Billing model for these instances is by committing to 24H on spinup. (Meaning if you kill the instance and release the host after 10 minutes from creation, you will still pay for a full 24H window). 

It is also worth saying that usually there is a limit on how many dedicated hosts you can create, so make sure you know your needs ahead of time so you will not discover while you go that you have reached a quota. 

After setting up and configuring the dedicated host with a name, instance family, and instance type, the next step is to launch an OSX instance on top of it.

(The instance can be created from AMI or as we did, a clean OSX instance which we provisioned from scratch to support running automation.)

The instance requires configuration. Since we manually provisioned it for a POC, we didn't use Packer or Terraform and had to manually configure all the necessary settings, such as security groups, relevant instance tags, networking, keypairs, storage type, etc.

Once the instance was launched, we had 2 main access options to provision it. Via SSH (Which we mostly used) and with a remote desktop connection (which was pretty cool since you actually see a Mac with an oSX desktop, and came in handy when we did some OS-level configuration). 

Once the instance was provisioned and ready we tested it via local run and then proceeded to connect it as a Jenkins slave. (Since most of the networking was resolved by assigning the appropriate VPC, and security groups it was pretty smooth sailing from there). 

We learned quite a bit about the limitations of Running Automations on Safari

Through this process, we have learned about the limitations of running on the Safari browser and using the architecture we have chosen. Here are several main things to consider before you go ahead and attempt to implement it in your infrastructure.

Dedicated host quota -  Since we are running on an actual hardware that is allocated for us on AWS (And the consequential substantial pricing model) there will be a limit on how many dedicated hosts you can have up and running at the same time. If you only need a single instance at any given time that should not bother you, but keep in mind that the quota is on an organization/account level so there could be a scenario where you might need to spin up a host and you will have no available quota. This leads me to the next point…

Long host release/shutdown time - While the AWS team is working on speeding up this process, the last time I checked it took more than 30 minutes to release a host. Take that velocity bump under consideration when you need availability. 

Pricing and 24H commitment - As I mentioned earlier, these instances are not cheap, and once you spin them up you are committed to a 24H price slot. Even if you release it / shut it down after a few minutes.

Does not support AWS instance-connect - It is important to note that as opposed to other instance types, macOS EC2 instances do not support AWS instance connect so you will have to find another secure way to connect to them. 

No execution concurrency - It is worth mentioning that Safari (for the last version I checked) does not support concurrent execution. So if you are dreaming about having 30 browser instances running simultaneously on the same EC2 instance, I have bad news for you…

No Pac files support - If you haven’t encountered the need to use them, PAC (Proxy Auto-Configuration) files are scripts written in JavaScript that define how web browsers should choose a proxy server for a given URL. These files are used to automate the process of proxy server selection based on specific rules and conditions. PAC files are commonly used in corporate networks or environments with proxy servers. They allow organizations to define complex routing logic, such as directing traffic to different proxy servers based on the URL, IP address, or time of day.  So, if your Automated tests rely on using PAC files, it's important to note that Safari browser does not support them. 

Limited Browser Version Availability - Safari updates are tied to macOS releases, which can lead to challenges in ensuring consistency with specific browser versions. Unlike other browsers that offer frequent updates across multiple operating systems, Safari updates are less frequent and may not always align with the latest web standards or features. It also means that once an update is released and you would like to upgrade - it would require an upgrade of the OS. Small yet important details can be easily overlooked and a lot of times lead to different types of issues with Automation. 

Restricted Browser Configuration - Safari may have limitations in terms of browser configuration options compared to other browsers, such as Chrome or Firefox. This could impact the ability to customize browser settings or install extensions/plugins necessary for certain testing scenarios, potentially limiting the flexibility of test automation workflows.

Limited Third-Party Tool Support - Some third-party testing tools and frameworks may have limited support or compatibility with Safari compared to other browsers. Organizations using specific testing tools should verify their compatibility with Safari and ensure that necessary integrations or workarounds are available.

In conclusion…

Knowing the limitations of running automation on Safari is important. Despite its challenges like limited browser version availability and dependencies on Operating System updates, having coverage for Safari is important if your company's end users predominantly use macOS and Safari. Having a limited, focused suite can be a good compromise to ensure that your company will provide a good and solid browsing experience.