A complete Selenium Tech Stack: Manage it with Open Source

Want to find out which software products and programming languages we use at Testbirds? Here we give you an insight into our tech stack for test automation. This tech stack is fully independant, and 100% Open Source.On 7th of June our colleague, Felix Kuperjans, Technical Lead Device Cloud, held a lecture at Germany Testing Day in Frankfurt in which he covered the topic and introduced our technology stack for Selenium tests to the audience.He started with the different challenges you normally must face during the development process of a test automation project.As he explained, every test automation project typically has 4 segments:

Test case design
Test case creation
Test execution
Test result evaluation

And, each of these steps has its own challenges and difficulties. If you want to know more details about the different challenges in each segment, have a look at our blog article: German Testing Day 2019 – we are there!

But now, we’d like to show you how we at Testbirds manage to overcome those difficulties. Let me give you a review of Felix’ lecture and guide you through our Selenium Technology Stack.

Testbirds’ Selenium Technology Stack – Open Source Based

Our Tech Stack has been developing over time. We executed a lot of test automation projects during the last years with a huge variety of clients and needs. Each one of those projects taught us something. During this time we more and more learned which requirements we have regarding the tools we use for test automation.

So the technology stack we show you today, is based on our experience with different projects and it shows the best solution for us personally. For sure, this is not the one and only solution for every test automation but this stack has proven to work out on a large scale with our clients. If you’re also planning automation on a large scale this tech stack works perfectly together with our Device Cloud.

In our Selenium tech stack we only use open source tools, so, this might work for you in a similar or even the same way, as well. Please see this as an example how to do it, but of course other tools can be used as well as long as they fulfill the same requirements.

Testbirds’ Tech Stack: Test Case Design with Gherkin

Communication as a common problem

We all are aware, that communication is a very common problem. In test automation it is communication between developers, product owners, analysts and other stakeholders in your company. So what happens mostly is: developers write a test case for themselves and no one else can understand what is tested and how. This might cause different problems not only, but also, when a test fails. Imagine an error occurring and the analyst has to decide what to do. Then at the very latest he needs to understand what was tested and if it’s a critical error for the product or not. Furthermore coverage and correctness of the passing tests are also not visible to the stakeholders.

BBD with Cucumber to ensure readability

To ensure readability for all stakeholders, we use BDD with Cucumber in our tech stack. It helps making the test specification readable and understood by the whole team. This is made possible through the Gherkin Syntax which uses a set of keywords to structure the specifications. That way, we can include the whole team reaching from testers, over developers to POs. For POs the “given -when – then” structure is easy to understand and still the developers have a structured test case so they know exactly what needs to be embedded in the test and when.

Don’t write the Syntax alone

Our Tipp: The Syntax for your test shouldn’t be written only by your developers neither only by product owners or other stakeholders. Take the time and develop the syntax together. That way you can make sure that POs are understanding it and it still being granular enough for your developers. This approach is often referred to as the 3 amigos approach. Which doesn’t mean that you always need exactly three people for your test case design but all three roles, developer, PO, and tester, should be represented. This saves you a lot of time after all.

Make test cases reusable

To establish test cases that are reusable, the step definitions in the Gherkin syntax should be designed with a single, more generic meaning. For example, instead of a step definition:“Given I have a todo list where the first item is checked and the second is not”, which is very specific and less reusable, I can use a more generic assembly of multiple reusable steps:“Given I have a todo list with 2 itemsand the 1st item is checkedand the 2nd item is not checked”With that you can design generic steps to have any number of items on the todo list and to check (or not check) any of them.This would also allow to build a scenario like the following, without further implementation.“Given I have a todo list with 3 itemsand the 1st item is not checkedand the 2nd item is checkedand the 3rd item is checked.”That way we can secure a flexible opportunity of combinations. In addition, Narratives can be used to give complementary explanations of the context and situation to the readers, if the steps themselves are a bit complex or not self-explanatory enough.But remember, it is very important that the Gherkin steps back away from the technical aspects of the test, like for example element selectors.

Test Case Development with Selenide

Stable and easy to use

You may ask yourself why our Selenium technology stack includes Selenide and not “just” Selenium. As we figured out, Selenide is much more stable and very easy to use.Selenide provides cross-browser proven framework methods that are much more resilient then the underlying basic Selenium actions. That way we don’t have to find workarounds to make our tests run in all targeted browsers on our own, we can concentrate on the test case specifications instead.Beyond that, Selenide comes with very neat session and driver management. The simple test runner configurations allow our test suite to be reused easily in various browsers and platforms.

Easy implementation of best practices

The framework furthermore implicitly implements best practices like a globally configurable timeout that gets executed before every operation. This timeout is by default configured to 4 seconds, this means the framework waits up to 4 seconds for any expected result or behavior to occur.Why do we do that? Because especially in web testing some things simply need time until they occur. Imagine a button that only appears after some time. Selenide can on the one hand make sure, that the button appears and on the other hand also wait until it does, and in web testing 4 seconds seem to be the “sweet spot” as several frameworks are using it now as default value.To make it short, Selenide takes care that elements are visible before a test case tries to interact with them. This reduces the flakiness of test cases immensely.Also, Selenide allows a native page object integration. Using page objects the selector isn’t hard coded into the test case anymore, you only need to use a reference to the element instead. That way we can avoid duplicate code and – as the selector exists only once in the code base – changes can be made quickly and not in multiple areas of the code.

Our advice: Choose frameworks with an active community

Always choose frameworks with an active community as that guarantees new developments in browser testing to be integrated into the framework. That makes life for your team a lot easier as you don’t have to build workarounds and fixes every time the most basic libraries change.Last but not least, Selenide offers screenshots and DOM-dumps which are very useful for your failure analysis. With them you can easily see where errors occured. Imagine a test running nightly over months and suddenly something fails. With help of the DOM-dumps and screenshots you can quickly find out what went wrong and where.

Test Execution with Jenkins

When automating tests we often had the conflict that on the one hand we wanted run the tests as often as possible but they also take a lot of time. That’s why we want to run a few selected tests after every change and the complete test suite on a schedule like nightly for example.As a consequence, we decided to add Jenkins with Jenkins Pipelines in our tech stack for continuous integration. One of its big advantages is the possibility to parameterize the pipelines and add timetables in the Jenkins file.If needed we also have our own projects for very specific test runs e.g. nightly tests in Firefox.Another advantage of using Jenkins is the loose coupling between product build and test execution. This is made possible as the Jenkinsfile is stored in every Git on every branch. Also, the parameterization ensures an easy coupling of the URL, like staging-123.test.local for example. That way we can couple the build (including deployment of the software product with the test execution.Talking about the background and advantages of this tool selection we can say that test development of unit tests belongs to every feature. We all know that E2E test development can lag behind or cover less, due to time or costs. The loose coupling of Selenium tests and product allows for this lag to exist and to separate the development efforts if needed.Compared to unit tests, E2E tests are long-running tests which need several platforms and may have failures because of the development lagging behind or platform compatibility issues. Therefore, we need to be able to continue developing while the test is still running.This can be achieved through pipeline stages or triggering, with the additional benefit that the configuration is stored in the Jenkinsfile and can be adjusted as needed together with the source code. That way we have everything together with the source code and don’t need different tools for the project configuration.

Test Result Evaluation Tool in our Tech Stack: reportportal.io

We usually have a lot of Selenium test cases and platforms to test on and therefore it is very hard to maintain an overview when it comes to evaluating the results. Let me show you this with an example:Imagine a web portal test with 200 test cases from 10 feature groups and you want to test it on all common desktop browsers (10 test systems respectively browsers). This means a 200×10 matrix with 2000 test results per test environment.Let’s say, in this test, you have 5 feature branches plus a staging branche. That makes 6 branches/ environments to test.In the end, we come to a total of 12.000 test results every night. An within those 12.000 results per night 100 errors occurred.Of course, you should have a look at all of those errors but that shouldn’t take forever. So, we thought it might help a lot if we would get those errors structured and analyzed with an automated evaluation.As you can imagine, the standard Jenkins plugins for JUnit or Cucumber is not the right tool for that number of tests and data. So, we decided to find another tool to evaluate all test runs and perform statistical correlation.Our main problem are recurring errors: You don’t have to take a look at them every time again. As it is always the same, once is enough. Additionally you need to classify the bugs – is it the product, the test or is the infrastructure unstable? While screening different tools we stumbled upon reportportal.io which offers data analysis based on machine learning. That way, tags, user feedback, and more data feed the AI for upcoming analysis. So we decided to add it to our technology stack for automated Selenium tests.Another very important feature is to be able to send additional data to the tool which let us not only analyse the test result with succesful/ unsuccessful but also add screenshots, log-info etc. and not only the junit-results.xml. Additionally, by connecting to the report system from the test code to the test code, it was possible to embed test or system specific data. Further advantages are the easy integration and dashboards with neat UI.

Testbirds’ Tech Stack: Overview

To give you an overview of our tech stack we use for test automation, we created this diagram. It shows the challenges and thoughts behind the selection of our software as well as the used tools for each step during automating tests.If you want to learn more, take a look in our Github repository about the Selenium Tech Stack. As everything in it is open source, we are more than happy to offer it to you as a template and hope you can adapt it to your needs.Curious to learn more about test automation and our Device Cloud?