Automated Testing

While Arcadier offers flexibility to customise through APIs and plug-ins, to ensure your customisations run smoothly through every release, you will have to include a test script together with your plug-ins to run through automated testing.

This will ensure that your custom plug-ins are testing for any issues before release to prevent any unwelcome surprises from happening

After 1st January 2021, Arcadier will no longer support custom plug-ins that do not come with automated testing

How do I do automated testing for my plug-ins?

Arcadier uses Katalon to do its automated testing for the plug-ins What we require from you as the plug-in developer is to develop Katalon test scripts that Arcadier will run before every release.

If there is an issue with the plug-in the developers will be notified so that they can attend to the issue immediately

In this document, we will detail how you can create a Katalon test script for your Arcadier plug-in

What is Katalon?

Katalon Studio is a free-license tool released in January 2015 with a Selenium-based engine. Mostly, Katalon is designed to create and reuse automated test scripts for UI without coding. Katalon Studio allows running automated tests of UI elements, including pop-ups, iFrames, and wait-time. The tool can be launched on Microsoft Windows, macOS, and Linux.

Why Katalon?

Katalon's main advantage is that it’s easy to deploy and has a wider set of integrations compared to Selenium, the market leader. Katalon has dual scripting interfaces for users with different programming skills. This means that testers with limited technical knowledge can use a simpler user interface that doesn’t require writing code.

Katalon Studio supports local and remote testing, as well as parallel and sequential executions. It runs on Groovy (Java) scripting language.

This solution has a quick setup and a number of pre-installed templates that allow repeating some testing patterns. Katalon Studio is a cross-browser tool that supports Web, mobile, and API testing. These solutions go with analytics and recording modules.


Read more about Katalon at these links:

To learn more about Katalon set up test script creation, please refer to Training-Material Katalon V1.1.pdf

Steps on creating a test script for your Plug-in?

  1. Decide what Test Cases to Automate
  2. Creating Test Cases Guide with Examples
  3. Create a Test Script (optional)
    1. Example is a simple test script from a sample plug-in
  4. Create Test Cases in Katalon
    1. Refer to QA Katalon guide
    2. Profiles
    3. Test Cases
    4. UI Test Cases
    5. Test Suites
    6. Test Suite Execution

1. Decide what Test Cases to Automate

It is not mandatory to automate All the testing; a plug-in may serve a lot of functionalities so it is important to determine which test cases should be automated first.

Test cases that should be automated:

  • Tests that tend to cause human error

  • Tests that require multiple data sets

  • Frequently used functions that introduces high-risk conditions

  • Tests that take a lot of time and effort if done manually

Automated testing requires careful planning and design work. Start out by creating manual test cases to identify the initial set of tests to automate and to serve as a guide for future tests.

2. Decide what Test Cases to Automate

A test case has components that describe input, action and an expected response in order to determine if a feature of an application is working correctly.

A test case is a set of instructions on "HOW" to validate a particular test objective/target, which when followed will tell us if the expected behavior of the system is satisfied or not.

Below we will provide a template on how to create your own test cases and how to use it

You can download our Sample Test Case Template here:

  • Developer – developer's name, unique code or team involved in the project
  • Plug-in name – complete plug in name preferably with version
  • Plug in description – specific description of the plug in. Brief significance e.g. (what it does, specific client to whom it was built for)
  • Status – current testing status of the plug in, (In Progress, For Revision, For Validation, Done) 
  • Page – specific page to be tested (admin page, user’s page, check out page)
  • Test Scenario – specific scenario to be tested, from the example below, our test scenario is to ‘verify if the plug-in is installed
  • Test Case Class – test case classification
  • UI – for verifying UI components
  • Simple – for verifying simple test scenarios which involves buttons clicks or navigation
  • Complex – verifying complex test case scenarios mostly if there are several affected pages
  • Actions – the pre requisite actions to take before proceeding to actual test page. From our example above, the have Admin Login and Plugin Installation before we can navigate to actual plugin page
  • Test Case ID – specific ID for each test case which will be used on creating automated test scripts
  • Test Case – variable name used in test scripts to identify the specific object to be tested
  • Test Data – variable name used in test scripts to identify the specific object to be tested
  • Objective – the main objective of the test case scenario Expected Result - specific expected result of the objective
  • Actual Result – test result which can be Passed or Failed if the test scenario did not meet expected results

3. Creating a simple test script from a Sample Plug-in

The figure below shows the front-end of a sample plug-in in the admin dashboard. The highlighted elements in red are those whose presence is required to be displayed.

  1. Plug-in description
  2. Plug-in context
  3. Link of plug-in documentation
  4. Main button of add functionality
  5. Data table / Table headers
  6. Data table / Table headers

You will have to create test cases classified as ‘UI’s’ to verify the presence of each. (Please refer to sample excel file PROF_INPUT.xls)

Once you create several simple test cases, you can group your tests into one, larger automated test. You can organize automated tests by the application’s functional area, major/minor division in the application, common functions or a base set of test data. If an automated test refers to other tests, you may need to create a test tree, where you can run tests in a specific order

4. Creating Test Cases in Katalon

If you are unfamiliar to Katalon, please refer to our Katalon Training Manual 1.1

After creating a new Project in Katalon, please be reminded that the plug-in could be tested on different environments, e.g. (Dev server, test, sandbox, staging) or mostly depends on the available environment for testing. With this, we have to create different Profiles.

- Profiles

Aside from the default, SANDBOX, STAGING and TEST were also set so we can switch and execute test suites easily.

For more information about Execution Profiles, you can refer to

- Test Cases

It is advisable to group and sort test cases by Pages (admin, user) and place repetitive test cases (log in, page navigations) under Utilities.

Inside the 'Test Cases' directory, test cases are classified to UI, Simple and Complex.

- UI Test Cases

From our sample test cases, (refer to sample excel file PROF_INPUT.xls) TC_UI_001 -007 focuses mostly on creating UI test cases.

Under the 'Plugin Header' directory, the first seven (7) UI test cases were grouped together in sequence.

Let's take the TC_UI_001 as the first example.

  • Test Case: 'Verify if plug-in icon is displayed on upper left section'

    After setting the element on the Object Repository (Please refer to Training-Material Katalon V1.1.pdf on how to spy elements), we will verify if the header icon is present or not.

    Simple script using 'Verify Element Present'

    Do the same with other test scripts.

- Test Suites

After creating sets of test cases / scripts, we have to compile and run to see results. A test suite is a container that has a set of tests which help in executing and reporting the test execution status. They are categorized in such a way that they match your planning and analysis needs or scenarios.

From the created test cases, let’s create a Test suite, name it Header. We can also group the entire test suite by scenarios, page, and test case classification inside a directory beforehand

Header test suite is inside the Admin Plugin Page directory.

The Header test suite contains the created test cases. "Admin Login" and "Navigate to the actual plugin" were created inside the 'Utilities' directory.

- Test Suite Execution

Run the test suite with the desired Browser (Chrome is best recommended) and select Profile Execution of the environment

Execution Result