spryker/robotframework-suite-tests

Automated tests for the Robot Framework

dev-master 2024-04-29 09:43 UTC

MIT

README

This repository contains sets of API and UI tests, built on the Robot Framework. API tests use the RequestsLibrary in conjunction with Robot Framework, while UI tests rely on the Browser library (powered by Playwright).

Installation

Prerequisites

Robot Framework is implemented using Python, and a precondition to install it is having Python or its alternative implementation PyPy installed. Another recommended precondition is having the pip package manager available. Robot Framework requires Python 3.6 or newer.

1. Install Robot Framework

python3 -m pip install -U robotframework

1. Install RequestsLibrary

python3 -m pip install -U robotframework-requests

1. Install DatabaseLibrary

python3 -m pip install -U robotframework-databaselibrary

1. Install Python SQL library, depending on your configuration
   • Engine: MySQL

    python3 -m pip install PyMySQL

   • Engine: PostgreSQL

   python3 -m pip install psycopg2-binary

Installation for UI tests

For UI testing installation requires Robot Framework, RequestsLibrary, DatabaseLibrary and Browser library powered by Playwright.

If you installed everything from the prerequisites, all you need to install is Node.js and the Browser library.

1. Install Node.js®
2. Install Browser library

python3 -m pip install -U robotframework-browser

1. Initialize the Browser library

rfbrowser init

Installation for API tests

For API testing installation requires Robot Framework, RequestsLibrary, JSONLibrary and DatabaseLibrary.

If you installed everything from the prerequisites, all you need to install is the JSONLibrary.

1. Install JSONLibrary

python3 -m pip install -U robotframework-jsonlibrary

Automated installation

You can also run all of the installation steps in one go by executing the shell script install.sh.

How to run tests

Robot Framework test cases are executed from the command line, and the end result is, by default, an output file in XML format and an HTML report and log. After the execution, output files can be combined and otherwise post-processed with the Rebot tool.

Note: If you prefer to run test using the default configuration of your local environment, you can navigate to the Helper section

Synopsis

robot [options] data
python -m robot [options] data
python path/to/robot/ [options] data

Execution is normally started using the robot command created as part of installation. Alternatively it is possible to execute the installed robot module using the selected Python interpreter. This is especially convenient if Robot Framework has been installed under multiple Python versions. Finally, if you know where the installed robot directory exists, it can be executed using Python as well.

Regardless of execution approach, the path (or paths) to the test data to be executed is given as an argument after the command. Additionally, different command line options can be used to alter the test execution or generated outputs in many ways.

Basic usage example: robot -v env:{ENVIRONMENT} {PATH}

Supported CLI Parameters

Parameter	Comment	Example	Required
-v env:{env}	Environment variable. Demo data, locators and hosts in tests depend on this variable value. It's crucial to pass the env variable as tests fully depend on it. Supported parameters are: api_b2b, api_b2c, api_mp_b2b, api_mp_b2c, api_suite, ui_b2b, ui_b2c, ui_mp_b2b, ui_mp_b2c, ui_suite	robot -v env:api_b2b -s '*'.tests.api.b2b.glue . / robot -v env:ui_b2c tests/ui/e2e/b2c.robot	yes
-v db_engine:{engine}	Depending on your system setup, you can run tests against MySQL or PostgreSQL. Possible values: mysql or postgresql. Default: mysql	robot -v env:api_b2b -v db_engine:postgresql -s '*'.tests.api.b2b.glue .	optional
-v db_host:{host}	Depending on your system setup, you can specify db_host if it differs from the default one. Default: 127.0.0.1	robot -v env:api_b2b -v db_host:127.2.3.4 -s '*'.tests.api.b2b.glue .	optional
-v db_port:{port}	Depending on your system setup, you can specify db_port if it differs from the default one. Default MariaDB: 3306 / Default PostgreSQL: 5432	robot -v env:api_b2b -v db_port:3390 -s '*'.tests.api.b2b.glue .	optional
-v db_user:{user}	Depending on your system setup, you can specify db_user if it differs from the default one. Default: spryker	robot -v env:api_b2b -v db_user:fake_user -s '*'.tests.api.b2b.glue .	optional
-v db_password:{pwd}	Depending on your system setup, you can specify db_password if it differs from the default one. Default: secret	robot -v env:api_b2b -v db_password:fake_password -s '*'.tests.api.b2b.glue .	optional
-v db_name:{name}	Depending on your system setup, you can specify db_name if it differs from the default one. Default: eu-docker	robot -v env:api_b2b -v db_name:fake_name -s '*'.tests.api.b2b.glue .	optional
-d {PATH}	Path to the folder to store the run report. Like results	robot -v env:api_b2b -d results tests/.../example.robot	optional
-s test_suite_name	Test suite name is the name of any subfolder in tests folder (without path) or filename (without extension). If specified, tests from that folder/file folder will be executed.	robot -v env:api_b2b -s '*'.tests.api.b2b.glue . / robot -v env:api_mp_b2b -s '*'.tests.api.mp_b2b.glue .	optional
-v yves_env:{URL}	You can specify Yves URL if you would like to run your tests on cloud environment	robot -v env:ui_b2c -v yves_env:http://example.com tests/ui/e2e/b2c.robot	optional
-v yves_at_env:{URL}	You can specify Yves AT store URL if you would like to run your tests on cloud environment	robot -v env:ui_b2c -v yves_env:http://example.com -v yves_at_env:http://at.example.com tests/ui/e2e/b2c.robot	optional
-v zed_env:{URL}	You can specify Zed URL if you would like to run your tests on cloud environment	robot -v env:ui_b2c -v yves_env:http://example.com -v zed_env:http://bo.example.com tests/ui/e2e/b2c.robot	optional
-v glue_env:{URL}	You can specify Glue URL if you would like to run your tests on cloud environment	robot -v env:api_b2c -v glue_env:http://glue.example.com -s '*'.tests.api.b2c.glue .	optional
-v bapi_env:{URL}	You can specify BAPI URL if you would like to run your tests on cloud environment	robot -v env:api_b2c -v glue_env:http://glue.example.com -v bapi_env:http://bapi.example.com -s '*'.tests.api.b2c.bapi .	optional
-v sapi_env:{URL}	You can specify SAPI URL if you would like to run your tests on cloud environment	robot -v env:api_b2c -v glue_env:http://glue.example.com -v sapi_env:http://sapi.example.com -s '*'.tests.api.b2c.sapi .	optional
-v mp_env:{URL}	You can specify Merchant Portal URL if you would like to run your tests on cloud environment	robot -v env:ui_mp_b2c -v yves_env:http://example.com -v zed_env:http://bo.example.com -v mp_env:http://mp.example.com tests/ui/e2e/mp_b2c.robot	optional
-v browser:{browser}	Defines in which browser run tests. Possible values: chromium,firefox, webkit. Default: chromium	robot -v env:ui_mp_b2c -v browser:firefox tests/ui/e2e/mp_b2c.robot	For UI tests only. optional
-v headless:{headless}	Defines if the browser should be launched in the headless mode. Possible values: true,false. Default: true	robot -v env:ui_mp_b2c -v headless:false tests/ui/e2e/mp_b2c.robot	For UI tests only. optional
-v browser_timeout:{timeout}	Default time for Implicit wait in UI tests. Default: 60s	robot -v env:ui_mp_b2c -v browser_timeout:30s tests/ui/e2e/mp_b2c.robot	For UI tests only. optional
-v api_timeout:${timeout}	Default time for Implicit wait of the response in API tests. Default: 60s	robot -v env:api_b2c -v api_timeout:30s -s '*'.tests.api.b2c.glue .	For API tests only. optional
-v verify_ssl:bool	Enables/Disables SSL verification in API and UI tests. Default: false	robot -v env:api_b2c -v verify_ssl:true -s '*'.tests.api.b2c.glue .	optional
-v docker:bool	Enables/Disables Docker compatibility. Default: false	robot -v env:api_b2c -v docker:True -s '*'.tests.api.b2c.glue .	yes for running tests via docker/sdk
-v ignore_console:bool	Enables/Disables docker/sdk commands during test execution, such as 'trigger oms' or 'trigger p&s'. Default: true, meaning console commands are not executed	robot -v env:ui_b2c -v ignore_console:False tests/ui/e2e/mp_b2c.robot	optional. If -v docker:true - this parameter is set to False by default, meaning docker/sdk commands are executable
-v project_location:${absolute_path}	Absolute project/application root folder location for docker/sdk cli commands	robot -v env:ui_b2c -v ignore_console:True -v project_location:/Users/your_user/projects/b2c tests/ui/e2e/mp_b2c.robot	optional. If -v docker:true - this parameter is ignored
{PATH}	Path to the file to execute	robot -v env:api_b2b tests/api/b2b/glue/cart_endpoints/carts/positive.robot / robot -v env:ui_b2c tests/ui/e2e/b2c.robot where tests/ui/e2e/b2c.robot and tests/api/b2b/glue/cart_endpoints/carts/positive.robot is a {PATH}	yes for UI tests
-v dms:bool	Enables/Disables Dynamic Multistore tests. Indicating that all tests are run on the regular shop by default. If you need to run all tests on the dynamic multistore, you should set True value. Default: false	robot -v env:ui_b2c -v dms:true tests/ui/e2e/b2c.robot	optional. If -v dms:false - this parameter is ignored

CLI Examples

• Execute all tests (positive and negative) in api/suite folder (all glue, bapi and sapi API tests that exist) via docker/sdk.

  docker/sdk exec robot-framework robot -v docker:True -v env:api_suite -d results -s '*'.tests.api.suite .

• Execute all tests in api/b2b folder (all glue, bapi and sapi API tests that exist).

  robot -v env:api_b2b -d results -s '*'.tests.api.b2b .

• Execute all tests in a specific folder (all API tests that exist inside the folder and sub-folders).

  robot -v env:api_b2b -d results -s '*'.tests.api.b2b.glue.access_token_endpoints .

• Execute only positive tests in api folder (all positive API tests that exist, from all folders).

  robot -v env:api_suite -d results -s positive .

• Execute all positive and negative API tests in tests/api/suite/glue/abstract_product_endpoints folder. Subfolders (other endpoints) will be executed as well.

  robot -v env:api_suite -d results -s '*'.tests.api.suite.glue.abstract_product_endpoints .

• Execute all positive and negative API tests in tests/api/suite/glue/abstract_product_endpoints/abstract_products

  robot -v env:api_suite -d results -s '*'.tests.api.suite.glue.abstract_product_endpoints.abstract_products .

• Execute all E2E UI tests for MP-B2B on specific cloud environment.

  robot -v env:ui_mp_b2b -v yves_env:http://yves.example.com -v zed_env:http://zed.example.com -v mp_env:http://mp.example.com -d results tests/ui/e2e/mp_b2b.robot

• Execute all API tests for B2B on specific cloud environment with custom DB configuration.

  robot -v env:api_b2b -v db_engine:postgresql -v db_host:124.1.2.3 -v db_port:5336 -v db_user:fake_user -v db_password:fake_password -v db_name:fake_name -s '*'.tests.api.b2b.glue .

Supported Browsers in UI tests

Since Playwright comes with a pack of builtin binaries for all browsers, no additional drivers e.g. geckodriver are needed.

All these browsers that cover more than 85% of the world wide used browsers, can be tested on Windows, Linux and MacOS. Theres is not need for dedicated machines anymore.

Browser	Browser with this engine
chromium	Google Chrome, Microsoft Edge (since 2020), Opera
firefox	Mozilla Firefox
webkit	Apple Safari, Mail, AppStore on MacOS and iOS

Helper

For local testing, all tests are commonly executed against default hosts. To avoid typos in execution commands, you can use the Makefile helper to quickly start your runs. Note: no installation is required on macOS and Linux systems. The make command is included in most Linux distributions by default. To run Makefile on Windows, you need to install a program called "make".

Supported Helper commands

Command	Comment	Optional arguments
make test_api_b2b	Run all API tests for B2B on default local environment	glue_env= / bapi_env= / sapi_env= / docker= / ignore_console= / project_location=
make test_api_b2c	Run all API tests for B2C on default local environment	glue_env= / bapi_env= / sapi_env= / docker= / ignore_console= / project_location=
make test_api_mp_b2b	Run all API tests for MP-B2B on default local environment	glue_env= / bapi_env= / sapi_env= / docker= / ignore_console= / project_location=
make test_api_mp_b2c	Run all API tests for MP-B2C on default local environment	glue_env= / bapi_env= / sapi_env= / docker= / ignore_console= / project_location=
make test_api_suite	Run all API tests for Suite on default local environment	glue_env= / bapi_env= / sapi_env= / docker= / ignore_console= / project_location=
make test_ui_suite	Run all UI tests for Suite on default local environment	glue_env= / bapi_env= / yves_env= / yves_at_env= / zed_env= / mp_env=/ docker= / ignore_console= / project_location=
make test_ui_b2b	Run all UI tests for B2B on default local environment	glue_env= / bapi_env= / yves_env= / yves_at_env= / zed_env= / mp_env= / docker= / ignore_console= / project_location=
make test_ui_b2c	Run all UI tests for B2C on default local environment	glue_env= / bapi_env= / yves_env= / yves_at_env= / zed_env= / mp_env= / docker= / ignore_console= / project_location=
make test_ui_mp_b2b	Run all UI tests for MP-B2B on default local environment	glue_env= / bapi_env= / yves_env= / yves_at_env= / zed_env= / mp_env= / docker= / ignore_console= / project_location=
make test_ui_mp_b2c	Run all UI tests for MP-B2C on default local environment	glue_env= / bapi_env= / yves_env= / yves_at_env= / zed_env= / mp_env= / docker= / ignore_console= / project_location=

Helper Examples

• Run all API tests for B2B on local environment

  make test_api_b2b

• Run all UI tests for MP-B2C on local environment with disabled docker/sdk commands

  make test_ui_mp_b2c ignore_console=true

• Run all UI tests for MP-B2C on local environment with enabled docker/sdk commands and specify your application location

  make test_ui_mp_b2c ignore_console=false project_location=/Users/your_user/projects/mp-b2b

• Run all API tests for B2B on cloud environment

  make test_api_b2c glue_env=http://glue.example.com bapi_env=http://bapi.example.com sapi_env=http://sapi.example.com

Built-in libraries

Name	Description	Keywords documentation
Builtin	Provides a set of often needed generic keywords. Always automatically available without imports.	Documentation
Collections	Provides a set of keywords for handling Python lists and dictionaries.	Documentation
DateTime	Library for date and time conversions.	Documentation
Dialogs	Provides means for pausing the execution and getting input from users.	Documentation
OperatingSystem	Enables various operating system related tasks to be performed in the system where Robot Framework is running.	Documentation
Process	Library for running processes in the system.	Documentation
Screenshot	Provides keywords to capture screenshots of the desktop.	Documentation
String	Library for generating, modifying and verifying strings.	Documentation

External libraries that can be installed based on your needs

The full list can be found on the official website

Name	Description	Keywords documentation
Browser	A modern web testing library powered by Playwright. Aiming for speed, reliability and visibility. Note: Already installed	Documentation
RequestsLibrary	Library for sending and API requests and receiving the responses.	Documentation
JSONLibrary	Library for parsing JSONs that come in API responses.	Documentation

Automatically re-executing failed tests

There is often a need to re-execute a subset of tests, for example, after fixing a bug in the system under test or in the tests themselves. This can be accomplished by selecting test cases by names (--test and --suite options), tags (--include and --exclude), or by previous status (--rerunfailed or --rerunfailedsuites).

Combining re-execution results with the original results using the default combining outputs approach does not work too well. The main problem is that you get separate test suites and possibly already fixed failures are also shown. In this situation it is better to use --merge (-R) option to tell Rebot to merge the results instead. In practice this means that tests from the latter test runs replace tests in the original.

Command	Description
robot -v env:ui_suite -d results tests/ui/e2e/suite.robot	first execute all tests
robot -v env:ui_suite -d results/rerun --runemptysuite --rerunfailed results/output.xml --output rerun.xml tests/ui/e2e/suite.robot	then re-execute failing
rebot -d results --merge results/output.xml results/rerun/rerun.xml	finally merge results
robot -v env:ui_suite -d results tests/ui/e2e/suite.robot; robot -v env:ui_suite -d results/rerun --runemptysuite --rerunfailed results/output.xml --output rerun.xml tests/ui/e2e/suite.robot; rebot -d results --merge results/output.xml results/rerun/rerun.xml	All in one

The message of the merged tests contains a note that results have been replaced. The message also shows the old status and message of the test.

Merged results must always have same top-level test suite. Tests and suites in merged outputs that are not found from the original output are added into the resulting output.

Viewing and Generating Keyword Documentation

Keywords used in the tests can and should be documented. If you add any new keyword into the files inside the 'common' folder, they should have [Documentation] tag in them that describes what the keyword does, what parameters mean and gives an example of the usage.

Documentation can be generated by these tags. For now only common_api.robot has documentation generated. If you added new keywords, you should re-generate the documentation and commit it together with the other changes you made.

To generate the documentation for api, bapi and sapi tests use this command: libdoc resources/common/common_api.robot API_Keyword_Documentation.html

To view the documentation just open the generated html file in any browser.

Output files

Several output files are created when tests are executed, and all of them are somehow related to test results.

Log files contain details about the executed test cases in HTML format. They have a hierarchical structure showing test suite, test case and keyword details. Log files are needed nearly every time when test results are to be investigated in detail. Even though log files also have statistics, reports are better for getting an higher-level overview.

The command line option --log (-l) determines where log files are created. Unless the special value NONE is used, log files are always created and their default name is log.html.