Why You Should Consider Postman API Automation and HTML Report Generation (Allure & Newman) with TestRail Integration

9 min readMay 27, 2023

In today’s rapidly evolving digital landscape, Application Programming Interfaces (APIs) play a crucial role in connecting different software systems and enabling seamless communication. As a result, the need for efficient API testing and automation has become paramount for software testers and quality assurance professionals. While there are various API automation frameworks available, Postman stands out as a popular choice among testers due to its versatility and user-friendly interface.

This article aims to explore the benefits and advantages of utilizing Postman’s capabilities to automate API testing, generate comprehensive HTML reports, seamlessly integrate with TestRail for effective test management and Continues Integration (CI) to automate the process of building, testing, and deploying applications.

One significant advantage of leveraging Postman for API automation is its familiarity with testers. Many testers already possess their own Postman API collections for manual testing. By converting these collections into automation scripts, testers can significantly enhance their automation efficiency while saving valuable time. This seamless transition from manual to automated testing empowers testers to leverage their existing knowledge and resources, ensuring a smooth and efficient testing process.

As mentioned above the integration of the Postman API automation suite with CI/CD pipelines brings numerous benefits. By incorporating Postman API automation into the pipeline, teams can ensure that API tests are executed consistently and automatically whenever code changes are made. This integration enhances the overall software development process by catching bugs and issues early on, allowing for faster feedback loops and more reliable releases.

In addition to CI integration, Allure reports and Newman reports are powerful reporting frameworks that can be integrated with the proposed automation workflow. Allure reports provide visually appealing and interactive reports, making it easier to analyze test results, track progress, and identify areas for improvement. Newman HTML reports, on the other hand, offer a convenient way to generate detailed HTML-based reports that can be easily shared and accessed by stakeholders.

Another significant advantage of using Postman for API automation is its ability to facilitate scheduled test executions. Postman has this feature inbuilt as well. But I will be discussing the scheduling with CI integration so that TestRail also becomes a part of the flow. TestRail allows teams to manage and organize test cases, track test execution progress, and generate comprehensive reports. By integrating Postman with TestRail, testers can seamlessly link their automated API tests to corresponding test cases, ensuring complete traceability and visibility throughout the testing lifecycle.

In summary, Postman provides the following benefits.

Simplified test case creation: Postman makes it easy to create test cases for your APIs.
Enhanced collaboration: Postman facilitates seamless collaboration among team members.
Integration with CI/CD: Postman supports integration with continuous integration and continuous deployment (CI/CD) pipelines.
Cost savings: Using Postman can lead to cost savings for your API development and testing processes.
Free and user-friendly: Postman provides a free version that is easy to get started with.
Comprehensive API and schema support: Postman offers extensive support for various APIs and schemas.
Extensibility: Postman can be extended to meet specific requirements through its flexible architecture.
Support and community: Postman has a strong support system and a thriving community that can assist you in your API-related endeavours.

Steps breakdown:

Step 01: Design the API test matrix
Step 02: Create testcases in TestRail
Step 03: Create a Postman collection with several requests
Step 04: Set the Postman Environment and define variables
Step 05: Write Test Scripts
Step 06: Configure Newman and the required packages
Step 07: TestRail Preparation
Step 08: Run the Postman collection with the CLI reporter
Step 09: Run the Postman collection with the HTML Extra reporter
Step 10: Run the Postman collection with the Allure reporter
Step 11: Run the Postman collection with the TestRail reporter
Step 12: Run the Postman collection with multiple reporters
Step 13: CI/CD Integration

Step 01: Design the API test matrix

As the first step, you need to design a test matrix for the API that needs to be automated. A test matrix is not necessary and whatever test design technique can be applied for this but make sure to cover both positive and negative scenarios.

Simple test matrix design of a GET request

Step 02: Create testcases in TestRail

Once the test design is completed and reviewed with the team, you can create test cases in TestRail for the scenarios. At the same time, you can create a test run by including the created testcases.

Step 03: Create a Postman collection with several requests

I have designed a collection consisting of two requests: Authentication and GetUserInformation. The execution order of these requests is crucial, as the GetUserInformation request relies on the successful completion of the Authentication request. To ensure this sequence, I have arranged the requests in the following order. Whenever the collection is executed, the Authentication request will be triggered first, followed by the remaining requests. I adhered to the test design mentioned above while creating the requests in order to encompass all possible scenarios.

Step 04: Set the Postman Environment and define variables

It is often necessary to conduct testing in multiple environments, making it beneficial to utilize separate environment files for managing environment variables across these environments. These files can store essential information such as URLs and user credentials. Additionally, for the purpose of test scripting, I have stored test data within the same environment file, including JSON schemas of the responses and authentication tokens for negative scenarios.

Step 05: Write Test Scripts

This is the step in which you are doing the assertion scripting of the negative and positive scenarios. The authentication request falls outside the testing scope and serves solely to obtain the token, which is then stored as an environment variable. The AUTH_TOKEN variable will be utilized for authenticating the endpoints within the testing scope.

Other requests are in the testing scope and testcases are also created on TestRail. So it is needed to map the TestRail testcase IDs in our scripting. The constant variable tc_id is storing the TestRail testcase ID and it should be the prefix of the Postman test name if you need to send execution results to TestRail.

Schema validation is an effective method for ensuring that the received response aligns with the expected format. To generate the schemas for validating JSON responses, you can use the online tool available at https://techbrij.com/brijpad/json. The JSON schemas are stored as environment variables so they can be accessed within the test scripts.

Whether you have multiple testcase IDs within the same Postman script or if you reuse the same testcase ID multiple times in a Postman script, it makes no difference. Ultimately, the status of each test will be recorded and reported to the TestRail run. Additionally, it is possible to have multiple testcase IDs in the same test as well.

Multiple Testcases belong to the same Postman test

Multiple Postman tests belong to the same test case

Multiple TestRail test cases belong to a single Postman test

Step 06: Configure Newman and the required packages

Please ensure that Node.js is installed on your machine by running the command “node -v” in the Command Prompt to verify the installation. Once you have confirmed Node.js is installed, proceed to execute the following command.

npm install -g newman newman-reporter-htmlextra newman-reporter-testrail newman-reporter-allure allure-commandline

Step 07: TestRail Preparation

To send the execution results to TestRail you should define the following environment variables in your shell. The following example shows how to do it on windows powershell.

$env:TESTRAIL_RUNID="<RUNID>"
$env:TESTRAIL_APIKEY="<APIKEY>"
$env:TESTRAIL_PROJECTID="<PROJECTID>"
$env:TESTRAIL_USERNAME="<USERNAME>"
$env:TESTRAIL_DOMAIN="<DOMAIN>"

Step 08: Run the Postman collection with the CLI reporter

To execute the test from the collection file, utilize the appropriate command line syntax based on your requirements. Always ensure that the environment file is exported and saved in the directory.

Run from the local collection (exported JSON file)

newman run <COLLECTION_FILE_NAME> --environment <ENVIRONMENT_FILE_NAME> 
--reporters <REPORTER_NAME>

Run from the shared collection URL

newman run <COLLECTION_URL> --environment <ENVIRONMENT_FILE_NAME>
--reporters <REPORTER_NAME>

Run Postman collection with URL and CLI reporter

newman run "https://api.postman.com/collections/<collection_id>?access_key=<your_api_key>" --environment '.\QA Environment.postman_environment.json' --reporters "cli"

At the end of execution, you will see a summary of the execution on CLI as below.

Step 09: Run the Postman collection with the HTML Extra reporter

Run Postman collection with URL and HTMLExtra reporter

newman run "https://api.postman.com/collections/<collection_id>?access_key=<your_api_key>" --environment '.\QA Environment.postman_environment.json' --reporters "htmlextra"

Upon completion of the execution, a directory named “newman” will be generated, containing a solitary HTML file. This file presents diverse views that facilitate the analysis of your test execution.

Step 10: Run the Postman collection with the Allure reporter

Run Postman collection with URL and Allure reporter

newman run "https://api.postman.com/collections/<collection_id>?access_key=<your_api_key>" --environment '.\QA Environment.postman_environment.json' --reporters "allure"

Upon completion of the execution, a directory named “allure-results” will be generated, containing a set of files which helps to generate the Allure HTML report. Now run the following command to generate the Allure HTML report.

allure generate --clean
allure serve

It will generate and open the HTML file on the browser within a few seconds.

Step 11: Run the Postman collection with the TestRail reporter

Run Postman collection with URL and TestRail reporter

newman run "https://api.postman.com/collections/<collection_id>?access_key=<your_api_key>" --environment '.\QA Environment.postman_environment.json' --reporters "testrail"

Once the execution is completed, it will show the direct URL to the TestRail test run which is updated by the collection execution.

Step 12: Run the Postman collection with multiple reports

Add multiple reporters to the command line syntax to execute the Postman collection with multiple reporters.

newman run "https://api.postman.com/collections/<collection_id>?access_key=<your_api_key>" --environment '.\QA Environment.postman_environment.json' --reporters "cli,htmlextra,testrail,allure"

Step 13: CI/CD Integration

To achieve the desired outcome, follow the aforementioned procedures within any CI environment to obtain the final outcomes. It is crucial to ensure that both the Postman collection and environment file are available prior to execution. Presented below is a YAML file that illustrates the integration with GitLab CI.

stages:
  - test

variables:
  COLLECTION_ID: "<collection_id>"
  API_KEY: "<postman_api_key>"
  TESTRAIL_RUNID: "<run_id>"
  TESTRAIL_APIKEY: "<testrail_api_key>"
  TESTRAIL_PROJECTID: "<testrail_project_key>"
  TESTRAIL_USERNAME: "<testrail_username_or_email>"
  TESTRAIL_DOMAIN: "<testrail_instance_domain_without_https>"

run_tests:
  stage: test
  image: node:latest
  before_script:
    - npm install -g newman newman-reporter-htmlextra newman-reporter-testrail newman-reporter-allure allure-commandline
  script:
    - newman run "https://api.postman.com/collections/$COLLECTION_ID?access_key=$API_KEY" --environment '.\QA Environment.postman_environment.json' --reporters "cli,htmlextra,testrail,allure"
    - allure generate --clean
  artifacts:
    paths:
      - newman/
      - allure-results/
      - allure-report/

Conclusion

In conclusion, incorporating Postman API automation, HTML report generation with Allure and Newman, and TestRail integration brings significant benefits to software testers. Postman’s user-friendly interface and familiarity streamline the transition from manual to automated testing, improving efficiency and saving time. Integration with CI/CD pipelines ensures consistent and automatic API testing, catching bugs early and enabling faster feedback loops. Allure and Newman reports provide visual and interactive analysis, while TestRail integration ensures traceability and visibility. Overall, adopting these tools enhances testing processes and contributes to high-quality software development.