Building a Test Automation Framework using Cypress.io — Reporting (Part 5)

Olá!

In Part-4 of this article series we added Page Object Model support to our Test Automation Framework. In this article, we will add a key component of an automation framework i.e. adding Test Reports. Vamos começar!

What is a Test Report?^[1]

Test Report is a document which contains a summary of all test activities and final test results of a testing project. Test report is an assessment of how well the Testing is performed. Based on the test report, stakeholders can evaluate the quality of the tested product and make a decision on the software release.

Key components of a good Test Report:

To achieve it's goals, a good Test Report should contain:

• Detail: You should provide a detailed description of the testing activity, show which testing you have performed. Do not put the abstract information into the report, because the reader will not understand what you said.
• Clear: All information in the test report should be short and clearly understandable.
• Standard: The Test Report should follow the standard template. It is easy for stakeholder to review and ensure the consistency between test reports in many projects.
• Specific: Do not write an essay about the project activity. Describe and summarize the test result specification and focus on the main point.

Adding Reporting to Cypress

Add Dependencies:

In order to add Test Reporting capabilities to our Test Automation Framework, we will be making use of an external library called Mochawesome. Mochawesome is a custom reporter to be used with the Javascript testing framework, Mocha.

So, in order for our reports to work, we would need following dependencies:

• mocha — Simple, flexible, fun JavaScript test framework for Node.js
• mochawesome — Mochawesome is a custom reporter to be used with Javascript testing framework, mocha.
• mochawesome-merge — Merges several Mochawesome JSON reports
• mochawesome-report-generator — It takes the JSON output from Mochawesome and generates a full fledged HTML/CSS report that helps visualize your test suites.

Essentially we wouldn't need last 2 dependencies, but  since the version 3.0.0, Cypress runs every spec(in our case features) separately, which leads to generating multiple Mochawesome reports, one for each spec.

We can add all these packages at once using following command:

npm i mocha mochawesome mochawesome-merge mochawesome-report-generator --save-dev

We will also update Cypress (5.2.0 --> 5.3.0)  [latest update by the time of writing]:

npm update -g cypress     

We also install a package called Minimist (which we'll need later for parsing CLI arguments)

npm install minimist --save-dev

The Code!!

Before we can start writing the necessary code for Mochawesome, we will have to specify with cypress which reporter to use setting up following configuration in cypress.json

{
"testFiles": "**/*.{feature,features}",
"ignoreTestFiles": ["**/stepDefinitions/*","*.js", "*.md"],
"reporter": "mochawesome",
"reporterOptions": {
    "reportFilename": "test-report"
  }
}  

We also would want to have screenshot of the failure after a test gets failed, so for achieving this reusable behavior we are gonna make use of Cypress Support file.

This file runs before every single spec file.

The support file is a great place to put reusable behavior such as custom commands or global overrides that you want applied and available to all of your spec files.

we are going to put our code in Cypress Event called test:after:run, so that attaching screenshot on failure code is run after each spec run.

import './commands'
const moment = require('moment')
const addContext = require('mochawesome/addContext');

// Alternatively you can use CommonJS syntax:
// require('./commands')

beforeEach(function () {
  cy.log('Test run started on : ' + new moment().format('DD-MM-YYYY HH:mm:ss'));
})

//Runs after a test completes
Cypress.on('test:after:run', (test, runnable) => {

  cy.log('Test run ended on : ' + new moment().format('DD-MM-YYYY HH:mm:ss'));

  const spec_title = runnable.parent.title;

  console.log("spec_title :", spec_title);
  console.log("test.state  :", test.state);
  console.log("Cypress.spec.name  :", Cypress.spec.name);
  console.log("test.title  :", test.title);

  if (test.state === 'failed') {
    addContext({ test }, {
      title: 'Failing Screenshot: ' + '>> screenshots/' + Cypress.spec.name + '/' + spec_title + ' -- ' + test.title + ' (failed)' + '.png <<',
      value: 'screenshots/' + Cypress.spec.name + '/' + spec_title + ' -- ' + test.title + ' (failed)' + '.png'
    })
  }
});

Here on the event we check if test.state === 'failed', in that case we are gonna append the failure screenshot from the screenshots folder to our report using Mochawesome addcontext().

Doing this will integrate Mochawesome to our testing framework.

Now if we run,

npx cypress run

We see following .html report has been generated in the directory ./mochawesome-report

Problem with this is we had two feature(spec) files, but we only see details for second spec file, i.e. because mochawesome overwrites the previous report

So to avoid this, we can provide a flag overwrite=false in cypress.json file under reporterOptions attribute.

which results in multiple report files like below:

Then we'll have to take these file and them merge is using mochawesome-merge and then generating a unified report with mochawesome-report-generator, which involves few CLI commands to be run, you can find them on respective npm listing pages.

WAY FORWARD??

We're gonna automate the whole process of joining the individual spec files and then generating a single report out of it programmatically.

We will also make sure that our past test run reports remain available to us.

To achieve this, we will:

• create a runner.js file in parent directory, which uses Cypress Module API
• write the logic to merge all individual reports' output.json
• create a unified report from the merged output.json
• write the logic to create a report directory ./reports and then separate test run folders with test run timestamps like "Test Run - 05-10-2020--13_52_22"

const cypress = require('cypress')
const marge = require('mochawesome-report-generator')
const { merge } = require('mochawesome-merge')
const moment = require('moment')

// get current run timestamp
const currRunTimestamp = getTimeStamp();

//Get cypress CLI options using 'minimist
const args = require('minimist')(process.argv.slice(3));

//source directory where individual test reports are created
const sourceReport = {
    files: ["./reports/" + "Test Run - " + currRunTimestamp + "/mochawesome-report/*.json"],
}

//destination directory where we want our unified .html and .json file to be placed
const finalReport = {
    reportDir: 'reports/' + "Test Run - " + currRunTimestamp,
    saveJson: true,
    reportFilename: 'Run-Report',
    reportTitle: 'Run-Report',
    reportPageTitle: 'Run-Report'
}

//Cypree Module API
cypress.run({   
    ...args,                           
    config: {
        pageLoadTimeout: 10000,
        screenshotsFolder: 'reports/' + "Test Run - " + currRunTimestamp + '/screenshots',
        video: true,
        videosFolder: 'reports/' + "Test Run - " + currRunTimestamp + '/videos'
    },
    reporter: 'mochawesome',
    reporterOptions: {
        reportDir: 'reports/' + "Test Run - " + currRunTimestamp + '/mochawesome-report',
        overwrite: false,
        html: true,
        json: true
    }
}).then(result => {
	
    // generate a unified report, once Cypress test run is done
    generateReport()
    .then(() => {
        console.log("Test Report has been generated");
    })
    .catch(err => {
        console.error("Getting error while generating reports: ", err.message)
    })
    .finally(() => {
        console.log("Test Run Completed at " + result.endedTestsAt);
        console.log("Total time taken " + new Date(result.totalDuration).toISOString().substr(11, 8));
    })
  
    })
.catch(err => {
    console.error(err.message)
    process.exit(1)
  })

//get current timestamp
function getTimeStamp() {
    var now = new moment().format('DD-MM-YYYY--HH_mm_ss')
    return now
}

//generate unified report from sourecReport.files directory and create a unified report and store it in finalReport.reportDir location
function generateReport() {
    return  merge(sourceReport).then(report => {
        marge.create(report, finalReport)
        .then(result => {
            console.log("Run report saved at " + result[0]);
        })
        .catch(err => {
            console.error("Getting error while merging reports: ", err.message)
        })
        .finally(() => {
            process.exit()
        });
    });
    
}

Key Points:

• We are providing Cypress reporter configuration(reporterOptions) from runner.js instead of cypress.json start
• We are using Cypress Module API to run our tests
• As Cypress.run() will be our new entry execution point we'll have to pass all the CLI options to runner.js
• We are using Minimist to recognize the CLI arguments and then pass it to Cypress.run() as runOptions along with reporter options
• generateReport() will merge the different individual run output.json and then generate a single unified report out of it  (it does this by using merge() from mochawesome-merge and marge() function from mochawesome-report-generator

Now that our entry execution point is runner.js; we will have to trigger our test run from here by following run statement:

node runner.js cypress run

//or with arguments,

node runner.js cypress run --env TAGS="@API"

Now if we go to ./reports/Test Run <latest timestamp>, we will see following directory and its content:

Here,

• mochawesome-reports contain individual test results for feature files
• videos/ screenshot contains test run videos and failure screenshots
• Run-Report.json is unified test results output file and Run-Report.html is final unified report

Final report looks like below:

See you next time, when we will see adding some other cool things to our Framework. So keep tuned in.


Tchau, Até logo!

GitHub repo: https://github.com/far11ven/Cypress-TestFramework/tree/develop/Part 05