Playwright test integration with TeamCity

Recently we were migrating our UI tests from Jest to Playwright test as it looked mature enough already and the API is very similar to Jest’s, so it did not require many changes to the tests them-self. And most importantly it includes some neat features baked in already I had to create for Jest on my own otherwise. But at the time of the migration, there was still one crucial integration missing for us - the TeamCity plugin. Because TeamCity is the central CI tool in our project I wanted it to expose as much information as possible to make debugging of failed tests as easy as possible. In TeamCity, each test run can be accompanied by metadata such as a count of passed/failed tests, test duration, console output, screenshots, logs, and so on. Specifically, I wanted the UI test build to display the test status, execution time, playwright traces, and in case of a failure we want to log the console output in TeamCity UI.

To make it work we need Playwright Reporter API to log the right data at right time and to visualize them in TeamCity UI we make use of the TeamCity’s service messages. We start by creating a tcReport.ts file and preparing the class:

import pwConfig from "./playwright.config"

class TCReporter implements Reporter {
    private readonly outputDir

    constructor() {
        this.outputDir = pwConfig.outputDir
    }
}

The pwConfig is your playwright config as we want to take the output directory from there.

As a next, step we create getTestName method, that will parse the test name:

  private getTestName(test: TestCase) {
        const testTitle = test.title
        const describeTitle = test.parent.title
        return `${describeTitle} * ${testTitle}`
    }

Now we can implement the onTestBegin method:

    onTestBegin(test: TestCase) {
        console.log(`##teamcity[testStarted name='${this.getTestName(test)}' captureStandardOutput='false']`)
    }

This piece of code gets triggered at the beginning of each test and will log the test start using TeamCity’s service messages format. Before we process the test result we need to create a method for parsing the errors in case of a test failure.

    private parseError(error) {
        if (error) {
            return stripAnsi(this.escape(error.stack.toString()))
        }
        return ""
    }

Please mind the stripAnsi that needs to be installed by npm i strip-ansi and imported at the very top of our tcReporter.ts file:

import stripAnsi from "strip-ansi"

And here goes the escape method implementation, that escape special characters, so they get properly interpreted by TeamCity.

    private escape(str) {
        // TC escape characters
        if (!str) {
            return ""
        }

        return str
            .toString()
            // eslint-disable-next-line no-control-regex
            .replace(/\x1B.*?m/g, "")
            .replace(/\|/g, "||")
            .replace(/\n/g, "|n")
            .replace(/\r/g, "|r")
            .replace(/\[/g, "|[")
            .replace(/\]/g, "|]")
            .replace(/'/g, "|'")
    }

Before putting it all together we still need to create one more method, that will append the traces to our test artifacts:

   private addTracesMeta(result: TestResult, testName: string) {
          const traces = result.attachments.filter(attachment => attachment.name === "trace")
          if (traces.length > 0) {
            const traceFullPath = traces[0].path
            const outDirCharLength = this.outputDir.length
            const tracePath = traceFullPath.substr(traceFullPath.indexOf(this.outputDir) + outDirCharLength)
            const traceFilePath = `${process.env.TRACE_ARTIFACTS_PATH}${tracePath}`
            console.log(`##teamcity[testMetadata testName='${testName}' type='artifact' value='${traceFilePath}']`)
  }
}

Now we are ready to craft the final onTestEnd method, where we amend all the meta information to a test result.

    onTestEnd(test: TestCase, result: TestResult) {
        const testName = this.getTestName(test)

        if (result.status === "failed") {
            const trace = this.parseError(result.error)
            console.log(`##teamcity[testFailed name='${testName}' details='${trace}']`)
        }
        if (result.status === "skipped") {
            console.log(`##teamcity[testIgnored name='${testName}']`)
        }
        this.addTracesMeta(result, testName)
        console.log(`##teamcity[testFinished name='${testName}' duration='${result.duration}']`)
    }

In this method, a test result is logged and playwright traces are appended. In case of a test failure, the error message is logged. And at the very end we also log the test execution time.

The very last thing to do to make it work is to initialize our newly created TeamCity Reporter in playwright config:

import { PlaywrightTestConfig } from "@playwright/test"

const playwrightConfig: PlaywrightTestConfig = { ... }

if (process.env.TEAMCITY_VERSION) {
    playwrightConfig.reporter = "./tcReporter.ts"
}

The TEAMCITY_VERSION environment variable is set in TC automatically, so it can serve as a CI environment detection. If everything was put correctly together, once a test fails, the build detail should look like this on the image below. Note the trace.zip link, upon clicking it will download the playwright traces. The reporter is easily adjustable to display more media content like video recordings or screenshots from the tests if wanted, but traces worked the best for us. Happy testing!