Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Integrating Integration with testing frameworks is achieved by processing the results contained in the reports test runners create (e.g., in XML, JSON). The results are mapped to the proper Test issues and, ; if possible, if they don't exist then they will be created. This flow is explained in Using Generic Tests for Automation, although . Xray also supports Cucumber tests natively.       

Besides Cucumber, there are currently many testing frameworks available for every language you may think of.

Many of these frameworks provide test runners that are able to output reports in the JUnit XML format. Therefore, in In the "worst" case" , your automated tests can be imported to JIRA Jira and mapped to "Generic" Test issues , by importing JUnit XML test result reports, as detailed herein Taking advantage of JUnit XML reports. Note that the JUnit report format is rather limited and sometimes it's is not supported by some testing frameworks. Java users may prefer to use TestNG instead. Xray supports TestNG reports as detailed in Taking advantage of TestNG XML reports.

Another way of integrating with testing frameworks is by using the NUnit test runner Nunit's XML report format. Nunit The NUnit report format has more features than JUnit 4.0 XML report, which allows Xray to do things such as automatic linking to issues (e.g., requirements) or assignment of labels to the newly created Test entities.

Besides this, Xray also provides specific integrations for some frameworks, such as Cucumber, Behave, Robot, Xamarin, and other frameworks.

Gliffy Diagram
namexray_integration_with_testingautomation_frameworks_new Copy Copy
pagePin8


Table of Contents

Summary of features per framework

The following table presents the available features , whenever when importing automated test results.

The Xray JSON format is more generic and its capabilities, if used for importing, are different.



Robot framework

Junit

JUnit 4

Nunit
TestNG

NUnit 2.6/3.x

CucumberXray JSON

Abstract automated test as a Test

(map an automated test to a Test issue in

JIRA

Jira)

Yes

(as a Generic Test)

Yes

(as a Generic Test)

Yes

(as a Generic Test)

Yes

(as a Generic Test)

Yes

(as a Cucumber Test)

No

Tests must exist beforehand

Make Test specification in

JIRA

Jira

(specify the Test itself in

JIRA

Jira)


NoNoNoNoYesYes

Create Tests from results

(create Tests whenever importing results)

Yes

Yes

Yes

Yes

No*

(

however, tehre

but there is and endpoint for importing Cucumber features, which will create Tests for the respective Scenarios/Scenario Outlines)

No

Uniquely identify Tests

(identify existing Tests whenever importing results, avoiding duplication of Test issues)

Yes

Yes

Yes

Yes

Yes

Yes*

(based on the provided Test issue keys)

Import results

(importing results by REST API or UI)

Yes

Yes

Yes

Yes

YesYes

Import “labels”

(create labels in the Test issues)

Yes

(labels may be specified in the test's source code)

No

Yes

(labels may be specified in the test's source code)

Yes

(labels may be specified in the test's source code)

Yes*

(this is only available

whenever

when using the endpoint for importing Cucumber features; it's not possible

whenever

when importing results)

No

Automatic linking to requirements

(create links to requirements)

Yes

(requirement's issue key may be specified in the test's source code)

No

Yes

(requirement's issue key may be specified in the test's source code)

Yes

(requirement's issue key may be specified in the test's source code)

Yes*

(this is only available

whenever

when using the endpoint for importing Cucumber features; it's not possible

whenever

when importing results)

No

Semantic on the results

(present

detailed

execution details in the execution screen)

Yes

(

keywords

keyword “steps”)

-

Yes

(

Test Suites & parameterized Tests)

Yes

(steps for scenario/scenario outline and background)

Integration by platform

...

Image result for java logoImage Removed

...

Image Removed

...

Image result for javascript logoImage Removed

...

Image Removed

...

Image Removed

 

...

Image Removed

...

Content by Label
showLabelsfalse
showSpacefalse
sorttitle
cqllabel = "java" and space = currentSpace()
labelsjava

...

Content by Label
showLabelsfalse
showSpacefalse
sorttitle
cqllabel = "csharp" and space = currentSpace()
labelscsharp

...

Content by Label
showLabelsfalse
showSpacefalse
sorttitle
cqllabel in ("c++","c") and space = currentSpace()
labelsc++,c

...

Content by Label
showLabelsfalse
showSpacefalse
sorttitle
cqllabel = "javascript" and space = currentSpace()
labelsjavascript

...

Content by Label
showLabelsfalse
showSpacefalse
sorttitle
cqllabel = "ruby" and space = currentSpace()
labelsruby

...

Content by Label
showLabelsfalse
showSpacefalse
sorttitle
cqllabel = "php" and space = currentSpace()
labelsphp

...

Content by Label
showLabelsfalse
showSpacefalse
sorttitle
cqllabel = "python" and space = currentSpace()
labelspython

...

Content by Label
showLabelsfalse
showSpacefalse
sorttitle
cqllabel = "perl" and space = currentSpace()
labelsperl

...

 

 

Integration by testing framework report

...

...

NUnitImage Removed

...

Content by Label
showLabelsfalse
showSpacefalse
sorttitle
cqllabel = "junit" and space = currentSpace()
labelsjunit

...

Content by Label
showLabelsfalse
showSpacefalse
sorttitle
cqllabel = "nunit" and space = currentSpace()
labelsnunit

...

Content by Label
showLabelsfalse
showSpacefalse
sorttitle
cqllabel = "cucumber" and space = currentSpace()
labelscucumber

 

 

Testing by target environment

...

Content by Label
showLabelsfalse
showSpacefalse
sorttitle
cqllabel in ("phantomjs","selenium","phantomcss") and space = currentSpace()
labelsselenium phantomcss phantomjs

...

Content by Label
showLabelsfalse
showSpacefalse
sorttitle
cqllabel = "mobile" and space = currentSpace()
labelsmobile

...

Content by Label
showLabelsfalse
showSpacefalse
sorttitle
cqllabel = "cloud" and space = currentSpace()
labelscloud

 

 

 

 

 

 

 

 

 

for every "test" section in the XML config file and parameterized Tests)

Yes

(Test Suites and parameterized Tests)

Yes

(steps for Scenario/Scenario Outline and Background)

Yes*

(semantic is implicit, because the Test must be created beforehand)