Page tree

This is the documentation of Xray Server. If you are looking for Xray Cloud, visit this page. Documentation of older versions of Xray Server is available in this page.

Skip to end of metadata
Go to start of metadata

In very simple terms, a Test is a sequence of steps coupled with conditions or variables, test inputs and an expected result. It is intended to establish the quality, performance or reliability of a piece of  system, i.e., test target. Usually, every requirement or objective the test target is expected to achieve needs at least one Test. The success of the Test is determined by comparing expected and actual results.

Test Types

Tests in Xray can be of two types: automated or manual.

Automated

Automated Tests are executed through an external tool which controls the execution of tests and the comparison of actual outcomes to predicted outcomes. They can automate some repetitive tasks in a formalized testing process already in place, or add additional testing that would be difficult do perform manually. The following automated Test types are available:


In Jira's administration, you may add other values for the "Test Type" custom field (see list below). For example, you may want to differentiate automated tests that may exist simultaneously in a project.

However, internally they will and must be handled as one of the two default Test Types for automated tests: "Cucumber" or "Generic".

Test Type custom field possible valuesInternally will be handled as Test Type...
"Cucumber", "Automated[Cucumber]", "Gherkin", "Behave", "SpecFlow", "Behat"Cucumber
"Generic", "Automated[Generic]"Generic


Note: "Test Type" custom field values not shown above are not supported.


Cucumber

Cucumber is a tool that executes plain-text functional descriptions as automated tests. The language that Cucumber understands is called Gherkin, which is a business-readable, domain-specific language that describes the software’s behavior without detailing how that behavior is implemented. It is line-oriented language and uses indentation to define structure. Line endings terminate statements (e.g., steps). Either spaces or tabs may be used for indentation (but spaces are more portable). Most lines start with a keyword.

For more information on the Cucumber Basics, please consult the Wiki and Tutorials at: http://cucumber.io

Fields

Cucumber Tests are divided in two Scenario Types:

  •  Scenario,  a set of instructions exclusively composed by constant values, which, when executed by a tool, will only give a static result (e.g., 2 + 3 = 5)
  •  Scenario Outline,  a generic set of instructions containing variables, which, when executed by a tool, will prompt varied results according to the specified input.

- The behavior to be reproduced by the Cucumber tool in entered in the Steps field.

Generic

A Generic Test is an automated agnostic Test that is executed by an external tool. Execution results will then be imported back to Jira. An example might be a JUnit test that is used for quality assurance.

Fields

The only Field that is related to a Generic Test is its Definition. Here, the automated external test may be defined or referenced through an open text field. 

The Generic Test Definition field supports the Jira markup wiki language.

Manual

Manual Tests are user-assisted tests, listing an assortment of guidelines the user must follow, and the results such behavior is expected to accomplish. If the actual test result matches the expected test result, the test is considered as passed; otherwise, it is marked as failed.

Fields

Manual Tests are segmented in Manual Steps. Each Manual Step is composed of 4 fields (StepDataExpected Result and Attachments) with Step being the only mandatory one.

  • Step - The action to be reproduced by the tester
  • Data - Any data the related step requests (e.g., login credentials) to be used by the tester 
  • Expected Result - The behavior the step should accomplish 
  • Attachments - External files that provide additional information related with the corresponding Manual Step. Attachments can only be added or removed in the Test Issue view page.

The StepData and Expected Result fields support the Jira markup wiki language.

Actions

Create

Using the top "Create" button"

In order to create a new Test Issue, 

Step 1: Click the Create Issue at the top of the screen to open the Create Issue dialog box / page.

Step 2: Select the  Project  and on  Issue Type,  select Test.

Step 3: Under the General tab, type a Summary for the test and fill all appropriate fields — at least, the mandatory ones marked with an asterisk.

Step 4: On the Test Details tab, select the Test Type and complete all appropriate fields;

Step 4a: If you are writing a Manual Test:
- Select the Test Type as Manual. 
- After filling out the Manual Steps fields, press the Add button to jump to the next Manual Step. 
- Use the Pencil button to edit a previously entered Manual Step, and press the Update button when done. 
- Use the Bin button to delete a previous entered Manual Step.
- To change a Manual Step position in the Test, drag the left area of the row and drop it to the desired place.

Step 4b: If you are writing a Cucumber Test:
- Select the Test Type as Cucumber. 
- On Cucumber Type, select either Scenario or Scenario Outline as test type. 
- On Cucumber Scenario, enter the Cucumber Scenario clauses in Gherkin.

Step 4c: If you are writing a Generic Test:
- Select the Test Type as Generic.
- On Generic Test Definition, enter the definition for the automated generic Test.

Optional:
- Click on the Test Sets tab to associate the current issue with Test Sets.
- Click on the Pre-Conditions tab to associate the current issue with Pre-Conditions.
- Click on the Test Plans tab to associate the current issue with Test Plans.
 
- Click on the Link Issues tab to link with other issues and specify the nature of the association.
To create a series of similar tests – within the same  Project  – check the  Create another box at the bottom of the dialog. 

Step 5: When you are satisfied with the content of your test, click the Create button.

Step 5a: If you checked the Create another box, a new Create Issue dialog appears. This issue is automatically pre-populated with your previous test details, while leaving the Summary field blank.

From the requirement issue screen

In order to create a new Test Issue directly from the requirement issue screen, and link it automatically to the requirement, 

Step 1: Click the Create Issue at the top of the screen to open the Create Issue dialog box / page.

Step 2: Select the  Project and on Issue Type make sure you have Test selected.

Step 3: Under the General tab, type a Summary for the test and fill all appropriate fields — at least, the mandatory ones marked with an asterisk.

Step 4: On the Test Details tab, select the Test Type and complete all appropriate fields;

Step 4a: If you are writing a Manual Test:
- Select the Test Type as Manual. 
- After filling out the Manual Steps fields, press the Add button to jump to the next Manual Step. 
- Use the Pencil button to edit a previously entered Manual Step, and press the Update button when done. 
- Use the Bin button to delete a previous entered Manual Step.
- To change a Manual Step position in the Test, drag the left area of the row and drop it to the desired place.

Step 4b: If you are writing a Cucumber Test: 
- Select the Test Type as Cucumber. 
- On Cucumber Type, select either Scenario or Scenario Outline as test type. 
- On Cucumber Scenario, enter the Cucumber Scenario clauses in Gherkin.

Step 4c: If you are writing a Generic Test: 
- Select the Test Type as Generic.
- On Generic Test Definition, enter the definition for the automated generic Test.

Optional:
- Click on the Test Sets tab to associate the current issue with Test Sets.
- Click on the Pre-Conditions tab to associate the current issue with Pre-Conditions.
- Click on the Test Plans tab to associate the current issue with Test Plans. 
- Click on the Link Issues tab to link with other issues and specify the nature of the association.
- To create a series of similar tests – within the same  Project  – check the  Create another box at the bottom of the dialog. 

Step 5: When you are satisfied with the content of your test, click the Create button.

Step 5a: If you checked the Create another box, a new Create Issue dialog appears. This issue is automatically pre-populated with your previous test details, while leaving the Summary field blank.

Edit

Editing an existing Test can be done in one of two ways:

Using the Edit Issue dialog box

The Edit Issue dialog box is similar to the Create Issue dialog box, and can be accessed through the following steps:

Step 1: Click the Edit button (at the top-left of the 'view issue' page) to open the Edit Issue dialog box.

Step 2: Modify your Test's details in the appropriate fields of the Edit Issue dialog box General tab.

Step 3: Click on the Test Details tab to edit the Test Type and any complementary fields.

Step 3a: If you are editing a Manual Test:
- To input a new Manual Steps field, use the empty areas at the bottom and click the Add button.
- Use the Pencil button to edit a previously entered Manual Step, and click the Update button when done.
- Use the Bin button to delete a previously entered Manual Step.
- To change a Manual Step position in the Test, drag the left area on the that row and drop it to the desired place.

Step 3b: If you are editing a Cucumber Test: 
- To change test type on Cucumber Type, select either Scenario or Scenario Outline.
- In the Steps field, you can edit a previously entered or create a new Cucumber Scenario clauses in Gherkin.

Step 3c: If you are editing a Generic Test:
-In the Steps field, you can edit a previously entered or create a new Definition for the automated generic Test.

Optional:
- Click on the Test Sets tab to edit Test Set associations.
- Click on the Pre-Conditions tab to edit Pre-Condition associations.
- Click on the Link Issues tab to edit Links with other issues.

Step 4: Click the Update button to save your changes.

Inline Editing Xray Tests

Xray Test issue sections support inline editing similar to Jira native and custom fields inline edit. This prevents full page reloads when modifying fields.

Inline Edit Fields

To inline edit a field, just hover over the field view area and click it.

The following Test fields support inline edit:

  • Type: The Test issue type
  • Definition: The automated generic test definition (only visible for Generic tests)
  • Cucumber Type: The Cucumber Scenario type (only visible for Cucumber tests)

To inline edit automated Test issues:

Step 1b: If you are inline editing a Cucumber Test: 
- You can hover over the actual Test Type, click it, and change it to another type that is available on the select box.
- You can hover over the actual Cucumber Type, click it, and change it to another type that is available on the select box.
- You can hover over the Steps field, click it, and edit its contents.

Step 1c: If you are inline editing a Generic Test:
- You can hover over the actual Test Type, click it, and change it to another type available on the select box.
- You can hover over the Definition field, click it, and edit its contents.

Notes: On Jira versions greater or equal to 7.3.0, the Definition field supports Jira's Rich Text Editor. To enable the Rich Text Editor, you have to set the renderer of the definition custom field to  Wiki Style Renderer.  It is also possible to use Rich Text Editor on Jira versions greater than 7.2.0 by enabling the experimental Rich Text Editor in Jira configurations, but this feature is not totally supported by Xray.


Manual Test Steps Table

In addition to inline editable fields, Xray provides a Restful table for inline editing manual Test steps direcly from the Test view issue screen.

From this table you can:

Add Steps: After entering the Test step details (StepData and Expected Result fields) using the bottom table row text boxes, click on the Plus icon for creating a new step. The new step will be added in the last position in the steps table. 

Remove Steps: The last column for each step contains a Bin button for removing steps. After clicking the button, you will be prompted with a confirmation message to delete the step. This operation cannot be undone.

Edit Steps: When hovering over StepData and Expected Result fields, they get highlighted and if you click them, the step row visualization will change to edit mode. From here, you can edit the fields and click on the Update button that appears in the last column to save changes or you can cancel the changes by clicking on the Cancel button. Either way, the row will be back in View mode.

Move Steps: You can move a Test step Up or Down in the Test steps table in one of two ways:

    • Click on the drag area in the first column of each step and drag the step to the desired position.
    • Click on the actions menu and choosing the "Move To.." option. This method allows you to directly move a step to a specific position.

Add Attachments: While in View or Edit modes, you can add attachments to a step using the Plus sign appearing in the Attachments column. After clicking the button, you will be prompted with a dialog to choose the attachments to add.

Remove Attachments: While in View or Edit modes, you can remove attachments of Test steps using the Bin button that appears when hovering over existing attachments. After clicking the button, you will be prompted with a confirmation message to remove the attachment or not.

Copy Step Fields: You can copy the StepData and Expected Result fields to the clipboard by hovering over these fields and choosing the Copy icon next to each highlighted field. If the copy is successful, a green border will appear momentarily around the Copy icon.

Expand Fields: If the text boxes are too small to show all the information, you can expand the StepData and Expected Result fields by clicking the Expand icon next to each highlighted field. An inline dialog, with a bigger area, will be opened with the contents of the field and a wiki notation toolbar to assist you in writing your information. This operation is available in both View and Edit modes for manual Test steps. 

Clone Steps: To clone an existing Test step, click on the actions menu and choose the "Clone" option. A confirmation message will appear. Once you confirm, a new step will be created in the last position, containing all information of the cloned step (Step, Data, Expected Result and Attachments).

Edit Permissions

You can only inline edit Xray fields if you have permission to edit the Test issue.

Changing the Type of a Test

In the Global Preferences, Xray allows you to configure if a test definition is deleted or not.

If enabled, the previous Info details will be deleted.

 

Otherwise, the Info details will not be deleted, and it'll be possible to undo the Test Type change operation and keep the previous information.

Additionally, in both cases, because each pre-condition must be of the same Type of the Test, any associated Pre-Conditions will be lost.

Link Test to requirement(s)

Covering some requirement issue with an existing Test can either be done from the Test issue screen or from the requirement issue screen.

From the Test issue screen

To link some existing Test case to some requirement directly from the Test issue screen, you just need to create a "tests" link between it and the requirement

 Step 1: Click on More and then Link


Step 2: Make sure that the created link is "tests" (i.e. the Test covers the given issues)

Step 3: Enter one, or more, issue key(s) of the requirement you wish to validate

Step 4: Click on "Link"

From the requirement issue screen

To link some existing Test case to some requirement directly from the requirement issue screen, we can do it using two different shortcuts as detailed in the first step below.

Step 1aUsing the "+" icon on the Test Coverage panel  - Click the Edit button (at the top-left of the 'view issue' page) to open the Edit Issue dialog box.

Step 1b: Using the standard issue link creation dialog - Click on More and then Link

Step 2: Make sure that the created link is "tested by" (i.e. the requirement will be tested by the given issues)

Step 3: Enter one, or more, issue key(s) of the Test cases that validate your requirement.



Please note

You may also specify Test Set issue keys; in this case, the link will be created between the requirement and the Test Set.

When you do this, you'll have a dynamic coverage of the requirement based on the Tests belonging to the Test Set. That mean that you can then add or remove Tests to/from the Test Set and they will be covering the requirement or not depending on being or not on the Test Set.


Step 4: Click on "Link"

Clone

Cloning a Test issue creates a new Test with all the information of the cloned Test, except its executions. Therefore, the cloned issue will not be associated with any Test Execution issue.

Step 1: Select More > Clone. The Clone Issue screen will appear.

Step 2You can edit the clone Test's Summary if you wish.

  - If the test contains links to other issue(s), you can select whether or not to include the links in the new clone test.

  - If the test contains sub-tasks, you can select whether or not to create the sub-tasks in the new clone test.

  - If the test contains attachments, you can select whether or not to include the attachments in the new clone test.

Step 3: Click Create.

Configure Test Run Columns

More info in Test Runs.

Configure Test Set Columns

Xray provides the ability to configure columns for the Test Sets table. This configuration is specific to each user and can be restored to the default configuration defined in the Default columns layout page in Xray app administration.

To configure columns for the Test Sets table,

Step 1: Open the Test issue.

Step 2: Click on Columns, select the dropdown that appears in the Test Sets section, above the table in the top right corner. A list of the current configure columns and all available columns appears.

Step 3: Add or remove columns by clicking the checkboxes next to each field.

Step 4: Click the Done button when finished or Cancel to abort this operation. The table refreshes with the configured columns.

Restoring Columns to Default Configuration

To restore columns in the Test Sets table to the default configuration,

Step 1: Open the Test issue.

Step 2: Click on the Columns select dropdown that appears in the Test Sets section, above the table in the top right corner. A list of the current configure columns and all available columns appears.

Step 3: Click the Restore Columns link above the columns list. The table refreshes with the default configured columns.

Change Columns Order

To reorder columns in the Test Sets table,

Step 1: Open the Test issue.

Step 2: Hover over the desired column header.

Step 3: Click on the column header and drag it to the desired position.

Configure Test Plan Columns

Xray provides the ability to configure columns for the Test Plans table. This configuration is specific to each user and can be restored to the default configuration defined in the Default columns layout page in Xray app administration.

To configure columns for the Test Plans table,

Step 1: Open the Test issue.

Step 2: Click on Columns, select the dropdown that appears in the Test Plans section, above the table in the top right corner. A list of the current configure columns and all available columns appears.

Step 3: Add or remove columns by clicking the checkboxes next to each field.

Step 4: Click the Done button when finished or Cancel to abort this operation. The table refreshes with the configured columns.

Restoring Columns to Default Configuration

To restore columns in the Test Plans table to the default configuration,

Step 1: Open the Test issue.

Step 2: Click on Columns, select the dropdown that appears in the Test Plans section, above the table in the top right corner. A list of the current configure columns and all available columns appears.

Step 3: Click the Restore Columns link above the columns list. The table refreshes with the default configured columns.

Change the Order of Columns

To reorder columns in the Test Plans table,

Step 1: Open the Test issue.

Step 2: Hover over the desired column header.

Step 3: Click on the column header and drag it to the desired position.


Create Pre-Condition

To create a Pre-Condition directly associated to the current Test from the Test issue view screen:

Step 1: Open the Test issue view screen you wish to create a Pre-Condition with.

Step 2: Click the Create Pre-Condition button (under the Pre-Condition section on the view page) to open the Create Issue dialog box.

Step 3: Type a Summary for the Pre-Condition and complete all appropriate fields — at least, the required ones marked by an asterisk.This issue is automatically pre-populated with your previous Test issue Fix Version/s.

Step 4: On the Pre-Condition Details tab, complete all appropriate fields. This issue is automatically pre-populated with your previous Test type.

Step 5Optional. Click on the Tests tab to edit Test associations to associate more Tests. This issue is automatically pre-populated with your previous Test key.

Step 6: Click Create.


Associate Pre-Conditions

To associate Tests with a Pre-Condition from the Test issue view screen:

Step 1: Open the Test issue view screen you wish to associate a Pre-Condition with.

Step 2: Click the Associate Pre-Condition button (under the Pre-Condition section on the view page) to open the Issue Picker Dialog.

Step 3Open the Search tab.

How to use de Issue Picker Dialog

This is an example of how to add tests, using some of the features available in the Search tab.

For more information, please visit the Issue Picker Dialog detailed overview.

Step 4: Configure the search.

-Use the Filters dropdown button to include extra fields in the search criteria.

-Fill in the filters values.

-Press the search button.

Step 5: Select and Add Pre-conditions.

- Once the results are obtained you can order, add/remove columns to the results list (the columns selection is saved as a preference).

Check the pre-conditions you wish to add from the listed results.

Step 6:  Add tests by pressing "Add selected".

Remove Pre-Conditions

To remove a Pre-Condition association from the Test issue view screen:

Step 1: Open the Test you wish to remove the Pre-Condition association.

Step 2: On the Pre-Conditions table, hover over the desired Pre-Condition row and click on the Trash button that appears in the end of the row. You will be prompted with a confirmation dialog.

Reorder Pre-Conditions

Pre-Conditions within a Test issue can be reordered. There are two ways to change the order of Pre-Conditions:

Drag-and-Drop

To reorder Pre-Conditions,

Step 1: Open the Test issue.

Step 2: Hover over the desired Pre-Condition row (under the Pre-Condition section on the view page) and the drag handle column.

Step 3: Click the drag handle and move the Pre-Condition up or down in the current page.

Change Rank Action

This action allows users to change the Rank of a Pre-Condition between pages.

To change the rank of a Pre-Condition through the action:

Step 1: Open the Test issue.

Step 2: Hover over the desired Pre-Condition (under the Pre-Condition section on the view page).

Step 3: Click the gear that appears in the last column. The actions menu should pop up with the available actions.

Step 4: Click the Move To action. The change rank dialog appears.

Step 5: Enter the desired index number (1-based) and click OK.










Export to Cucumber

Cucumber Tests in Jira can be exported to feature files ready to be executed in Cucumber. The generated feature files follow the rules defined in Export Cucumber Features and take into account the context of the Jira Test issue.

A single Cucumber feature file should be generated and prompted for download, unless the target Test is associated with multiple Requirements, which, in that case, will prompt a FeatureBundle zip file containing a single Cucumber feature file for every Requirement associated with the target Test.

To export a Cucumber Test issue to feature files, 

Step 1: Open the Cucumber Test issue view page. 

Step 2: Select More > Export to Cucumber. The file will be downloaded to the browser.

Export Test to XML

Xray Test issues can be exported to XML. This export differs from the Jira default action for exporting to XML. Rather than exporting all fields, the Xray export action will only export to XML the most relevant fields for Tests, such as the Test definition fields, Pre-Conditions and some issue-tracking information.

To export a Test issue to XML, 

Step 1Open the Test issue view page. 

Step 2: Select More > Export Test to XML. The resulting XML file will be downloaded to the browser.

Export Test Runs to CSV

Xray Test Runs can be exported to CSV. Xray export action will export to CSV the most relevant fields for Test Runs such as:

    • Test Key
    • Test Execution Key
    • Executed By
    • Assignee
    • Start
    • Finish
    • Defects issue keys (list separated by ",")
    • Attachment JIRA links (list separated by ",")
    • Comment
    • Status

To export a Test Run to CSV, 

Step 1Open the Test issue view page. 

Step 2: Select More > Export Test Runs to CSV. The resulting CSV file will be downloaded to the browser.

Export Test as Text

Automated Generic Test issues can be exported to a plain text file. Only the Test Definition field will be included in the plain text file. This action can be useful when executing a Generic Test outside of Jira.

To export an Automated Generic Test definition to plain text, 

Step 1Open the Automated Generic Test issue view page. 

Step 2: Select More > Export Test as Text. The resulting plain text file will be downloaded to the browser.

From the Test issue view screen

Import Steps

Test Steps can be imported from external sources such as:

    • Test
    • CSV
    • JSON
    • Clipboard

You can configure which columns you want to import, including if you want to override the steps already present in the Test.

From Test

The import from tests dialog allows you to choose which steps you wants to clone from a given test and a checkbox to choose if you want to override the steps already present in the Test.

From CSV

The import steps from the CSV dialog allows you to choose the file that you wants to import the steps from, the delimiter that is used in the CSV, a dropdown to specify which column layout the steps present in the file have, a checkbox to specify if the file has header or not, and a checkbox to allow you to choose if you wants to override the steps already present in the Test.


Try it

Example of CSV contained Step, Data and (Expected) Result fields delimited by semicolon; it also contains the header that should be ignored whenever importing: example_steps.csv .

From JSON

The import from JSON dialog allows you to specify which file you wants to import the steps from and provides a checkbox if you want to override the steps already present in the Test.

The JSON file should be an array of step objects. The objects should have the field step and can optionally have the fields data and result. Below is an example of a valid file:

Example Json
[
   {
      "step":"Step 1",
      "data":"Data for Step 1",
      "result":"Result for Steps 1"
   },
   {
      "step":"Step 3",
      "data":"Data for Step 3"
   },
   {
      "step":"Step 4",
      "result":"Result for Step 4"
   },
   {
      "step":"Step 2"
   }
]

From Clipboard

The import from clipboard allows you to copy the steps from Excel or CSV and paste them in the dialog. You can choose if you want to use the default Excel delimiter or use a custom one, if you want to copy steps from CSV. You can choose which column layout the steps in the file have and provides a checkbox if you want to override the steps already present in the Test.

After the steps are pasted in the dialog, it will display the previous version of the steps in the same dialog.

Export Steps

Manual Test Steps can be exported to CSV by clicking the “Export Steps” button in the Test Details section of the Test.

The following dialog will pop up.

The CSV delimiter can be changed and optionally, the attachment links can also be included.

The result will be a CSV file containing the Step, Data, Expected Result columns and, if the option is selected, the Attachments column that contains the attachments links.


  • No labels