Table of Contents

Excerpt

Testing

To ensure reliability and stability of Safety Analysis a variety of tests are performed. We are using JUnit, Pi-test and other Java extensons connected with tests. Actually we started to introduce Cucumber as a new tool for acceptance tests.

During execution of all tests bugs and issues are found. Analysis of tests results is available here.

Acceptance tests

Software testing method conducted to determine if the requirements of a specification or contract are met. It may involve User Acceptance Testing (UAT), Operational Acceptance Testing (OAT), Acceptance testing in extreme programming, alpha and beta testing. You write acceptance tests to check if your code is passing the requirements of project. You should run these tests in integration-test phase.Naming param is *prefix*AT.java.

Component tests

Component testing is a method where testing of each component in an application is done separately. Suppose, in an application there are 5 components. Testing of each 5 components separately and efficiently is called as component testing. It finds the defects in the module and verifies the functioning of software. Component tests use spring context or in memory database. Naming param is *className*IT.java.

Unit tests

Software testing method by which individual units of source code, sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures, are tested to determine whether they are fit for use.Unit testing is commonly automated, but may still be performed manually. The objective in unit testing is to isolate a unit and validate its correctness. You should write JUnit to check every function that you use in your code. JUnit must be fast, it should be matters of seconds to run that test. JUnit should test only define unit of system. Naming param is *className*Test.java.

Command Line Use Case Tests Specification

This page contains documentation for CLI Use Case Tests in order of appearance in code.

Test	Parser test - unrecognized argument
Description	This test checks GNU Parser for use of unrecognized argument.
Conditions	Argument must be one of given in tables above.
Test procedure	Pass the argument to CLI "-l" Compare result with expected result
Expected results	Output string: "Encountered exception while parsing using GnuParser:\nUnrecognized option: -l"

Test	Analyze Fault Tree - no tree loaded
Description	This test checks correctness of Fault Tree Analysis.
Conditions	No tree loaded.
Test procedure	Pass the argument to CLI "-a" Call the method Compare result with expected result
Expected results	Output string: "There is no tree that I could analyze."

Test	Check the Cohesion of Event Tree - no tree loaded
Description	This test checks cohesion of Event Tree.
Conditions	No tree loaded.
Test procedure	Pass the argument to CLI "-c" Call the method Compare result with expected result
Expected results	Output string: "There is no tree that I could analyze."

Test	Get Fuzzy Importance Index of Fault Tree - no tree loaded
Description	This test gets fuzzy importance index of fault tree.
Conditions	No tree loaded.
Test procedure	Pass the argument to CLI "-i" Call the method Compare result with expected result
Expected results	Output string: "There is no tree that I could analyze."

Test	Load Event Tree - wrong tree
Description	This test checks Event Tree loading from file.
Conditions	-
Test procedure	Pass the argument to CLI "-e wrong.xml" Call the method Compare result with expected result
Expected results	Output string: "\nRead Event Tree file from: wrong.xml Fail"

Test	Load Fault Tree - wrong tree
Description	This test checks Fault Tree loading from file.
Conditions	-
Test procedure	Pass the argument to CLI "-f wrong.xml" Call the method Compare result with expected result
Expected results	Output string: "\nRead Fault Tree file from: wrong.xml Fail"

Test	Load Event Tree - correct tree
Description	This test checks Event Tree loading from file.
Conditions	-
Test procedure	Pass the argument to CLI "-e treeXmltest2.xml" Call the method Compare result with expected result
Expected results	Output string: "\nRead Event Tree file from: treeXmltest2.xml Success"

Test	Check Cohesion of Event Tree
Description	This test checks cohesion of Event Tree.
Conditions	Sucesfully load Event Tree from file.
Test procedure	Pass the argument to CLI "-e treeXmltest2.xml -c" Call the method Compare result with expected result
Expected results	Output string: "\nRead Event Tree file from: treeXmltest2.xml Success\n"Result of the analysis Events Tree: \nEvents tree is ok"

Test	Check Report generation
Description	This test checks report generation.
Conditions	-
Test procedure	Pass the argument to CLI "-r Report.odt" Call the method Compare result with expected result
Expected results	Output string: "\nCreated file: Report.odt"

Test	Load Fault Tree - correct tree
Description	This test checks Fault Tree loading from file.
Conditions	-
Test procedure	Pass the argument to CLI "-f exampleFaultTree.xml" Call the method Compare result with expected result
Expected results	Output string: "\nRead Fault Tree file from: exampleFaultTree.xml Success"

Test	Get Fuzzy Importance Index of Fault Tree
Description	This test gets fuzzy importance index of fault tree.
Conditions	Sucesfully loaded Fault Tree.
Test procedure	Pass the argument to CLI "-f exampleFaultTree.xml -i" Call the method Compare result with expected result
Expected results	Output string: "\nRead Fault Tree file from: exampleFaultTree.xml Success\nResult of the analysis Fault Tree: \nKey: root = Value: "

Cucumber

Cucumber is a tool that we use for write acceptable tests. It runs automated acceptance tests written in a behaviour-driven development (BDD) style. Cucumber defines application behaviour using simple English text, defined by a language called Gherkin, which make tests easy to read and understand for people that don't have much experience with programming. Another advantage of this tests environment is, that it automatically generates report for the conducted test, which might help with analysing tests results.

More informations about Cucumber are available in documentation.

Testing with cucumber

Note: Our project is already prepared to work with Cucumber, so you don't have to re-configure your environment.

To make a executable test you have to prepare Feature,Steps Definition and Runnerfiles.

In our project Feature file should be located in safetyanalysis/moduleName/src/test/resources/cucumber/FileName.feature, Step Definition file in safetyanalysis/moduleName/src/test/java/steps/FileNameSteps.java and Runner file in safetyanalysis/moduleName/src/test/java/FileName.java

Feature files

Feature file includes test scenario description written in Gherkin language. Using basic English language we can describe every single step of our test. It includes feature which is tested, concrete scenarios for this features and test steps.

Example of feature file (source):

Code Block

language	text
firstline	1
title	Example code for feature
linenumbers	true

Feature: Withdraw Money from ATM

    A user with an account at a bank would like to withdraw money from an ATM.
    Provided he has a valid account and debit or credit card, he should be allowed to make the transaction. The ATM will tend the requested amount of money, return his card, and subtract amount of the withdrawal from the user's account.

    Scenario: First scenario title
        Given preconditions
        When actions
        Then results

    Scenario: Second scenario title
        ...

A feature file consists of 2 main parts:

Feature - Feature contains detailed description of tested feature and test reason. This part is not required, but it makes our test more understandable and helps to analyse test result.
Scenario - Each scenario consist of steps, which have to be passed to finish test with success.

Every step has to start in new line with one of the keywords:

- Given - It describes the pre-requisite for the test to be executed. Example - GIVEN I am a Facebook user.
- When - It defines the trigger point for any test scenario execution. Example − WHEN I enter "<username>".
- Then - It holds the expected result for the test to be executed. Example − THEN login should be successful.
- And - It provides the logical AND condition between any two statements. AND can be used in conjunction with GIVEN, WHEN and THEN statement. Example − WHEN I enter my "<username>" AND I enter my "<password>".
- But - It signifies logical OR condition between any two statements. OR can be used in conjunction with GIVEN, WHEN and THEN statement. Example − THEN login should be successful. BUT home page should not be missing.

Using those keywords you can create any test you want.

Step definitions file

It is a file which contains steps definitions. Without him, steps defined in Feature file are just simple words. It contains source code which should be done within a test step. Parameters defined in feature files are passed to step definitions with regular expressions. To make them work correctly, you should configure methods for each step defined in Feature file. Each step is preceded by an annotation connected to step keywords defined in feature file (e.g. @Then )

Here's example code for step descriptions (first part is Feature code, second is step definitions) (source):

Code Block

language	java
firstline	1
title	Example - Part of Feature file
linenumbers	true

/*------------------------------- Feature file ----------------------------*/
Feature: F1
	Scenario: S1
		Given first value 2
		Given second value 3
		When first value is 2
		And second value is 3
		Then multiplication result is 6

Code Block

language	java
firstline	1
title	Example - Part of Step definition file
linenumbers	true

/*-------------------------- Step definitions file ------------------------*/
/*packages and imports*/

public class MathMull {

    @Before
    public void beforeScenario() {
        math = new MathMull;
    }

    @After
    public void afterScenario() {
    }

    @Given("^first value (-?\\d)$")
    public void setFirstValue(int var) throws Throwable {
        var1 = var;
    }

    @Given("^second value (-?\\d)$")
    public void setSecondValue(int var) throws Throwable {
        var2 = var;

    @When("^first value is (-?\\d)$")
    public void compareFirst(int var) throws Throwable {
        var1 == var;


    @And("^second value is (-?\\d)$")
    public void compareSecond(int var) throws Throwable {
        var2 == var;


	@Then("^multiplication result is (-?\\d)$")
	public void compareMultiplication(int var) throws Throwable {
    	var1*var2 == var;
	}
}

In this example you can see, how every steps from Feature file (upper part) have to be written in Step definition file (lower part). Even if you make the same method, but written with different words you will have to add another definition to step definition file. This is very important, because every missed step definition make that your test does not be complete, and in consequence it will not end with success.

Another thing is, that in Step definition file, there are two steps that are not in Feature file, @Before and @After. Both are used to do things before and after tests. It might be helpful, if we have for example to create class which is require to tests.

You can see below more concrete example:

Code Block

language	java
firstline	1
title	Example code for Step description
linenumbers	true

/*here are packages and imports*/

public class EventTreeAcceptanceTestSteps {

    private EventTree tree;
    private BufferedReader br;
    private String filepath;
    private EventTree newTree;

    @Before
    public void beforeScenario() {
        tree = new EventTree();
    }

    @After
    public void afterScenario() {
    }

    @Given("^I have an event tree$")
    public void iHaveAnEventTree() throws Throwable {
        assertNotNull(tree);
    }

    @When("^I set its ID to (-?\\d)$")
    public void iSetItsIDTo(int ID) throws Throwable {
        tree.setId(ID);


	@Then("^The event tree should be defined with string \"([^\"]*)\"$")
	public void theEventTreeShouldBeDefinedWithString(String eventTreeString) throws Throwable {
    	assertEquals(eventTreeString, tree.toString());
	}

	@When("^I add an init event$")
	public void iAddAnInitEvent() throws Throwable {
   	Event e1 = new Event();
    	tree.addInitEvent(e1);
	}
(...)
}

Runner file

In Runner file you have to set all options about tests, like which features are tested or where steps are located. There's few more options to configure but they are optional, the main thing of this file is that he making our test code working.

Required options are:

format - It describes how cucumber will format test case output and if it create reports.
glue - It contains path to package containing Step definition files
features - It contains path to feature file.

Example code for running tests

...

:

Code Block

language	java
firstline	1
title	Example code for Run
linenumbers	true

package pl.wroc.pwr.sa.et;
import cucumber.api.CucumberOptions;
import cucumber.api.junit.Cucumber;
import org.junit.runner.RunWith;

@RunWith(Cucumber.class)
@CucumberOptions(
        format = { "pretty", "html:target/cucumber" },
        glue = "pl.wroc.pwr.sa.et.steps",
        features = "classpath:cucumber/EventTree.feature"
)
public class EventTreeAcceptanceTest {
}

To run our test all we have to do is just run our test
Image Added

If everything was configured correct, we should got message about completed tests.

Image Added

After completed tests Cucumber automatically generate report file, where you can check what was tested, or if something went wrong where was the problem. Report file is generated as file safetyanalysis/moduleName/target/cucumber/index.html.

Scenario outline - alternative scenario type

In some situations, when we want to test for example results of some mathematic function making the same scenario with different values doesn't make sense. Alternatively you can use Scenario outline in place of normal Scenario to test multiple values at the same scenario. All what you have to do is write tests where instead static values you have to write <variableName>. Later, after describing scenario all you have to do is make after Examples: keyword table, with values for each variant of this scenario. Here you have example of code which use this feature (source):

Code Block

language	text
firstline	1
title	Example code for feature
linenumbers	true

Feature: (...)


Scenario Outline: A user withdraws money from an ATM
    Given <Name> has a valid Credit or Debit card
    And their account balance is <OriginalBalance>
    When they insert their card
    And withdraw <WithdrawalAmount>
    Then the ATM should return <WithdrawalAmount>
    And their account balance is <NewBalance>

    Examples:
      | Name   | OriginalBalance | WithdrawalAmount | NewBalance |
      | Eric   | 100             | 45               | 55         |
      | Pranav | 100             | 40               | 60         |
      | Ed     | 1000            | 200              | 800        |

Example test

Here you can see example test of events tree class. Below is code for all three files, which allow to run correctly this test.

Feature file

File located in safetyanalysis/safety-eventstree/src/test/resources/cucumber/EventTree.feature

Code Block

language	text
firstline	1
title	Example - Feature
linenumbers	true

Feature: EventTree
  As a user
  I want to create an event tree with three events and save it to xml file
  So that I don't need to create it once again
  Scenario: Create an Event Tree, set its description, add three events and save it to XML file
    Given I have an event tree
    When I set its description to "Drzewo zdarzen"
    And I set its ID to 5
    And I add an init event
    And I add a second event to tree
    And I add a third event to tree
    Then The event tree should be defined with string "(5,3,7|0:0|1:1,2|2:3,4,5,6)"
    And Its description should be "Drzewo zdarzen"
    When I save my event tree to XML file called "eventTree.xml"
    Then XML file should be available
    When I load stored xml file
    Then File should not be empty
    And XML file should contain my event tree
    When I load an event tree from stored xml file
    Then A new event tree should be created
    And New events tree description should be "Drzewo zdarzen"
    And New event tree should be defined with string "(5,3,7|0:0|1:1,2|2:3,4,5,6)"
    But New event tree should not be defined with string "(5,2,3|0:0|1:1,2)"

Step Definition file

File located in safetyanalysis/safety-eventstree/src/test/java/steps/EventTreeATSteps.java

Code Block

language	java
firstline	1
title	Example - Steps definition
linenumbers	true

package pl.wroc.pwr.sa.et.steps;

import cucumber.api.java.Before;
import cucumber.api.java.en.*;
import pl.wroc.pwr.sa.et.Event;
import pl.wroc.pwr.sa.et.EventTree;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathConstants;
import javax.xml.xpath.XPathExpression;
import javax.xml.xpath.XPathFactory;
import org.w3c.dom.Document;
import org.w3c.dom.NodeList;
import static org.junit.Assert.*;

/**
 * Created by Patryk on 11.05.2017.
 */
public class EventTreeATSteps {

    private EventTree tree;
    private BufferedReader bufferedReader;
    private String filepath;
    private EventTree newTree;

    @Before
    public void beforeScenario() {
        tree = new EventTree();
    }

    @Given("^I have an event tree$")
    public void iHaveAnEventTree() throws Exception {
        assertNotNull(tree);
    }

    @When("^I set its ID to (-?\\d)$")
    public void iSetItsIDTo(int ID) throws Exception {
        tree.setId(ID);
    }

    @Then("^The event tree should be defined with string \"([^\"]*)\"$")
    public void theEventTreeShouldBeDefinedWithString(String eventTreeString) throws Exception {
        assertEquals(eventTreeString, tree.toString());
    }

    @And("^I add an init event$")
    public void iAddAnInitEvent() throws Exception {
        tree.addInitEvent(new Event());
    }

    @And("^I set its description to \"([^\"]*)\"$")
    public void iSetItsDescriptionTo(String description) throws Exception {
        tree.setDescription(description);
    }

    @And("^Its description should be \"([^\"]*)\"$")
    public void itsDescriptionShouldBe(String description) throws Exception {
        assertEquals(description, tree.getDescription());
    }

    @And("^I add a second event to tree$")
    public void iAddASecondEventToTree() throws Exception {
        tree.addEvent(new Event());
    }

    @And("^I add a third event to tree$")
    public void iAddAThirdEventToTree() throws Exception {
        tree.addEvent(new Event());
    }

    @When ("^I save my event tree to XML file called \"([^\"]*)\"$")
    public void iSaveMyEventTreeToXMLFileCalled(String filename) throws Exception {
        EventTree.saveTreeToXML(filename, tree);
        filepath = filename;
    }

    @Then("^A new event tree should be created$")
    public void aNewEventTreeShouldBeCreated() throws Exception {
        assertNotNull(newTree);
    }

    @Then("^XML file should be available$")
    public void xmlFileShouldBeAvailable() throws Exception {
        assertTrue(new File(filepath).exists());
    }

    @When("^I load stored xml file$")
    public void iLoadStoredXmlFile() throws Exception {
        bufferedReader = new BufferedReader(new FileReader(filepath));
    }

    @Then("^File should not be empty$")
    public void fileShouldNotBeEmpty() throws Exception {
        assertNotNull(bufferedReader.readLine());
        bufferedReader.close();
    }

    @And("^XML file should contain my event tree$")
    public void xmlFileShouldContainMyEventTree() throws Exception {
        DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
        Document doc = factory.newDocumentBuilder().parse(new File(filepath));

        XPathFactory xFactory = XPathFactory.newInstance();
        XPath xpath = xFactory.newXPath();
        XPathExpression expr = xpath.compile("//eventTree//description[contains(.,'Drzewo zdarzen')]");
        Object result = expr.evaluate(doc,XPathConstants.NODESET);
        NodeList nodes = (NodeList)result;
        assertTrue(nodes.getLength() > 0);
    }

    @When("^I load an event tree from stored xml file$")
    public void iLoadAnEventTreeFromStoredXmlFile() throws Exception {
        newTree = EventTree.loadFromXML(filepath);
    }

    @And("^New events tree description should be \"([^\"]*)\"$")
    public void newEventsTreeDescriptionShouldBe(String description) throws Exception {
        assertEquals(description, newTree.getDescription());
    }

    @And("^New event tree should be defined with string \"([^\"]*)\"$")
    public void newEventTreeShouldBeDefinedWithString(String eventTreeString) throws Exception {
        assertEquals(eventTreeString, newTree.toString());
    }

    @But("^New event tree should not be defined with string \"([^\"]*)\"$")
    public void newEventTreeShouldNotBeDefinedWithString(String eventTreeString) throws Exception {
        assertFalse(eventTreeString.equals(newTree.toString()));
    }
}

Runner file

File located in safetyanalysis/safety-eventstree/src/test/java/EventTreeAcceptanceTest.java

Code Block

language	java
firstline	1
title	Example - Run
linenumbers	true

package pl.wroc.pwr.sa.et;
import cucumber.api.CucumberOptions;
import cucumber.api.junit.Cucumber;
import org.junit.runner.RunWith;

/**
 * Created by Patryk on 11.05.2017.
 */

@RunWith(Cucumber.class)
@CucumberOptions(
        monochrome = true,
        format = { "pretty", "html:target/cucumber" },
        glue = "pl.wroc.pwr.sa.et.steps",
        features = "classpath:cucumber/EventTree.feature"
)
public class EventTreeAT {
}

Test Results

After run our test we can see in console that it was finished:
Image Added

Below we can see automatically generated report with results of this test. As we can see, everything is green-coloured, which means that test was completed successfully. In other case, red colour will fill all steps that wasn't passed.
This report is located in safetyanalysis/safety-eventstree/target/cucumber/index.html

Image Added

Version	Old Version 5	New Version Current
Changes made by	Patryk Pluskota	Patryk Pluskota
Saved on	Apr 06, 2017	Jun 01, 2017

Content Comparison

Versions Compared

Key

Testing

Acceptance tests

Component tests

Unit tests

Command Line Use Case Tests Specification

Cucumber

Testing with cucumber

Feature files

Step definitions file

Runner file

Scenario outline - alternative scenario type

Example test

Feature file

Step Definition file

Runner file

Test Results