Skip to end of metadata
Go to start of metadata

You are viewing an old version of this content. View the current version.

Compare with Current View Version History

« Previous Version 17 Next »

Testing

To ensure reliability and stability of Safety Analysis a variety of tests are performed. We are using JUnit, Pi-test and other Java extensons connected with tests. Actually we started to introduce Cucumber as a new tool for acceptable tests.

During execution of all tests bugs and issues are found. Analysis of tests results is available here.

Acceptance tests

Software testing method conducted to determine if the requirements of a specification or contract are met. It may involve User Acceptance Testing (UAT), Operational Acceptance Testing (OAT), Acceptance testing in extreme programming, alpha and beta testing. You write acceptance tests to check if your code is passing the requirements of project. You should run these tests in integration-test phase. Naming param is *className*ATest.java.

Unit tests

Software testing method by which individual units of source code, sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures, are tested to determine whether they are fit for use.Unit testing is commonly automated, but may still be performed manually. The objective in unit testing is to isolate a unit and validate its correctness. You should write JUnit to check every function that you use in your code. JUnit must be fast, it should be matters of seconds to run that test. JUnit should test only define unit of system. Naming param is *className*JUTest.java.

Command Line Use Case Tests Specification

This page contains documentation for CLI Use Case Tests in order of appearance in code.

TestParser test - unrecognized argument
DescriptionThis test checks GNU Parser for use of unrecognized argument.
ConditionsArgument must be one of given in tables above.
Test procedure
      1. Pass the argument to CLI "-l"
      2. Compare result with expected result
Expected resultsOutput string: "Encountered exception while parsing using GnuParser:\nUnrecognized option: -l"

TestAnalyze Fault Tree - no tree loaded
DescriptionThis test checks correctness of Fault Tree Analysis.
ConditionsNo tree loaded.
Test procedure
      1. Pass the argument to CLI "-a"
      2. Call the method
      3. Compare result with expected result
Expected resultsOutput string: "There is no tree that I could analyze."

TestCheck the Cohesion of Event Tree - no tree loaded
DescriptionThis test checks cohesion of Event Tree.
ConditionsNo tree loaded.
Test procedure
      1. Pass the argument to CLI "-c"
      2. Call the method
      3. Compare result with expected result
Expected resultsOutput string: "There is no tree that I could analyze."

TestGet Fuzzy Importance Index of Fault Tree - no tree loaded
DescriptionThis test gets fuzzy importance index of fault tree.
ConditionsNo tree loaded.
Test procedure
      1. Pass the argument to CLI "-i"
      2. Call the method
      3. Compare result with expected result
Expected resultsOutput string: "There is no tree that I could analyze."


TestLoad Event Tree - wrong tree
DescriptionThis test checks Event Tree loading from file.
Conditions-
Test procedure
      1. Pass the argument to CLI "-e wrong.xml"
      2. Call the method
      3. Compare result with expected result
Expected resultsOutput string: "\nRead Event Tree file from: wrong.xml Fail"

TestLoad Fault Tree - wrong tree
DescriptionThis test checks Fault Tree loading from file.
Conditions-
Test procedure
      1. Pass the argument to CLI "-f wrong.xml"
      2. Call the method
      3. Compare result with expected result
Expected resultsOutput string: "\nRead Fault Tree file from: wrong.xml Fail"

TestLoad Event Tree - correct tree
DescriptionThis test checks Event Tree loading from file.
Conditions-
Test procedure
      1. Pass the argument to CLI "-e treeXmltest2.xml"
      2. Call the method
      3. Compare result with expected result
Expected resultsOutput string: "\nRead Event Tree file from: treeXmltest2.xml Success"

TestCheck Cohesion of Event Tree
DescriptionThis test checks cohesion of Event Tree.
ConditionsSucesfully load Event Tree from file.
Test procedure
      1. Pass the argument to CLI "-e treeXmltest2.xml -c"
      2. Call the method
      3. Compare result with expected result
Expected resultsOutput string: "\nRead Event Tree file from: treeXmltest2.xml Success\n"Result of the analysis Events Tree: \nEvents tree is ok"

TestCheck Report generation
DescriptionThis test checks report generation.
Conditions-
Test procedure
      1. Pass the argument to CLI "-r Report.odt"
      2. Call the method
      3. Compare result with expected result
Expected resultsOutput string: "\nCreated file: Report.odt"

TestLoad Fault Tree - correct tree
DescriptionThis test checks Fault Tree loading from file.
Conditions-
Test procedure
      1. Pass the argument to CLI "-f exampleFaultTree.xml"
      2. Call the method
      3. Compare result with expected result
Expected resultsOutput string: "\nRead Fault Tree file from: exampleFaultTree.xml Success"

TestGet Fuzzy Importance Index of Fault Tree
DescriptionThis test gets fuzzy importance index of fault tree.
ConditionsSucesfully loaded Fault Tree.
Test procedure
      1. Pass the argument to CLI "-f exampleFaultTree.xml -i"
      2. Call the method
      3. Compare result with expected result
Expected resultsOutput string: "\nRead Fault Tree file from: exampleFaultTree.xml Success\nResult of the analysis Fault Tree: \nKey: root = Value: "


Note: Our project configuration for cucumber tests for now is in /SA-99/Pluskota branch, but it should be merged in the feature.

Cucumber

Cucumber is a tool that we used for write acceptable tests. It runs automated acceptance tests written in a behavior-driven development (BDD) style. Cucumber defines application behavior using simple English text, defined by a language called Gherkin, which make tests easy to read and understood for peoples that doesn't have much experience with programming. Another advantage of this tests environment is, that it automatically generate report for the conducted test, which might help with analyzing tests results.

More informations about Cucumber are available in documentation.

Testing with cucumber

Note: Project is already prepared to work with Cucumber, so you don't have to configure this environment.

To make a executable test you have to prepare Feature,Steps Definition and Run files.

In our project Feature file should be located in safetyanalysis/moduleName/src/test/resources/cucumber/FileName.feature, Step Definition file in safetyanalysis/moduleName/src/test/java/steps/FileNameSteps.java and Run file in safetyanalysis/moduleName/src/test/java/FileName.java

Feature files

Feature file include whole description written in gherkin language. Using basic English language we can describe every single step a description of every single step of our test. It's include feature which is tested, concrete scenarios for this features and test steps.

Example of feature file:

Example code for feature
Feature: Withdraw Money from ATM

    A user with an account at a bank would like to withdraw money from an ATM.
    Provided he has a valid account and debit or credit card, he should be allowed to make the transaction. The ATM will tend the requested amount of money, return his card, and subtract amount of the withdrawal from the user's account.

    Scenario: First scenario title
        Given preconditions
        When actions
        Then results

    Scenario: Second scenario title
        ...


As you can see in this example, in first line is written tested feature after Feature: keyword . Under feature name we can write detailed description of tested feature, where we can write how the whole situation looks like etc. This step is not required, but might help later with test analyzing. In 6th line we have name of first scenario after Scenario: keyword. Each scenario consist of steps, which have to be passed to finish test correctly.

Every step have to start in new line with one of the keywords:

    • Given - It describes the pre-requisite for the test to be executed. Example - GIVEN I am a Facebook user.

    • When - It defines the trigger point for any test scenario execution. Example − WHEN I enter "<username>".

    • Then - Then holds the expected result for the test to be executed. Example − THEN login should be successful.

    • And - It provides the logical AND condition between any two statements. AND can be used in conjunction with GIVEN, WHEN and THEN statement. Example − WHEN I enter my "<username>" AND I enter my "<password>".

    • But - It signifies logical OR condition between any two statements. OR can be used in conjunction with GIVEN, WHEN and THEN statement. Example − THEN login should be successful. BUT home page should not be missing.

Using those keywords you can create any test what you want.

Step definitions file

File which description steps definitions i really important. Without him, steps in Feature file is just a words. Those steps in feature file are searched with regex and compared to definitions in step definition file. To make them work correctly, you should configure functions for each sentence that is used in Feature file. Each step, which is started by correct keyword (When, Then etc.) have to have code, how this step works.


Here's example code for step descriptions (first part is Feature code, second is step definitions): 

Example - Part of Feature file
/*------------------------------- Feature file ----------------------------*/
Feature: F1
	Scenario: S1
		Given first value 2
		Given second value 3
		When first value is 2
		And second value is 3
		Then multiplication result is 6
Example - Part of Step definition file
/*-------------------------- Step definitions file ------------------------*/
/*packages and imports*/

public class MathMull {

    @Before
    public void beforeScenario() {
        math = new MathMull;
    }

    @After
    public void afterScenario() {
    }

    @Given("^first value (-?\\d)$")
    public void setFirstValue(int var) throws Throwable {
        var1 = var;
    }

    @Given("^second value (-?\\d)$")
    public void setSecondValue(int var) throws Throwable {
        var2 = var;

    @When("^first value is (-?\\d)$")
    public void compareFirst(int var) throws Throwable {
        var1 == var;


    @And("^second value is (-?\\d)$")
    public void compareSecond(int var) throws Throwable {
        var2 == var;


	@Then("^multiplication result is (-?\\d)$")
	public void compareMultiplication(int var) throws Throwable {
    	var1*var2 == var;
	}
}

In this example you can see, how every steps from Feature file (upper part) have to be written in Step definition file (lower part). Even if you make the same function, but written with different words you will have to add another definition to step definition file. This is very important, because every missed step definition make that your test does not be complete, and in consequential will not work.

Another thing is, that in Step definition file, there are two steps that are not in Feature file, @Before and @After. Both are used to do things before and after tests. It might be helpful, if we have for example to create class which is require to tests.

You can see below more concrete example: 

Example code for Step description
/*here are packages and imports*/

public class EventTreeAcceptanceTestSteps {

    private EventTree tree;
    private BufferedReader br;
    private String filepath;
    private EventTree newTree;

    @Before
    public void beforeScenario() {
        tree = new EventTree();
    }

    @After
    public void afterScenario() {
    }

    @Given("^I have an event tree$")
    public void iHaveAnEventTree() throws Throwable {
        assertNotNull(tree);
    }

    @When("^I set its ID to (-?\\d)$")
    public void iSetItsIDTo(int ID) throws Throwable {
        tree.setId(ID);


	@Then("^The event tree should be defined with string \"([^\"]*)\"$")
	public void theEventTreeShouldBeDefinedWithString(String eventTreeString) throws Throwable {
    	assertEquals(eventTreeString, tree.toString());
	}

	@When("^I add an init event$")
	public void iAddAnInitEvent() throws Throwable {
   	Event e1 = new Event();
    	tree.addInitEvent(e1);
	}
(...)
}

Run file

In Run file you have to set all options about tests, like which features are tested or where steps are located. There's few more options to configure but they are optional, the main thing of this file is that he making our test code working.

Example code for running tests:

Example code for Run
package pl.wroc.pwr.sa.et;
import cucumber.api.CucumberOptions;
import cucumber.api.junit.Cucumber;
import org.junit.runner.RunWith;

@RunWith(Cucumber.class)
@CucumberOptions(
        format = { "pretty", "html:target/cucumber" },
        glue = "pl.wroc.pwr.sa.et.steps",
        features = "classpath:cucumber/EventTree.feature"
)
public class EventTreeAcceptanceTest {
}

 
To run our test all we have to do is just run our test
 

If everything was configured correct, we should got message about completed tests.

After completed tests Cucumber automatically generate report file, where you can check what was tested, or if something went wrong where was the problem. Report file is generated as file safetyanalysis/moduleName/target/cucumber/index.html.

Scenario outline - alternative scenario type

In some situations, when we want to test for example results of some mathematic function making the same scenario with different values doesn't make sense. Alternatively you can use Scenario outline in place of normal Scenario to test multiple values at the same scenario. All what you have to do is write tests where instead static values you have to write <variableName>. Later, after describing scenario all you have to do is make after Examples: keyword table, with values for each variant of this scenario. Here you have example of code which use this feature:

Example code for feature
Feature: (...)


Scenario Outline: A user withdraws money from an ATM
    Given <Name> has a valid Credit or Debit card
    And their account balance is <OriginalBalance>
    When they insert their card
    And withdraw <WithdrawalAmount>
    Then the ATM should return <WithdrawalAmount>
    And their account balance is <NewBalance>

    Examples:
      | Name   | OriginalBalance | WithdrawalAmount | NewBalance |
      | Eric   | 100             | 45               | 55         |
      | Pranav | 100             | 40               | 60         |
      | Ed     | 1000            | 200              | 800        |


Example test

Here you can see example test of events tree class. As a scenario in events tree we'll test, does our events tree will create, modify, save and load correctly. Below is code for all three files, which allow to run correctly this test.

Feature file

File located in safetyanalysis/safety-eventstree/src/test/resources/cucumber/EventTree.feature

Example - Feature
Feature: EventTree

	As a user
    I want to create an event tree with three events and save it to xml file
    So that I don't need to create it once again

    Scenario: Create an Event Tree, set its description, add three events and save it to XML file
    	Given I have an event tree
    	When I set its description to "Drzewo zdarzen"
    	Then Its description should be "Drzewo zdarzen"
   		When I set its ID to 5
    	And I add an init event
    	Then The event tree should be defined with string "(5,1,1|0:0)"
    	When I add a second event to tree
    	Then The event tree should be defined with string "(5,2,3|0:0|1:1,2)"
    	When I add a third event to tree
    	Then The event tree should be defined with string "(5,3,7|0:0|1:1,2|2:3,4,5,6)"
    	When I save my event tree to XML file called "eventTree.xml"
    	Then XML file should be available
    	When I load stored xml file
    	Then File should not be empty
    	And XML file should contain my event tree
    	When I load an event tree from stored xml file
    	Then A new event tree should be created
    	And New events tree description should be "Drzewo zdarzen"
    	And New event tree should be defined with string "(5,3,7|0:0|1:1,2|2:3,4,5,6)"
    	But New event tree should not be defined with string "(5,2,3|0:0|1:1,2)"


Step Definition file

File located in safetyanalysis/safety-eventstree/src/test/java/steps/EventTreeAcceptanceTestSteps.java

Example - Steps definition
package pl.wroc.pwr.sa.et.steps;

import /*(...)*/

/**
 * Created by Patryk on 11.05.2017.
 */
public class EventTreeAcceptanceTestSteps {

    private EventTree tree;
    private BufferedReader br;
    private String filepath;
    private EventTree newTree;

    @Before
    public void beforeScenario() {
        tree = new EventTree();
    }

    @After
    public void afterScenario() {
    }

    @Given("^I have an event tree$")
    public void iHaveAnEventTree() throws Throwable {
        assertNotNull(tree);
    }

    @When("^I set its ID to (-?\\d)$")
    public void iSetItsIDTo(int ID) throws Throwable {
        tree.setId(ID);
    }

    @Then("^The event tree should be defined with string \"([^\"]*)\"$")
    public void theEventTreeShouldBeDefinedWithString(String eventTreeString) throws Throwable {
        assertEquals(eventTreeString, tree.toString());
    }

    @When("^I add an init event$")
    public void iAddAnInitEvent() throws Throwable {
        Event e1 = new Event();
        tree.addInitEvent(e1);
    }

    @When("^I set its description to \"([^\"]*)\"$")
    public void iSetItsDescriptionTo(String description) throws Throwable {
        tree.setDescription(description);
    }

    @Then("^Its description should be \"([^\"]*)\"$")
    public void itsDescriptionShouldBe(String description) throws Throwable {
        assertEquals(description, tree.getDescription());
    }

    @When("^I add a second event to tree$")
    public void iAddASecondEventToTree() throws Throwable {
        Event e2 = new Event();
        tree.addEvent(e2);
    }

    @When("^I add a third event to tree$")
    public void iAddAThirdEventToTree() throws Throwable {
        Event e3 = new Event();
        tree.addEvent(e3);
    }

    @When ("^I save my event tree to XML file called \"([^\"]*)\"$")
    public void iSaveMyEventTreeToXMLFileCalled(String filename) throws Throwable {
        EventTree.saveTreeToXML(filename, tree);
        filepath = filename;
    }

    @Then("^A new event tree should be created$")
    public void aNewEventTreeShouldBeCreated() throws Throwable {
        assertNotNull(newTree);
    }

    @Then("^XML file should be available$")
    public void xmlFileShouldBeAvailable() throws Throwable {
        File createdfile = new File(filepath);
        assertTrue(createdfile.exists());
    }

    @When("^I load stored xml file$")
    public void iLoadStoredXmlFile() throws Throwable {
        br = new BufferedReader(new FileReader(filepath));
    }

    @Then("^File should not be empty$")
    public void fileShouldNotBeEmpty() throws Throwable {
        assertNotNull(br.readLine());
        br.close();
    }

    @And("^XML file should contain my event tree$")
    public void xmlFileShouldContainMyEventTree() throws Throwable {
        DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
        Document doc = factory.newDocumentBuilder().parse(new File(filepath));

        XPathFactory xFactory = XPathFactory.newInstance();
        XPath xpath = xFactory.newXPath();
        XPathExpression expr = xpath.compile("//eventTree//description[contains(.,'Drzewo zdarzen')]");
        Object result = expr.evaluate(doc,XPathConstants.NODESET);
        NodeList nodes = (NodeList)result;
        assertTrue(nodes.getLength() > 0);
    }

    @When("^I load an event tree from stored xml file$")
    public void iLoadAnEventTreeFromStoredXmlFile() throws Throwable {
        newTree = EventTree.loadFromXML(filepath);
    }

    @And("^New events tree description should be \"([^\"]*)\"$")
    public void newEventsTreeDescriptionShouldBe(String description) throws Throwable {
        assertEquals(description, newTree.getDescription());
    }

    @And("^New event tree should be defined with string \"([^\"]*)\"$")
    public void newEventTreeShouldBeDefinedWithString(String eventTreeString) throws Throwable {
        assertEquals(eventTreeString, newTree.toString());
    }

    @But("^New event tree should not be defined with string \"([^\"]*)\"$")
    public void newEventTreeShouldNotBeDefinedWithString(String eventTreeString) throws Throwable {
        assertFalse(eventTreeString.equals(newTree.toString()));
    }
}

Run file

File located in safetyanalysis/safety-eventstree/src/test/java/EventTreeAcceptanceTest.java

Example - Run
package pl.wroc.pwr.sa.et;
import cucumber.api.CucumberOptions;
import cucumber.api.junit.Cucumber;
import org.junit.runner.RunWith;

/**
 * Created by Patryk on 11.05.2017.
 */

@RunWith(Cucumber.class)
@CucumberOptions(
        monochrome = true,
        format = { "pretty", "html:target/cucumber" },
        glue = "pl.wroc.pwr.sa.et.steps",
        features = "classpath:cucumber/EventTree.feature"
)
public class EventTreeAcceptanceTest {
}

Test Results

 After run our test we can see in console that it was finished:
 

Below we can see automatically generated report with results of this test. As we can see, everything have green color, which means that test was completed successfully. In other case, red color will fill all steps that wasn't passed.
This report is located in  safetyanalysis/safety-eventstree/target/cucumber/index.html

  • No labels