TestingTo ensure reliability and stability of Safety Analysis a variety of tests are performed. We are using JUnit, Pi-test and other Java extensons connected with tests. Actually we started to introduce Cucumber as a new tool for acceptable tests. During execution of all tests bugs and issues are found. Analysis of tests results is available here. Acceptance testsSoftware testing method conducted to determine if the requirements of a specification or contract are met. It may involve User Acceptance Testing (UAT), Operational Acceptance Testing (OAT), Acceptance testing in extreme programming, alpha and beta testing. You write acceptance tests to check if your code is passing the requirements of project. You should run these tests in integration-test phase.Naming param is *className*ATest.java. Unit testsSoftware testing method by which individual units of source code, sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures, are tested to determine whether they are fit for use.Unit testing is commonly automated, but may still be performed manually. The objective in unit testing is to isolate a unit and validate its correctness. You should write JUnit to check every function that you use in your code. JUnit must be fast, it should be matters of seconds to run that test. JUnit should test only define unit of system. Naming param is *className*JUTest.java. Command Line Use Case Tests SpecificationThis page contains documentation for CLI Use Case Tests in order of appearance in code. Test | Parser test - unrecognized argument |
---|
Description | This test checks GNU Parser for use of unrecognized argument. |
---|
Conditions | Argument must be one of given in tables above. |
---|
Test procedure | - Pass the argument to CLI "-l"
- Compare result with expected result
|
---|
Expected results | Output string: "Encountered exception while parsing using GnuParser:\nUnrecognized option: -l" |
---|
| Test | Analyze Fault Tree - no tree loaded |
---|
Description | This test checks correctness of Fault Tree Analysis. |
---|
Conditions | No tree loaded. |
---|
Test procedure | - Pass the argument to CLI "-a"
- Call the method
- Compare result with expected result
|
---|
Expected results | Output string: "There is no tree that I could analyze." |
---|
| Test | Check the Cohesion of Event Tree - no tree loaded |
---|
Description | This test checks cohesion of Event Tree. |
---|
Conditions | No tree loaded. |
---|
Test procedure | - Pass the argument to CLI "-c"
- Call the method
- Compare result with expected result
|
---|
Expected results | Output string: "There is no tree that I could analyze." |
---|
| Test | Get Fuzzy Importance Index of Fault Tree - no tree loaded |
---|
Description | This test gets fuzzy importance index of fault tree. |
---|
Conditions | No tree loaded. |
---|
Test procedure | - Pass the argument to CLI "-i"
- Call the method
- Compare result with expected result
|
---|
Expected results | Output string: "There is no tree that I could analyze." |
---|
|
| Test | Load Event Tree - wrong tree |
---|
Description | This test checks Event Tree loading from file. |
---|
Conditions | - |
---|
Test procedure | - Pass the argument to CLI "-e wrong.xml"
- Call the method
- Compare result with expected result
|
---|
Expected results | Output string: "\nRead Event Tree file from: wrong.xml Fail" |
---|
| Test | Load Fault Tree - wrong tree |
---|
Description | This test checks Fault Tree loading from file. |
---|
Conditions | - |
---|
Test procedure | - Pass the argument to CLI "-f wrong.xml"
- Call the method
- Compare result with expected result
|
---|
Expected results | Output string: "\nRead Fault Tree file from: wrong.xml Fail" |
---|
| Test | Load Event Tree - correct tree |
---|
Description | This test checks Event Tree loading from file. |
---|
Conditions | - |
---|
Test procedure | - Pass the argument to CLI "-e treeXmltest2.xml"
- Call the method
- Compare result with expected result
|
---|
Expected results | Output string: "\nRead Event Tree file from: treeXmltest2.xml Success" |
---|
| Test | Check Cohesion of Event Tree |
---|
Description | This test checks cohesion of Event Tree. |
---|
Conditions | Sucesfully load Event Tree from file. |
---|
Test procedure | - Pass the argument to CLI "-e treeXmltest2.xml -c"
- Call the method
- Compare result with expected result
|
---|
Expected results | Output string: "\nRead Event Tree file from: treeXmltest2.xml Success\n"Result of the analysis Events Tree: \nEvents tree is ok" |
---|
| Test | Check Report generation |
---|
Description | This test checks report generation. |
---|
Conditions | - |
---|
Test procedure | - Pass the argument to CLI "-r Report.odt"
- Call the method
- Compare result with expected result
|
---|
Expected results | Output string: "\nCreated file: Report.odt" |
---|
| Test | Load Fault Tree - correct tree |
---|
Description | This test checks Fault Tree loading from file. |
---|
Conditions | - |
---|
Test procedure | - Pass the argument to CLI "-f exampleFaultTree.xml"
- Call the method
- Compare result with expected result
|
---|
Expected results | Output string: "\nRead Fault Tree file from: exampleFaultTree.xml Success" |
---|
| Test | Get Fuzzy Importance Index of Fault Tree |
---|
Description | This test gets fuzzy importance index of fault tree. |
---|
Conditions | Sucesfully loaded Fault Tree. |
---|
Test procedure | - Pass the argument to CLI "-f exampleFaultTree.xml -i"
- Call the method
- Compare result with expected result
|
---|
Expected results | Output string: "\nRead Fault Tree file from: exampleFaultTree.xml Success\nResult of the analysis Fault Tree: \nKey: root = Value: " |
---|
Note: Our project configuration for cucumber tests for now is in /SA-99/Pluskota branch, but it should be merged in the feature. CucumberCucumber is a tool that we used use for write acceptable tests. It runs automated acceptance tests written in a behaviorbehaviour-driven development (BDD) style. Cucumber defines application behavior behaviour using simple English text, defined by a language called Gherkin, which make tests easy to read and understood understand for peoples people that doesndon't have much experience with programming. Another advantage of this tests environment is, that it automatically generate generates report for the conducted test, which might help with analyzing analysing tests results. More informations about Cucumber are available in documentation. Testing with cucumberNote: Project Our project is already prepared to work with Cucumber, so you don't have to re-configure this your environment. To make a executable test you have to prepare Feature,Steps Definition and Run Runnerfiles. In our project Feature file should be located in safetyanalysis/moduleName/src/test/resources/cucumber/FileName.feature, Step Definition file in safetyanalysis/moduleName/src/test/java/steps/FileNameSteps.java and Run Runner file in safetyanalysis/moduleName/src/test/java/FileName.java Feature filesFeature file include whole includes test scenario description written in gherkin Gherkin language. Using basic English language we can describe every single step a description of every single step of our test. It 's include includes feature which is tested, concrete scenarios for this features and test steps. Example of feature file: Code Block |
---|
language | text |
---|
firstline | 1 |
---|
title | Example code for feature |
---|
linenumbers | true |
---|
| Feature: Withdraw Money from ATM
A user with an account at a bank would like to withdraw money from an ATM.
Provided he has a valid account and debit or credit card, he should be allowed to make the transaction. The ATM will tend the requested amount of money, return his card, and subtract amount of the withdrawal from the user's account.
Scenario: First scenario title
Given preconditions
When actions
Then results
Scenario: Second scenario title
... |
As you can see in this example, in first line is written tested feature after Feature: keyword. Under feature name we can write A feature file consists of 2 main parts: - Feature - Feature contains detailed description of tested feature
, where we can write how the whole situation looks like etc step - part is not required, but
might help later with test analyzing. In 6th line we have name of first scenario after Scenario: keyword. - it makes our test more understandable and helps to analyse test result.
- Scenario - Each scenario consist of steps, which have to be passed to finish test
correctlyEvery step have has to start in new line with one of the keywords: Given - It describes the pre-requisite for the test to be executed. Example - GIVEN I am a Facebook user. When - It defines the trigger point for any test scenario execution. Example − WHEN I enter "<username>". Then - Then It holds the expected result for the test to be executed. Example − THEN login should be successful. And - It provides the logical AND condition between any two statements. AND can be used in conjunction with GIVEN, WHEN and THEN statement. Example − WHEN I enter my "<username>" AND I enter my "<password>".
- But - It signifies logical OR condition between any two statements. OR can be used in conjunction with GIVEN, WHEN and THEN statement. Example − THEN login should be successful. BUT home page should not be missing.
Using those keywords you can create any test what you want. Step definitions fileFile which description It is a file which contains steps definitions i really important. Without him, steps defined in Feature file is just a words. Those steps in feature file are searched with regex and compared to definitions in step definition file are just simple words. It contains source code which should be done within a test step. Parameters defined in feature files are passed to step definitions with regular expressions. To make them work correctly, you should configure functions methods for each sentence that is used step defined in Feature file. Each step , which is started by correct keyword (When, Then etc.) have to have code, how this step works.preceded by an annotation connected to step keywords defined in feature file (e.g. @Then ) Here's example code for step descriptions (first part is Feature code, second is step definitions):
Code Block |
---|
language | java |
---|
firstline | 1 |
---|
title | Example - Part of Feature file |
---|
linenumbers | true |
---|
| /*------------------------------- Feature file ----------------------------*/
Feature: F1
Scenario: S1
Given first value 2
Given second value 3
When first value is 2
And second value is 3
Then multiplication result is 6 |
Code Block |
---|
language | java |
---|
firstline | 1 |
---|
title | Example - Part of Step definition file |
---|
linenumbers | true |
---|
| /*-------------------------- Step definitions file ------------------------*/
/*packages and imports*/
public class MathMull {
@Before
public void beforeScenario() {
math = new MathMull;
}
@After
public void afterScenario() {
}
@Given("^first value (-?\\d)$")
public void setFirstValue(int var) throws Throwable {
var1 = var;
}
@Given("^second value (-?\\d)$")
public void setSecondValue(int var) throws Throwable {
var2 = var;
@When("^first value is (-?\\d)$")
public void compareFirst(int var) throws Throwable {
var1 == var;
@And("^second value is (-?\\d)$")
public void compareSecond(int var) throws Throwable {
var2 == var;
@Then("^multiplication result is (-?\\d)$")
public void compareMultiplication(int var) throws Throwable {
var1*var2 == var;
}
}
|
In this example you can see, how every steps from Feature file (upper part) have to be written in Step definition file (lower part). Even if you make the same functionmethod, but written with different words you will have to add another definition to step definition file. This is very important, because every missed step definition make that your test does not be complete, and in consequential consequence it will not workend with success. Another thing is, that in Step definition file, there are two steps that are not in Feature file, @Before and @After. Both are used to do things before and after tests. It might be helpful, if we have for example to create class which is require to tests. You can see below more concrete example: Code Block |
---|
language | java |
---|
firstline | 1 |
---|
title | Example code for Step description |
---|
linenumbers | true |
---|
| /*here are packages and imports*/
public class EventTreeAcceptanceTestSteps {
private EventTree tree;
private BufferedReader br;
private String filepath;
private EventTree newTree;
@Before
public void beforeScenario() {
tree = new EventTree();
}
@After
public void afterScenario() {
}
@Given("^I have an event tree$")
public void iHaveAnEventTree() throws Throwable {
assertNotNull(tree);
}
@When("^I set its ID to (-?\\d)$")
public void iSetItsIDTo(int ID) throws Throwable {
tree.setId(ID);
@Then("^The event tree should be defined with string \"([^\"]*)\"$")
public void theEventTreeShouldBeDefinedWithString(String eventTreeString) throws Throwable {
assertEquals(eventTreeString, tree.toString());
}
@When("^I add an init event$")
public void iAddAnInitEvent() throws Throwable {
Event e1 = new Event();
tree.addInitEvent(e1);
}
(...)
}
| Run Runner fileIn Run Runner file you have to set all options about tests, like which features are tested or where steps are located. There's few more options to configure but they are optional, the main thing of this file is that he making our test code working. Required options are: - format - It describes how cucumber will format test case output and if it create reports.
- glue - It contains path to package containing Step definition files
- features - It contains path to feature file.
Example code for running tests:
Code Block |
---|
language | java |
---|
firstline | 1 |
---|
title | Example code for Run |
---|
linenumbers | true |
---|
| package pl.wroc.pwr.sa.et;
import cucumber.api.CucumberOptions;
import cucumber.api.junit.Cucumber;
import org.junit.runner.RunWith;
@RunWith(Cucumber.class)
@CucumberOptions(
format = { "pretty", "html:target/cucumber" },
glue = "pl.wroc.pwr.sa.et.steps",
features = "classpath:cucumber/EventTree.feature"
)
public class EventTreeAcceptanceTest {
}
|
To run our test all we have to do is just run our test
If everything was configured correct, we should got message about completed tests. 
After completed tests Cucumber automatically generate report file, where you can check what was tested, or if something went wrong where was the problem. Report file is generated as file safetyanalysis/moduleName/target/cucumber/index.html. Scenario outline - alternative scenario typeIn some situations, when we want to test for example results of some mathematic function making the same scenario with different values doesn't make sense. Alternatively you can use Scenario outline in place of normal Scenario to test multiple values at the same scenario. All what you have to do is write tests where instead static values you have to write <variableName>. Later, after describing scenario all you have to do is make after Examples: keyword table, with values for each variant of this scenario. Here you have example of code which use this feature:
Code Block |
---|
language | text |
---|
firstline | 1 |
---|
title | Example code for feature |
---|
linenumbers | true |
---|
| Feature: (...)
Scenario Outline: A user withdraws money from an ATM
Given <Name> has a valid Credit or Debit card
And their account balance is <OriginalBalance>
When they insert their card
And withdraw <WithdrawalAmount>
Then the ATM should return <WithdrawalAmount>
And their account balance is <NewBalance>
Examples:
| Name | OriginalBalance | WithdrawalAmount | NewBalance |
| Eric | 100 | 45 | 55 |
| Pranav | 100 | 40 | 60 |
| Ed | 1000 | 200 | 800 | |
Example testHere you can see example test of events tree class. As a scenario in events tree we'll test, does our events tree will create, modify, save and load correctly. Below is code for all three files, which allow to run correctly this test. Feature fileFile located in safetyanalysis/safety-eventstree/src/test/resources/cucumber/EventTree.feature Code Block |
---|
language | text |
---|
firstline | 1 |
---|
title | Example - Feature |
---|
linenumbers | true |
---|
| Feature: EventTree
As a user
I want to create an event tree with three events and save it to xml file
So that I don't need to create it once again
Scenario: Create an Event Tree, set its description, add three events and save it to XML file
Given I have an event tree
When I set its description to "Drzewo zdarzen"
Then Its description should be "Drzewo zdarzen"
When I set its ID to 5
And I add an init event
Then The event tree should be defined with string "(5,1,1|0:0)"
When I add a second event to tree
Then The event tree should be defined with string "(5,2,3|0:0|1:1,2)"
When I add a third event to tree
Then The event tree should be defined with string "(5,3,7|0:0|1:1,2|2:3,4,5,6)"
When I save my event tree to XML file called "eventTree.xml"
Then XML file should be available
When I load stored xml file
Then File should not be empty
And XML file should contain my event tree
When I load an event tree from stored xml file
Then A new event tree should be created
And New events tree description should be "Drzewo zdarzen"
And New event tree should be defined with string "(5,3,7|0:0|1:1,2|2:3,4,5,6)"
But New event tree should not be defined with string "(5,2,3|0:0|1:1,2)" |
Step Definition file File located in safetyanalysis/safety-eventstree/src/test/java/steps/EventTreeAcceptanceTestSteps.java Code Block |
---|
language | java |
---|
firstline | 1 |
---|
title | Example - Steps definition |
---|
linenumbers | true |
---|
| package pl.wroc.pwr.sa.et.steps;
import /*(...)*/
/**
* Created by Patryk on 11.05.2017.
*/
public class EventTreeAcceptanceTestSteps {
private EventTree tree;
private BufferedReader br;
private String filepath;
private EventTree newTree;
@Before
public void beforeScenario() {
tree = new EventTree();
}
@After
public void afterScenario() {
}
@Given("^I have an event tree$")
public void iHaveAnEventTree() throws Throwable {
assertNotNull(tree);
}
@When("^I set its ID to (-?\\d)$")
public void iSetItsIDTo(int ID) throws Throwable {
tree.setId(ID);
}
@Then("^The event tree should be defined with string \"([^\"]*)\"$")
public void theEventTreeShouldBeDefinedWithString(String eventTreeString) throws Throwable {
assertEquals(eventTreeString, tree.toString());
}
@When("^I add an init event$")
public void iAddAnInitEvent() throws Throwable {
Event e1 = new Event();
tree.addInitEvent(e1);
}
@When("^I set its description to \"([^\"]*)\"$")
public void iSetItsDescriptionTo(String description) throws Throwable {
tree.setDescription(description);
}
@Then("^Its description should be \"([^\"]*)\"$")
public void itsDescriptionShouldBe(String description) throws Throwable {
assertEquals(description, tree.getDescription());
}
@When("^I add a second event to tree$")
public void iAddASecondEventToTree() throws Throwable {
Event e2 = new Event();
tree.addEvent(e2);
}
@When("^I add a third event to tree$")
public void iAddAThirdEventToTree() throws Throwable {
Event e3 = new Event();
tree.addEvent(e3);
}
@When ("^I save my event tree to XML file called \"([^\"]*)\"$")
public void iSaveMyEventTreeToXMLFileCalled(String filename) throws Throwable {
EventTree.saveTreeToXML(filename, tree);
filepath = filename;
}
@Then("^A new event tree should be created$")
public void aNewEventTreeShouldBeCreated() throws Throwable {
assertNotNull(newTree);
}
@Then("^XML file should be available$")
public void xmlFileShouldBeAvailable() throws Throwable {
File createdfile = new File(filepath);
assertTrue(createdfile.exists());
}
@When("^I load stored xml file$")
public void iLoadStoredXmlFile() throws Throwable {
br = new BufferedReader(new FileReader(filepath));
}
@Then("^File should not be empty$")
public void fileShouldNotBeEmpty() throws Throwable {
assertNotNull(br.readLine());
br.close();
}
@And("^XML file should contain my event tree$")
public void xmlFileShouldContainMyEventTree() throws Throwable {
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
Document doc = factory.newDocumentBuilder().parse(new File(filepath));
XPathFactory xFactory = XPathFactory.newInstance();
XPath xpath = xFactory.newXPath();
XPathExpression expr = xpath.compile("//eventTree//description[contains(.,'Drzewo zdarzen')]");
Object result = expr.evaluate(doc,XPathConstants.NODESET);
NodeList nodes = (NodeList)result;
assertTrue(nodes.getLength() > 0);
}
@When("^I load an event tree from stored xml file$")
public void iLoadAnEventTreeFromStoredXmlFile() throws Throwable {
newTree = EventTree.loadFromXML(filepath);
}
@And("^New events tree description should be \"([^\"]*)\"$")
public void newEventsTreeDescriptionShouldBe(String description) throws Throwable {
assertEquals(description, newTree.getDescription());
}
@And("^New event tree should be defined with string \"([^\"]*)\"$")
public void newEventTreeShouldBeDefinedWithString(String eventTreeString) throws Throwable {
assertEquals(eventTreeString, newTree.toString());
}
@But("^New event tree should not be defined with string \"([^\"]*)\"$")
public void newEventTreeShouldNotBeDefinedWithString(String eventTreeString) throws Throwable {
assertFalse(eventTreeString.equals(newTree.toString()));
}
} | Run
Runner file File located in safetyanalysis/safety-eventstree/src/test/java/EventTreeAcceptanceTest.java Code Block |
---|
language | java |
---|
firstline | 1 |
---|
title | Example - Run |
---|
linenumbers | true |
---|
| package pl.wroc.pwr.sa.et;
import cucumber.api.CucumberOptions;
import cucumber.api.junit.Cucumber;
import org.junit.runner.RunWith;
/**
* Created by Patryk on 11.05.2017.
*/
@RunWith(Cucumber.class)
@CucumberOptions(
monochrome = true,
format = { "pretty", "html:target/cucumber" },
glue = "pl.wroc.pwr.sa.et.steps",
features = "classpath:cucumber/EventTree.feature"
)
public class EventTreeAcceptanceTest {
} |
Test Results After run our test we can see in console that it was finished:
Below we can see automatically generated report with results of this test. As we can see, everything have is green color-coloured, which means that test was completed successfully. In other case, red color colour will fill all steps that wasn't passed. This report is located in safetyanalysis/safety-eventstree/target/cucumber/index.html 
|