99. Testing
Testing
To ensure reliability and stability of Safety Analysis a variety of tests are performed. We are using JUnit, Pi-test and other Java extensons connected with tests. Actually we started to introduce Cucumber as a new tool for acceptance tests.
During execution of all tests bugs and issues are found. Analysis of tests results is available here.
Acceptance tests
Software testing method conducted to determine if the requirements of a specification or contract are met. It may involve User Acceptance Testing (UAT), Operational Acceptance Testing (OAT), Acceptance testing in extreme programming, alpha and beta testing. You write acceptance tests to check if your code is passing the requirements of project. You should run these tests in integration-test phase. Naming param is *prefix*AT.java.
Component tests
Component testing is a method where testing of each component in an application is done separately. Suppose, in an application there are 5 components. Testing of each 5 components separately and efficiently is called as component testing. It finds the defects in the module and verifies the functioning of software. Component tests use spring context or in memory database. Naming param is *className*IT.java.
Unit tests
Software testing method by which individual units of source code, sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures, are tested to determine whether they are fit for use.Unit testing is commonly automated, but may still be performed manually. The objective in unit testing is to isolate a unit and validate its correctness. You should write JUnit to check every function that you use in your code. JUnit must be fast, it should be matters of seconds to run that test. JUnit should test only define unit of system. Naming param is *className*Test.java.
Command Line Use Case Tests Specification
This page contains documentation for CLI Use Case Tests in order of appearance in code.
Test | Parser test - unrecognized argument |
---|---|
Description | This test checks GNU Parser for use of unrecognized argument. |
Conditions | Argument must be one of given in tables above. |
Test procedure |
|
Expected results | Output string: "Encountered exception while parsing using GnuParser:\nUnrecognized option: -l" |
Test | Analyze Fault Tree - no tree loaded |
Description | This test checks correctness of Fault Tree Analysis. |
Conditions | No tree loaded. |
Test procedure |
|
Expected results | Output string: "There is no tree that I could analyze." |
Test | Check the Cohesion of Event Tree - no tree loaded |
Description | This test checks cohesion of Event Tree. |
Conditions | No tree loaded. |
Test procedure |
|
Expected results | Output string: "There is no tree that I could analyze." |
Test | Get Fuzzy Importance Index of Fault Tree - no tree loaded |
Description | This test gets fuzzy importance index of fault tree. |
Conditions | No tree loaded. |
Test procedure |
|
Expected results | Output string: "There is no tree that I could analyze." |
Test | Load Event Tree - wrong tree |
Description | This test checks Event Tree loading from file. |
Conditions | - |
Test procedure |
|
Expected results | Output string: "\nRead Event Tree file from: wrong.xml Fail" |
Test | Load Fault Tree - wrong tree |
Description | This test checks Fault Tree loading from file. |
Conditions | - |
Test procedure |
|
Expected results | Output string: "\nRead Fault Tree file from: wrong.xml Fail" |
Test | Load Event Tree - correct tree |
Description | This test checks Event Tree loading from file. |
Conditions | - |
Test procedure |
|
Expected results | Output string: "\nRead Event Tree file from: treeXmltest2.xml Success" |
Test | Check Cohesion of Event Tree |
Description | This test checks cohesion of Event Tree. |
Conditions | Sucesfully load Event Tree from file. |
Test procedure |
|
Expected results | Output string: "\nRead Event Tree file from: treeXmltest2.xml Success\n"Result of the analysis Events Tree: \nEvents tree is ok" |
Test | Check Report generation |
Description | This test checks report generation. |
Conditions | - |
Test procedure |
|
Expected results | Output string: "\nCreated file: Report.odt" |
Test | Load Fault Tree - correct tree |
Description | This test checks Fault Tree loading from file. |
Conditions | - |
Test procedure |
|
Expected results | Output string: "\nRead Fault Tree file from: exampleFaultTree.xml Success" |
Test | Get Fuzzy Importance Index of Fault Tree |
Description | This test gets fuzzy importance index of fault tree. |
Conditions | Sucesfully loaded Fault Tree. |
Test procedure |
|
Expected results | Output string: "\nRead Fault Tree file from: exampleFaultTree.xml Success\nResult of the analysis Fault Tree: \nKey: root = Value: " |
Cucumber
Cucumber is a tool that we use for write acceptable tests. It runs automated acceptance tests written in a behaviour-driven development (BDD) style. Cucumber defines application behaviour using simple English text, defined by a language called Gherkin, which make tests easy to read and understand for people that don't have much experience with programming. Another advantage of this tests environment is, that it automatically generates report for the conducted test, which might help with analysing tests results.
More informations about Cucumber are available in documentation.
Testing with cucumber
Note: Our project is already prepared to work with Cucumber, so you don't have to re-configure your environment.
To make a executable test you have to prepare Feature,Steps Definition and Runner files.
In our project Feature file should be located in safetyanalysis/moduleName/src/test/resources/cucumber/FileName.feature, Step Definition file in safetyanalysis/moduleName/src/test/java/steps/FileNameSteps.java and Runner file in safetyanalysis/moduleName/src/test/java/FileName.java
Feature files
Feature file includes test scenario description written in Gherkin language. Using basic English language we can describe every single step of our test. It includes feature which is tested, concrete scenarios for this features and test steps.
Example of feature file (source):
Feature: Withdraw Money from ATM A user with an account at a bank would like to withdraw money from an ATM. Provided he has a valid account and debit or credit card, he should be allowed to make the transaction. The ATM will tend the requested amount of money, return his card, and subtract amount of the withdrawal from the user's account. Scenario: First scenario title Given preconditions When actions Then results Scenario: Second scenario title ...
A feature file consists of 2 main parts:
- Feature - Feature contains detailed description of tested feature and test reason. This part is not required, but it makes our test more understandable and helps to analyse test result.
- Scenario - Each scenario consist of steps, which have to be passed to finish test with success.
Every step has to start in new line with one of the keywords:
Given - It describes the pre-requisite for the test to be executed. Example - GIVEN I am a Facebook user.
When - It defines the trigger point for any test scenario execution. Example − WHEN I enter "<username>".
Then - It holds the expected result for the test to be executed. Example − THEN login should be successful.
And - It provides the logical AND condition between any two statements. AND can be used in conjunction with GIVEN, WHEN and THEN statement. Example − WHEN I enter my "<username>" AND I enter my "<password>".
- But - It signifies logical OR condition between any two statements. OR can be used in conjunction with GIVEN, WHEN and THEN statement. Example − THEN login should be successful. BUT home page should not be missing.
Using those keywords you can create any test you want.
Step definitions file
It is a file which contains steps definitions. Without him, steps defined in Feature file are just simple words. It contains source code which should be done within a test step. Parameters defined in feature files are passed to step definitions with regular expressions. To make them work correctly, you should configure methods for each step defined in Feature file. Each step is preceded by an annotation connected to step keywords defined in feature file (e.g. @Then )
Here's example code for step descriptions (first part is Feature code, second is step definitions) (source):
/*------------------------------- Feature file ----------------------------*/ Feature: F1 Scenario: S1 Given first value 2 Given second value 3 When first value is 2 And second value is 3 Then multiplication result is 6
/*-------------------------- Step definitions file ------------------------*/ /*packages and imports*/ public class MathMull { @Before public void beforeScenario() { math = new MathMull; } @After public void afterScenario() { } @Given("^first value (-?\\d)$") public void setFirstValue(int var) throws Throwable { var1 = var; } @Given("^second value (-?\\d)$") public void setSecondValue(int var) throws Throwable { var2 = var; @When("^first value is (-?\\d)$") public void compareFirst(int var) throws Throwable { var1 == var; @And("^second value is (-?\\d)$") public void compareSecond(int var) throws Throwable { var2 == var; @Then("^multiplication result is (-?\\d)$") public void compareMultiplication(int var) throws Throwable { var1*var2 == var; } }
In this example you can see, how every steps from Feature file (upper part) have to be written in Step definition file (lower part). Even if you make the same method, but written with different words you will have to add another definition to step definition file. This is very important, because every missed step definition make that your test does not be complete, and in consequence it will not end with success.
Another thing is, that in Step definition file, there are two steps that are not in Feature file, @Before and @After. Both are used to do things before and after tests. It might be helpful, if we have for example to create class which is require to tests.
You can see below more concrete example:
/*here are packages and imports*/ public class EventTreeAcceptanceTestSteps { private EventTree tree; private BufferedReader br; private String filepath; private EventTree newTree; @Before public void beforeScenario() { tree = new EventTree(); } @After public void afterScenario() { } @Given("^I have an event tree$") public void iHaveAnEventTree() throws Throwable { assertNotNull(tree); } @When("^I set its ID to (-?\\d)$") public void iSetItsIDTo(int ID) throws Throwable { tree.setId(ID); @Then("^The event tree should be defined with string \"([^\"]*)\"$") public void theEventTreeShouldBeDefinedWithString(String eventTreeString) throws Throwable { assertEquals(eventTreeString, tree.toString()); } @When("^I add an init event$") public void iAddAnInitEvent() throws Throwable { Event e1 = new Event(); tree.addInitEvent(e1); } (...) }
Runner file
In Runner file you have to set all options about tests, like which features are tested or where steps are located. There's few more options to configure but they are optional, the main thing of this file is that he making our test code working.
Required options are:
- format - It describes how cucumber will format test case output and if it create reports.
- glue - It contains path to package containing Step definition files
- features - It contains path to feature file.
Example code for running tests:
package pl.wroc.pwr.sa.et; import cucumber.api.CucumberOptions; import cucumber.api.junit.Cucumber; import org.junit.runner.RunWith; @RunWith(Cucumber.class) @CucumberOptions( format = { "pretty", "html:target/cucumber" }, glue = "pl.wroc.pwr.sa.et.steps", features = "classpath:cucumber/EventTree.feature" ) public class EventTreeAcceptanceTest { }
To run our test all we have to do is just run our test
If everything was configured correct, we should got message about completed tests.
After completed tests Cucumber automatically generate report file, where you can check what was tested, or if something went wrong where was the problem. Report file is generated as file safetyanalysis/moduleName/target/cucumber/index.html.
Scenario outline - alternative scenario type
In some situations, when we want to test for example results of some mathematic function making the same scenario with different values doesn't make sense. Alternatively you can use Scenario outline in place of normal Scenario to test multiple values at the same scenario. All what you have to do is write tests where instead static values you have to write <variableName>. Later, after describing scenario all you have to do is make after Examples: keyword table, with values for each variant of this scenario. Here you have example of code which use this feature (source):
Feature: (...) Scenario Outline: A user withdraws money from an ATM Given <Name> has a valid Credit or Debit card And their account balance is <OriginalBalance> When they insert their card And withdraw <WithdrawalAmount> Then the ATM should return <WithdrawalAmount> And their account balance is <NewBalance> Examples: | Name | OriginalBalance | WithdrawalAmount | NewBalance | | Eric | 100 | 45 | 55 | | Pranav | 100 | 40 | 60 | | Ed | 1000 | 200 | 800 |
Example test
Here you can see example test of events tree class. Below is code for all three files, which allow to run correctly this test.
Feature file
File located in safetyanalysis/safety-eventstree/src/test/resources/cucumber/EventTree.feature
Feature: EventTree As a user I want to create an event tree with three events and save it to xml file So that I don't need to create it once again Scenario: Create an Event Tree, set its description, add three events and save it to XML file Given I have an event tree When I set its description to "Drzewo zdarzen" And I set its ID to 5 And I add an init event And I add a second event to tree And I add a third event to tree Then The event tree should be defined with string "(5,3,7|0:0|1:1,2|2:3,4,5,6)" And Its description should be "Drzewo zdarzen" When I save my event tree to XML file called "eventTree.xml" Then XML file should be available When I load stored xml file Then File should not be empty And XML file should contain my event tree When I load an event tree from stored xml file Then A new event tree should be created And New events tree description should be "Drzewo zdarzen" And New event tree should be defined with string "(5,3,7|0:0|1:1,2|2:3,4,5,6)" But New event tree should not be defined with string "(5,2,3|0:0|1:1,2)"
Step Definition file
File located in safetyanalysis/safety-eventstree/src/test/java/steps/EventTreeATSteps.java
package pl.wroc.pwr.sa.et.steps; import cucumber.api.java.Before; import cucumber.api.java.en.*; import pl.wroc.pwr.sa.et.Event; import pl.wroc.pwr.sa.et.EventTree; import java.io.BufferedReader; import java.io.File; import java.io.FileReader; import javax.xml.parsers.DocumentBuilderFactory; import javax.xml.xpath.XPath; import javax.xml.xpath.XPathConstants; import javax.xml.xpath.XPathExpression; import javax.xml.xpath.XPathFactory; import org.w3c.dom.Document; import org.w3c.dom.NodeList; import static org.junit.Assert.*; /** * Created by Patryk on 11.05.2017. */ public class EventTreeATSteps { private EventTree tree; private BufferedReader bufferedReader; private String filepath; private EventTree newTree; @Before public void beforeScenario() { tree = new EventTree(); } @Given("^I have an event tree$") public void iHaveAnEventTree() throws Exception { assertNotNull(tree); } @When("^I set its ID to (-?\\d)$") public void iSetItsIDTo(int ID) throws Exception { tree.setId(ID); } @Then("^The event tree should be defined with string \"([^\"]*)\"$") public void theEventTreeShouldBeDefinedWithString(String eventTreeString) throws Exception { assertEquals(eventTreeString, tree.toString()); } @And("^I add an init event$") public void iAddAnInitEvent() throws Exception { tree.addInitEvent(new Event()); } @And("^I set its description to \"([^\"]*)\"$") public void iSetItsDescriptionTo(String description) throws Exception { tree.setDescription(description); } @And("^Its description should be \"([^\"]*)\"$") public void itsDescriptionShouldBe(String description) throws Exception { assertEquals(description, tree.getDescription()); } @And("^I add a second event to tree$") public void iAddASecondEventToTree() throws Exception { tree.addEvent(new Event()); } @And("^I add a third event to tree$") public void iAddAThirdEventToTree() throws Exception { tree.addEvent(new Event()); } @When ("^I save my event tree to XML file called \"([^\"]*)\"$") public void iSaveMyEventTreeToXMLFileCalled(String filename) throws Exception { EventTree.saveTreeToXML(filename, tree); filepath = filename; } @Then("^A new event tree should be created$") public void aNewEventTreeShouldBeCreated() throws Exception { assertNotNull(newTree); } @Then("^XML file should be available$") public void xmlFileShouldBeAvailable() throws Exception { assertTrue(new File(filepath).exists()); } @When("^I load stored xml file$") public void iLoadStoredXmlFile() throws Exception { bufferedReader = new BufferedReader(new FileReader(filepath)); } @Then("^File should not be empty$") public void fileShouldNotBeEmpty() throws Exception { assertNotNull(bufferedReader.readLine()); bufferedReader.close(); } @And("^XML file should contain my event tree$") public void xmlFileShouldContainMyEventTree() throws Exception { DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance(); Document doc = factory.newDocumentBuilder().parse(new File(filepath)); XPathFactory xFactory = XPathFactory.newInstance(); XPath xpath = xFactory.newXPath(); XPathExpression expr = xpath.compile("//eventTree//description[contains(.,'Drzewo zdarzen')]"); Object result = expr.evaluate(doc,XPathConstants.NODESET); NodeList nodes = (NodeList)result; assertTrue(nodes.getLength() > 0); } @When("^I load an event tree from stored xml file$") public void iLoadAnEventTreeFromStoredXmlFile() throws Exception { newTree = EventTree.loadFromXML(filepath); } @And("^New events tree description should be \"([^\"]*)\"$") public void newEventsTreeDescriptionShouldBe(String description) throws Exception { assertEquals(description, newTree.getDescription()); } @And("^New event tree should be defined with string \"([^\"]*)\"$") public void newEventTreeShouldBeDefinedWithString(String eventTreeString) throws Exception { assertEquals(eventTreeString, newTree.toString()); } @But("^New event tree should not be defined with string \"([^\"]*)\"$") public void newEventTreeShouldNotBeDefinedWithString(String eventTreeString) throws Exception { assertFalse(eventTreeString.equals(newTree.toString())); } }
Runner file
File located in safetyanalysis/safety-eventstree/src/test/java/EventTreeAcceptanceTest.java
package pl.wroc.pwr.sa.et; import cucumber.api.CucumberOptions; import cucumber.api.junit.Cucumber; import org.junit.runner.RunWith; /** * Created by Patryk on 11.05.2017. */ @RunWith(Cucumber.class) @CucumberOptions( monochrome = true, format = { "pretty", "html:target/cucumber" }, glue = "pl.wroc.pwr.sa.et.steps", features = "classpath:cucumber/EventTree.feature" ) public class EventTreeAT { }
Test Results
After run our test we can see in console that it was finished:
Below we can see automatically generated report with results of this test. As we can see, everything is green-coloured, which means that test was completed successfully. In other case, red colour will fill all steps that wasn't passed.
This report is located in safetyanalysis/safety-eventstree/target/cucumber/index.html