Testing
To ensure reliability and stability of Safety Analysis a variety of tests are performed. We are using JUnit, Pi-test and other Java extensons connected with tests. Actually we started to introduce Cucumber as a new tool for acceptable tests.
During execution of all tests bugs and issues are found. Analysis of tests results is available here.
Acceptance tests
Software testing method conducted to determine if the requirements of a specification or contract are met. It may involve User Acceptance Testing (UAT), Operational Acceptance Testing (OAT), Acceptance testing in extreme programming, alpha and beta testing. You write acceptance tests to check if your code is passing the requirements of project. You should run these tests in integration-test phase. Naming param is *className*ATest.java.
Unit tests
Software testing method by which individual units of source code, sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures, are tested to determine whether they are fit for use.Unit testing is commonly automated, but may still be performed manually. The objective in unit testing is to isolate a unit and validate its correctness. You should write JUnit to check every function that you use in your code. JUnit must be fast, it should be matters of seconds to run that test. JUnit should test only define unit of system. Naming param is *className*JUTest.java.
Command Line Use Case Tests Specification
This page contains documentation for CLI Use Case Tests in order of appearance in code.
Test | Parser test - unrecognized argument |
---|---|
Description | This test checks GNU Parser for use of unrecognized argument. |
Conditions | Argument must be one of given in tables above. |
Test procedure |
|
Expected results | Output string: "Encountered exception while parsing using GnuParser:\nUnrecognized option: -l" |
Test | Analyze Fault Tree - no tree loaded |
Description | This test checks correctness of Fault Tree Analysis. |
Conditions | No tree loaded. |
Test procedure |
|
Expected results | Output string: "There is no tree that I could analyze." |
Test | Check the Cohesion of Event Tree - no tree loaded |
Description | This test checks cohesion of Event Tree. |
Conditions | No tree loaded. |
Test procedure |
|
Expected results | Output string: "There is no tree that I could analyze." |
Test | Get Fuzzy Importance Index of Fault Tree - no tree loaded |
Description | This test gets fuzzy importance index of fault tree. |
Conditions | No tree loaded. |
Test procedure |
|
Expected results | Output string: "There is no tree that I could analyze." |
Test | Load Event Tree - wrong tree |
Description | This test checks Event Tree loading from file. |
Conditions | - |
Test procedure |
|
Expected results | Output string: "\nRead Event Tree file from: wrong.xml Fail" |
Test | Load Fault Tree - wrong tree |
Description | This test checks Fault Tree loading from file. |
Conditions | - |
Test procedure |
|
Expected results | Output string: "\nRead Fault Tree file from: wrong.xml Fail" |
Test | Load Event Tree - correct tree |
Description | This test checks Event Tree loading from file. |
Conditions | - |
Test procedure |
|
Expected results | Output string: "\nRead Event Tree file from: treeXmltest2.xml Success" |
Test | Check Cohesion of Event Tree |
Description | This test checks cohesion of Event Tree. |
Conditions | Sucesfully load Event Tree from file. |
Test procedure |
|
Expected results | Output string: "\nRead Event Tree file from: treeXmltest2.xml Success\n"Result of the analysis Events Tree: \nEvents tree is ok" |
Test | Check Report generation |
Description | This test checks report generation. |
Conditions | - |
Test procedure |
|
Expected results | Output string: "\nCreated file: Report.odt" |
Test | Load Fault Tree - correct tree |
Description | This test checks Fault Tree loading from file. |
Conditions | - |
Test procedure |
|
Expected results | Output string: "\nRead Fault Tree file from: exampleFaultTree.xml Success" |
Test | Get Fuzzy Importance Index of Fault Tree |
Description | This test gets fuzzy importance index of fault tree. |
Conditions | Sucesfully loaded Fault Tree. |
Test procedure |
|
Expected results | Output string: "\nRead Fault Tree file from: exampleFaultTree.xml Success\nResult of the analysis Fault Tree: \nKey: root = Value: " |
Note: Our project configuration for cucumber tests for now is in /SA-99/Pluskota branch, but it should be merged in the feature.
Cucumber
Cucumber is a tool that we used for write acceptable tests. It runs automated acceptance tests written in a behavior-driven development (BDD) style. Cucumber defines application behavior using simple English text, defined by a language called Gherkin, which make tests easy to read and understood for peoples that doesn't have much experience with programming. Another advantage of this tests environment is, that it automatically generate report for the conducted test, which might help with analyzing tests results.
More informations about Cucumber are available in documentation.
Testing with cucumber
Note: Project is already prepared to work with Cucumber, so you don't have to configure this environment.
To make a executable test you have to prepare Feature,Steps Definition and Run files.
In our project Feature file should be located in safetyanalysis/moduleName/src/test/resources/cucumber/FileName.feature, Step Definition file in safetyanalysis/moduleName/src/test/java/steps/FileNameSteps.java and Run file in safetyanalysis/moduleName/src/test/java/FileName.java
Feature files
Feature file include whole description written in gherkin language. Using basic English language we can describe every single step a description of every single step of our test. It's include feature which is tested, concrete scenarios for this features and test steps.
Example of feature file:
Feature: Withdraw Money from ATM A user with an account at a bank would like to withdraw money from an ATM. Provided he has a valid account and debit or credit card, he should be allowed to make the transaction. The ATM will tend the requested amount of money, return his card, and subtract amount of the withdrawal from the user's account. Scenario: First scenario title Given preconditions When actions Then results Scenario: Second scenario title ...
As you can see in this example, in first line is written tested feature after Feature: keyword . Under feature name we can write detailed description of tested feature, where we can write how the whole situation looks like etc. This step is not required, but might help later with test analyzing. In 6th line we have name of first scenario after Scenario: keyword. Each scenario consist of steps, which have to be passed to finish test correctly.
Every step have to start in new line with one of the keywords:
Given - It describes the pre-requisite for the test to be executed. Example - GIVEN I am a Facebook user.
When - It defines the trigger point for any test scenario execution. Example − WHEN I enter "<username>".
Then - Then holds the expected result for the test to be executed. Example − THEN login should be successful.
And - It provides the logical AND condition between any two statements. AND can be used in conjunction with GIVEN, WHEN and THEN statement. Example − WHEN I enter my "<username>" AND I enter my "<password>".
- But - It signifies logical OR condition between any two statements. OR can be used in conjunction with GIVEN, WHEN and THEN statement. Example − THEN login should be successful. BUT home page should not be missing.
Using those keywords you can create any test what you want.
Step definitions file
File which description steps definitions i really important. Without him, steps in Feature file is just a words. Those steps in feature file are searched with regex and compared to definitions in step definition file. To make them work correctly, you should configure functions for each sentence that is used in Feature file. Each step, which is started by correct keyword (When, Then etc.) have to have code, how this step works.
Here's example code for step descriptions (first part is Feature code, second is step definitions):
/*------------------------------- Feature file ----------------------------*/ Feature: F1 Scenario: S1 Given first value 2 Given second value 3 When first value is 2 And second value is 3 Then multiplication result is 6
/*-------------------------- Step definitions file ------------------------*/ /*packages and imports*/ public class MathMull { @Before public void beforeScenario() { math = new MathMull; } @After public void afterScenario() { } @Given("^first value (-?\\d)$") public void setFirstValue(int var) throws Throwable { var1 = var; } @Given("^second value (-?\\d)$") public void setSecondValue(int var) throws Throwable { var2 = var; @When("^first value is (-?\\d)$") public void compareFirst(int var) throws Throwable { var1 == var; @And("^second value is (-?\\d)$") public void compareSecond(int var) throws Throwable { var2 == var; @Then("^multiplication result is (-?\\d)$") public void compareMultiplication(int var) throws Throwable { var1*var2 == var; } }
In this example you can see, how every steps from Feature file (upper part) have to be written in Step definition file (lower part). Even if you make the same function, but written with different words you will have to add another definition to step definition file. This is very important, because every missed step definition make that your test does not be complete, and in consequential will not work.
Another thing is, that in Step definition file, there are two steps that are not in Feature file, @Before and @After. Both are used to do things before and after tests. It might be helpful, if we have for example to create class which is require to tests.
You can see below more concrete example:
/*here are packages and imports*/ public class EventTreeAcceptanceTestSteps { private EventTree tree; private BufferedReader br; private String filepath; private EventTree newTree; @Before public void beforeScenario() { tree = new EventTree(); } @After public void afterScenario() { } @Given("^I have an event tree$") public void iHaveAnEventTree() throws Throwable { assertNotNull(tree); } @When("^I set its ID to (-?\\d)$") public void iSetItsIDTo(int ID) throws Throwable { tree.setId(ID); @Then("^The event tree should be defined with string \"([^\"]*)\"$") public void theEventTreeShouldBeDefinedWithString(String eventTreeString) throws Throwable { assertEquals(eventTreeString, tree.toString()); } @When("^I add an init event$") public void iAddAnInitEvent() throws Throwable { Event e1 = new Event(); tree.addInitEvent(e1); } (...) }
Run file
In Run file you have to set all options about tests, like which features are tested or where steps are located. There's few more options to configure but they are optional, the main thing of this file is that he making our test code working.
Example code for running tests:
package pl.wroc.pwr.sa.et; import cucumber.api.CucumberOptions; import cucumber.api.junit.Cucumber; import org.junit.runner.RunWith; @RunWith(Cucumber.class) @CucumberOptions( format = { "pretty", "html:target/cucumber" }, glue = "pl.wroc.pwr.sa.et.steps", features = "classpath:cucumber/EventTree.feature" ) public class EventTreeAcceptanceTest { }
To run our test all we have to do is just run our test
If everything was configured correct, we should got message about completed tests.
After completed tests Cucumber automatically generate report file, where you can check what was tested, or if something went wrong where was the problem. Report file is generated as file safetyanalysis/moduleName/target/cucumber/index.html.
Scenario outline - alternative scenario type
In some situations, when we want to test for example results of some mathematic function making the same scenario with different values doesn't make sense. Alternatively you can use Scenario outline in place of normal Scenario to test multiple values at the same scenario. All what you have to do is write tests where instead static values you have to write <variableName>. Later, after describing scenario all you have to do is make after Examples: keyword table, with values for each variant of this scenario. Here you have example of code which use this feature:
Feature: (...) Scenario Outline: A user withdraws money from an ATM Given <Name> has a valid Credit or Debit card And their account balance is <OriginalBalance> When they insert their card And withdraw <WithdrawalAmount> Then the ATM should return <WithdrawalAmount> And their account balance is <NewBalance> Examples: | Name | OriginalBalance | WithdrawalAmount | NewBalance | | Eric | 100 | 45 | 55 | | Pranav | 100 | 40 | 60 | | Ed | 1000 | 200 | 800 |
Example test
Here you can see example test of events tree class. As a scenario in events tree we'll test, does our events tree will create, modify, save and load correctly. Below is code for all three files, which allow to run correctly this test.
Feature file
File located in safetyanalysis/safety-eventstree/src/test/resources/cucumber/EventTree.feature
Feature: EventTree As a user I want to create an event tree with three events and save it to xml file So that I don't need to create it once again Scenario: Create an Event Tree, set its description, add three events and save it to XML file Given I have an event tree When I set its description to "Drzewo zdarzen" Then Its description should be "Drzewo zdarzen" When I set its ID to 5 And I add an init event Then The event tree should be defined with string "(5,1,1|0:0)" When I add a second event to tree Then The event tree should be defined with string "(5,2,3|0:0|1:1,2)" When I add a third event to tree Then The event tree should be defined with string "(5,3,7|0:0|1:1,2|2:3,4,5,6)" When I save my event tree to XML file called "eventTree.xml" Then XML file should be available When I load stored xml file Then File should not be empty And XML file should contain my event tree When I load an event tree from stored xml file Then A new event tree should be created And New events tree description should be "Drzewo zdarzen" And New event tree should be defined with string "(5,3,7|0:0|1:1,2|2:3,4,5,6)" But New event tree should not be defined with string "(5,2,3|0:0|1:1,2)"
Step Definition file
File located in safetyanalysis/safety-eventstree/src/test/java/steps/EventTreeAcceptanceTestSteps.java
package pl.wroc.pwr.sa.et.steps; import /*(...)*/ /** * Created by Patryk on 11.05.2017. */ public class EventTreeAcceptanceTestSteps { private EventTree tree; private BufferedReader br; private String filepath; private EventTree newTree; @Before public void beforeScenario() { tree = new EventTree(); } @After public void afterScenario() { } @Given("^I have an event tree$") public void iHaveAnEventTree() throws Throwable { assertNotNull(tree); } @When("^I set its ID to (-?\\d)$") public void iSetItsIDTo(int ID) throws Throwable { tree.setId(ID); } @Then("^The event tree should be defined with string \"([^\"]*)\"$") public void theEventTreeShouldBeDefinedWithString(String eventTreeString) throws Throwable { assertEquals(eventTreeString, tree.toString()); } @When("^I add an init event$") public void iAddAnInitEvent() throws Throwable { Event e1 = new Event(); tree.addInitEvent(e1); } @When("^I set its description to \"([^\"]*)\"$") public void iSetItsDescriptionTo(String description) throws Throwable { tree.setDescription(description); } @Then("^Its description should be \"([^\"]*)\"$") public void itsDescriptionShouldBe(String description) throws Throwable { assertEquals(description, tree.getDescription()); } @When("^I add a second event to tree$") public void iAddASecondEventToTree() throws Throwable { Event e2 = new Event(); tree.addEvent(e2); } @When("^I add a third event to tree$") public void iAddAThirdEventToTree() throws Throwable { Event e3 = new Event(); tree.addEvent(e3); } @When ("^I save my event tree to XML file called \"([^\"]*)\"$") public void iSaveMyEventTreeToXMLFileCalled(String filename) throws Throwable { EventTree.saveTreeToXML(filename, tree); filepath = filename; } @Then("^A new event tree should be created$") public void aNewEventTreeShouldBeCreated() throws Throwable { assertNotNull(newTree); } @Then("^XML file should be available$") public void xmlFileShouldBeAvailable() throws Throwable { File createdfile = new File(filepath); assertTrue(createdfile.exists()); } @When("^I load stored xml file$") public void iLoadStoredXmlFile() throws Throwable { br = new BufferedReader(new FileReader(filepath)); } @Then("^File should not be empty$") public void fileShouldNotBeEmpty() throws Throwable { assertNotNull(br.readLine()); br.close(); } @And("^XML file should contain my event tree$") public void xmlFileShouldContainMyEventTree() throws Throwable { DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance(); Document doc = factory.newDocumentBuilder().parse(new File(filepath)); XPathFactory xFactory = XPathFactory.newInstance(); XPath xpath = xFactory.newXPath(); XPathExpression expr = xpath.compile("//eventTree//description[contains(.,'Drzewo zdarzen')]"); Object result = expr.evaluate(doc,XPathConstants.NODESET); NodeList nodes = (NodeList)result; assertTrue(nodes.getLength() > 0); } @When("^I load an event tree from stored xml file$") public void iLoadAnEventTreeFromStoredXmlFile() throws Throwable { newTree = EventTree.loadFromXML(filepath); } @And("^New events tree description should be \"([^\"]*)\"$") public void newEventsTreeDescriptionShouldBe(String description) throws Throwable { assertEquals(description, newTree.getDescription()); } @And("^New event tree should be defined with string \"([^\"]*)\"$") public void newEventTreeShouldBeDefinedWithString(String eventTreeString) throws Throwable { assertEquals(eventTreeString, newTree.toString()); } @But("^New event tree should not be defined with string \"([^\"]*)\"$") public void newEventTreeShouldNotBeDefinedWithString(String eventTreeString) throws Throwable { assertFalse(eventTreeString.equals(newTree.toString())); } }
Run file
File located in safetyanalysis/safety-eventstree/src/test/java/EventTreeAcceptanceTest.java
package pl.wroc.pwr.sa.et; import cucumber.api.CucumberOptions; import cucumber.api.junit.Cucumber; import org.junit.runner.RunWith; /** * Created by Patryk on 11.05.2017. */ @RunWith(Cucumber.class) @CucumberOptions( monochrome = true, format = { "pretty", "html:target/cucumber" }, glue = "pl.wroc.pwr.sa.et.steps", features = "classpath:cucumber/EventTree.feature" ) public class EventTreeAcceptanceTest { }
Test Results
After run our test we can see in console that it was finished:
Below we can see automatically generated report with results of this test. As we can see, everything have green color, which means that test was completed successfully. In other case, red color will fill all steps that wasn't passed.
This report is located in safetyanalysis/safety-eventstree/target/cucumber/index.html