Black Box Testing and JUnit

Objectives

In this laboratory, you will:

learn what black box testing is all about.
improve your debugging and black box testing skills.
be exposed to a way of implementing a stack 'from scratch' using Java.
learn the basics of using JUnit through Eclipse to test your Java implementation. JUnit provides a set of tools that make it easier to write test drivers for your code.

Before the Lab

Read over the black box testing primer below, as well as the rest of this lab.
Look over Stack.java.
Write down a series of black box tests that, if successfully run, would indicate a program implementing the Stack.java interface is working correctly.
Check out the JUnit site. Poke around the JavaDocs there and figure out the difference between the various Assert class methods.

Black box testing: A primer

In black box testing, we don't know (or pretend we don't know) the internal workings of the item under test. All we know (or can figure out) is what output should be generated given a series of inputs. We never examine the programming code, and only need the specifications of the program to do this type of testing. Black box testing has the following advantages [according to the webopedia]:

The test is unbiased because the designer and the tester are independent of each other.
The tester does not need knowledge of any specific programming language.
The test is done from the point of view of the user, not the designer.
Test cases can be designed as soon as the specifications are complete.

Although we can't always believe everything we read on the web, this is actually not a bad perspective. In our case, the code under test is in a file called StackImpl.java, but you should design your test cases without checking the code in that file. You should carefully examine Stack.java, which specifies what each method should do.

A Stack is a data structure that works a little bit like a pile of dishes: whenever you insert an element, it goes on top of the stack. When you retrieve an element from the stack, you always get the topmost element (that is, the latest element inserted and not yet deleted). Likewise, the only element you can delete from the stack is the element that is currently on top. The insertion operation is called push, the deletion operation is called pop, and the one that returns the element currently on top is called top.

For instance, here is a sequence of representations of a stack to which we have done the following operations (in this order): push(5), push(8), push(3), pop(), push (5), push(3), pop(), pop(), pop(), push(9), pop(), pop() and push (2).

                           3
            3         5    5    5
       8    8    8    8    8    8    8         9
  5    5    5    5    5    5    5    5    5    5    5    2

States

A state can be thought of as what condition an object is in. For example, your bathtub can be full of water or empty of water. Your stomach can be full or empty. Your TA's hair could be behaving or not behaving.

States do not have to be binary; an object can have multiple states. For example, your TA could be in a happy state, angry state, frustrated state, or a generally sorry state. Objects may have multiple concurrent states; perhaps at the moment your TA is frustrated AND has misbehaving hair (maybe your TA is frustrated because he/she has misbehaving hair; however we have no way to know that for sure).

When conducting a black box test, it is often useful to consider the different states the objects instantiated from a class might have. After reading through the methods of theStack class, we notice that it sometimes behaves differently if it is empty than when it is not. So, our stack has 2 states: empty and not-empty. We will want to test every method while the stack is in the empty state, and also while the stack is in the not-empty state.

Generally, it is useful to test 'in between' states, also known as boundary conditions. What state would be on or near the dividing line between an empty and a non-empty stack? Stop reading for a second and think about that. Write it down here: "A stack containing elements".

If you entered the number between zero and two above, you have got it. We have thus identified three states; a stack with no elements, with one element, and with two or more elements.

Now, let us consider that empty state. Are all empty states created equal? For example, is the empty state right after we initialize our stack the same as the state our stack is in when we add 3 elements and then remove them all? Well, it should be, but is it feasible that they would produce different results if something in the code was screwy? Well... yes. We will thus split the empty state into two states: 'empty initial' and 'empty non-initial'. We are up to 4 states now.

Ask yourself some more questions while doing this 'state splitting'. Would the stack act differently if there were 5 elements in it as compared to 80? How about 30? Think about it. Probably not. If you do not agree with this statement, by all means, add another state. It is up to you to decide what states you need.

Generally, you will want to test each method once for every single state that you identify.

Properties

Let us consider now what a stack is. It is a collection that guarantees last in first out (LIFO) semantics. We should test that, too. Perhaps we should add 3 or 4 elements to our stack and then pop them off one by one and make sure the ordering is right. Are there any other properties of a stack? If you can think of any, test them as well.

Special Cases

Let us say that I was in a really bad mood and wanted to make my friend's program crash. What would I do? This sort of thing comes with experience, but you may be able to come up with some additional tests to make sure this stack can handle any valid input. Does it say anywhere in the specification that you can not add a new stack as an element in another stack? How about adding a null object to the stack? How about zeros, negative numbers, fractions, pi, or your Uncle Floyd's photo album? Choose cases that are valid in regards to the specification, but which you feel the programmer may have overlooked.

Write down your testing scheme

Congratulations! You are well on your way to becoming a proficient black box tester.

Another resource for black box testing

If you are interested in learning more about black box testing, want to see a different example, or are hankering for a more formal definition and algorithm than presented above, please refer to the slides discussed in class.

During the Lab

1. Getting Started

Create the directory ~/cs211/labs/junit
Download Stack.java and StackImpl.java and save them to the new directory you created. Do NOT look at the code for StackImpl until you have written and run tests for every method. The idea behind blackbox testing is that you do not have access to the code, so try to find the three bugs in StackImpl based only on your test results, and then fix the bugs only AFTER you have found ALL three of them.
Create a new project in Eclipse called JUnitLab. Choose Create project from existing source and fill in the directory field with ~/cs211/labs/junit.

Note: this lab assumes that you are working with a version of Eclipse which includes the JUnit plugin. This plugin greatly facilitates using JUnit within Eclipse, is included in all versions of Eclipse 3.0 or greater, and is installed in the lab. If you do not have the JUnit plugin at home, you should upgrade to a newer version of Eclipse.

2. Creating a JUnit Test Case

Instead of writing a test driver, with JUnit you create a class and write tests as methods of the class. You use annotations to provide instructions to JUnit. A public void method annotated with the @Test tag can be run by JUnit as a test case. Instead of using print statements for output, JUnit tests call various assert functions to compare expected and actual results.

Eclipse has built-in support for writing unit tests with the JUnit framework. For instance, to create a new test case that tests the StackImpl class:

Select your JUnitLab project and then, from the menu bar, select File->New->JUnit Test Case.
The New JUnit Test Case dialog should pop up. Ensure that the "New JUnit 4 test" radio button is selected. Near the bottom of the dialog may be a warning announcing that JUnit 4 is not on the build path of the project. Click on the "click here" link to add JUnit 4 to your project, and then click "OK" in the dialog window that opens. You should see the JUnit 4 library added to your project.
We're now back to the New JUnit Test Case dialog. Enter StackTest under the Name field.
Under Class Under Test click on Browse. In the Class Under Test window, enter StackImpl. The bottom pane will show 2 options: one that mentions "corba" and one that mentions the default package - select the default package one and click 'OK'.
Under method stubs, ensure everything is unchecked. Click Next >.
Now, click on the makeEmpty() method to have Eclipse automatically generate the appropriate code stub. Click Finish to create the class.
Open StackTest.java if Eclipse has not already opened it for you. In the testMakeEmpty() method stub, delete any of Eclipse's auto-generated code, add the following code, and save the file:

Stack testStack = new StackImpl();
testStack.push(new Integer(3));
testStack.push(new Integer(4));
testStack.makeEmpty();
assertEquals(0, testStack.size());

Method testMakeEmpty() pushes two items onto testStack and then uses makeEmpty() to empty testStack. The call to assertEquals() will generate a failure if testStack.size() (the actual value) is not equal to zero (the expected value). For more information on assert comparisons, see the Assert class in JUnit's Javadoc.

3. The JUnit View

In Eclipse, JUnit test classes do not need a main() method to run because the test classes are run in a special JUnit mode. When running in JUnit mode the output/results of the test run are reported via the JUnit view (you will see how to open it in section 3.1).
In the JUnit view there are two panes. In the Hierarchy pane (at the top of the JUnit window) the hierarchy of the tests being run are displayed. The Failure Trace pane (at the bottom of the JUnit window) displays information about the particular test case failure selected in the hierarchy pane.

3.1 Running StackTest

Select StackTest.java in Package Explorer, then Run -> Run As...-> JUnit Test. Remember that because StackTest has no main() it cannot be run as a Java Application. The JUnit view will open, sharing space with Package Explorer.
No failures are reported because size() returned zero.

3.2 Failure Traces

To demonstrate the failure tracing abilities of JUnit, add assertTrue(false); to testMakeEmpty(), and then save your changes. This guarantees that a failure will occur the next time we run the test.
Re-run StackTest. Under the Failures tab of JUnit, the test case testMakeEmpty is listed. Double click on the item to jump to the corresponding code for testMakeEmpty() in the text editor (you are probably already there unless you closed the file since the previous step).
Statements that contributed to the failure of the test case are listed in the Failure Trace window. When you double click on items in the Failure Trace that refer to lines in your code, the text editor will go to those statements.
Delete the line assertTrue(false); to get rid of this failure.

3.3. Sharing Fixtures

A Fixture is the set of objects that a test case uses. When test cases use a set of common objects, we should share them to avoid rewriting the same code multiple times. We can share fixtures by creating a method annotated with a @Before tag, and placing initialization code for shared objects (or all objects) there. JUnit recognizes a method annotated with the @Before tag as a special method and runs it prior to each test case method. As a result, the values of the variables initialized in such method are reset after every test.

Create a method setUp() in StackTest, annotate it with the @Before tag, and move the initialization code for testStack (i.e. testStack = new StackImpl();) from the test case methods to setUp() (you only need one initialization statement per object). Declare Stack testStack; at the beginning of the class, giving it class scope so that every test method can use it. The beginning of your code should look something like this:
```
import static org.junit.Assert.*;
import org.junit.*;
 
public class StackTest 
{ 
   private Stack testStack;

   @Before
   public void setUp()     
   {         
      testStack = new StackImpl();     
   } 

   @Test
   public void testMakeEmpty() 
   {     	
      testStack.push(new Integer(3));     
      testStack.push(new Integer(4));
      testStack.makeEmpty();
      assertEquals(0, testStack.size());     
   } ... 
}
```
There may be a syntax error displayed on the @Before line. Click on the red x and select "import Before (org.junit)". This will import the required annotation. Another option is to get rid of the "import org.junit.Test;" statement in your code and replace it with "import org.junit.*;" (as we have done in the sample code above).
There should be no failures when you run your code.

Write a simple test case for isEmpty() in StackTest called testIsEmpty. Share testIsEmpty()'s fixture. (Hint: Check if a newly created stack is empty). Run your tests and check if there are any failures for this method. For any failure you discover, make the necessary corrections and re-run the tests again until all tests succeed.

4 Fixing StackImpl

Write a test case named testPop(). It should check that popping a value off of a newly created stack returns null. Then it should try pushing a couple of new integer objects onto testStack, and then popping them off. After every pop(), you need to compare the stack contents to the expected values.
Add additional test cases to give the stack implementation a good workout. There are three very distinct bugs within the StackImpl class; you should be convinced that your testing has identified where all three bugs occur. You can assume that StackImpl's size() and toString() methods are bug free (as these are trivial to implement) and work from there.
Once you have identified the location of the three bugs, open StackImpl.java and correct them.
After fixing the suspected bugs, run the JUnit test once more to make sure that everything is running correctly.

5. Sanity Check

PLEASE DON'T READ THIS SECTION UNTIL YOU THINK YOU ARE COMPLETELY DONE WITH THE LAB

If you made it all the way down here without ever adding several items to the stack and checking that they were added successfully, you probably have not found all the errors inStackImpl. Try pushing 3 items on to the stack, and then popping them off, one at a time. During each pop, check to make sure that the item returned is what you expected. Consider reading the Primer again.

We all know that TAs are your friends, but in this case think of your TA as an evil adversary. Is there anything the TA could come up with that would make your program crash and/or malfunction? Some personal favourites of mine include pushing null onto a stack and calling pop on an empty stack. Can your implementation handle this? What about other things your evil adversary could come up with?

When you feel that your stack implementation is as robust as it is going to get, call over that evil TA of yours, and show him/her that no one can touch your testing and debugging skills.

6. Can't get enough JUnit?

There is a huge amount of information on the JUnit site http://junit.sourceforge.net/. What you learned so far would only be useful for testing small programs. As you add more and more tests, your test case would quickly become disorganized and unmanagable. You would really want more than one test case to help clean things up. Conveniently, JUnit allows us to group test cases together into test suites. JUnit can also be used outside of the Eclipse environment. More details on test suites and using JUnit outside of Eclipse can be found here. Feel free to play around a bit.

	Black Box Testing and JUnit
	Objectives In this laboratory, you will: learn what black box testing is all about. improve your debugging and black box testing skills. be exposed to a way of implementing a stack 'from scratch' using Java. learn the basics of using JUnit through Eclipse to test your Java implementation. JUnit provides a set of tools that make it easier to write test drivers for your code. Before the Lab Read over the black box testing primer below, as well as the rest of this lab. Look over Stack.java. Write down a series of black box tests that, if successfully run, would indicate a program implementing the Stack.java interface is working correctly. Check out the JUnit site. Poke around the JavaDocs there and figure out the difference between the various Assert class methods. Black box testing: A primer In black box testing, we don't know (or pretend we don't know) the internal workings of the item under test. All we know (or can figure out) is what output should be generated given a series of inputs. We never examine the programming code, and only need the specifications of the program to do this type of testing. Black box testing has the following advantages [according to the webopedia]: The test is unbiased because the designer and the tester are independent of each other. The tester does not need knowledge of any specific programming language. The test is done from the point of view of the user, not the designer. Test cases can be designed as soon as the specifications are complete. Although we can't always believe everything we read on the web, this is actually not a bad perspective. In our case, the code under test is in a file called StackImpl.java, but you should design your test cases without checking the code in that file. You should carefully examine Stack.java, which specifies what each method should do. A Stack is a data structure that works a little bit like a pile of dishes: whenever you insert an element, it goes on top of the stack. When you retrieve an element from the stack, you always get the topmost element (that is, the latest element inserted and not yet deleted). Likewise, the only element you can delete from the stack is the element that is currently on top. The insertion operation is called push, the deletion operation is called pop, and the one that returns the element currently on top is called top. For instance, here is a sequence of representations of a stack to which we have done the following operations (in this order): push(5), push(8), push(3), pop(), push (5), push(3), pop(), pop(), pop(), push(9), pop(), pop() and push (2). 3 3 5 5 5 8 8 8 8 8 8 8 9 5 5 5 5 5 5 5 5 5 5 5 2 States A state can be thought of as what condition an object is in. For example, your bathtub can be full of water or empty of water. Your stomach can be full or empty. Your TA's hair could be behaving or not behaving. States do not have to be binary; an object can have multiple states. For example, your TA could be in a happy state, angry state, frustrated state, or a generally sorry state. Objects may have multiple concurrent states; perhaps at the moment your TA is frustrated AND has misbehaving hair (maybe your TA is frustrated because he/she has misbehaving hair; however we have no way to know that for sure). When conducting a black box test, it is often useful to consider the different states the objects instantiated from a class might have. After reading through the methods of theStack class, we notice that it sometimes behaves differently if it is empty than when it is not. So, our stack has 2 states: empty and not-empty. We will want to test every method while the stack is in the empty state, and also while the stack is in the not-empty state. Generally, it is useful to test 'in between' states, also known as boundary conditions. What state would be on or near the dividing line between an empty and a non-empty stack? Stop reading for a second and think about that. Write it down here: "A stack containing elements". If you entered the number between zero and two above, you have got it. We have thus identified three states; a stack with no elements, with one element, and with two or more elements. Now, let us consider that empty state. Are all empty states created equal? For example, is the empty state right after we initialize our stack the same as the state our stack is in when we add 3 elements and then remove them all? Well, it should be, but is it feasible that they would produce different results if something in the code was screwy? Well... yes. We will thus split the empty state into two states: 'empty initial' and 'empty non-initial'. We are up to 4 states now. Ask yourself some more questions while doing this 'state splitting'. Would the stack act differently if there were 5 elements in it as compared to 80? How about 30? Think about it. Probably not. If you do not agree with this statement, by all means, add another state. It is up to you to decide what states you need. Generally, you will want to test each method once for every single state that you identify. Properties Let us consider now what a stack is. It is a collection that guarantees last in first out (LIFO) semantics. We should test that, too. Perhaps we should add 3 or 4 elements to our stack and then pop them off one by one and make sure the ordering is right. Are there any other properties of a stack? If you can think of any, test them as well. Special Cases Let us say that I was in a really bad mood and wanted to make my friend's program crash. What would I do? This sort of thing comes with experience, but you may be able to come up with some additional tests to make sure this stack can handle any valid input. Does it say anywhere in the specification that you can not add a new stack as an element in another stack? How about adding a null object to the stack? How about zeros, negative numbers, fractions, pi, or your Uncle Floyd's photo album? Choose cases that are valid in regards to the specification, but which you feel the programmer may have overlooked. Write down your testing scheme Congratulations! You are well on your way to becoming a proficient black box tester. Another resource for black box testing If you are interested in learning more about black box testing, want to see a different example, or are hankering for a more formal definition and algorithm than presented above, please refer to the slides discussed in class. During the Lab 1. Getting Started Create the directory ~/cs211/labs/junit Download Stack.java and StackImpl.java and save them to the new directory you created. Do NOT look at the code for StackImpl until you have written and run tests for every method. The idea behind blackbox testing is that you do not have access to the code, so try to find the three bugs in StackImpl based only on your test results, and then fix the bugs only AFTER you have found ALL three of them. Create a new project in Eclipse called JUnitLab. Choose Create project from existing source and fill in the directory field with ~/cs211/labs/junit. Note: this lab assumes that you are working with a version of Eclipse which includes the JUnit plugin. This plugin greatly facilitates using JUnit within Eclipse, is included in all versions of Eclipse 3.0 or greater, and is installed in the lab. If you do not have the JUnit plugin at home, you should upgrade to a newer version of Eclipse. 2. Creating a JUnit Test Case Instead of writing a test driver, with JUnit you create a class and write tests as methods of the class. You use annotations to provide instructions to JUnit. A public void method annotated with the @Test tag can be run by JUnit as a test case. Instead of using print statements for output, JUnit tests call various assert functions to compare expected and actual results. Eclipse has built-in support for writing unit tests with the JUnit framework. For instance, to create a new test case that tests the StackImpl class: Select your JUnitLab project and then, from the menu bar, select File->New->JUnit Test Case. The New JUnit Test Case dialog should pop up. Ensure that the "New JUnit 4 test" radio button is selected. Near the bottom of the dialog may be a warning announcing that JUnit 4 is not on the build path of the project. Click on the "click here" link to add JUnit 4 to your project, and then click "OK" in the dialog window that opens. You should see the JUnit 4 library added to your project. We're now back to the New JUnit Test Case dialog. Enter StackTest under the Name field. Under Class Under Test click on Browse. In the Class Under Test window, enter StackImpl. The bottom pane will show 2 options: one that mentions "corba" and one that mentions the default package - select the default package one and click 'OK'. Under method stubs, ensure everything is unchecked. Click Next >. Now, click on the makeEmpty() method to have Eclipse automatically generate the appropriate code stub. Click Finish to create the class. Open StackTest.java if Eclipse has not already opened it for you. In the testMakeEmpty() method stub, delete any of Eclipse's auto-generated code, add the following code, and save the file: Stack testStack = new StackImpl(); testStack.push(new Integer(3)); testStack.push(new Integer(4)); testStack.makeEmpty(); assertEquals(0, testStack.size()); Method testMakeEmpty() pushes two items onto testStack and then uses makeEmpty() to empty testStack. The call to assertEquals() will generate a failure if testStack.size() (the actual value) is not equal to zero (the expected value). For more information on assert comparisons, see the Assert class in JUnit's Javadoc. 3. The JUnit View In Eclipse, JUnit test classes do not need a main() method to run because the test classes are run in a special JUnit mode. When running in JUnit mode the output/results of the test run are reported via the JUnit view (you will see how to open it in section 3.1). In the JUnit view there are two panes. In the Hierarchy pane (at the top of the JUnit window) the hierarchy of the tests being run are displayed. The Failure Trace pane (at the bottom of the JUnit window) displays information about the particular test case failure selected in the hierarchy pane. 3.1 Running StackTest Select StackTest.java in Package Explorer, then Run -> Run As...-> JUnit Test. Remember that because StackTest has no main() it cannot be run as a Java Application. The JUnit view will open, sharing space with Package Explorer. No failures are reported because size() returned zero. 3.2 Failure Traces To demonstrate the failure tracing abilities of JUnit, add assertTrue(false); to testMakeEmpty(), and then save your changes. This guarantees that a failure will occur the next time we run the test. Re-run StackTest. Under the Failures tab of JUnit, the test case testMakeEmpty is listed. Double click on the item to jump to the corresponding code for testMakeEmpty() in the text editor (you are probably already there unless you closed the file since the previous step). Statements that contributed to the failure of the test case are listed in the Failure Trace window. When you double click on items in the Failure Trace that refer to lines in your code, the text editor will go to those statements. Delete the line assertTrue(false); to get rid of this failure. 3.3. Sharing Fixtures A Fixture is the set of objects that a test case uses. When test cases use a set of common objects, we should share them to avoid rewriting the same code multiple times. We can share fixtures by creating a method annotated with a @Before tag, and placing initialization code for shared objects (or all objects) there. JUnit recognizes a method annotated with the @Before tag as a special method and runs it prior to each test case method. As a result, the values of the variables initialized in such method are reset after every test. Create a method setUp() in StackTest, annotate it with the @Before tag, and move the initialization code for testStack (i.e. testStack = new StackImpl();) from the test case methods to setUp() (you only need one initialization statement per object). Declare Stack testStack; at the beginning of the class, giving it class scope so that every test method can use it. The beginning of your code should look something like this: import static org.junit.Assert.; import org.junit.; public class StackTest { private Stack testStack; @Before public void setUp() { testStack = new StackImpl(); } @Test public void testMakeEmpty() { testStack.push(new Integer(3)); testStack.push(new Integer(4)); testStack.makeEmpty(); assertEquals(0, testStack.size()); } ... } There may be a syntax error displayed on the @Before line. Click on the red x and select "import Before (org.junit)". This will import the required annotation. Another option is to get rid of the "import org.junit.Test;" statement in your code and replace it with "import org.junit.;" (as we have done in the sample code above). There should be no failures when you run your code. Write a simple test case for isEmpty() in StackTest called testIsEmpty. Share testIsEmpty()'s fixture. (Hint: Check if a newly created stack is empty). Run your tests and check if there are any failures for this method. For any failure you discover, make the necessary corrections and re-run the tests again until all tests succeed. 4 Fixing StackImpl Write a test case named testPop(). It should check that popping a value off of a newly created stack returns null. Then it should try pushing a couple of new integer objects onto testStack, and then popping them off. After every pop(), you need to compare the stack contents to the expected values. Add additional test cases to give the stack implementation a good workout. There are three very distinct bugs within the StackImpl class; you should be convinced that your testing has identified where all three bugs occur. You can assume that StackImpl's size() and toString() methods are bug free (as these are trivial to implement) and work from there. Once you have identified the location of the three bugs, open StackImpl.java and correct them. After fixing the suspected bugs, run the JUnit test once more to make sure that everything is running correctly. 5. Sanity Check PLEASE DON'T READ THIS SECTION UNTIL YOU THINK YOU ARE COMPLETELY DONE WITH THE LAB* If you made it all the way down here without ever adding several items to the stack and checking that they were added successfully, you probably have not found all the errors inStackImpl. Try pushing 3 items on to the stack, and then popping them off, one at a time. During each pop, check to make sure that the item returned is what you expected. Consider reading the Primer again. We all know that TAs are your friends, but in this case think of your TA as an evil adversary. Is there anything the TA could come up with that would make your program crash and/or malfunction? Some personal favourites of mine include pushing null onto a stack and calling pop on an empty stack. Can your implementation handle this? What about other things your evil adversary could come up with? When you feel that your stack implementation is as robust as it is going to get, call over that evil TA of yours, and show him/her that no one can touch your testing and debugging skills. 6. Can't get enough JUnit? There is a huge amount of information on the JUnit site http://junit.sourceforge.net/. What you learned so far would only be useful for testing small programs. As you add more and more tests, your test case would quickly become disorganized and unmanagable. You would really want more than one test case to help clean things up. Conveniently, JUnit allows us to group test cases together into test suites. JUnit can also be used outside of the Eclipse environment. More details on test suites and using JUnit outside of Eclipse can be found here. Feel free to play around a bit.

	©University of British Columbia	Last updated: June 21, 2007 14:43