Mockito Verify Inconsistency

When I initially set out to blog about Mockito I wanted to write a post where I explained the Verify API. Though there wasn't much in the realm of posts around this topic, I think that the javadocs have ample examples. Maybe the only topic that would be worth covering for a second post would be good places to use Verify. I recently have been working a lot with the Verify API and found some inconsistencies that I wanted to share so that if you are using Mockito you can avoid the pain I had. So here it is two weeks later and here is my post about some caveats while using the Verify API.

The Setup

I can't exactly share the original problem classes I had, yet, so instead I have recreated my problem in a set of examples. First, let's assume we have an interface called CallbackB (original name, I know) and it takes SomeModel as a parameter that we will say is only made up of a String.

public interface CallbackB{
        public void someMethod(SomeModel model);
    }

    public class SomeModel{
        private String text;

        public SomeModel(String text){
            this.text = text;
        }
    }

This interface defines the signatures for any class implementing it with the method someMethod. Now let's say we have the following class that uses CallbackB and we will call it ClassA (again, super original):

public class ClassA {

        private CallbackB mCallbacks;

        public ClassA(CallbackB callbacks){
            mCallbacks = callbacks;
        }

        public void doSomething(String text){
            if(text.contains("Artoo")) {
                mCallbacks.someMethod(new SomeModel(text));
            }
        }
    }

You can see that ClassA expects CallbackB to be passed to it from the constructor, but I wouldn't be too focused on that, the important part is the usage of the method doSomething.

You will notice that doSomething takes a string and if the string contains the text Artoo (my dogs name... sorry, only thing that came to mind as an example string) then we will call someMethod on our callbacks which is CallbackB.

Ok, so that’s the setup. We have a class which will act as callbacks for us, as well as a class doing some business logic which triggers the callback.

Testing

So now let's say we want to test that our callback is called at the correct time and for the correct reasons. In our case, because we are writing an interface that we expect someone else to utilize to listen for things happening in our code we don't care as much about testing what happens when the concrete versions of the interface are called, but the fact that they are called. To do this we can utilize Mockito and a tool through it called verify. If you have never used it I would highly suggest it. I will link to the docs for it but I don't want to spend time talking too much about how to use it. I may save that for another post if people actually want that.

So to test our interface our test could look something like this:

@Test
    public void doSomethingIsCalledOnlyOnceAny(){

      CallbackB mockedCallbackB = mock(CallbackB.class);
      ClassA classA = new ClassA(mockedCallbackB);
      String testStringHappy = "Artoo rocks!";
      String testStringSad = "Rocks!";

      classA.doSomething(testStringHappy);
      classA.doSomething(testStringSad);

      verify(mockedCallbackB).someMethod(any(SomeModel.class));

    }

Hopefully, this test is straightforward, basically we want to verify that someMethod is called only once and we don't care what the parameter passed to it is. This test will pass because of the way we built ClassA, however if we take out:

if(text.contains("Artoo")) {
    }

From the mix the test will fail because Mockito expects that the number of calls for that specific method should only be one. Now comes the oddity. let's say we use a custom argument matcher and also let's assume SomeModel has a getter to get the string. Here is what our ArgumentMatcher could look like:

class IsValidSomeModel extends ArgumentMatcher<SomeModel> {
        private String mExpected;
        public IsValidSomeModel(String expected){
            mExpected = expected;
        }
        public boolean matches(Object object) {
            String actual = ((SomeModel) object).getText();
            return actual.contains(mExpected);
        }
    }

Now that we have created the ArgumentMatcher the new test should look like this:

@Test
    public void doSomethingIsCalledOnlyOnceSpecific(){

      CallbackB mockedCallbackB = mock(CallbackB.class);
      ClassA classA = new ClassA(mockedCallbackB);
      String testStringHappy = "Artoo rocks!";
      String testStringSad = "Rocks!";

      classA.doSomething(testStringHappy);
      classA.doSomething(testStringSad);

      verify(mockedCallbackB).someMethod(argThat(new IsValidSomeModel("Artoo"));

    }

False Positive

So now we re-run the test suite. The first test will fail with the following error:

org.mockito.exceptions.verification.TooManyActualInvocations:callbackB.someMethod(<any>);
    Wanted 1 time:-> at com.omitneedlesscode.mockitoverifyexample.VerifyTests.doSomethingIsCalledOnlyOnceAny(VerifyTests.java:26)
    But was 2 times. Undesired invocation:-> at com.omitneedlesscode.mockitoverifyexample.ClassA.doSomething(ClassA.java:14)

However, the new test, mysteriously passes. Queue Twilight Zone music. I am still unsure why it is passing. However, I have come up with a temporary solution to the problem.

First and foremost, I would suggest not using the ArgumentMatcher while using verify. Instead I would suggest using the ArgumentCaptor it allows you to do assertions directly on the parameters themselves and seems to not be ailed by the mysterious passing test issue.

So let's say we want to go down this route. Here is a test with the ArgumentCaptor that will now properly fail due to too many invocations:

@Test
    public void doSomethingIsCalledOnlyOnceSpecificCaptor() {
                ArgumentCaptor<SomeModel> argument = ArgumentCaptor.forClass(SomeModel.class);
                CallbackB mockedCallbackB = mock(CallbackB.class);
                ClassA classA = new ClassA(mockedCallbackB);
                String testStringHappy = "Artoo rocks!";
                String testStringSad = "Rocks!";
                String expected  = "Artoo";

                classA.doSomething(testStringHappy);
                classA.doSomething(testStringSad);

                verify(mockedCallbackB).someMethod(argument.capture());

                assertTrue(argument.getValue().getText().contains(expected));
        }

So hopefully if you are using Mockito, or thinking about using it, this helps make things clear. The Verify API is a great asset in your testing toolbox, but be weary of some inconsistencies. Use ArgumentCaptors instead of ArgumentMatchers and don’t give up on tests because you can’t get them to run. The work you put in will save you regressions to your code, new bugs and will always make your code better.

Elliott Chenger