When `py.test` was run with a folder as argument, it would not only search for tests in that folder, but also create a .pytest-cache folder. Not only is this state we don't want, but it was also causing collisions.