Clash Royale CLAN TAG#URR8PPP

How do I run all Python unit tests in a directory?

I have a directory that contains my Python unit tests. Each unit test module is of the form test_*.py. I am attempting to make a file called all_test.py that will, you guessed it, run all files in the aforementioned test form and return the result. I have tried two methods so far; both have failed. I will show the two methods, and I hope someone out there knows how to actually do this correctly.

For my first valiant attempt, I thought "If I just import all my testing modules in the file, and then call this unittest.main() doodad, it will work, right?" Well, turns out I was wrong.

unittest.main()

import glob
import unittest

testSuite = unittest.TestSuite()
test_file_strings = glob.glob('test_*.py')
module_strings = [str[0:len(str)-3] for str in test_file_strings]

if __name__ == "__main__":
unittest.main()


This did not work, the result I got was:

$ python all_test.py

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK


For my second try, I though, ok, maybe I will try to do this whole testing thing in a more "manual" fashion. So I attempted to do that below:

import glob
import unittest

testSuite = unittest.TestSuite()
test_file_strings = glob.glob('test_*.py')
module_strings = [str[0:len(str)-3] for str in test_file_strings]
[__import__(str) for str in module_strings]
suites = [unittest.TestLoader().loadTestsFromName(str) for str in module_strings]
[testSuite.addTest(suite) for suite in suites]
print testSuite

result = unittest.TestResult()
testSuite.run(result)
print result

#Ok, at this point I have a result
#How do I display it as the normal unit test command line output?
if __name__ == "__main__":
unittest.main()


This also did not work, but it seems so close!

$ python all_test.py
<unittest.TestSuite tests=[<unittest.TestSuite tests=[<unittest.TestSuite tests=[<test_main.TestMain testMethod=test_respondes_to_get>]>]>]>
<unittest.TestResult run=1 errors=0 failures=0>

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK


I seem to have a suite of some sort, and I can execute the result. I am a little concerned about the fact that it says I have only run=1, seems like that should be run=2, but it is progress. But how do I pass and display the result to main? Or how do I basically get it working so I can just run this file, and in doing so, run all the unit tests in this directory?

run=1

run=2

14 Answers
14

You could use a test runner that would do this for you. nose is very good for example. When run, it will find tests in the current tree and run them.

Updated:

Here's some code from my pre-nose days. You probably don't want the explicit list of module names, but maybe the rest will be useful to you.

testmodules = [
'cogapp.test_makefiles',
'cogapp.test_whiteutils',
'cogapp.test_cogapp',
]

suite = unittest.TestSuite()

for t in testmodules:
try:
# If the module defines a suite() function, call it to get the suite.
mod = __import__(t, globals(), locals(), ['suite'])
suitefn = getattr(mod, 'suite')
suite.addTest(suitefn())
except (ImportError, AttributeError):
# else, just load all the test cases from the module.
suite.addTest(unittest.defaultTestLoader.loadTestsFromName(t))

unittest.TextTestRunner().run(suite)


With Python 2.7 and higher you don't have to write new code or use third-party tools to do this; recursive test execution via the command line is built-in.

python -m unittest discover <test_directory>
# or
python -m unittest discover -s <directory> -p '*_test.py'


You can read more in the python 2.7
or python 3.x unittest documentation.

This is now possible directly from unittest: unittest.TestLoader.discover.

import unittest
loader = unittest.TestLoader()
start_dir = 'path/to/your/test/files'
suite = loader.discover(start_dir)

runner = unittest.TextTestRunner()
runner.run(suite)


.... ---------------------------------------------------------------------- Ran 4 tests in 0.000s OK

Well by studying the code above a bit (specifically using TextTestRunner and defaultTestLoader), I was able to get pretty close. Eventually I fixed my code by also just passing all test suites to a single suites constructor, rather than adding them "manually", which fixed my other problems. So here is my solution.

TextTestRunner

defaultTestLoader

import glob
import unittest

test_files = glob.glob('test_*.py')
module_strings = [test_file[0:len(test_file)-3] for test_file in test_files]
suites = [unittest.defaultTestLoader.loadTestsFromName(test_file) for test_file in module_strings]
test_suite = unittest.TestSuite(suites)
test_runner = unittest.TextTestRunner().run(test_suite)


Yeah, it is probably easier to just use nose than to do this, but that is besides the point.

If you want to run all the tests from various test case classes and you're happy to specify them explicitly then you can do it like this:

from unittest import TestLoader, TextTestRunner, TestSuite
from uclid.test.test_symbols import TestSymbols
from uclid.test.test_patterns import TestPatterns

if __name__ == "__main__":

loader = TestLoader()
tests = [
loader.loadTestsFromTestCase(test)
for test in (TestSymbols, TestPatterns)
]
suite = TestSuite(tests)

runner = TextTestRunner(verbosity=2)
runner.run(suite)


where uclid is my project and TestSymbols and TestPatterns are subclasses of TestCase.

uclid

TestSymbols

TestPatterns

TestCase

TestSuite

loader.loadTestsFromTestCase

In python 3, if you're using unittest.TestCase and you have an empty (or otherwise) __init__.py file in your test directory, then you can run all the tests with

unittest.TestCase

__init__.py

python -m unittest


Done! A solution less than 100 lines. Hopefully another python beginner saves time by finding this.

I have used the discover method and an overloading of load_tests to achieve this result in a (minimal, I think) number lines of code:

discover

load_tests

def load_tests(loader, tests, pattern):
''' Discover and load all unit tests in all files named ``*_test.py`` in ``./src/``
'''
suite = TestSuite()
for all_test_suite in unittest.defaultTestLoader.discover('src', pattern='*_tests.py'):
for test_suite in all_test_suite:
suite.addTests(test_suite)
return suite

if __name__ == '__main__':
unittest.main()


Execution on fives something like

Ran 27 tests in 0.187s
OK


I tried various approaches but all seem flawed or I have to makeup some code, that's annoying. But there's a convinient way under linux, that is simply to find every test through certain pattern and then invoke them one by one.

find . -name 'Test*py' -exec python '' ;


and most importantly, it definitely works.

In case of a packaged library or application, you don't want to do it. setuptools will do it for you.

setuptools

To use this command, your project’s tests must be wrapped in a unittest test suite by either a function, a TestCase class or method, or a module or package containing TestCase classes. If the named suite is a module, and the module has an additional_tests() function, it is called and the result (which must be a unittest.TestSuite) is added to the tests to be run. If the named suite is a package, any submodules and subpackages are recursively added to the overall test suite.

unittest

TestCase

additional_tests()

unittest.TestSuite

Just tell it where your root test package is, like:

setup(
# ...
test_suite = 'somepkg.test'
)


And run python setup.py test.

python setup.py test

File-based discovery may be problematic in Python 3, unless you avoid relative imports in your test suite, because discover uses file import. Even though it supports optional top_level_dir, but I had some infinite recursion errors. So a simple solution for a non-packaged code is to put the following in __init__.py of your test package (see load_tests Protocol).

discover

top_level_dir

__init__.py

import unittest

from . import foo, bar


def load_tests(loader, tests, pattern):
suite = unittest.TestSuite()
suite.addTests(loader.loadTestsFromModule(foo))
suite.addTests(loader.loadTestsFromModule(bar))

return suite


I use PyDev/LiClipse and haven't really figured out how to run all tests at once from the GUI. (edit: you right click the root test folder and choose Run as -> Python unit-test

Run as -> Python unit-test

This is my current workaround:

import unittest

def load_tests(loader, tests, pattern):
return loader.discover('.')

if __name__ == '__main__':
unittest.main()


I put this code in a module called all in my test directory. If I run this module as a unittest from LiClipse then all tests are run. If I ask to only repeat specific or failed tests then only those tests are run. It doesn't interfere with my commandline test runner either (nosetests) -- it's ignored.

all

You may need to change the arguments to discover based on your project setup.

discover

Based on the answer of Stephen Cagle I added support for nested test modules.

import fnmatch
import os
import unittest

def all_test_modules(root_dir, pattern):
test_file_names = all_files_in(root_dir, pattern)
return [path_to_module(str) for str in test_file_names]

def all_files_in(root_dir, pattern):
matches =

for root, dirnames, filenames in os.walk(root_dir):
for filename in fnmatch.filter(filenames, pattern):
matches.append(os.path.join(root, filename))

return matches

def path_to_module(py_file):
return strip_leading_dots(
replace_slash_by_dot(
strip_extension(py_file)))

def strip_extension(py_file):
return py_file[0:len(py_file) - len('.py')]

def replace_slash_by_dot(str):
return str.replace('\', '.').replace('/', '.')

def strip_leading_dots(str):
while str.startswith('.'):
str = str[1:len(str)]
return str

module_names = all_test_modules('.', '*Tests.py')
suites = [unittest.defaultTestLoader.loadTestsFromName(mname) for mname
in module_names]

testSuite = unittest.TestSuite(suites)
runner = unittest.TextTestRunner(verbosity=1)
runner.run(testSuite)


The code searches all subdirectories of . for *Tests.py files which are then loaded. It expects each *Tests.py to contain a single class *Tests(unittest.TestCase) which is loaded in turn and executed one after another.

.

*Tests.py

*Tests.py

*Tests(unittest.TestCase)

This works with arbitrary deep nesting of directories/modules, but each directory in between needs to contain an empty __init__.py file at least. This allows the test to load the nested modules by replacing slashes (or backslashes) by dots (see replace_slash_by_dot).

__init__.py

replace_slash_by_dot

Because Test discovery seems to be a complete subject, there is some dedicated framework to test discovery :

More reading here : https://wiki.python.org/moin/PythonTestingToolsTaxonomy

This BASH script will execute the python unittest test directory from ANYWHERE in the file system, no matter what working directory you are in: its working directory always be where that test directory is located.

test

ALL TESTS, independent $PWD

unittest Python module is sensitive to your current directory, unless you tell it where (using discover -s option).

discover -s

This is useful when staying in the ./src or ./example working directory and you need a quick overall unit test:

./src

./example

#!/bin/bash
this_program="$0"
dirname="`dirname $this_program`"
readlink="`readlink -e $dirname`"

python -m unittest discover -s "$readlink"/test -v


SELECTED TESTS, independent $PWD

I name this utility file: runone.py and use it like this:

runone.py

runone.py <test-python-filename-minus-dot-py-fileextension>


#!/bin/bash
this_program="$0"
dirname="`dirname $this_program`"
readlink="`readlink -e $dirname`"

(cd "$dirname"/test; python -m unittest $1)


No need for a test/__init__.py file to burden your package/memory-overhead during production.

test/__init__.py

Here is my approach by creating a wrapper to run tests from the command line:

#!/usr/bin/env python3
import os, sys, unittest, argparse, inspect, logging

if __name__ == '__main__':
# Parse arguments.
parser = argparse.ArgumentParser(add_help=False)
parser.add_argument("-?", "--help", action="help", help="show this help message and exit" )
parser.add_argument("-v", "--verbose", action="store_true", dest="verbose", help="increase output verbosity" )
parser.add_argument("-d", "--debug", action="store_true", dest="debug", help="show debug messages" )
parser.add_argument("-h", "--host", action="store", dest="host", help="Destination host" )
parser.add_argument("-b", "--browser", action="store", dest="browser", help="Browser driver.", choices=["Firefox", "Chrome", "IE", "Opera", "PhantomJS"] )
parser.add_argument("-r", "--reports-dir", action="store", dest="dir", help="Directory to save screenshots.", default="reports")
parser.add_argument('files', nargs='*')
args = parser.parse_args()

# Load files from the arguments.
for filename in args.files:
exec(open(filename).read())

# See: http://codereview.stackexchange.com/q/88655/15346
def make_suite(tc_class):
testloader = unittest.TestLoader()
testnames = testloader.getTestCaseNames(tc_class)
suite = unittest.TestSuite()
for name in testnames:
suite.addTest(tc_class(name, cargs=args))
return suite

# Add all tests.
alltests = unittest.TestSuite()
for name, obj in inspect.getmembers(sys.modules[__name__]):
if inspect.isclass(obj) and name.startswith("FooTest"):
alltests.addTest(make_suite(obj))

# Set-up logger
verbose = bool(os.environ.get('VERBOSE', args.verbose))
debug = bool(os.environ.get('DEBUG', args.debug))
if verbose or debug:
logging.basicConfig( stream=sys.stdout )
root = logging.getLogger()
root.setLevel(logging.INFO if verbose else logging.DEBUG)
ch = logging.StreamHandler(sys.stdout)
ch.setLevel(logging.INFO if verbose else logging.DEBUG)
ch.setFormatter(logging.Formatter('%(asctime)s %(levelname)s: %(name)s: %(message)s'))
root.addHandler(ch)
else:
logging.basicConfig(stream=sys.stderr)

# Run tests.
result = unittest.TextTestRunner(verbosity=2).run(alltests)
sys.exit(not result.wasSuccessful())


For sake of simplicity, please excuse my non-PEP8 coding standards.

Then you can create BaseTest class for common components for all your tests, so each of your test would simply look like:

from BaseTest import BaseTest
class FooTestPagesBasic(BaseTest):
def test_foo(self):
driver = self.driver
driver.get(self.base_url + "/")


To run, you simply specifying tests as part of the command line arguments, e.g.:

./run_tests.py -h http://example.com/ tests/**/*.py


By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.