Wikidata/Development/Testing

There are three different frameworks

• Selenium tests (mw:Selenium Framework, old page)
• QUnit tests (mw:Manual:JavaScript unit testing)
• PHPunit tests (mw:Manual:PHP unit testing)

Also note the page at mw:Requests for comment/Unit testing.

Wikidata Test Server

The Wikidata test server runs all the mentioned tests and takes care of other QA related jobs. It's jobs are documented here.

Selenium tests

Selenium Grid Infrastructure

This is how the current setup of the Selenium Grid Infrastructure in the Wikidata office looks like.

• 

  Selenium Grid Infrastructure

Setup

Install Ruby and RubyGems (>=1.3.6) then run

sudo gem install bundler

cd ./path/to/wikibase/selenium/
sudo bundle

Copy "configuration.yml.sample", rename it to "configuration.yml" and set configuration parameters to your needs before running the tests!

Prerequisites for the MW instance you want to test

• You need to have an administrator user, an user which can be blocked during testing and an ordinary user (without admin rights) registered for the wikidata repo instance you want to test. You have to set username and password of those users in your local configuration.yml.
• Make sure you clean the MW instance (no data items should be left) before running the tests. You can do this by running "php /lib/maintenance/deleteAllData.php" in the Wikibase extension.

Running the tests on your local machine

• In configuration.yml set the WIKI_REPO_URL to point to your local MW instance you want to test and WIKI_CLIENT_URL to the connected client wiki. Don't forget the slash in the end!
• Go to the selenium directory in the Wikibase extension and run

  rake run_tests[<taskname>,<browser>]
  

  whereas <taskname> specifies the task to run (you can look into the Rakefile to see a list of available tasks) and <browser> specifies the browser you want to use for the tests (either "firefox" || "chrome" || "ie").
• Firefox (if installed) should work out of the box. For other browsers you should check the corresponding Webdriver on http://code.google.com/p/selenium/w/list. See also:
  • http://code.google.com/p/selenium/wiki/OperaDriver
  • http://code.google.com/p/selenium/wiki/InternetExplorerDriver
  • http://code.google.com/p/selenium/wiki/SafariDriver
  • http://code.google.com/p/selenium/wiki/FirefoxDriver
  • http://code.google.com/p/selenium/wiki/ChromeDriver

Running the tests remotely via Selenium Grid2

• In configuration.yml set WIKI_REPO_URL to point to your local MW instance from outside LOCALHOST and WIKI_CLIENT_URL to the connected client wiki. Don't forget a slash in the end!
• Set REMOTE_SELENIUM_HUB to point to the Selenium Hub instance (should not change that often)
• Then run

  rake[<taskname>,<browser>,remote,<os>]
  

  whereas <taskname> specifies the task to run (you can look into the Rakefile to see a list of available tasks), <browser> specifies the browser you want to use for the tests (either "firefox" || "chrome" || "ie") and <os> specifies the operating system you want to use (either "windows" || "linux" || "mac")

Examples

• this will run all tests locally in the browser you specified in configuration.yml as DEFAULT_BROWSER

rake
rake run_tests
rake run_tests[all]

• this will run all tests locally in chrome

rake run_tests[all,chrome]

• this will run the label tests locally in firefox

rake run_tests[label,firefox]

• this will run all tests on remote selenium grid hub in firefox

rake run_tests[all,firefox,remote]

• this will run the description tests on a remote linux box (if available) in chrome

rake run_tests[description,chrome,remote,linux]

• this will run all tests on a remote windows box (if available) in internet explorer

rake run_tests[all,ie,remote,windows]

Links

• Page object documentation @ rubydocs: http://rubydoc.info/github/cheezy/page-object/master/frames
• Introduction, examples and a lot of more information on Page Obejct: http://www.cheezyworld.com/2011/07/29/introducing-page-object-gem/ (also other blog posts there are really interesting)
• Watir-Webdriver (the underlying framework). We use it with Page Object on the top: http://watirwebdriver.com
• RSpec (Ruby's testing tool) documentation: http://rspec.info/ & http://rubydoc.info/gems/rspec-core/frames
• Pointer to the Selenium README on gerrit: https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/extensions/Wikibase.git;a=blob_plain;f=selenium/README
• Link to the Selenium Grid Hub on our office test server (only accessible via LAN): http://192.168.2.251:4444/grid/console

QUnit tests

Before being able to run QUnit tests, they have to be enabled in the LocalSettings.php by adding:

$wgEnableJavaScriptTest = true;

As of now, Wikibase specific QUnit tests only exist for the repo extension. To run all QUnit tests simply point your browser to the following URL:

http://localhost/<path to MediaWiki installation>/index.php/Special:JavaScriptTest/qunit

You might want to run the Wikibase tests exclusively which can be achieved by applying a filter:

http://localhost/<path to MediaWiki installation>/index.php/Special:JavaScriptTest/qunit?filter=wikibase

PHPunit tests

Installing on Ubuntu

Install PEAR if its not already installed, that makes everything a lot easier. On an Ubuntu box this is done with a

sudo apt-get install php-pear

or you could do a

wget http://pear.php.net/go-pear.phar
php go-pear.phar

After its installed do a

sudo pear upgrade pear

Then your at mw:Manual:PHP unit testing/Installing PHPUnit#Using PEAR. Only difference is that you probably should use --alldeps to save some griefs

sudo pear install --alldeps phpunit/PHPUnit

To be able to run tests I had to set absolute paths in LocalSettings.php, this could perhaps be better solved in suite.xml

Running tests

MediaWiki provides a custom wrapper for the standard phpunit command, which is located in tests/phpunit/phpunit.php. It supports all command line options and parameters the original phpunit command does (plus a few arcane ones we don't need).

To run all tests, including tests for any extension you have configured in LocalSettings.php:

 cd tests/phpunit
 php phpunit.php

To run only tests for the Wikibase group (provided you have the Wikibase extensions configured in your LocalSettings.php):

 php phpunit.php --group Wikibase

To run one specific test class:

 php phpunit.php ../../extensions/Wikibase/repo/tests/phpunit/includes/api/ApiGetItemId.php

To run one specific test function:

 php phpunit.php --filter testMyThing

(note that ---filter actually takes a regular expression).

Advanced Options and Configuration

In order to catch more problems when running tests, enable strict error reporting in your LocalSettings.php:

 error_reporting( E_ALL | E_STRICT );
 ini_set( 'display_errors', 1 );
 
 $wgShowExceptionDetails = true;
 $wgShowSQLErrors = true;
 $wgDebugDumpSql  = true;
 $wgShowDBErrorBacktrace = true;

There are several things you may want to change in test/phpunit/suite.xml:

Turning off verbose output, removing all the annoying details about skipped and incomplete tests:

 verbose="false"

Disabling test timeouts (and other strict checks):

 strict="false"

Or increasing the timeout:

 timeoutForSmallTests="8"

Note: make sure never to check in your modified suite.xml! you could also place your modified suit.xml in a different location and tell phpunit where to find it:

 php phpunit.php --configuration /path/to/phpunit/suite.xml  

For testing different setups (e.g. different database engines), you can tell phpunit to load a different LocalSettings.php file:

 php phpunit.php --conf /path/to/server/LocalSettings.php

Writing PHPUnit Tests

Test case classes should use the following code skeleton:

/**
 * ...
 *
 * @ingroup Wikibase
 * @ingroup Test
 *
 * @group Wikibase
 * @group Stuff
 *
 * ...
 */
class MyStuffTest extends MediaWikiTestCase { 
    function testFoo() {
        ...
    }
}

The important bits are:

• extend MediaWikiTestCase
• use a class name (and matching file name) that ends in "Test".
• use method names that start with "test" for actual test functions. You can have other (helper) functions that don't use that prefix.
• use @ingroup in the class comment to indicate to doxygen into which group documentation about this class should go. "Wikibase" and "Test" should be there.
• use @group in the class comment to indicate to phpunit to which test group this test belongs. "Wikibase" should be there.

There are some special groups that trigger special behavior if you apply them:

• @group Database causes phpunit to set up temporary database tables for use by the test, so that modifications performed by the test are not visible in the actual wiki database. This must be done for all tests that need the database, because Jenkins will run tests without @group Database without a valid database connection.
• @group medium causes phpunit to consider the test to be "medium heavy" instead of the default "small". This will apply a greater timeout to the test when phpunit runs in strict mode.

Registering Extension Test Cases

Update the registerUnitTests method in the Wikibase.hooks.php file (resp. WikibaseLib.hooks.php or WikibaseClient.hooks.php) with file paths to the individual test files. This is all that is necessary to get it up and running. To test this from the command line use

php phpunit.php --group Wikibase

When everything is in place on a central test server it should be possible to run tests for the Wikidata extension just like any other tests.

See also mw:Manual:PHP unit testing/Writing unit tests for extensions

File structure

Test cases should be placed in the tests/phpunit directory under the extension's directory. Below that, follow the directory structure used by the php files under test: if the file to test is in includes/api, put the test case in tests/phpunit/includes/api, and so on.

Resources

• Michael Hunter's You Are Not Done Yet (pdf), a comprehensive checklist of what can and should be checked. It seems to be aimed at Windows based desktop applications, but many points still apply to web applications.