WP Acceptance

(Beta) A team scalable solution for reliable WordPress acceptance testing.

WP Acceptance is a toolkit that empowers developers and CI pipelines to test codebases using version controlled acceptance tests and sharable, defined file and database snapshots.


Requirements

WP Local Docker is highly recommended as the local development environment but not required.

Installation

WP Acceptance is easiest to use as a project-level Composer package:

1. Since WP Acceptance is in beta, you will need to set your project minimum stability to dev:

composer config minimum-stability dev

2. Next, require the WP Acceptance package:

composer require 10up/wpacceptance:dev-master --dev

3. Finally, verify and run WP Acceptance by calling the script in the Composer bin directory:

./vendor/bin/wpacceptance

After installation, you will want to setup WP Acceptance on a project.

Project Setup

After installing WP Acceptance, you need to setup your project and development workflow.

1. Spin up your local environment. WP Acceptance will use your local if run with the --local flag. We highly recommend WP Local Docker.

2. Decide where you want to initialize WP Acceptance (create wpacceptance.json) which must be the root of your version controlled repository. This is usually in wp-content/, a theme, or a plugin. wp-content/ might make most sense if you are developing an entire website. Let's assume we are initializing our WP Acceptance project in wp-content/ and have installed WP Acceptance in the same directory.

Navigate to wp-content in the command line. Run the following command:

./vendor/bin/wpacceptance init

3. You will be presented with some command prompts. Choose a project slug and select the defaults for the other options. When the command is finished, there will be a wpacceptance.json file in wp-content as well as a tests directory and an example test, tests/ExampleTest.php.

WP Acceptance reads wpacceptance.json every time tests are run. The file must contain both name and tests properties in JSON format. name is the name of your test suite, and it must be unique. snapshot_id is optional and is explained in Workflow and Snapshots. tests points to your test files. WP Acceptance tests are written in PHP and PHPUnit based.

There are a few important rules for wpacceptance.json:

4. Now let's run our tests to make sure everything works:

./vendor/bin/wpacceptance run --local

You should see your tests passing.

5. Now let's create a primary snapshot to commit to our repository. In order to do this, we will need WP Snapshots configured. WP Snapshots handles the transportation and storage of snapshots. Run the following command to configure WP Snapshots (if it's not already configured):

./vendor/bin/wpsnapshots configure <repository-name>

You will need to create a repository if you don't have one. At 10up, we use 10up as our <repository-name>.

6. Now that we are ready with WP Snapshots, let's run our tests again but this time saving the snapshot ID to our wpacceptance.json and pushing the snapshot to our remote repository:

./vendor/bin/wpacceptance run --local --save

After our tests pass, you will see the snapshot get pushed upstream.

7. Commit the snapshot ID to your repository and push the new commit upstream.

8. Now that you have a snapshot ID in your wpacceptance.json, you can run your test suite without having a local environment running:

./vendor/bin/wpacceptance run

You should create new snapshots when new features, plugins, content types, etc. are added to your web application.

Note: Make sure you run WP Acceptance on your HOST machine, not within another Docker environment.

Workflow and Snapshots

There are two scenarios or workflows for running WP Acceptance:

  1. Testing a codebase using your local environment (files and database).
  2. Testing a codebase against a "primary" snapshot.

The power of WP Acceptance is working with a team that is all testing it's code against one primary snapshot. Of course, in order for this to be successful the primary snapshot must be kept relevant which is the responsiblity of the development team. For example, when new content types are added, content should be added and a new primary snapshot created.

To test a codebase on your local environment, you would run the following command in the directory of wpacceptance.json:

wpacceptance run --local --save

The --local flag will force WP Acceptance to ignore a snapshot ID defined in wpacceptance.json. The --save flag will make WP Acceptance create a new snapshot from your local and save the ID to wpacceptance.json (overwritting any old ID). After saving a new primary snapshot to wpacceptance.json, you will want to commit and push the change upstream.

To test a codebase on a primary snapshot, you would simply run the following command in the directory of wpacceptance.json:

wpacceptance run

You can only run WP Acceptance against snapshots that contain some version of the codebase your are testing. This means the snapshot must contain wpacceptance.json with the same name as the one you are running.

wpacceptance.json File

wpacceptance.json is the "configuration" file read by WP Acceptance. This file is required for each project. Whenever a test suite is run via the run command, wpacceptance.json is processed.

Here's what wpacceptance.json looks like

{
    "name": "example-suite",
    "tests": [
        "tests\/*.php"
    ],
    "snapshot_id": "...",
    "exclude": [
        "%REPO_ROOT%/node_modules",
        "%REPO_ROOT%/vendor"
    ],
    "enforce_clean_db": true,
    "disable_clean_db": false,
    "repository": "10up",
    "bootstrap": "./bootstrap.php",
    "repo_path": "%WP_ROOT%/wp-content",
    "before_scripts": [
        "composer install",
        "npm run build"
    ]
}

Writing Tests

WP Acceptance tests are based on PHPUnit. Here is a simple example of a test:

class ExampleTest extends \WPAcceptance\PHPUnit\TestCase {

    /**
     * Example test
     */
    public function testExample() {
        $this->assertTrue( true );
    }
}

You can place tests in whatever files you choose (as long as tests points to the right place in wpacceptance.json). However, per PHPUnit defaults, your test code must be in a class (or classes across multiple files) with name(s) that end in Test. Inside your test class(es), each test method must begin with test.

All your tests must extend \WPAcceptance\PHPUnit\TestCase. \WPAcceptance\PHPUnit\TestCase extends \PHPUnit\Framework\TestCase so you will have access to all the standard PHPUnit methods e.g. assertTrue, assertEquals, etc.

Along with standard PHPUnit functionality, you have WP Acceptance special methods/functions/classes:

Actor

The most poweful WP Acceptance functionality is provided by the Actor class and let's you interact as a Chrome browser user with your website.

A new Actor must be initialized for each test and is done like so:

public function testExample() {
    $I = $this->getAnonymousUser();
}

getAnonymousUser does take an optional array of arguments. In particular, you can choose the browser size: getAnonymousUser( [ 'screen_size' => 'small' ] ).

With $I we can navigate to sections of our website:

$I->moveTo( 'page-two' );

When you ask the browser to take an action that isn't instant, you will need to wait:

$I->moveTo( 'page-two' );

$I->waitUntilElementVisible( 'body.page-two' );

The Actor can login to the WordPress admin:

$I->loginAs( 'wpsnapshots' );

We can fill in form fields:

$I->fillField( '.field-name', 'value' );
$I->checkOptions( '.checkbox-or-radio' );
$I->selectOptions( '.select', 'value-to-select' );

Since WP Acceptance is built on WP Snapshots, the wpsnapshots user is always available as a super admin with password, password.

Since $I is literally interacting with a browser, we can do anything a browser can: click on elements, follow links, get specific DOM elements, run JavaScript, resize the browser, refresh the page, interact with forms, move the mouse, interact with the keyboard, etc.

The Actor also contains methods for making assertions:

$I->seeText( 'text' );
$I->dontSeeText( 'text' );
$I->seeText( 'text', '.element-to-search-within' );

$I->seeElement( '.element-path' );
$I->dontSeeElement( '.element-path' );

$I->seeLink( 'Link Text', 'http://url' );
$I->dontSeeLink( 'Link Text', 'http://url' );

$I->seeTextInUrl( 'Title Text' );
$I->dontSeeTextInUrl( 'Title Text' );

Database

WP Acceptance not only let's you test UI elements but how your web application interacts with the database as well.

We can assert that new posts (or other custom post types) were created:

$last_id = self::getLastPostId( [ 'post_type' => 'post' ] );

// Interact with website....

self::assertNewPostsExist( $last_id, [ 'post_type' => 'post' ], 'No new post.' );

self::assertNewPostsExist checks for new database items after $last_id.

Full API Documentation

To read about all WP Acceptance testing related methods, look at the source code Actor class.

Examples

For detailed test examples, look at the example test suite.

Commands

Speed of Testing

Unfortunately, good test suites can take awhile to run. WP Acceptance has to do a lot of work in order to setup an environment for testing. Here are some tips for getting your test suite to run faster:

Local Test Development

Here are some tips for writing tests locally:

Continuous Integration

WP Acceptance is a great addition to your CI process.

Travis CI

Here is an example .travis.yml that includes WP Acceptance:

language: php
php:
  - 7.2
env:
  global:
  - WP_VERSION=master
  - WP_VERSION=4.7
before_script:
  - composer install
script:
  - if [ -n "$AWS_ACCESS_KEY" ]; then ./vendor/bin/wpsnapshots configure 10up --aws_key=$AWS_ACCESS_KEY --aws_secret=$SECRET_ACCESS_KEY --user_name=Travis --user_email=travis@10up.com; fi
  - if [ -n "$AWS_ACCESS_KEY" ]; then bash run-wpacceptance.sh; fi
sudo: required
services: docker

Make sure you replace REPO_NAME with your WP Snapshots repository name. You will also need to define AWS_ACCESS_KEY and SECRET_ACCESS_KEY as hidden Travis environmental variables in your Travis project settings.

Here is run-wpacceptance.sh which will retry WP Acceptance up to 3 times if environment errors occur:

for i in 1 2 3; do
    ./vendor/bin/wpacceptance run

    EXIT_CODE=$?

    if [ $EXIT_CODE -gt 1 ]; then
        echo "Retrying..."
        sleep 3
    else
        break
    fi
done

exit $EXIT_CODE

GitLab

WP Acceptance works well with GitLab as well. The only difference is when running wpsnapshots configure, you need to prefix the command with an environmental variable WPSNAPSHOTS_DIR: WPSNAPSHOTS_DIR=/builds/${CI_PROJECT_NAMESPACE}/.wpsnapshots/ wpsnapshots configure.