WP Acceptance

(Beta) A team scalable solution for reliable WordPress acceptance testing.

WP Acceptance is a toolkit that empowers developers and CI pipelines to test codebases using version controlled acceptance tests and sharable environments.


Requirements

WP Local Docker is highly recommended as the local development environment but not required.

Installation

WP Acceptance is easiest to use as a project-level Composer package:

1. Since WP Acceptance is in beta, you will need to set your project minimum stability to dev:

composer config minimum-stability dev

2. Next, require the WP Acceptance package:

composer require 10up/wpacceptance:dev-master --dev

3. Finally, verify and run WP Acceptance by calling the script in the Composer bin directory:

./vendor/bin/wpacceptance

After installation, you will want to setup WP Acceptance on a project.

Project Setup

After installing WP Acceptance, you need to setup your project and development workflow.

1. Spin up your local environment. WP Acceptance will use your local if run with the --local flag. We highly recommend WP Local Docker.

2. Decide where you want to initialize WP Acceptance (create wpacceptance.json) which must be the root of your version controlled repository. This is usually in wp-content/, a theme, or a plugin. wp-content/ might make most sense if you are developing an entire website. Let's assume we are initializing our WP Acceptance project in wp-content/ and have installed WP Acceptance in the same directory.

Navigate to wp-content in the command line. Run the following command:

./vendor/bin/wpacceptance init

3. You will be presented with some command prompts. Choose a project slug and select the defaults for the other options. When the command is finished, there will be a wpacceptance.json file in wp-content as well as a tests directory and an example test, tests/ExampleTest.php.

WP Acceptance reads wpacceptance.json every time tests are run. The file must contain both name and tests properties in JSON format. name is the name of your test suite, and it must be unique. wpacceptance.json can define environment_instructions OR snapshot_id. This is explained in Workflow, Environment Instructions, and Snapshots. tests points to your test files. WP Acceptance tests are written in PHP and PHPUnit based.

There are a few important rules for wpacceptance.json:

4. Now let's run our tests to make sure everything works:

./vendor/bin/wpacceptance run --local

You should see your tests passing.

If you just want to run tests locally, you are done. If you want to have a teammate run your test suite or integrate with a CI process, you will need to decide on using either environment instrutions or snapshots (check out Snapshots vs. Environment Instructions).

Environment Instructions

5. In wpacceptance.json, create a property named environment_instructions. environment_instructions takes an array of "instructions". Instructions are processed via WP Instructions. Here's a simple example:

{
    "environment_instructions": [
        "install wordpress where site url is http://wpacceptance.test and home url is http://wpacceptance.test",
        "install theme where theme name is twentynineteen"
    ]
}

The first instruction MUST be installing WordPress. Install wordpress must include a site and home url which can be anything. For documentation and usage on supported instructions, see WP Instructions.

environment_instructions can also take an array of arrays of instructions. This is useful if you want to run your tests across multiple environments:

{
    "environment_instructions": [
        [
            "install wordpress where version is 4.9 and site url is http://wpacceptance.test and home url is http://wpacceptance.test"
        ],
        [
            "install wordpress where version is latest and site url is http://wpacceptance.test and home url is http://wpacceptance.test"
        ]
    ]
}

6. Next you need to tell WP Acceptance where your project should be mounted. For example, if my wpacceptance.json is the root of wp-content, I would set project_path like so:

{
    "project_path": "%WP_ROOT%/wp-content"
}

Snapshots

5. Let's create a primary snapshot to commit to our repository. In order to do this, we will need WP Snapshots configured. WP Snapshots handles the transportation and storage of snapshots. Run the following command to configure WP Snapshots (if it's not already configured):

./vendor/bin/wpsnapshots configure <repository-name>

You will need to create a repository if you don't have one. At 10up, we use 10up as our <repository-name>.

6. Now that we are ready with WP Snapshots, let's run our tests again but this time saving the snapshot ID to our wpacceptance.json and pushing the snapshot to our remote repository:

./vendor/bin/wpacceptance run --local --save_snapshot

After our tests pass, you will see the snapshot get pushed upstream.

7. Commit the snapshot ID to your repository and push the new commit upstream.

8. Now that you have a snapshot ID in your wpacceptance.json, you can run your test suite without having a local environment running:

./vendor/bin/wpacceptance run

You should create new snapshots when new features, plugins, content types, etc. are added to your web application.

Note: Make sure you run WP Acceptance on your HOST machine, not within another Docker environment.

Workflow, Environment Instructions, and Snapshots

There are three scenarios or workflows for running WP Acceptance:

  1. Testing a codebase using your local environment (files and database).
  2. Testing a codebase against a set of environment instructions.
  3. Testing a codebase against a "primary" snapshot.

The power of WP Acceptance is working with a team or CI process that is testing it's code against a set of environment instructions or snapshot (you can also test against multiple sets of environment instructions or snapshots). Of course, in order for this to be successful the environment instructions or snapshot(s) must be kept relevant which is the responsiblity of the development team. For example, when new content types are added, content should be added via new environment instructions or a new snapshot created.

Environment instructions are a simple set of instructions for defining an environment e.g. install WordPress, download twentynineteen theme, activate plugin, etc (supported instructions are documented in WP Instructions). A snapshot is a database/file package where everything e.g. WP version, theme, plugins, are preset. You can use environment instructions OR snapshots, not both. Read Snapshots vs. Environment Instructions.

To test a codebase on your local environment, you would run the following command in the directory of wpacceptance.json:

wpacceptance run --local

The --local flag will force WP Acceptance to ignore environment instructions or snapshot ID defined in wpacceptance.json. If you are using a snapshot workflow, you can use the --save_snapshot flag which will make WP Acceptance create a new snapshot from your local and save the ID to wpacceptance.json. After saving a new snapshot to wpacceptance.json, you will want to commit and push the change upstream.

To test a codebase against environment instructions (assuming environment_instructions and project_path are defined in wpacceptance.json), you would run the following command in the directory of wpacceptance.json:

wpacceptance run

To test a codebase against snapshot(s) (assuming snapshot_id or snapshots is defined in wpacceptance.json), you would simply run the following command in the directory of wpacceptance.json:

wpacceptance run

You can only run WP Acceptance against snapshots that contain some version of the codebase your are testing. This means snapshots must contain wpacceptance.json with the same name as the one you are running.

wpacceptance.json File

wpacceptance.json is the "configuration" file read by WP Acceptance. This file is required for each project. Whenever a test suite is run via the run command, wpacceptance.json is processed.

Here's what wpacceptance.json looks like

{
    "name": "example-suite",
    "tests": [
        "tests\/*.php"
    ],
    "snapshot_id": "...",
    "snapshots": [
        {
            "snapshot_name": "name",
            "snapshot_id": "ID"
        }
    ],
    "environment_instructions": [
        [
            "..."
        ]
    ],
    "project_path": "%WP_ROOT%/wp-content",
    "exclude": [
        "node_modules",
        "vendor"
    ],
    "enforce_clean_db": true,
    "disable_clean_db": false,
    "repository": "10up",
    "bootstrap": "./bootstrap.php",
    "before_scripts": [
        "composer install",
        "npm run build"
    ]
}

Writing Tests

WP Acceptance tests are based on PHPUnit. Here is a simple example of a test:

class ExampleTest extends \WPAcceptance\PHPUnit\TestCase {

    /**
     * Example test
     */
    public function testExample() {
        $this->assertTrue( true );
    }
}

You can place tests in whatever files you choose (as long as tests points to the right place in wpacceptance.json). However, per PHPUnit defaults, your test code must be in a class (or classes across multiple files) with name(s) that end in Test. Inside your test class(es), each test method must begin with test.

All your tests must extend \WPAcceptance\PHPUnit\TestCase. \WPAcceptance\PHPUnit\TestCase extends \PHPUnit\Framework\TestCase so you will have access to all the standard PHPUnit methods e.g. assertTrue, assertEquals, etc.

Along with standard PHPUnit functionality, you have WP Acceptance special methods/functions/classes:

Actor

The most poweful WP Acceptance functionality is provided by the Actor class and let's you interact as a Chrome browser user with your website.

A new Actor must be initialized for each test and is done like so:

public function testExample() {
    $I = $this->openBrowserPage();
}

openBrowserPage does take an optional array of arguments. In particular, you can choose the browser size: openBrowserPage( [ 'screen_size' => 'small' ] ).

With $I we can navigate to sections of our website:

$I->moveTo( 'page-two' );

When you ask the browser to take an action that isn't instant, you will need to wait:

$I->moveTo( 'page-two' );

$I->waitUntilElementVisible( 'body.page-two' );

The Actor can login to the WordPress admin:

$I->loginAs( 'wpsnapshots' );

We can fill in form fields:

$I->fillField( '.field-name', 'value' );
$I->checkOptions( '.checkbox-or-radio' );
$I->selectOptions( '.select', 'value-to-select' );

If using snapshots, the wpsnapshots user is always available as a super admin with password, password. If using environment instructions, the admin user is always available as a super admin with password, password.

Since $I is literally interacting with a browser, we can do anything a browser can: click on elements, follow links, get specific DOM elements, run JavaScript, resize the browser, refresh the page, interact with forms, move the mouse, interact with the keyboard, etc.

The Actor also contains methods for making assertions:

$I->seeText( 'text' );
$I->dontSeeText( 'text' );
$I->seeText( 'text', '.element-to-search-within' );

$I->seeElement( '.element-path' );
$I->dontSeeElement( '.element-path' );

$I->seeLink( 'Link Text', 'http://url' );
$I->dontSeeLink( 'Link Text', 'http://url' );

$I->seeTextInUrl( 'Title Text' );
$I->dontSeeTextInUrl( 'Title Text' );

Database

WP Acceptance not only let's you test UI elements but how your web application interacts with the database as well.

We can assert that new posts (or other custom post types) were created:

$last_id = self::getLastPostId( [ 'post_type' => 'post' ] );

// Interact with website....

self::assertNewPostsExist( $last_id, [ 'post_type' => 'post' ], 'No new post.' );

self::assertNewPostsExist checks for new database items after $last_id.

Full API Documentation

To read about all WP Acceptance testing related methods, look at the source code Actor class.

Examples

For detailed test examples, look at the example test suite.

Commands

Speed of Testing

Unfortunately, good test suites can take awhile to run. WP Acceptance has to do a lot of work in order to setup an environment for testing. Here are some tips for getting your test suite to run faster:

Local Test Development

Here are some tips for writing tests locally:

Continuous Integration

WP Acceptance is a great addition to your CI process.

Travis CI

Here are examples of .travis.yml that includes WP Acceptance:

Here is run-wpacceptance.sh which will retry WP Acceptance up to 3 times if environment errors occur:

for i in 1 2 3; do
    ./vendor/bin/wpacceptance run

    EXIT_CODE=$?

    if [ $EXIT_CODE -gt 1 ]; then
        echo "Retrying..."
        sleep 3
    else
        break
    fi
done

exit $EXIT_CODE

Travis with Environment Instructions

language: php
php:
  - 7.2
env:
  global:
  - WP_VERSION=master
  - WP_VERSION=4.7
before_script:
  - composer install
script:
  - bash run-wpacceptance.sh; fi
sudo: required
services: docker

Travis with Snapshots

language: php
php:
  - 7.2
env:
  global:
  - WP_VERSION=master
  - WP_VERSION=4.7
before_script:
  - composer install
script:
  - if [ -n "$AWS_ACCESS_KEY" ]; then ./vendor/bin/wpsnapshots configure 10up --aws_key=$AWS_ACCESS_KEY --aws_secret=$SECRET_ACCESS_KEY --user_name=Travis --user_email=travis@10up.com; fi
  - if [ -n "$AWS_ACCESS_KEY" ]; then bash run-wpacceptance.sh; fi
sudo: required
services: docker

Make sure you replace REPO_NAME with your WP Snapshots repository name. You will also need to define AWS_ACCESS_KEY and SECRET_ACCESS_KEY as hidden Travis environmental variables in your Travis project settings.

GitLab

WP Acceptance works well with GitLab as well. The only difference is, if using snapshots, when running wpsnapshots configure, you need to prefix the command with an environmental variable WPSNAPSHOTS_DIR: WPSNAPSHOTS_DIR=/builds/${CI_PROJECT_NAMESPACE}/.wpsnapshots/ wpsnapshots configure.

Snapshots vs. Environment Instructions

Snapshots and environmental instructions are two different tools for creating shareable environments that empower team members and/or CI processes to test consistently against the same environment. Read more about the two workflows above. Here are some reasons to use one workflow over the other: