Frontend Testing with Socket

— Written by bcomnes

We've been cooking up several exciting and useful testing helpers inside the Socket runtime to make it easier and more productive to test browser javascript which we are excited to share with you! But first, let's take a quick look at the problem space to understand the landscape better.

Front-end unit testing remains an elusive testing target for many projects. There are many open source and commercial offerings that try to help solve this problem, but it's still extremely common to find even popular frontend libraries only running their tests inside of runtimes that merely simulate the browser environment, like Jest + JSDom running in Node.js. Why is this?

Browser Testing is Hard

Here's a non-exhaustive list of issues that makes browser testing challenging:

  • Browsers are notoriously challenging to automate, given their strong security concerns. Running them in a quick, headless manner requires special wrappers and procedures.
  • Modern browsers are heavy. If a tool is slow to start and shut down, it's less likely that developers want to incorporate a browser test harness into their development loop.
  • Maintaining real browser testing tools is challenging, because browsers change all the time, frequently breaking even diligently maintained projects.
  • Commercial offerings don't fit well in open source. Setting up commercial tools might work for some of the project contributors, but not for everyone. This is on top of the overhead commercial products typically require, like account signups, integrations, and licensing.
  • Each browser is different, so maintaining a common testing surface for each browser quickly becomes a full-time job.
  • Some projects ship tools based on Electron which installs and runs with the convenience of a normal project dependency, but it's heavy, updates all the time, and notoriously ships breaking changes often.
  • Communicating test code and receiving results back out of a real browser presents a challenge as well.
  • Running real browsers inside of CI environments offers additional challenges, as they can often lack the necessary dependencies required to run a browser and require additional knowledge and setup to get working in the various headless testing environments. Once they are set up, real browser tests running in CI are notoriously slow and flaky, often encountering spurious unexplained timeouts that require manual intervention.

To summarize: testing in a real browser is a pain. Browsers are big and complex, and these concerns make writing and maintaining tests that run like unit tests inside of browsers out of reach for many open-source projects.

Browser testing with Socket

First and foremost, Socket's testing tools make it easy to write tests for apps written to run in the Socket runtime. It can also act as a helpful tool when testing module code that targets browser-specific APIs that could benefit from testing in a real browser environment.

  • Socket apps run inside of a system-provided webview, and therefore have full access to browser APIs because it's running in a real browser! No simulations or polyfills are needed.
  • Socket is small and starts fast and has a small dependency footprint. This removes the perceived weight of browser testing tools, making it more of a joy to run unit tests in a browser during a development loop.
  • Socket ships built-in test tools that include helpful DOM helpers that make it easy to perform common tests, like selecting elements, waiting for content, etc.
  • Tests can run in isolation or the context of the running application. Writing isolated unit tests has its benefits, but a lot of UI code tends to have complex dependencies or relationships to other parts of the application. Running tests in the context of the running app is often easier than deconstructing UI elements to run tests against mocked data but still provides many of the same benefits of unit testing.

An example

ssc includes a --test=./src/test/index.js flag that offers a unique front-end testing environment.

When run with this flag, your Socket application launches normally, except it also begins running your tests within a testing context.

This lets you run any kind of test against your fully built, fully running app. You can simulate clicks, typing, visibility tests, and add and remove nodes all from your running application.

import { test } from 'socket:test'

test('app starts', async t => {
  await t.waitFor('.app-view', null, 'wait for app to load')

  const lobbyChannelButton = t.querySelector('.lobby-button')

  t.equal(lobbyChannelButton.getAttribute('is-selected'), 'false', 'lobby channel is selected')

  await t.sleep(100)

  await t.elementVisible('.app-channels', 'app-channels is visible')
  await t.elementVisible('.app-messages', 'app-messages is visible')

  await t.click(lobbyChannelButton, 'Click socket channel')

  await t.elementInvisible('app-profile', 'app-profile is not visible')
  await t.click('#profile-button button')
  await t.elementVisible('app-profile', 'app-profile is made visible')

  const app = t.querySelector('.app-view')
  t.equal(app.getAttribute('mode'), 'profile', 'app switched to profile mode')
})

Combining various DOM interactions with test assertions lets you quickly write lots of small tests that you would otherwise need to validate by hand.

You can even include or omit the --headless flag when launching your tests depending on if you want to watch the tests run in a semi-interactive manner or not.

Note: There are differences in browser behavior between running --headless, especially around animation APIs or other APIs that tie into the rendering of a WebView. While it's nice not to see the browser window pop open when you are running tests locally, sometimes the introduced environment differences can outweigh hiding the testing window. The good news is you can run --headless or not, even in CI depending on your preference.

DOM test assertions and helpers

The socket:test library (which is based on socketsupply/tapzero), has added a few additional DOM assertions and helpers that help streamline testing frontend application code:

  • await t.sleep(ms, [msg]) - Sleep the test for ms milliseconds. Adding timings to tests is generally best avoided, but sometimes its the only option when waiting for certain browser repaints to happen. When an optional message is provided, the sleep also becomes a test assertion that is reported in the TAP output.
  • await t.requestAnimationFrame([msg]) - Works similarly to t.sleep, but instead uses window.requestAnimationFrame instead of setTimeout. When tests run in headless mode, this switches to process.nextTick() which works similarly to requestAnimationFrame because requestAnimationFrame never resolves when a WebView is run heedlessly.
  • await t.click(selector, [msg]) - Run the .click() method on the querySelector string or element, and wait for a repaint.
  • await t.eventClick(selector, [msg]) - Works the same as t.click except it uses window.MouseEvent.
  • await dispatchEvent(event, target, [msg]) - Dispatch a custom event on a target element or querySelector string and print a pass assertion.
  • await t.focus(selector, [msg]) - Call the .focus() method on a target element and assert success.
  • await t.blur(selector, [msg]) - Call the .blur() method on a target element and assert success.
  • await t.type(selector, str, [msg]) - Set the value of a target element or querySelector string one additional character at a time, while dispatching an input event, and waiting for a repaint between each character, simulating typing.
  • await t.appendChild(parentSelector, el, [msg]) - Append an element el to a parent querySelector string or element followed by a passing assertion.
  • await t.removeElement(selector, [msg]) - Remove a querySelector string or element from the DOM by calling the .remove() method on it, followed by a passing assertion.
  • await t.elementVisible(selector, [msg]) - Asserts if a querySelector string or element is visible in the DOM.
  • await t.elementInvisible(selector, [msg]) - Asserts if a querySelector string or element is invisible in the DOM.
  • await t.waitFor(querySelectorOrFn, [opts, msg]) - Wait with a timeout for a function or querySelector to resolve to a truthy value.
  • await t.waitForText(selector, [opts, msg]) - Wait with a timeout for a selector to contain a regex or text in its innerText.
  • const selectedElement = t.querySelector(selector, [msg]) - The same as document.querySelector, but as a test assertion. This is surprisingly useful! Returns the selected node as well.
  • const selectedElements = t.querySelectorAll(selector, [msg]) - The same as document.querySelectorAll, but as a test assertion. Returns the elements in an array, which tends to be more practical for most circumstances.
  • const computedStyle = t.getComputedStyle(selector, [msg]) - Returns the defaultView.getComputedStyle of the selected element, with a pass assertion.

Do you have more ideas for other helpful front-end testing assertions? This is what we've come up for now, but there is a good chance we will add more. For the most up to date API docs for socket:test, please refer to the Docs.

Unit testing with socket

If you want to run isolated unit tests that are separate from running your full Socket application, simply create a second test application inside of your project, include a similar build step that the main application uses, and run it with the test flag.

import { test } from 'socket:test'
import { html } from '../_test/util'

test('Unit tests your application', async t => {
  const testEl = html`
    <section id="relative-time">
      <h2>Relative Time</h2>

      <div class="test-container">
        <tonic-relative-time
          id="relative-time-now"
          date="${new Date()}">
        </tonic-relative-time>
      </div>
    </section>`

  await t.appendChild(document.body, testEl)

  const now = t.querySelector('#relative-time-now')
  t.equal(now.textContent, 'now')

  await t.removeElement(testEl)
})

More is coming

These APIs are still early. There are several features we are still looking to add:

  • Default test file discovery and globbing
  • Test coverage results
  • Interactive test running (step through testing)
  • Auto-Reloading tests
  • Additional DOM assertion helpers
  • API alignment with the Node.js test runner and other popular test frameworks.

If you have ideas that you are interested in, we would love to chat with you on our Discord. We hope that Socket can help make writing front-end tests feel as light and painless as standard unit tests do.
Prev — Next