In this publish, we’re excited to introduce SafeTest, a revolutionary library that provides a contemporary perspective on End-To-End (E2E) assessments for web-based User Interface (UI) purposes.
Traditionally, UI assessments have been carried out by both unit testing or integration testing (additionally known as End-To-End (E2E) testing). However, every of those strategies presents a novel trade-off: it’s a must to select between controlling the take a look at fixture and setup, or controlling the take a look at driver.
For occasion, when utilizing react-testing-library, a unit testing answer, you keep full management over what to render and the way the underlying companies and imports ought to behave. However, you lose the flexibility to work together with an precise web page, which may result in a myriad of ache factors:
- Difficulty in interacting with advanced UI components like <Dropdown /> parts.
- Inability to check CORS setup or GraphQL calls.
- Lack of visibility into z-index points affecting click-ability of buttons.
- Complex and unintuitive authoring and debugging of assessments.
Conversely, utilizing integration testing instruments like Cypress or Playwright gives management over the web page, however sacrifices the flexibility to instrument the bootstrapping code for the app. These instruments function by remotely controlling a browser to go to a URL and work together with the web page. This method has its personal set of challenges:
- Difficulty in making calls to another API endpoint with out implementing customized community layer API rewrite guidelines.
- Inability to make assertions on spies/mocks or execute code throughout the app.
- Testing one thing like darkish mode entails clicking the theme switcher or realizing the localStorage mechanism to override.
- Inability to check segments of the app, for instance if a element is simply seen after clicking a button and ready for a 60 second timer to countdown, the take a look at might want to run these actions and shall be no less than a minute lengthy.
Recognizing these challenges, options like E2E Component Testing have emerged, with choices from Cypress and Playwright. While these instruments try to rectify the shortcomings of conventional integration testing strategies, they produce other limitations on account of their structure. They begin a dev server with bootstrapping code to load the element and/or setup code you need, which limits their skill to deal with advanced enterprise purposes that may have OAuth or a fancy construct pipeline. Moreover, updating TypeScript utilization might break your assessments till the Cypress/Playwright group updates their runner.
SafeTest goals to handle these points with a novel method to UI testing. The important concept is to have a snippet of code in our utility bootstrapping stage that injects hooks to run our assessments (see the How Safetest Works sections for more information on what that is doing). Note that how this works has no measurable influence on the common utilization of your app since SafeTest leverages lazy loading to dynamically load the assessments solely when operating the assessments (within the README instance, the assessments aren’t within the manufacturing bundle in any respect). Once that’s in place, we are able to use Playwright to run common assessments, thereby attaining the best browser management we wish for our assessments.
This method additionally unlocks some thrilling options:
- Deep linking to a selected take a look at while not having to run a node take a look at server.
- Two-way communication between the browser and take a look at (node) context.
- Access to all of the DX options that include Playwright (excluding those that include @playwright/take a look at).
- Video recording of assessments, hint viewing, and pause web page performance for attempting out completely different web page selectors/actions.
- Ability to make assertions on spies within the browser in node, matching snapshot of the decision throughout the browser.
SafeTest is designed to really feel acquainted to anybody who has carried out UI assessments earlier than, because it leverages the very best elements of present options. Here’s an instance of easy methods to take a look at a whole utility:
import { describe, it, count on } from 'safetest/jest';
import { render } from 'safetest/react';describe('my app', () => {
it('hundreds the primary web page', async () => {
const { web page } = await render();
await count on(web page.getByText('Welcome to the app')).toBeVisible();
count on(await web page.screenshot()).toMatchImageSnapshot();
});
});
We can simply as simply take a look at a selected element
import { describe, it, count on, browserMock } from 'safetest/jest';
import { render } from 'safetest/react';describe('Header element', () => {
it('has a traditional mode', async () => {
const { web page } = await render(<Header />);
await count on(web page.getByText('Admin')).not.toBeVisible();
});
it('has an admin mode', async () => {
const { web page } = await render(<Header admin={true} />);
await count on(web page.getByText('Admin')).toBeVisible();
});
it('calls the logout handler when signing out', async () => {
const spy = browserMock.fn();
const { web page } = await render(<Header handleLogout={fn} />);
await web page.getByText('logout').click on();
count on(await spy).toHaveBeenCalledWith();
});
});
SafeTest makes use of React Context to permit for worth overrides throughout assessments. For an instance of how this works, let’s assume we’ve a fetchPeople perform utilized in a element:
import { useAsync } from 'react-use';
import { fetchPerson } from './api/individual';export const People: React.FC = () => {
const { knowledge: individuals, loading, error } = useAsync(fetchPeople);
if (loading) return <Loader />;
if (error) return <ErrorWeb page error={error} />;
return <Table knowledge={knowledge} rows=[...] />;
}
We can modify the People element to make use of an Override:
import { fetchPerson } from './api/individual';
+import { createOverride } from 'safetest/react';+const FetchPerson = createOverride(fetchPerson);
export const People: React.FC = () => {
+ const fetchPeople = FetchPerson.useValue();
const { knowledge: individuals, loading, error } = useAsync(fetchPeople);
if (loading) return <Loader />;
if (error) return <ErrorWeb page error={error} />;
return <Table knowledge={knowledge} rows=[...] />;
}
Now, in our take a look at, we are able to override the response for this name:
const pending = new Promise(r => { /* Do nothing */ });
const resolved = [{name: 'Foo', age: 23], {title: 'Bar', age: 32]}];
const error = new Error('Whoops');describe('People', () => {
it('has a loading state', async () => {
const { web page } = await render(
<FetchPerson.Override with={() => () => pending}>
<People />
</FetchPerson.Override>
);
await count on(web page.getByText('Loading')).toBeVisible();
});
it('has a loaded state', async () => {
const { web page } = await render(
<FetchPerson.Override with={() => async () => resolved}>
<People />
</FetchPerson.Override>
);
await count on(web page.getByText('User: Foo, title: 23')).toBeVisible();
});
it('has an error state', async () => {
const { web page } = await render(
<FetchPerson.Override with={() => async () => { throw error }}>
<People />
</FetchPerson.Override>
);
await count on(web page.getByText('Error getting customers: "Whoops"')).toBeVisible();
});
});
The render perform additionally accepts a perform that shall be handed the preliminary app element, permitting for the injection of any desired components anyplace within the app:
it('has a individuals loaded state', async () => {
const { web page } = await render(app =>
<FetchPerson.Override with={() => async () => resolved}>
{app}
</FetchPerson.Override>
);
await count on(web page.getByText('User: Foo, title: 23')).toBeVisible();
});With overrides, we are able to write advanced take a look at circumstances akin to making certain a service technique which mixes API requests from /foo, /bar, and /baz, has the right retry mechanism for simply the failed API requests and nonetheless maps the return worth accurately. So if /bar takes 3 makes an attempt to resolve the tactic will make a complete of 5 API calls.
Overrides aren’t restricted to only API calls (since we are able to use additionally use web page.route), we are able to additionally override particular app degree values like function flags or altering some static worth:
+const UseFlags = createOverride(useFlags);
export const Admin = () => {
+ const useFlags = UseFlags.useValue();
const { isAdmin } = useFlags();
if (!isAdmin) return <div>Permission error</div>;
// ...
}+const Language = createOverride(navigator.language);
export const LanguageChanger = () => {
- const language = navigator.language;
+ const language = Language.useValue();
return <div>Current language is { language } </div>;
}
describe('Admin', () => {
it('works with admin flag', async () => {
const { web page } = await render(
<UseIsAdmin.Override with={oldHook => {
const oldFlags = oldHook();
return { ...oldFlags, isAdmin: true };
}}>
<MyComponent />
</UseIsAdmin.Override>
);
await count on(web page.getByText('Permission error')).not.toBeVisible();
});
});
describe('Language', () => {
it('shows', async () => {
const { web page } = await render(
<Language.Override with={previous => 'abc'}>
<MyComponent />
</Language.Override>
);
await count on(web page.getByText('Current language is abc')).toBeVisible();
});
});
Overrides are a robust function of SafeTest and the examples right here solely scratch the floor. For extra info and examples, consult with the Overrides part on the README.
SafeTest comes out of the field with highly effective reporting capabilities, akin to computerized linking of video replays, Playwright hint viewer, and even deep hyperlink on to the mounted examined element. The SafeTest repo README hyperlinks to all of the instance apps in addition to the reviews
Many giant companies want a type of authentication to make use of the app. Typically, navigating to localhost:3000 simply leads to a perpetually loading web page. You must go to a unique port, like localhost:8000, which has a proxy server to examine and/or inject auth credentials into underlying service calls. This limitation is among the important causes that Cypress/Playwright Component Tests aren’t appropriate to be used at Netflix.
However, there’s normally a service that may generate take a look at customers whose credentials we are able to use to log in and work together with the applying. This facilitates creating a lightweight wrapper round SafeTest to robotically generate and assume that take a look at consumer. For occasion, right here’s principally how we do it at Netflix:
import { setup } from 'safetest/setup';
import { createTestUser, addCookies } from 'netflix-test-helper';sort Setup = Parameters<typeof setup>[0] & {
additionalUserChoices?: UserChoices;
};
export const setupNetflix = (choices: Setup) => {
setup({
...choices,
hooks: { beforeNavigate: [async page => addCookies(page)] },
});
beforeAll(async () => {
createTestUser(choices.additionalUserChoices)
});
};
After setting this up, we merely import the above bundle rather than the place we’d have used safetest/setup.
While this publish centered on how SafeTest works with React, it’s not restricted to only React. SafeTest additionally works with Vue, Svelte, Angular, and even can run on NextJS or Gatsby. It additionally runs utilizing both Jest or Vitest based mostly on which take a look at runner your scaffolding began you off with. The examples folder demonstrates easy methods to use SafeTest with completely different tooling combos, and we encourage contributions so as to add extra circumstances.
At its core, SafeTest is an clever glue for a take a look at runner, a UI library, and a browser runner. Though the most typical utilization at Netflix employs Jest/React/Playwright, it’s straightforward so as to add extra adapters for different choices.
SafeTest is a robust testing framework that’s being adopted inside Netflix. It permits for straightforward authoring of assessments and gives complete reviews when and the way any failures occurred, full with hyperlinks to view a playback video or manually run the take a look at steps to see what broke. We’re excited to see the way it will revolutionize UI testing and stay up for your suggestions and contributions.