309 lines
11 KiB
XML
309 lines
11 KiB
XML
<section xmlns="http://docbook.org/ns/docbook"
|
||
xmlns:xlink="http://www.w3.org/1999/xlink"
|
||
xmlns:xi="http://www.w3.org/2001/XInclude"
|
||
version="5.0"
|
||
xml:id="sec-writing-nixos-tests">
|
||
|
||
<title>Writing Tests</title>
|
||
|
||
<para>A NixOS test is a Nix expression that has the following structure:
|
||
|
||
<programlisting>
|
||
import ./make-test.nix {
|
||
|
||
# Either the configuration of a single machine:
|
||
machine =
|
||
{ config, pkgs, ... }:
|
||
{ <replaceable>configuration…</replaceable>
|
||
};
|
||
|
||
# Or a set of machines:
|
||
nodes =
|
||
{ <replaceable>machine1</replaceable> =
|
||
{ config, pkgs, ... }: { <replaceable>…</replaceable> };
|
||
<replaceable>machine2</replaceable> =
|
||
{ config, pkgs, ... }: { <replaceable>…</replaceable> };
|
||
…
|
||
};
|
||
|
||
testScript =
|
||
''
|
||
<replaceable>Perl code…</replaceable>
|
||
'';
|
||
}
|
||
</programlisting>
|
||
|
||
The attribute <literal>testScript</literal> is a bit of Perl code that
|
||
executes the test (described below). During the test, it will start
|
||
one or more virtual machines, the configuration of which is described
|
||
by the attribute <literal>machine</literal> (if you need only one
|
||
machine in your test) or by the attribute <literal>nodes</literal> (if
|
||
you need multiple machines). For instance, <filename
|
||
xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/login.nix">login.nix</filename>
|
||
only needs a single machine to test whether users can log in on the
|
||
virtual console, whether device ownership is correctly maintained when
|
||
switching between consoles, and so on. On the other hand, <filename
|
||
xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nfs.nix">nfs.nix</filename>,
|
||
which tests NFS client and server functionality in the Linux kernel
|
||
(including whether locks are maintained across server crashes),
|
||
requires three machines: a server and two clients.</para>
|
||
|
||
<para>There are a few special NixOS configuration options for test
|
||
VMs:
|
||
|
||
<!-- FIXME: would be nice to generate this automatically. -->
|
||
|
||
<variablelist>
|
||
|
||
<varlistentry>
|
||
<term><option>virtualisation.memorySize</option></term>
|
||
<listitem><para>The memory of the VM in
|
||
megabytes.</para></listitem>
|
||
</varlistentry>
|
||
|
||
<varlistentry>
|
||
<term><option>virtualisation.vlans</option></term>
|
||
<listitem><para>The virtual networks to which the VM is
|
||
connected. See <filename
|
||
xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nat.nix">nat.nix</filename>
|
||
for an example.</para></listitem>
|
||
</varlistentry>
|
||
|
||
<varlistentry>
|
||
<term><option>virtualisation.writableStore</option></term>
|
||
<listitem><para>By default, the Nix store in the VM is not
|
||
writable. If you enable this option, a writable union file system
|
||
is mounted on top of the Nix store to make it appear
|
||
writable. This is necessary for tests that run Nix operations that
|
||
modify the store.</para></listitem>
|
||
</varlistentry>
|
||
|
||
</variablelist>
|
||
|
||
For more options, see the module <filename
|
||
xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/qemu-vm.nix">qemu-vm.nix</filename>.</para>
|
||
|
||
<para>The test script is a sequence of Perl statements that perform
|
||
various actions, such as starting VMs, executing commands in the VMs,
|
||
and so on. Each virtual machine is represented as an object stored in
|
||
the variable <literal>$<replaceable>name</replaceable></literal>,
|
||
where <replaceable>name</replaceable> is the identifier of the machine
|
||
(which is just <literal>machine</literal> if you didn’t specify
|
||
multiple machines using the <literal>nodes</literal> attribute). For
|
||
instance, the following starts the machine, waits until it has
|
||
finished booting, then executes a command and checks that the output
|
||
is more-or-less correct:
|
||
|
||
<programlisting>
|
||
$machine->start;
|
||
$machine->waitForUnit("default.target");
|
||
$machine->succeed("uname") =~ /Linux/;
|
||
</programlisting>
|
||
|
||
The first line is actually unnecessary; machines are implicitly
|
||
started when you first execute an action on them (such as
|
||
<literal>waitForUnit</literal> or <literal>succeed</literal>). If you
|
||
have multiple machines, you can speed up the test by starting them in
|
||
parallel:
|
||
|
||
<programlisting>
|
||
startAll;
|
||
</programlisting>
|
||
|
||
</para>
|
||
|
||
<para>The following methods are available on machine objects:
|
||
|
||
<variablelist>
|
||
|
||
<varlistentry>
|
||
<term><methodname>start</methodname></term>
|
||
<listitem><para>Start the virtual machine. This method is
|
||
asynchronous — it does not wait for the machine to finish
|
||
booting.</para></listitem>
|
||
</varlistentry>
|
||
|
||
<varlistentry>
|
||
<term><methodname>shutdown</methodname></term>
|
||
<listitem><para>Shut down the machine, waiting for the VM to
|
||
exit.</para></listitem>
|
||
</varlistentry>
|
||
|
||
<varlistentry>
|
||
<term><methodname>crash</methodname></term>
|
||
<listitem><para>Simulate a sudden power failure, by telling the VM
|
||
to exit immediately.</para></listitem>
|
||
</varlistentry>
|
||
|
||
<varlistentry>
|
||
<term><methodname>block</methodname></term>
|
||
<listitem><para>Simulate unplugging the Ethernet cable that
|
||
connects the machine to the other machines.</para></listitem>
|
||
</varlistentry>
|
||
|
||
<varlistentry>
|
||
<term><methodname>unblock</methodname></term>
|
||
<listitem><para>Undo the effect of
|
||
<methodname>block</methodname>.</para></listitem>
|
||
</varlistentry>
|
||
|
||
<varlistentry>
|
||
<term><methodname>screenshot</methodname></term>
|
||
<listitem><para>Take a picture of the display of the virtual
|
||
machine, in PNG format. The screenshot is linked from the HTML
|
||
log.</para></listitem>
|
||
</varlistentry>
|
||
|
||
<varlistentry>
|
||
<term><methodname>getScreenText</methodname></term>
|
||
<listitem><para>Return a textual representation of what is currently
|
||
visible on the machine's screen using optical character
|
||
recognition.</para>
|
||
<note><para>This requires passing <option>enableOCR</option> to the test
|
||
attribute set.</para></note></listitem>
|
||
</varlistentry>
|
||
|
||
<varlistentry>
|
||
<term><methodname>sendMonitorCommand</methodname></term>
|
||
<listitem><para>Send a command to the QEMU monitor. This is rarely
|
||
used, but allows doing stuff such as attaching virtual USB disks
|
||
to a running machine.</para></listitem>
|
||
</varlistentry>
|
||
|
||
<varlistentry>
|
||
<term><methodname>sendKeys</methodname></term>
|
||
<listitem><para>Simulate pressing keys on the virtual keyboard,
|
||
e.g., <literal>sendKeys("ctrl-alt-delete")</literal>.</para></listitem>
|
||
</varlistentry>
|
||
|
||
<varlistentry>
|
||
<term><methodname>sendChars</methodname></term>
|
||
<listitem><para>Simulate typing a sequence of characters on the
|
||
virtual keyboard, e.g., <literal>sendKeys("foobar\n")</literal>
|
||
will type the string <literal>foobar</literal> followed by the
|
||
Enter key.</para></listitem>
|
||
</varlistentry>
|
||
|
||
<varlistentry>
|
||
<term><methodname>execute</methodname></term>
|
||
<listitem><para>Execute a shell command, returning a list
|
||
<literal>(<replaceable>status</replaceable>,
|
||
<replaceable>stdout</replaceable>)</literal>.</para></listitem>
|
||
</varlistentry>
|
||
|
||
<varlistentry>
|
||
<term><methodname>succeed</methodname></term>
|
||
<listitem><para>Execute a shell command, raising an exception if
|
||
the exit status is not zero, otherwise returning the standard
|
||
output.</para></listitem>
|
||
</varlistentry>
|
||
|
||
<varlistentry>
|
||
<term><methodname>fail</methodname></term>
|
||
<listitem><para>Like <methodname>succeed</methodname>, but raising
|
||
an exception if the command returns a zero status.</para></listitem>
|
||
</varlistentry>
|
||
|
||
<varlistentry>
|
||
<term><methodname>waitUntilSucceeds</methodname></term>
|
||
<listitem><para>Repeat a shell command with 1-second intervals
|
||
until it succeeds.</para></listitem>
|
||
</varlistentry>
|
||
|
||
<varlistentry>
|
||
<term><methodname>waitUntilFails</methodname></term>
|
||
<listitem><para>Repeat a shell command with 1-second intervals
|
||
until it fails.</para></listitem>
|
||
</varlistentry>
|
||
|
||
<varlistentry>
|
||
<term><methodname>waitForUnit</methodname></term>
|
||
<listitem><para>Wait until the specified systemd unit has reached
|
||
the “active” state.</para></listitem>
|
||
</varlistentry>
|
||
|
||
<varlistentry>
|
||
<term><methodname>waitForFile</methodname></term>
|
||
<listitem><para>Wait until the specified file
|
||
exists.</para></listitem>
|
||
</varlistentry>
|
||
|
||
<varlistentry>
|
||
<term><methodname>waitForOpenPort</methodname></term>
|
||
<listitem><para>Wait until a process is listening on the given TCP
|
||
port (on <literal>localhost</literal>, at least).</para></listitem>
|
||
</varlistentry>
|
||
|
||
<varlistentry>
|
||
<term><methodname>waitForClosedPort</methodname></term>
|
||
<listitem><para>Wait until nobody is listening on the given TCP
|
||
port.</para></listitem>
|
||
</varlistentry>
|
||
|
||
<varlistentry>
|
||
<term><methodname>waitForX</methodname></term>
|
||
<listitem><para>Wait until the X11 server is accepting
|
||
connections.</para></listitem>
|
||
</varlistentry>
|
||
|
||
<varlistentry>
|
||
<term><methodname>waitForText</methodname></term>
|
||
<listitem><para>Wait until the supplied regular expressions matches
|
||
the textual contents of the screen by using optical character recognition
|
||
(see <methodname>getScreenText</methodname>).</para>
|
||
<note><para>This requires passing <option>enableOCR</option> to the test
|
||
attribute set.</para></note></listitem>
|
||
</varlistentry>
|
||
|
||
<varlistentry>
|
||
<term><methodname>waitForWindow</methodname></term>
|
||
<listitem><para>Wait until an X11 window has appeared whose name
|
||
matches the given regular expression, e.g.,
|
||
<literal>waitForWindow(qr/Terminal/)</literal>.</para></listitem>
|
||
</varlistentry>
|
||
|
||
<varlistentry>
|
||
<term><methodname>copyFileFromHost</methodname></term>
|
||
<listitem><para>Copies a file from host to machine, e.g.,
|
||
<literal>copyFileFromHost("myfile", "/etc/my/important/file")</literal>.</para>
|
||
<para>The first argument is the file on the host. The file needs to be
|
||
accessible while building the nix derivation. The second argument is
|
||
the location of the file on the machine.</para>
|
||
</listitem>
|
||
</varlistentry>
|
||
|
||
<varlistentry>
|
||
<term><methodname>systemctl</methodname></term>
|
||
<listitem>
|
||
<para>Runs <literal>systemctl</literal> commands with optional support for
|
||
<literal>systemctl --user</literal></para>
|
||
<para>
|
||
<programlisting>
|
||
$machine->systemctl("list-jobs --no-pager"); // runs `systemctl list-jobs --no-pager`
|
||
$machine->systemctl("list-jobs --no-pager", "any-user"); // spawns a shell for `any-user` and runs `systemctl --user list-jobs --no-pager`
|
||
</programlisting>
|
||
</para>
|
||
</listitem>
|
||
</varlistentry>
|
||
|
||
</variablelist>
|
||
|
||
</para>
|
||
|
||
<para>
|
||
To test user units declared by <literal>systemd.user.services</literal> the optional <literal>$user</literal>
|
||
argument can be used:
|
||
|
||
<programlisting>
|
||
$machine->start;
|
||
$machine->waitForX;
|
||
$machine->waitForUnit("xautolock.service", "x-session-user");
|
||
</programlisting>
|
||
|
||
This applies to <literal>systemctl</literal>, <literal>getUnitInfo</literal>,
|
||
<literal>waitForUnit</literal>, <literal>startJob</literal>
|
||
and <literal>stopJob</literal>.
|
||
</para>
|
||
|
||
</section>
|