Tutorial 1.8
VNUML Tutorial
Authors: Fermín Galán (galan at dit.upm.es) David Fernández (david at dit.upm.es) version 1.8, May 14th, 2007
Contents
Introduction
VNUML (Virtual Network User Mode Linux) is a general purpose virtualization tool designed to easily create complex network simulation scenarios based on User Mode Linux (UML) virtualization software. It has been initially developed in the context of Euro6IX research project to simulate IPv6 IX scenarios based on Linux and zebra routing daemon. However, it is a useful tool that can be used to simulate general Linux based network scenarios.
VNUML is aimed to help in testing network applications and services over complex testbeds made of several nodes (even tens) and networks inside one Linux machine, without involving the investment and management complexity needed to create it using real equipment.
This tutorial explains in detail how to use VNUML tool to simulate a simple networking scenario. It is intended as a hands-on approach for people that wants to start using VNUML tool, as well as an overview of the philosophy and features behind VNUML tool for people that do not want to install it yet.
Although VNUML hides from the user most of the details needed to use UML virtualization software, a minimum knowledge about UML and how it works is needed. The reader is referred to UML Web site to gain that knowledge.
Throughout this document and the rest of the documentation, we refer to the machine where VNUML is running as the "host". The Linux virtual machines that run over the host using UML virtualization software are typically referred as "virtual machines" or simply "UMLs".
This tutorial covers VNUML language version 1.8 and vnumlparser.pl 1.8.X.
Tutorial Requirements
In order to properly follow this tutorial you need:
- VNUML properly installed in your system. Use the most recent stable release. Please, refer to the VNUML Installation Guide for more details.
- UML kernel (we recomend the most recent stable one provided by the VNUML Project).
- Root_fs filesystem (we recomend the most recent stable one provided by the VNUML Project).
- X-window environment. Some steps in the tutorial will launch xterm terminals, so you need a X server.
- Be able to log as root in the host (for step 3 and step 4).
- The 10.0.x.x and 10.250.x.x address ranges are used in the VNUML specifications shown in this tutorial. If these ranges collide with assigned addresses in your host environment you must to change them to other prefixes.
Important note: we are assuming along this document that you installed VNUML through the .deb package. Therefore, /usr/share is used as base of for pathnames (e.g., kernels, fileystems, examples directory, etc.). Otherwise (i.e., if you installed VNUML using the source .tar.gz), you shouls consider /usr/local/share instead.
A quick way of getting a proper VNUML experimentation environment without worrying about the issues above, is to use the VNUML LiveDVD (except maybe for the last one, in the case you were using the VNUML in a not network isolated way).
Privileges levels
Old versions of VNUML parser (previous to 1.6.0) were very root-focused. Fortunately, VNUML support for conventional users was highly improved in recent versions.
There are three ways of using VNUML, depending of what privileges you want to use:
- No Root Privileges at All. To run vnumlparser.pl without root privileges. The main limitation is that there is no network access to the virtual machines from/to host and, therefore, no access to the external network the host is connected to (if any). You can interact with the virtual machines through direct login (typically, a xterm), but not using a management network. Command sequences can be run by vnumlparser.pl in the virtual machines.
- Some Root Privileges. To run vnumlparser.pl without root privileges, but use root privileges to enable host network access through a management network and access to external networks using the host as router/NAT (layer 3 interconnection).
- Full Root Privileges.To run vnumlparser.pl as root. This allow host configuration performed by vnumlparser.pl, private network management, mapping of virtual machine names in /etc/hosts and direct connection of virtual machines to external networks the host is connected to (layer 2 interconnection).
In conclusion, the most of the functionality of VNUML is available with the "No Root Privileges at All" level: you only need the other two levels if need special features, like the management network (which importance is much lesser in VNUML 1.8 than in previous versions because now command execution doesn't need it), or layer 2/3 interconnection with external network.
The rest of the tutorial describe these three ways in detail, starting with No Root Privileges at All, going to Some Root Privileges and finally, Full Root Privileges.
No Root Privileges at All
Writing the VNUML Specification
VNUML tool is made of two main components: the VNUML language used for describing simulations in XML; and the interpreter of the language (vnumlparser.pl), that builds and manages the simulation, hiding all UML complex details to the user.
If you managed to successfully install VNUML parser and all related software packages, one of the hardest steps is over :). Now it is time to create the XML description of the scenario to be simulated using VNUML language (see VNUML Language Reference for a complete description of the language).
The specification of the tutorial scenario in VNUML language follows:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE vnuml SYSTEM "/usr/share/xml/vnuml/vnuml.dtd">
<vnuml>
<global>
<version>1.8</version>
<simulation_name>tutorial-lu</simulation_name>
<automac/>
<vm_mgmt type="none" />
<vm_defaults exec_mode="mconsole">
<filesystem type="cow">/usr/share/vnuml/filesystems/root_fs_tutorial</filesystem>
<kernel>/usr/share/vnuml/kernels/linux</kernel>
<console id="0">xterm</console>
</vm_defaults>
</global>
<net name="Net0" mode="uml_switch" />
<net name="Net1" mode="uml_switch" />
<net name="Net2" mode="uml_switch" />
<vm name="uml1">
<if id="1" net="Net0">
<ipv4>10.0.0.1</ipv4>
</if>
<route type="ipv4" gw="10.0.0.3">default</route>
<exec seq="start" type="verbatim">nohup /usr/bin/hello </dev/null >/dev/null 2>&1 &</exec>
<exec seq="stop" type="verbatim">killall hello</exec>
</vm>
<vm name="uml2">
<if id="1" net="Net0">
<ipv4>10.0.0.2</ipv4>
</if>
<route type="ipv4" gw="10.0.0.3">default</route>
</vm>
<vm name="uml3">
<if id="1" net="Net0">
<ipv4>10.0.0.3</ipv4>
</if>
<if id="2" net="Net1">
<ipv4>10.0.1.1</ipv4>
</if>
<route type="ipv4" gw="10.0.1.2">10.0.2.0/24</route>
<forwarding type="ip" />
</vm>
<vm name="uml4">
<if id="1" net="Net1">
<ipv4>10.0.1.2</ipv4>
</if>
<if id="2" net="Net2">
<ipv4>10.0.2.1</ipv4>
</if>
<route type="ipv4" gw="10.0.1.1">default</route>
<forwarding type="ip" />
</vm>
<vm name="uml5">
<if id="1" net="Net2">
<ipv4>10.0.2.2</ipv4>
</if>
<route type="ipv4" gw="10.0.2.1">default</route>
</vm>
</vnuml>
Just copy the above XML code to a file named tutorial_limited_user.xml and save it (if you don't want to do the copy-paste, you can find it in the examples directory after VNUML installation in /usr/share/examples).
Some initial comments about this description file:
- The <global> section defines certain parameters and options that apply to the whole simulation. Within it, the <vm_defaults> tag defines some defaults for all virtual machines, like the kernel and filesystem. Note the <console> tag, that defines that we want a xterm to interact with each virtual machine.
- Virtual networks are created using the <net> tag. Each network is identified by a name (e.g., "Net0") which is used later in virtual machines descriptions to define the network each virtual interface is connected to.
- Virtual machines (or UMLs) are created and described using the <vm> tag. Inside <vm> section, several different tags can be included to define each UML characteristics. There are lot of tags that can be used in a virtual machine but, well, this is a tutorial :), so only the basic ones are shown here:
- <if> tag is used to create and describe network interfaces (<ipv4> tag is used to set up the IP address; <ipv6> is supported also),
- <route> tag is used to add static routes, and
- <forwarding> to enable forwarding. Note we use this tag in uml3 and uml4, the virtual machines that will act as routers (the others are end-systems).
- The <exec> tags in uml1 will be explained later in this document, when dealing with automatic command execution.
Building the Simulation
Once you have the VNUML specification written, you can build the scenario using "-t" switch of VNUML parser. That is:
bauer@host$ vnumlparser.pl -t tutorial_limited_user.xml -v
This command builds the virtual network topology described in tutorial_limited_user.xml file and boots all the virtual machines defined inside it.
After the command execution, you will see a lot of output messages, showing what the parser is exactly doing. That is because "-v" (verbose) switch is being used (we recommend always using "-v" to see how things are going).
By default, the output of each UML is redirected to the null device; if you want to see that output (getting an even more verbose output) use "-o" switch (for example "-o /tmp/boot").
After a few seconds, you should see five xterms popping up in sequence: each one correspond to one of the virtual machines defined in the simulation. After the booting process you would identify each one by the name in the login prompt (uml1, uml2, uml3, uml4 and uml5). VNUML allows several modes to interact with the virtual machines, xterm is the only way allowed if you don't have root privileges.
Be patient, because the boot process of each UML takes a while (think about how long it takes a conventional Linux kernel to boot; virtualized ones can take even more). It can take easily several minutes to boot each UML, so it could take from 1 minute (or less!) to around 10-15 minutes to boot the tutorial scenario (time depends greatly on the host machine characteristics). Maybe even you get the following prompt:
Boot timeout for virtual machine uml1 reached. Abort, Retry, or Continue? [A/r/c]:
This use to mean that vnumlparser.pl has got tired of waitting for a virtual machine. It could be that the virtual machine has not finishing booting yet (that means, the login prompt has not appeared yet). Try 'r' in that case. If the problem persists, contact the vnuml-user mailing list.
Testing the Simulation
You can log in the virtual machines as root user using xxxx (for times the 'x' letter) as password. After logging, you can execute commands as in a conventional GNU/Linux machine. For example try: 'ls /', 'ifconfig', 'df -h' or 'uname -ra'.
To check connectivity between virtual machines you can use ping or traceroute, as shown in the examples below. For example, from uml1 try (you should see a similar trace, if everything is going right):
uml1:~ # traceroute -n 10.0.2.2 traceroute to 10.0.2.2 (10.0.2.2), 30 hops max, 38 byte packets 1 10.0.0.3 1.282 ms 0.726 ms 0.678 ms 2 10.0.1.2 1.243 ms 1.462 ms 1.010 ms 3 10.0.2.2 1.396 ms 1.345 ms 1.303 ms
From uml5 try:
uml5:~ # traceroute -n 10.0.0.1 traceroute to 10.0.0.1 (10.0.0.1), 30 hops max, 38 byte packets 1 10.0.2.1 0.916 ms 0.743 ms 0.681 ms 2 10.0.1.1 1.072 ms 1.019 ms 1.009 ms 3 10.0.0.1 1.420 ms 1.363 ms 1.352 ms
You will see (using ifconfig) that the interfaces in the virtual machines starts with eth1, not with eth0 as usual. Actually the eth0 interface is reserved to the management network, that will be described later.
Releasing the Simulation Scenario
Once you have finished playing around with the simulation scenario, you can release it in a "polite way" by means of "-d" switch of VNUML parser.
bauer@host$ vnumlparser.pl -d tutorial_limited_user.xml -v
This sends a CAD (Ctrl+Alt+Delete) to each virtual machine, then you will see how the halting process begins. After waiting for a while (think that a conventional kernel can take a couple of minutes to halt, so a set of virtualized ones take quite a bit longer), the simulation scenario should be released (all virtual machines should have been halted, xterms closed and virtual networks released).
After releasing the scenario in this way, you can recreate it with "-t". Note that releasing a scenario does not mean that filesystems changes are lost. On the contrary, any change performed on them remains after the scenario has been released and restarted again. You can check it: 'touch /root/foo.bar' and you will see how the file /root/foo.bar remains next time you start using "-t" (the magic behind this resides in certain files under ~/.vnuml that stores user additions to the virtual machine filesystems: this files can be erased using the "-P", keep reading :).
You can also use the -F switch in the releasing mode, as follows:
bauer@host$ vnumlparser.pl -d tutorial_limited_user.xml -v -F
Instead of sending the CAD signal to the virtual machines, this sends a HALT signal, that is stronger: the virtual kernel halts immediately. This could corrupt the virtual filesystems so you only should use it when plain -d has failed.
There is even a stronger way to release the simulation: the purge mode (-P), but it should only be used as a last-resort measure. See VNUML User Manual for more details when -d mode fails (including the purge mode -P).
When you get familiar starting and releasing simulation, continue to next section about how to execute command sequences in the virtual machines.
Command Sequences
Once the simulated scenario is up and running, you can login in each virtual machine and execute the commands you desire. You could, for example, manually start a web or ftp server inside uml1 and uml5 virtual machines. You could also start routing daemons in uml3 and uml4 in order to set-up dynamic routing for the scenario. You could even start automatically such services from the init scripts in the root filesystem being used, although that would probably require the use of a different filesystem for each UML.
However, in order to be able to easily create and reproduce simulation scenarios, a simple and powerful way to specify the commands that each virtual machine has to execute to start the scenario is needed, as well as the commands to stop the simulation and return the virtual machine to a known state.
That is basically what VNUML offers: a way to automate the execution of commands in the virtual machines and host using the execute commands mode (-x switch). By means of the <exec> tag included inside each virtual machine definition, you can specify the commands that will be executed. Commands are grouped using a command sequence identifier, specified in the seq attribute of the <exec> tag. In this tutorial we will consider two command sequences: "start" and "stop".
To show this functionality in this tutorial we will use the /usr/bin/hello program included in the tutorial root filesystem. It is just a quite simple program whose only function is to output the 'hello' word to file /tmp/hello every 10 seconds (it is only for testing purposes, we suppose you'll use VNUML for more useful tasks :).
In order to test it, look at the lines at the end of uml1 description in tutorial_user2.xml:
<exec seq="start" type="verbatim"> nohup /usr/bin/hello </dev/null >/dev/null 2>&1 & </exec> <exec seq="stop" type="verbatim">killall hello</exec>
Basically, we have specified the following command to be executed in uml1 whenever the simulation is started (note that <, > and & characters had to be escaped in XML files):
nohup /usr/bin/hello </dev/null >/dev/null 2>&1 &
(Why not simply use 'nohup /usr/bin/hello &'? There are ssh related problems that force to redirect standard input and output; look at VNUML User Manual for more details).
And the following one to be executed in uml1 whenever the simulation is stopped:
killall hello
Start the simulation using:
bauer@host$ vnumlparser.pl -x start@tutorial_limited_user.xml -v
Note that the process is much shorter than when building the simulation scenario (mode -t). Check the existence and growing of /tmp/hello file in uml1: the hello daemon is working.
uml1:~ # tail -f /tmp/hello Hello! Current date: Thu Jul 20 10:14:43 UTC 2006 Hello! Current date: Thu Jul 20 10:14:53 UTC 2006 Hello! Current date: Thu Jul 20 10:15:04 UTC 2006 Hello! Current date: Thu Jul 20 10:15:14 UTC 2006 Hello! Current date: Thu Jul 20 10:15:25 UTC 2006 Hello! Current date: Thu Jul 20 10:15:35 UTC 2006 Hello! Current date: Thu Jul 20 10:15:45 UTC 2006
In a similar way you can stop the simulation (killing the hello daemon) with:
bauer@host$ vnumlparser.pl -x stop@tutorial_limited_user.xml -v
Access again to uml1 and check that hello daemon is not working now ('ps axw | grep hello' for example) and that /tmp/hello file has stopped growing.
Once you have finished playing around with the simulation scenario, release it as usual:
bauer@host$ vnumlparser.pl -d tutorial_limited_user.xml -v
There are several executions modes. In this case, where are specifying exec_mode="mconsole" in <vm_defaults>. The mconsole mode is the recommended one: it is quick and available without any root privileges. There is another, the "net" mode (the only one available in pre-1.8 versions), which is slower and needs root privileges (and example is discussed later in this document) so it should be avoided except in very rare exceptions (these reasons are too complex to be explained in a tutorial :)
Some Root Privileges
In this case, you are supposed to run vnumparser.pl as a conventional user as in "No Root Privileges at All" case, but you also have root access in order to run some commands as superuser (maybe using sudo). You will be able to connect the simulation to the host (this way virtual machines could reach Internet, if the host acts as router/NAT) and to create a management network to access virtual machines through it (typically, using SSH) without needing xterm. This features will be described following.
Connecting Simulations to Host
The following is a variation of the tutorial_user_limited.xml file. Copy it to a file named tutorial_user1.xml and save it (if you don't want to do the copy-paste, you can find it in the examples directory after VNUML installation in /usr/share/examples).
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE vnuml SYSTEM "/usr/share/xml/vnuml/vnuml.dtd">
<vnuml>
<global>
<version>1.8</version>
<simulation_name>tutorial-u1</simulation_name>
<automac/>
<vm_mgmt type="none" />
<vm_defaults exec_mode="mconsole">
<filesystem type="cow">/usr/share/vnuml/filesystems/root_fs_tutorial</filesystem>
<kernel>/usr/share/vnuml/kernels/linux</kernel>
<console id="0">xterm</console>
</vm_defaults></dev/null >/dev/null 2>&1 &
</global>
<net name="Net0" mode="uml_switch" />
<net name="Net1" mode="uml_switch" />
<net name="Net2" mode="uml_switch" />
<net name="Net3" mode="uml_switch" sock="/var/run/vnuml/Net3.ctl" />
<vm name="uml1">
<if id="1" net="Net0">
<ipv4>10.0.0.1</ipv4>
</if>
<route type="ipv4" gw="10.0.0.3">default</route>
<exec seq="start" type="verbatim">nohup /usr/bin/hello </dev/null >/dev/null 2>&1 &</exec>
<exec seq="stop" type="verbatim">killall hello</exec>
</vm>
<vm name="uml2">
<if id="1" net="Net0">
<ipv4>10.0.0.2</ipv4>
</if>
<route type="ipv4" gw="10.0.0.3">default</route>
</vm>
<vm name="uml3">
<if id="1" net="Net0">
<ipv4>10.0.0.3</ipv4>
</if>
<if id="2" net="Net1">
<ipv4>10.0.1.1</ipv4>
</if>
<if id="3" net="Net3">
<ipv4>10.0.3.1</ipv4>
</if>
<route type="ipv4" gw="10.0.1.2">10.0.2.0/24</route>
<forwarding type="ip" />
</vm>
<vm name="uml4">
<if id="1" net="Net1">
<ipv4>10.0.1.2</ipv4>
</if>
<if id="2" net="Net2">
<ipv4>10.0.2.1</ipv4>
</if>
<route type="ipv4" gw="10.0.1.1">default</route>
<forwarding type="ip" />
</vm>
<vm name="uml5">
<if id="1" net="Net2">
<ipv4>10.0.2.2</ipv4>
</if>
<route type="ipv4" gw="10.0.2.1">default</route>
</vm>
</vnuml>
The difference is the Net3 network in a <net> tag, linked to the virtual machine uml3 in a <if> tag. This is the network that will be used to get connectivity to the host.
In order to have access to the host machine when running vnumlparser.pl as a non-privileged user, certain configurations must be performed by a privileged user prior to building the simulation scenario. In this example, the non-privileged user that will run vnumlparser.pl is bauer.
As part of the VNUML install process, a non-privileged user vnuml was created (check it now with 'id vnuml'). This user is a member of the uml-net group, which owns the /dev/net/tun cloning device. It will own the uml_switch process that will be started for the Net3 network, so that this process won't need to be run by root.
Warning: we have been reported about some distributions where the ownership configuration of /dev/net/tun during VNUML installation doesn't work properly. In that case, you can try to fix it manually or use 'chmod 666 /dev/net/tun' (however, giving permissions to everyone should be avoided as a permanent solution, due to its security implications).
- The user bauer must be a member of the vnuml group which owns the directory /var/run/vnuml, where the UNIX sockets for the uml_switch processes will be stored. (The command below looks intimidating, but really all it does is add the user bauer to the vnuml group, while preserving its current group membership; maybe you prefer simple edit the /etc/group file). Note: as group membership is not updated in running shells, after adding user bauer to vnuml group, user bauer has to start a new shell or xterm session before executing the simulation.
- Create a TUN/TAP device owned by the vnuml user. This TUN/TAP device acts as the termination point for the host connection in the Net3 virtual network.
- Set an address for the host in the Net3 network (for example, 10.0.3.2) and configure a static route in the host to reach all the virtual machines using uml3 as gateway.
- Start a uml_switch process, linked with the TUN/TAP device created previously (tap0) and owned by the vnuml user. The socket location is arbitrary. However, the /var/run/vnuml directory was created at installation for this purpose. The socket location will be used as the sock attribute in the <net> tag for Net3.
- Give the vnuml group read/write privileges to the socket that is used. Note that the UNIX socket for the uml_switch process isn't created instantly, so if these commands are scripted, a minor wait (sleep) is necessary before setting permissions on the file. (Of course, the 'ls' commands are not really needed, but they are useful to check the chmod commands has been sucessfull).
root@host# usermod -G vnuml,`groups bauer | sed 's/^.\+:[[:space:]]*//;s/[[:space:]]\\+/,/g'` bauer
root@host# tunctl -u vnuml -t tap0 Set 'tap0' persistent and owned by uid 1037
root@host# ifconfig tap0 10.0.3.2 netmask 255.255.255.0 up root@host# route -A inet add -net 10.0.0.0/16 gw 10.0.3.1
root@host# su -pc 'uml_switch -tap tap0 -unix /var/run/vnuml/Net3.ctl < /dev/null > /dev/null &' vnuml
root@host# sleep 1 root@host# ls -l /var/run/vnuml/Net3.ctl srwxr-xr-x 1 vnuml vnuml 0 2005-07-29 11:48 /var/run/vnuml/Net3.ctl root@host# chmod g+rw /var/run/vnuml/Net3.ctl root@host# ls -l /var/run/vnuml/Net3.ctl srwxrwxr-x 1 vnuml vnuml 0 2005-07-29 11:48 /var/run/vnuml/Net3.ctl
Now that the /var/run/vnuml/Net3.ctl exists and is readable and writeable by bauer, you can start the simulation.
bauer@host$ vnumlparser.pl -t tutorial_user1.xml -v
You should see the xterms starting as usual. After they have finished booting, you could check that now the host is part of the simulation. From uml1 you can try:
uml1:~ # traceroute -n 10.0.3.2 traceroute to 10.0.3.2 (10.0.3.2), 30 hops max, 38 byte packets 1 10.0.0.3 1.779 ms 2.572 ms 1.359 ms 2 10.0.3.2 1.785 ms 1.641 ms 1.744 ms
From uml5 you can try:
uml5:~ # traceroute -n 10.0.3.2 traceroute to 10.0.3.2 (10.0.3.2), 30 hops max, 38 byte packets 1 10.0.2.1 1.407 ms 0.857 ms 1.243 ms 2 10.0.1.1 1.265 ms 1.019 ms 0.980 ms 3 10.0.3.2 1.545 ms 3.156 ms 2.033 ms
From host try:
bauer@host$ traceroute -n 10.0.0.1 traceroute to 10.0.0.1 (10.0.0.1), 30 hops max, 40 byte packets 1 10.0.3.1 0.000 ms 0.000 ms 0.000 ms 2 10.0.0.1 0.000 ms 0.000 ms 0.000 ms bauer@host$ traceroute -n 10.0.2.2 traceroute to 10.0.2.2 (10.0.2.2), 30 hops max, 40 byte packets 1 10.0.3.1 0.000 ms 0.000 ms 0.000 ms 2 10.0.1.2 0.000 ms 0.000 ms 0.000 ms 3 10.0.2.2 0.000 ms 0.000 ms 0.000 ms
Virtual machines can reach the host this way. The scenario could be improved by configuring routing and NATing properly in the host, in order for virtual machines to reach any host on the Internet. Such a setup would be a layer 3 interconnection: the host acts as router/NAT between virtual machines and the external network. Details to configure NAT this way can be found later in this document. A stronger interconnection (layer 2) can be implemented, so virtual machines are connected to the external network directly, but root privileges are needed. We describe layer 2 interconnection later.
To release the simulation:
bauer@host$ vnumlparser.pl -d tutorial_user1.xml -v
When the simulation has been released, note that the uml_switch process for Net3 is still running, and its accompanying UNIX socket still exists in /var/run/vnuml/Net3.ctl. Also, the tap0 device is still configured, and its gateway still exists. As long as this setup remains, new simulation scenarios may be built using the same configuration without requiring it to be set up again.
When the user has finished running simulations that involve the network set up for Net3, the uml_switch process can be killed, and the socket removed:
root@host# kill `lsof -t /var/run/vnuml/Net3.ctl` /var/run/vnuml/Net3.ctl: 6280 root@host# rm /var/run/vnuml/Net3.ctl
Now the tap0 device may be deconfigured:
root@host# ifconfig tap0 down root@host# tunctl -d tap0 Set 'tap0' nonpersistent
Management network
VNUML allows configuring a management network that can be used to provide a direct network access from host to virtual machines. This management network has to main goals:
- Login into virtual machines when direct login is not possible (for example, because no X enviroment is available to run xterm)
- Execute command sequences using "net" execution mode, when "mconsole" method (described previously) can not be used.
Both features are based on SSH interactions, so, previously to describe them we need to generate a SSH key.
Generating SSH key
To avoid being asked for a password whenever accessing virtual machines trough SSH in the management network, you must generate a public key. Later, vnumlparser.pl will deal with the automatic installation of this key in the virtual machines (though the <ssh_key> tag).
To create the public key just type:
bauer@host$ ssh-keygen -t rsa1
Select the default file to save the key (~/.ssh/identity). The passphrase is not relevant (although we recommend to use an empty passphrase, in order no user input required whenever the key is used).
Anyway, the first time (and only once) that SSH is used, you will have to confirm the server key answering 'yes' to the question you will be asked.
Since 1.8.5: typing 'yes' during the first SSH interaction is no longer required (because now VNUML parser uses the 'StrictHostKeyChecking no' option).
Management Network
The following is a variation of the tutorial_user_limited.xml file. Copy it to a file named tutorial_user2.xml and save it (if you don't want to do the copy-paste, you can find it in the examples directory after VNUML installation in /usr/share/examples).
<?xml version="1.0" encoding="UTF-8">
<!DOCTYPE vnuml SYSTEM "/usr/share/xml/vnuml/vnuml.dtd">
<vnuml>
<global>
<version>1.8</version>
<simulation_name>tutorial-u2</simulation_name>
<ssh_version>1</ssh_version>
<ssh_key>~/.ssh/identity.pub</ssh_key>
<automac/>
<vm_mgmt type="net" network="10.250.0.0" mask="24">
<mgmt_net sock="/var/run/vnuml/Mgmt_net.ctl" hostip="10.250.0.1"/>
</vm_mgmt>
<vm_defaults exec_mode="net">
<filesystem type="cow">/usr/share/vnuml/filesystems/root_fs_tutorial</filesystem>
<kernel>/usr/share/vnuml/kernels/linux</kernel>
<console id="0">xterm</console>
</vm_defaults>
</global>
<net name="Net0" mode="uml_switch" />
<net name="Net1" mode="uml_switch" />
<net name="Net2" mode="uml_switch" />
<vm name="uml1">
<if id="1" net="Net0">
<ipv4>10.0.0.1</ipv4>
</if>
<route type="ipv4" gw="10.0.0.3">default</route>
<exec seq="start" type="verbatim">nohup /usr/bin/hello </dev/null >/dev/null 2>&1 &</exec>
<exec seq="stop" type="verbatim">killall hello</exec>
</vm>
<vm name="uml2">
<if id="1" net="Net0">
<ipv4>10.0.0.2</ipv4>
</if>
<route type="ipv4" gw="10.0.0.3">default</route>
</vm>
<vm name="uml3">
<if id="1" net="Net0">
<ipv4>10.0.0.3</ipv4>
</if>
<if id="2" net="Net1">
<ipv4>10.0.1.1</ipv4>
</if>
<route type="ipv4" gw="10.0.1.2">10.0.2.0/24</route>
<forwarding type="ip" />
</vm>
<vm name="uml4">
<if id="1" net="Net1">
<ipv4>10.0.1.2</ipv4>
</if>
<if id="2" net="Net2">
<ipv4>10.0.2.1</ipv4>
</if>
<route type="ipv4" gw="10.0.1.1">default</route>
<forwarding type="ip" />
</vm>
<vm name="uml5">
<if id="1" net="Net2">
<ipv4>10.0.2.2</ipv4>
</if>
<route type="ipv4" gw="10.0.2.1">default</route>
</vm>
</vnuml>
VNUML uses SSH to send command sequences to the virtual machines. The SSH protocol version and the SSH public key to be used are specified in the <ssh_version> and <ssh_key> tags, respectively. Note that the SSH key file is the one generated in the previous step.
There are two kind of management networks: private--establishes a peer-to-peer connection between the host and each one of the virtual machines (this requires that vnumlparser.pl be run as root); and net--a uml_switched network is used (the one herein described). This is set up with the type attribute in the <vm_mgmt> tag.
When using type="net" the network and mask attributes specify a network address range that vnumlparser.pl will use to assign addresses to the virtual machines. The hostip attribute in the <mgmt_net> tag will be skipped in that range, reserved for use by the host.
Let's see how the whole thing is working. As root, set up a tap device and start the uml_switch process (similar to what was done in the previous example):
root@host# tunctl -u vnuml -t tap0 Set 'tap0' persistent and owned by uid 1037 root@host# ifconfig tap0 10.250.0.1 netmask 255.255.255.0 up root@host# su -pc 'uml_switch -tap tap0 -unix /var/run/vnuml/Mgmt_net.ctl < /dev/null > /dev/null &' vnuml root@host# sleep 1 root@host# chmod g+rw /var/run/vnuml/Mgmt_net.ctl
Next, build the simulation:
bauer@host$ vnumlparser.pl -t tutorial_user2.xml -v
After logging in to the xterms, you can check using ifconfig that an additional interface is configured in each machine: the eth0 interface. All virtual machines are connected to the host (and between them, as side effect), with the following addresses:
| host | 10.250.0.1 | assigned with the ifconfig command, before building simulation |
| uml1 | 10.250.0.2 | assigned automatically by vnumlparser.pl |
| uml2 | 10.250.0.3 | assigned automatically by vnumlparser.pl |
| uml3 | 10.250.0.4 | assigned automatically by vnumlparser.pl |
| uml4 | 10.250.0.5 | assigned automatically by vnumlparser.pl |
| uml5 | 10.250.0.6 | assigned automatically by vnumlparser.pl |
You can login in each virtual machine from host using SSH. For example, to log in uml3:
bauer@host$ ssh -1 root@10.250.0.4 Last login: Mon Oct 16 00:53:47 2006 from 10.250.0.1 Linux (none) 2.6.16.27-bs2-xt-1m #2 Tue Aug 1 03:39:24 CEST 2006 i686 The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. uml3:~#
Note that, given that you can SSH to each virtual machine from host, you don't need xterms (just remove the <console> tag in the VNUML file) if you configure the management network properly. This is very useful when you have a lot of virtual machines in your simulation (ten or more) and don't want to be overwhelmed by the xterms.
Command Sequences Through Management Network
Command sequences can be also executed using the management network, as alternative to the procedure described in previously. The procedure is exactly the same.
Note that the only difference is that XML file is that we are using now exec_mode="net" instead of "mconsole" in <vm_defaults> tag.
To start:
bauer@host$ vnumlparser.pl -x start@tutorial_user2.xml -v
To stop:
bauer@host$ vnumlparser.pl -x stop@tutorial_user2.xml -v
Releasing the Management Network
Once you have finished playing around with the simulation scenario, release it as usual:.
bauer@host$ vnumlparser.pl -d tutorial_user2.xml -v
As with the previous example, in order to reset to the original state, the uml_switch process will need to be killed and its accompanying UNIX socket removed. Also the tap device will need will need to be deconfigured and removed.
root@host# kill `lsof -t /var/run/vnuml/Mgmt_net.ctl` /var/run/vnuml/Mgmt_net.ctl: 13746 root@host# rm /var/run/vnuml/Mgmt_net.ctl root@host# ifconfig tap0 down root@host# tunctl -d tap0 Set 'tap0' nonpersistent
Full Root Privileges
Host Configuration
The <host> tag, that can only be used if root runs vnumlparser.pl allows to automate host configuration. For example, the following example (tutorial_root1.xml) is functionally equivalent to tutorial_user1.xml (described here), but vnumlparser.pl deals with the tap0 creation and uml_switch starting. Note also that sock attribute in <host> Net3 is not used.
<?xml version="1.0" encoding="UTF-8">
<!DOCTYPE vnuml SYSTEM "/usr/share/xml/vnuml/vnuml.dtd">
<vnuml>
<global>
<version>1.8</version>
<simulation_name>tutorial-r1</simulation_name>
<automac/>
<vm_mgmt type="none" />
<vm_defaults exec_mode="mconsole">
<filesystem type="cow">/usr/share/vnuml/filesystems/root_fs_tutorial</filesystem>
<kernel>/usr/share/vnuml/kernels/linux</kernel>
<console id="0">xterm</console>
</vm_defaults>
</global>
<net name="Net0" mode="uml_switch" />
<net name="Net1" mode="uml_switch" />
<net name="Net2" mode="uml_switch" />
<net name="Net3" mode="uml_switch" />
<vm name="uml1">
<if id="1" net="Net0">
<ipv4>10.0.0.1</ipv4>
</if>
<route type="ipv4" gw="10.0.0.3">default</route>
<exec seq="start" type="verbatim">nohup /usr/bin/hello </dev/null >/dev/null 2>&1 &</exec>
<exec seq="stop" type="verbatim">killall hello</exec>
</vm>
<vm name="uml2">
<if id="1" net="Net0">
<ipv4>10.0.0.2</ipv4>
</if>
<route type="ipv4" gw="10.0.0.3">default</route>
</vm>
<vm name="uml3">
<if id="1" net="Net0">
<ipv4>10.0.0.3</ipv4>
</if>
<if id="2" net="Net1">
<ipv4>10.0.1.1</ipv4>
</if>
<if id="3" net="Net3">
<ipv4>10.0.3.1</ipv4>
</if>
<route type="ipv4" gw="10.0.1.2">10.0.2.0/24</route>
<forwarding type="ip" />
</vm>
<vm name="uml4">
<if id="1" net="Net1">
<ipv4>10.0.1.2</ipv4>
</if>
<if id="2" net="Net2">
<ipv4>10.0.2.1</ipv4>
</if>
<route type="ipv4" gw="10.0.1.1">default</route>
<forwarding type="ip" />
</vm>
<vm name="uml5">
<if id="1" net="Net2">
<ipv4>10.0.2.2</ipv4>
</if>
<route type="ipv4" gw="10.0.2.1">default</route>
</vm>
<host>
<hostif net="Net3">
<ipv4>10.0.3.2</ipv4>
</hostif>
<route type="ipv4" gw="10.0.3.1">10.0.0.0/16</route>
</host>
</vnuml>
To build, test and release the example use the usual commands:
root@host# vnumlparser.pl -t tutorial_root1.xml -v (...) root@host# traceroute -n 10.0.0.1 traceroute to 10.0.0.1 (10.0.0.1), 30 hops max, 40 byte packets 1 10.0.3.1 0.000 ms 0.000 ms 0.000 ms 2 10.0.0.1 0.000 ms 0.000 ms 0.000 ms root@host# traceroute -n 10.0.2.2 traceroute to 10.0.2.2 (10.0.2.2), 30 hops max, 40 byte packets 1 10.0.3.1 0.000 ms 0.000 ms 0.000 ms 2 10.0.1.2 0.000 ms 0.000 ms 0.000 ms 3 10.0.2.2 0.000 ms 0.000 ms 0.000 ms root@host# vnumlparser.pl -d tutorial_root1.xml -v (...)
Private Management
You can use type="private" in <vm_mgmt> to establish peer-to-peer connection between the host and each one of the virtual machines (/30 networks). The <mgmt_net> tag is not used in this case. The following example (tutorial_root2.xml) is functionally equivalent to tutorial_user2.xml (described here), but using private management.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE vnuml SYSTEM "/usr/share/xml/vnuml/vnuml.dtd">
<vnuml>
<global>
<version>1.8</version>
<simulation_name>tutorial-r2</simulation_name>
<ssh_version>1</ssh_version>
<ssh_key>~/.ssh/identity.pub</ssh_key>
<automac/>
<vm_mgmt type="private" network="10.250.0.0" mask="24">
<host_mapping />
</vm_mgmt>
<vm_defaults exec_mode="net">
<filesystem type="cow">/usr/local/vnuml/filesystems/root_fs_tutorial</filesystem>
<kernel>/usr/share/vnuml/kernels/linux</kernel>
<console id="0">xterm</console>
</vm_defaults>
</global>
<net name="Net0" mode="uml_switch" />
<net name="Net1" mode="uml_switch" />
<net name="Net2" mode="uml_switch" />
<vm name="uml1">
<if id="1" net="Net0">
<ipv4>10.0.0.1</ipv4>
</if>
<route type="ipv4" gw="10.0.0.3">default</route>
<exec seq="start" type="verbatim">nohup /usr/bin/hello </dev/null >/dev/null 2>&1 &</exec>
<exec seq="stop" type="verbatim">killall hello</exec>
</vm>
<vm name="uml2">
<if id="1" net="Net0">
<ipv4>10.0.0.2</ipv4>
</if>
<route type="ipv4" gw="10.0.0.3">default</route>
</vm>
<vm name="uml3">
<if id="1" net="Net0">
<ipv4>10.0.0.3</ipv4>
</if>
<if id="2" net="Net1">
<ipv4>10.0.1.1</ipv4>
</if>
<route type="ipv4" gw="10.0.1.2">10.0.2.0/24</route>
<forwarding type="ip" />
</vm>
<vm name="uml4">
<if id="1" net="Net1">
<ipv4>10.0.1.2</ipv4>
</if>
<if id="2" net="Net2">
<ipv4>10.0.2.1</ipv4>
</if>
<route type="ipv4" gw="10.0.1.1">default</route>
<forwarding type="ip" />
</vm>
<vm name="uml5">
<if id="1" net="Net2">
<ipv4>10.0.2.2</ipv4>
</if>
<route type="ipv4" gw="10.0.2.1">default</route>
</vm>
</vnuml>
The address assignment is as follows:
| uml connection | host-side | uml-side |
| uml1 | 10.250.0.1 | 10.250.0.2 |
| uml2 | 10.250.0.5 | 10.250.0.6 |
| uml3 | 10.250.0.9 | 10.250.0.10 |
| uml4 | 10.250.0.13 | 10.250.0.14 |
| uml5 | 10.250.0.17 | 10.250.0.18 |
Previously to build the simulation, you have to generate the SSH public key for the root user, as described in the generate ssh key section of this tutorial. Otherwise, vnumlparser.pl will complaint about the ~/.ssh/idenity.pub file.
The example uses the <host_mapping> tag, only allowed to root. This tag enables virtual machine name mapping in /etc/hosts while the simulation is running:
root@host# vnumlparser.pl -t tutorial_root2.xml -v ... root@host# cat /etc/hosts ... # BEGIN: tutorial-r2 # topology built: lun jul 4 21:48:08 CEST 2005 10.250.0.2 uml1 10.250.0.6 uml2 10.250.0.10 uml3 10.250.0.14 uml4 10.250.0.18 uml5 # END: tutorial-r2 ... root@host# vnumlparser.pl -d tutorial_root2.xml -v ... root@host# cat /etc/hosts ... # BEGIN: tutorial-r2 # topology destroyed: lun jul 4 22:04:16 CEST 2005 # END: tutorial-r2
Therefore, you can use 'ssh -1 uml1' instead of 'ssh -1 10.250.0.2', for example: more intuitive!
Layer 2 interconnection
Previously, we stated that is possible to connect a virtual machine directly to the host external network (layer 2 interconnection). To do so, a special kind of virtual network is needed, specified with attribute type="virtual_bridge" in <net> tag. In addition, the name of the physical interface in the host is specified with the external attribute.
The following VNUML description (tutorial_root3.xml) is a variation of tutorial_user1.xml (described here), that directly connects uml3 with the host external network, using IP 192.168.0.150 (of course, you have to change this address to one with meaning in your host external network).
<?xml version="1.0" encoding="UTF-8"?> <!ENTITY REDIR "</dev/null >/dev/null 2>&1 &">
<!DOCTYPE vnuml SYSTEM "/usr/share/xml/vnuml/vnuml.dtd">
<vnuml>
<global>
<version>1.8</version>
<simulation_name>tutorial-r3</simulation_name>
<automac/>
<vm_mgmt type="none" />
<vm_defaults exec_mode="mconsole">
<filesystem type="cow">/usr/share/vnuml/filesystems/root_fs_tutorial</filesystem>
<kernel>/usr/share/vnuml/kernels/linux</kernel>
<console id="0">xterm</console>
</vm_defaults>
</global>
<net name="Net0" mode="uml_switch" />
<net name="Net1" mode="uml_switch" />
<net name="Net2" mode="uml_switch" />
<net name="Net3" mode="virtual_bridge" external="eth0"/>
<vm name="uml1">
<if id="1" net="Net0">
<ipv4>10.0.0.1</ipv4>
</if>
<route type="ipv4" gw="10.0.0.3">default</route>
<exec seq="start" type="verbatim">nohup /usr/bin/hello </dev/null >/dev/null 2>&1 &</exec>
<exec seq="stop" type="verbatim">killall hello</exec>
</vm>
<vm name="uml2">
<if id="1" net="Net0">
<ipv4>10.0.0.2</ipv4>
</if>
<route type="ipv4" gw="10.0.0.3">default</route>
</vm>
<vm name="uml3">
<if id="1" net="Net0">
<ipv4>10.0.0.3</ipv4>
</if>
<if id="2" net="Net1">
<ipv4>10.0.1.1</ipv4>
</if>
<if id="3" net="Net3">
<ipv4>192.168.0.150</ipv4>
</if>
<route type="ipv4" gw="10.0.1.2">10.0.2.0/24</route>
<forwarding type="ip" />
</vm>
<vm name="uml4">
<if id="1" net="Net1">
<ipv4>10.0.1.2</ipv4>
</if>
<if id="2" net="Net2">
<ipv4>10.0.2.1</ipv4>
</if>
<route type="ipv4" gw="10.0.1.1">default</route>
<forwarding type="ip" />
</vm>
<vm name="uml5">
<if id="1" net="Net2">
<ipv4>10.0.2.2</ipv4>
</if>
<route type="ipv4" gw="10.0.2.1">default</route>
</vm>
<host>
<hostif net="Net3">
<ipv4>192.168.0.39</ipv4>
</hostif>
<physicalif name="eth0" ip="192.168.0.39" mask="255.255.255.0" gw="192.168.0.1" />
<route type="ipv4" gw="192.168.0.1">default</route>
</host>
</vnuml>
The <host> section is not mandatory, but very recommended due to the virtual bridging set up process during "-t" deconfigures the physical interface in the host. The <ipv4> and <route> tags ensures the host remains properly configured by vnumlparser.pl after "-t" mode (we are supposing that the host address in eth0 is 192.168.0.39 and the default gateway in the external network 192.168.0.1).
During virtual bridge releasing during "-d", the external connection is also deconfigured. To restore configuration the <physicalif> tag is needed.
After building the simulation you can check that uml3 reach the gateway and the host interface in the external network:
uml3:~# ping -c 1 192.168.0.1 PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data. 64 bytes from 192.168.0.1: icmp_seq=1 ttl=150 time=0.629 ms --- 192.168.0.1 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.629/0.629/0.629/0.000 ms uml3:~# ping -c 1 192.168.0.39 PING 192.168.0.39 (192.168.0.39) 56(84) bytes of data. 64 bytes from 192.168.0.39: icmp_seq=1 ttl=64 time=31.1 ms --- 192.168.0.39 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 31.162/31.162/31.162/0.000 ms
Configuring NAT
This section describes a procedure to configure NAT in the host for the scenarios described in section 3.1 of this tutorial, although it could be easily adapted to others VNUML scenarios using layer 3 interconnection at host. The procedure has been provided thanks to Carlos Sanmartín.
How to configure NAT? Well, we can use iptables(8), included in linux kernels from 2.4.x We can have included into kernel or load it with a module:
root@host:~# modprobe ipt_MASQUERADE
It will load ip_tables, ip_conntrack and iptable_nat.
Let's follow next steps:
- We have to enable forwarding into kernel, for managing packets which target isn't our host.
root@host:~# echo 1 > /proc/sys/net/ipv4/ip_forward
For same configuration at any startup, we can add to /etc/sysctl.conf:
net.ipv4.ip_forward=1
- We have to enable NAT rule, for translating virtual machines addresses to host output interface IP address (we suppose eth0 is the output interface):
root@host:~# iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
- Finally, we have to add a default route for managing outgoing packages, at uml3:
root@uml3:~# route add default gw 10.0.3.2
It looks like this:
uml3:~# route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 10.0.0.0 * 255.255.255.0 U 0 0 0 eth1 10.0.1.0 * 255.255.255.0 U 0 0 0 eth2 10.0.2.0 10.0.1.2 255.255.255.0 UG 0 0 0 eth2 10.0.3.0 * 255.255.255.0 U 0 0 0 eth3 default 10.0.3.2 0.0.0.0 UG 0 0 0 eth3
Now, we can try:
uml1:~# ping -c 1 www.google.es PING www.l.google.com (66.249.93.104) 56(84) bytes of data. 64 bytes from 66.249.93.104: icmp_seq=1 ttl=235 time=56.7 ms --- www.l.google.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 56.781/56.781/56.781/0.000 ms
Enjoy it!
For further information, please visit: the Simple HOWTO.
Additional Tips
In addition to the basic tutorial you can try the following tips:
- Improve performance. A virtualized kernel runs slower than a conventional one. Anyway, UML provides a mechanism of improving efficiency using SKAS mode. Different from previous versions of vnumlparser.pl, version 1.3.0 and beyond works fine with SKAS: you just need to compile a host kernel with SKAS support enabled. More information in the UML Project Web site.
- Capture traffic in virtual networks. You can use a patched version of uml_switch (look in the contrib/uml_router/ directory inside VNUML package) that, once compiled and installed, can be used to capture traffic in selected virtual networks. This is a powerfull feature, allowing capturing the traffic in simulation experiments for later study (capturing to a file) or even in real time with software tools like Ethereal (although this needs root privilegies). Please have a look to the capture_* attributes of <net> tag to know how to configure it. In addition, you will find a variation of the tutorial for limited user using capture features in the tutorial_limited_user_capture.xml that comes with the VNUML examples).
... <net name="Net1" mode="uml_switch" uml_switch_binary="/usr/local/bin/uml_switch4capture" capture_expression="icmp" capture_file="/tmp/net1.pcap" /> ... - Adding users and groups in the virtual machines. In the examples shown so far, there is only one user in the virtual machines: the root user. However, vnumlparser.pl can add users to the virtual machines using <user> tag. Try replacing the following vm section for uml1 in the tutorial_limited_user.xml file (or use the tutorial_limited_user2.xml file that comes with the VNUML examples):
... <vm name="uml1"> <if id="1" net="Net0"> <ipv4>10.0.0.1</ipv4> </if> <route type="ipv4" gw="10.0.0.3">default</route> <user username="yoda" group="jedi"> <group>jedi</group> <group>lightside</group> </user> <user username="anakin" group="sith"> <group>sith</group> <group>darkside</group> </user> </vm> ...Once built the simulation, you can check how the yoda and anakin users are created in uml1:
uml1:~# id yoda uid=1002(yoda) gid=1003(jedi) groups=1003(jedi),1004(lightside) uml1:~# id anakin uid=1001(anakin) gid=1001(sith) groups=1001(sith),1002(darkside)
- Socket autoconfiguration. You can avoid the management network socket configuration and unconfiguration steps described in Connecting Simulations to Host section when vnumlparser.pl is run by root and you use the autoconfigure attribute of <mgmt_net> tag, specifying the name of the TAP interface (typically, "tap0"). For example (see tutorial_user2_autoconfigure.xml file that comes with the VNUML examples for the complete XML):
... <vm_mgmt type="net" network="10.250.0.0" mask="24"> <mgmt_net sock="/var/run/vnuml/Mgmt_net.ctl" hostip="10.250.0.1" autoconfigure="tap0"/> </vm_mgmt> ... - UML kernel and filesystem selection. In the examples shown so far, the kernel and UML filesystem for all the virtual machines is set up with the <vm_defaults>. However, you can specify a different UML kernel using <kernel> and <filesystem> tags in a per-vm basis.
Further Reading
This tutorial does not show all features of VNUML tool. Concurrent simulation execution, filetree copying, hostfs filesystems, blocking modes, VLAN integration and other interesting capabilities are available in VNUML. To know about them in detail you should next read VNUML User Manual and VNUML Language Reference. If you need to hack vnumparser.pl to adapt it to your needs, you should read the source code (we think it's very clear! :). A VNUML Developers Manual could be released in the future if needed.
