diff options
Diffstat (limited to 'Documentation')
21 files changed, 5101 insertions, 3 deletions
diff --git a/Documentation/automake.mk b/Documentation/automake.mk index 6f3df0770..3fade9492 100644 --- a/Documentation/automake.mk +++ b/Documentation/automake.mk @@ -9,9 +9,25 @@ EXTRA_DIST += \ Documentation/contents.rst \ Documentation/intro/index.rst \ Documentation/intro/install/index.rst \ + Documentation/intro/install/bash-completion.rst \ + Documentation/intro/install/debian.rst \ + Documentation/intro/install/dpdk-advanced.rst \ + Documentation/intro/install/dpdk.rst \ + Documentation/intro/install/fedora.rst \ + Documentation/intro/install/general.rst \ + Documentation/intro/install/netbsd.rst \ + Documentation/intro/install/rhel.rst \ + Documentation/intro/install/userspace.rst \ + Documentation/intro/install/windows.rst \ + Documentation/intro/install/xenserver.rst \ Documentation/tutorials/index.rst \ Documentation/topics/index.rst \ Documentation/howto/index.rst \ + Documentation/howto/docker.rst \ + Documentation/howto/kvm.rst \ + Documentation/howto/libvirt.rst \ + Documentation/howto/selinux.rst \ + Documentation/howto/ssl.rst \ Documentation/ref/index.rst \ Documentation/faq/index.rst \ Documentation/internals/index.rst \ diff --git a/Documentation/howto/docker.rst b/Documentation/howto/docker.rst new file mode 100644 index 000000000..e23ca7575 --- /dev/null +++ b/Documentation/howto/docker.rst @@ -0,0 +1,319 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + + Convention for heading levels in Open vSwitch documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + +=================================== +Open Virtual Networking With Docker +=================================== + +This document describes how to use Open Virtual Networking with Docker 1.9.0 +or later. + +.. important:: + + Requires Docker version 1.9.0 or later. Only Docker 1.9.0+ comes with support + for multi-host networking. Consult www.docker.com for instructions on how to + install Docker. + +.. note:: + + You must build and install Open vSwitch before proceeding with the below + guide. Refer to :doc:`/intro/install/index` for more information. + +Setup +----- + +For multi-host networking with OVN and Docker, Docker has to be started with a +destributed key-value store. For example, if you decide to use consul as your +distributed key-value store and your host IP address is ``$HOST_IP``, start +your Docker daemon with:: + + $ docker daemon --cluster-store=consul://127.0.0.1:8500 \ + --cluster-advertise=$HOST_IP:0 + +OVN provides network virtualization to containers. OVN's integration with +Docker currently works in two modes - the "underlay" mode or the "overlay" +mode. + +In the "underlay" mode, OVN requires a OpenStack setup to provide container +networking. In this mode, one can create logical networks and can have +containers running inside VMs, standalone VMs (without having any containers +running inside them) and physical machines connected to the same logical +network. This is a multi-tenant, multi-host solution. + +In the "overlay" mode, OVN can create a logical network amongst containers +running on multiple hosts. This is a single-tenant (extendable to multi-tenants +depending on the security characteristics of the workloads), multi-host +solution. In this mode, you do not need a pre-created OpenStack setup. + +For both the modes to work, a user has to install and start Open vSwitch in +each VM/host that they plan to run their containers on. + +.. _docker-overlay: + +The "overlay" mode +------------------ + +.. note:: + + OVN in "overlay" mode needs a minimum Open vSwitch version of 2.5. + +1. Start the central components. + + OVN architecture has a central component which stores your networking intent + in a database. On one of your machines, with an IP Address of + ``$CENTRAL_IP``, where you have installed and started Open vSwitch, you will + need to start some central components. + + Start ovn-northd daemon. This daemon translates networking intent from Docker + stored in the OVN\_Northbound database to logical flows in ``OVN_Southbound`` + database. For example:: + + $ /usr/share/openvswitch/scripts/ovn-ctl start_northd + +2. One time setup + + On each host, where you plan to spawn your containers, you will need to run + the below command once. You may need to run it again if your OVS database + gets cleared. It is harmless to run it again in any case:: + + $ ovs-vsctl set Open_vSwitch . \ + external_ids:ovn-remote="tcp:$CENTRAL_IP:6642" \ + external_ids:ovn-nb="tcp:$CENTRAL_IP:6641" \ + external_ids:ovn-encap-ip=$LOCAL_IP \ + external_ids:ovn-encap-type="$ENCAP_TYPE" + + where: + + ``$LOCAL_IP`` + is the IP address via which other hosts can reach this host. This acts as + your local tunnel endpoint. + + ``$ENCAP_TYPE`` + is the type of tunnel that you would like to use for overlay networking. + The options are ``geneve`` or ``stt``. Your kernel must have support for + your chosen ``$ENCAP_TYPE``. Both ``geneve`` and ``stt`` are part of the + Open vSwitch kernel module that is compiled from this repo. If you use the + Open vSwitch kernel module from upstream Linux, you will need a minumum + kernel version of 3.18 for ``geneve``. There is no ``stt`` support in + upstream Linux. You can verify whether you have the support in your kernel + as follows:: + + $ lsmod | grep $ENCAP_TYPE + + In addition, each Open vSwitch instance in an OVN deployment needs a unique, + persistent identifier, called the ``system-id``. If you install OVS from + distribution packaging for Open vSwitch (e.g. .deb or .rpm packages), or if + you use the ovs-ctl utility included with Open vSwitch, it automatically + configures a system-id. If you start Open vSwitch manually, you should set + one up yourself. For example:: + + $ id_file=/etc/openvswitch/system-id.conf + $ test -e $id_file || uuidgen > $id_file + $ ovs-vsctl set Open_vSwitch . external_ids:system-id=$(cat $id_file) + +3. Start the ``ovn-controller``. + + You need to run the below command on every boot:: + + $ /usr/share/openvswitch/scripts/ovn-ctl start_controller + +4. Start the Open vSwitch network driver. + + By default Docker uses Linux bridge for networking. But it has support for + external drivers. To use Open vSwitch instead of the Linux bridge, you will + need to start the Open vSwitch driver. + + The Open vSwitch driver uses the Python's flask module to listen to Docker's + networking api calls. So, if your host does not have Python's flask module, + install it:: + + $ sudo pip install Flask + + Start the Open vSwitch driver on every host where you plan to create your + containers. Refer to the note on ``$OVS_PYTHON_LIBS_PATH`` that is used below + at the end of this document:: + + $ PYTHONPATH=$OVS_PYTHON_LIBS_PATH ovn-docker-overlay-driver --detach + + .. note:: + + The ``$OVS_PYTHON_LIBS_PATH`` variable should point to the directory where + Open vSwitch Python modules are installed. If you installed Open vSwitch + Python modules via the Debian package of ``python-openvswitch`` or via pip + by running ``pip install ovs``, you do not need to specify the PATH. If + you installed it by following the instructions in + :doc:`/intro/install/general`, then you should specify the PATH. In this + case, the PATH depends on the options passed to ``./configure``. It is + usually either ``/usr/share/openvswitch/python`` or + ``/usr/local/share/openvswitch/python`` + +Docker has inbuilt primitives that closely match OVN's logical switches and +logical port concepts. Consult Docker's documentation for all the possible +commands. Here are some examples. + +Create a logical switch +~~~~~~~~~~~~~~~~~~~~~~~ + +To create a logical switch with name 'foo', on subnet '192.168.1.0/24', run:: + + $ NID=`docker network create -d openvswitch --subnet=192.168.1.0/24 foo` + +List all logical switches +~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + $ docker network ls + +You can also look at this logical switch in OVN's northbound database by +running the following command:: + + $ ovn-nbctl --db=tcp:$CENTRAL_IP:6640 ls-list + +Delete a logical switch +~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + $ docker network rm bar + + +Create a logical port +~~~~~~~~~~~~~~~~~~~~~ + +Docker creates your logical port and attaches it to the logical network in a +single step. For example, to attach a logical port to network ``foo`` inside +container busybox, run:: + + $ docker run -itd --net=foo --name=busybox busybox + +List all logical ports +~~~~~~~~~~~~~~~~~~~~~~ + +Docker does not currently have a CLI command to list all logical ports but you +can look at them in the OVN database by running:: + + $ ovn-nbctl --db=tcp:$CENTRAL_IP:6640 lsp-list $NID + +Create and attach a logical port to a running container +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + $ docker network create -d openvswitch --subnet=192.168.2.0/24 bar + $ docker network connect bar busybox + +Detach and delete a logical port from a running container +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can delete your logical port and detach it from a running container +by running: + +:: + + $ docker network disconnect bar busybox + +.. _docker-underlay: + +The "underlay" mode +------------------- + +.. note:: + + This mode requires that you have a OpenStack setup pre-installed with + OVN providing the underlay networking. + +1. One time setup + + A OpenStack tenant creates a VM with a single network interface (or multiple) + that belongs to management logical networks. The tenant needs to fetch the + port-id associated with the interface via which he plans to send the container + traffic inside the spawned VM. This can be obtained by running the below + command to fetch the 'id' associated with the VM:: + + $ nova list + + and then by running:: + + $ neutron port-list --device_id=$id + + Inside the VM, download the OpenStack RC file that contains the tenant + information (henceforth referred to as ``openrc.sh``). Edit the file and add the + previously obtained port-id information to the file by appending the following + line:: + + $ export OS_VIF_ID=$port_id + + After this edit, the file will look something like:: + + #!/bin/bash + export OS_AUTH_URL=http://10.33.75.122:5000/v2.0 + export OS_TENANT_ID=fab106b215d943c3bad519492278443d + export OS_TENANT_NAME="demo" + export OS_USERNAME="demo" + export OS_VIF_ID=e798c371-85f4-4f2d-ad65-d09dd1d3c1c9 + +2. Create the Open vSwitch bridge + + If your VM has one ethernet interface (e.g.: 'eth0'), you will need to add + that device as a port to an Open vSwitch bridge 'breth0' and move its IP + address and route related information to that bridge. (If it has multiple + network interfaces, you will need to create and attach an Open vSwitch + bridge for the interface via which you plan to send your container + traffic.) + + If you use DHCP to obtain an IP address, then you should kill the DHCP + client that was listening on the physical Ethernet interface (e.g. eth0) and + start one listening on the Open vSwitch bridge (e.g. breth0). + + Depending on your VM, you can make the above step persistent across reboots. + For example, if your VM is Debian/Ubuntu-based, read + `openvswitch-switch.README.Debian` found in `debian` folder. If your VM is + RHEL-based, refer to :doc:`/intro/install/rhel`. + +3. Start the Open vSwitch network driver + + The Open vSwitch driver uses the Python's flask module to listen to Docker's + networking api calls. The driver also uses OpenStack's + ``python-neutronclient`` libraries. If your host does not have Python's + ``flask`` module or ``python-neutronclient`` you must install them. For + example:: + + $ pip install python-neutronclient + $ pip install Flask + + Once installed, source the ``openrc`` file:: + + $ . ./openrc.sh + + Start the network driver and provide your OpenStack tenant password when + prompted:: + + $ PYTHONPATH=$OVS_PYTHON_LIBS_PATH ovn-docker-underlay-driver \ + --bridge breth0 --detach + +From here-on you can use the same Docker commands as described in +`docker-overlay`_. + +Refer the the ovs-architecture man pages (``man ovn-architecture``) to +understand OVN's architecture in detail. diff --git a/Documentation/howto/index.rst b/Documentation/howto/index.rst index 1c4d9d2e3..81b2931d7 100644 --- a/Documentation/howto/index.rst +++ b/Documentation/howto/index.rst @@ -32,3 +32,9 @@ topics covered herein, refer to :doc:`/topics/index`. .. toctree:: :maxdepth: 2 + + kvm + selinux + libvirt + ssl + docker diff --git a/Documentation/howto/kvm.rst b/Documentation/howto/kvm.rst new file mode 100644 index 000000000..4b43af334 --- /dev/null +++ b/Documentation/howto/kvm.rst @@ -0,0 +1,103 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + + Convention for heading levels in Open vSwitch documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + +===================== +Open vSwitch with KVM +===================== + +This document describes how to use Open vSwitch with the Kernel-based Virtual +Machine (KVM). + +.. note:: + + This document assumes that you have Open vSwitch set up on a Linux system. + +Setup +----- + +KVM uses tunctl to handle various bridging modes, which you can install with +the Debian/Ubuntu package ``uml-utilities``:: + + $ apt-get install uml-utilities + +Next, you will need to modify or create custom versions of the ``qemu-ifup`` +and ``qemu-ifdown`` scripts. In this guide, we'll create custom versions that +make use of example Open vSwitch bridges that we'll describe in this guide. + +Create the following two files and store them in known locations. For example:: + + echo << EOF > /etc/ovs-ifup + #!/bin/sh + + switch='br0' + /sbin/ifconfig $1 0.0.0.0 up + ovs-vsctl add-port ${switch} $1 + EOF + +:: + + echo << EOF > /etc/ovs-ifdown + #!/bin/sh + + switch='br0' + /sbin/ifconfig $1 0.0.0.0 down + ovs-vsctl del-port ${switch} $1 + +The basic usage of Open vSwitch is described at the end of +:doc:`/intro/install/general`. If you haven't already, create a bridge named +``br0`` with the following command:: + + $ ovs-vsctl add-br br0 + +Then, add a port to the bridge for the NIC that you want your guests to +communicate over (e.g. ``eth0``):: + + $ ovs-vsctl add-port br0 eth0 + +Refer to ovs-vsctl(8) for more details. + +Next, we'll start a guest that will use our ifup and ifdown scripts:: + + $ kvm -m 512 -net nic,macaddr=00:11:22:EE:EE:EE -net \ + tap,script=/etc/ovs-ifup,downscript=/etc/ovs-ifdown -drive \ + file=/path/to/disk-image,boot=on + +This will start the guest and associate a tap device with it. The ``ovs-ifup`` +script will add a port on the br0 bridge so that the guest will be able to +communicate over that bridge. + +To get some more information and for debugging you can use Open vSwitch +utilities such as ovs-dpctl and ovs-ofctl, For example:: + + $ ovs-dpctl show + $ ovs-ofctl show br0 + +You should see tap devices for each KVM guest added as ports to the bridge +(e.g. tap0) + +Refer to ovs-dpctl(8) and ovs-ofctl(8) for more details. + +Bug Reporting +------------- + +Please report problems to bugs@openvswitch.org. diff --git a/Documentation/howto/libvirt.rst b/Documentation/howto/libvirt.rst new file mode 100644 index 000000000..0b303d54c --- /dev/null +++ b/Documentation/howto/libvirt.rst @@ -0,0 +1,92 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + + Convention for heading levels in Open vSwitch documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + +========================= +Open vSwitch with Libvirt +========================= + +This document describes how to use Open vSwitch with Libvirt 0.9.11 or later. +This document assumes that you followed :doc:`/intro/install/general` or +installed Open vSwitch from distribution packaging such as a .deb or .rpm. The +Open vSwitch support is included by default in Libvirt 0.9.11. Consult +www.libvirt.org for instructions on how to build the latest Libvirt, if your +Linux distribution by default comes with an older Libvirt release. + +Limitations +----------- + +Currently there is no Open vSwitch support for networks that are managed by +libvirt (e.g. NAT). As of now, only bridged networks are supported (those where +the user has to manually create the bridge). + +Setup +----- + +First, create the Open vSwitch bridge by using the ovs-vsctl utility (this must +be done with administrative privileges):: + + $ ovs-vsctl add-br ovsbr + +Once that is done, create a VM, if necessary, and edit its Domain XML file:: + + $ virsh edit <vm> + +Lookup in the Domain XML file the ``<interface>`` section. There should be one +such XML section for each interface the VM has:: + + <interface type='network'> + <mac address='52:54:00:71:b1:b6'/> + <source network='default'/> + <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> + </interface> + +And change it to something like this:: + + <interface type='bridge'> + <mac address='52:54:00:71:b1:b6'/> + <source bridge='ovsbr'/> + <virtualport type='openvswitch'/> + <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> + </interface> + +The interface type must be set to ``bridge``. The ``<source>`` XML element +specifies to which bridge this interface will be attached to. The +``<virtualport>`` element indicates that the bridge in ``<source>`` element is +an Open vSwitch bridge. + +Then (re)start the VM and verify if the guest's vnet interface is attached to +the ovsbr bridge:: + + $ ovs-vsctl show + +Troubleshooting +--------------- + +If the VM does not want to start, then try to run the libvirtd process either +from the terminal, so that all errors are printed in console, or inspect +Libvirt/Open vSwitch log files for possible root cause. + +Bug Reporting +------------- + +Report problems to bugs@openvswitch.org. diff --git a/Documentation/howto/selinux.rst b/Documentation/howto/selinux.rst new file mode 100644 index 000000000..ad556da58 --- /dev/null +++ b/Documentation/howto/selinux.rst @@ -0,0 +1,174 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + + Convention for heading levels in Open vSwitch documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + +========================= +Open vSwitch with SELinux +========================= + +Security-Enhanced Linux (SELinux) is a Linux kernel security module that limits +"the malicious things" that certain processes, including OVS, can do to the +system in case they get compromised. In our case SELinux basically serves as +the "second line of defense" that limits the things that OVS processes are +allowed to do. The "first line of defense" is proper input validation that +eliminates code paths that could be used by attacker to do any sort of "escape +attacks", such as file name escape, shell escape, command line argument escape, +buffer escape. Since developers don't always implement proper input validation, +then SELinux Access Control's goal is to confine damage of such attacks, if +they turned out to be possible. + +Besides Type Enforcement there are other SELinux features, but they are out of +scope for this document. + +Currently there are two SELinux policies for Open vSwitch: + +- the one that ships with your Linux distribution (i.e. + selinux-policy-targeted package) + +- the one that ships with OVS (i.e. openvswitch-selinux-policy package) + +Limitations +----------- + +If Open vSwitch is directly started from command line, then it will run under +``unconfined_t`` SELinux domain that basically lets daemon to do whatever it +likes. This is very important for developers to understand, because they might +introduced code in OVS that invokes new system calls that SELinux policy did +not anticipate. This means that their feature may have worked out just fine +for them. However, if someone else would try to run the same code when Open +vSwitch is started through systemctl, then Open vSwitch would get Permission +Denied errors. + +Currently the only distributions that enforce SELinux on OVS by default are +RHEL, CentOS and Fedora. While Ubuntu and Debian also have some SELinux +support, they run Open vSwitch under the unrestricted ``unconfined`` domain. +Also, it seems that Ubuntu is leaning towards Apparmor that works slightly +differently than SELinux. + +SELinux and Open vSwitch are moving targets. What this means is that, if you +solely rely on your Linux distribution's SELinux policy, then this policy might +not have correctly anticipated that a newer Open vSwitch version needs extra +white list rules. However, if you solely rely on SELinux policy that ships +with Open vSwitch, then Open vSwitch developers might not have correctly +anticipated the feature set that your SELinux implementation supports. + +Installation +------------ + +Refer to :doc:`/intro/install/fedora` for instructions on how to build all Open +vSwitch rpm packages. + +Once the package is built, install it on your Linux distribution:: + + $ dnf install openvswitch-selinux-policy-2.4.1-1.el7.centos.noarch.rpm + +Restart Open vSwitch:: + + $ systemctl restart openvswitch + +Troubleshooting +--------------- + +When SELinux was implemented some of the standard system utilities acquired +``-Z`` flag (e.g. ``ps -Z``, ``ls -Z``). For example, to find out under which +SELinux security domain process runs, use:: + + $ ps -AZ | grep ovs-vswitchd + system_u:system_r:openvswitch_t:s0 854 ? ovs-vswitchd + +To find out the SELinux label of file or directory, use:: + + $ ls -Z /etc/openvswitch/conf.db + system_u:object_r:openvswitch_rw_t:s0 /etc/openvswitch/conf.db + +If, for example, SELinux policy for Open vSwitch is too strict, then you might +see in Open vSwitch log files "Permission Denied" errors:: + + $ cat /var/log/openvswitch/ovs-vswitchd.log + vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log + ovs_numa|INFO|Discovered 2 CPU cores on NUMA node 0 + ovs_numa|INFO|Discovered 1 NUMA nodes and 2 CPU cores + reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... + reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected + netlink_socket|ERR|fcntl: Permission denied + dpif_netlink|ERR|Generic Netlink family 'ovs_datapath' does not exist. + The Open vSwitch kernel module is probably not loaded. + dpif|WARN|failed to enumerate system datapaths: Permission denied + dpif|WARN|failed to create datapath ovs-system: Permission denied + +However, not all "Permission denied" errors are caused by SELinux. So, before +blaming too strict SELinux policy, make sure that indeed SELinux was the one +that denied OVS access to certain resources, for example, run: + + $ grep "openvswitch_t" /var/log/audit/audit.log | tail + type=AVC msg=audit(1453235431.640:114671): avc: denied { getopt } for pid=4583 comm="ovs-vswitchd" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_generic_socket permissive=0 + +If SELinux denied OVS access to certain resources, then make sure that you have +installed our SELinux policy package that "loosens" up distribution's SELinux +policy:: + + $ rpm -qa | grep openvswitch-selinux + openvswitch-selinux-policy-2.4.1-1.el7.centos.noarch + +Then verify that this module was indeed loaded:: + + # semodule -l | grep openvswitch + openvswitch-custom 1.0 + openvswitch 1.1.1 + +If you still see Permission denied errors, then take a look into +``selinux/openvswitch.te`` file in the OVS source tree and try to add white +list rules. This is really simple, just run SELinux audit2allow tool:: + + $ grep "openvswitch_t" /var/log/audit/audit.log | audit2allow -M ovslocal + +Contributing SELinux policy patches +----------------------------------- + +Here are few things to consider before proposing SELinux policy patches to Open +vSwitch developer mailing list: + +1. The SELinux policy that resides in Open vSwitch source tree amends SELinux + policy that ships with your distributions. + + Implications of this are that it is assumed that the distribution's Open + vSwitch SELinux module must be already loaded to satisfy dependencies. + +2. The SELinux policy that resides in Open vSwitch source tree must work on all + currently relevant Linux distributions. + + Implications of this are that you should use only those SELinux policy + features that are supported by the lowest SELinux version out there. + Typically this means that you should test your SELinux policy changes on the + oldest RHEL or CentOS version that this OVS version supports. Refer to + :doc:`/intro/install/fedora` to find out this. + +3. The SELinux policy is enforced only when state transition to + ``openvswitch_t`` domain happens. + + Implications of this are that perhaps instead of loosening SELinux policy + you can do certain things at the time rpm package is installed. + +Reporting Bugs +-------------- + +Report problems to bugs@openvswitch.org. diff --git a/Documentation/howto/ssl.rst b/Documentation/howto/ssl.rst new file mode 100644 index 000000000..3085206fb --- /dev/null +++ b/Documentation/howto/ssl.rst @@ -0,0 +1,338 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + + Convention for heading levels in Open vSwitch documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + +===================== +Open vSwitch with SSL +===================== + +If you plan to configure Open vSwitch to connect across the network to an +OpenFlow controller, then we recommend that you build Open vSwitch with +OpenSSL. SSL support ensures integrity and confidentiality of the OpenFlow +connections, increasing network security. + +This document describes how to configure an Open vSwitch to connect to an +OpenFlow controller over SSL. Refer to :doc:`/intro/install/general`. for +instructions on building Open vSwitch with SSL support. + +Open vSwitch uses TLS version 1.0 or later (TLSv1), as specified by RFC 2246, +which is very similar to SSL version 3.0. TLSv1 was released in January 1999, +so all current software and hardware should implement it. + +This document assumes basic familiarity with public-key cryptography and +public-key infrastructure. + +SSL Concepts for OpenFlow +------------------------- + +This section is an introduction to the public-key infrastructure architectures +that Open vSwitch supports for SSL authentication. + +To connect over SSL, every Open vSwitch must have a unique private/public key +pair and a certificate that signs that public key. Typically, the Open vSwitch +generates its own public/private key pair. There are two common ways to obtain +a certificate for a switch: + +* Self-signed certificates: The Open vSwitch signs its certificate with its own + private key. In this case, each switch must be individually approved by the + OpenFlow controller(s), since there is no central authority. + + This is the only switch PKI model currently supported by NOX + (http://noxrepo.org). + +* Switch certificate authority: A certificate authority (the "switch CA") signs + each Open vSwitch's public key. The OpenFlow controllers then check that any + connecting switches' certificates are signed by that certificate authority. + + This is the only switch PKI model supported by the simple OpenFlow controller + included with Open vSwitch. + +Each Open vSwitch must also have a copy of the CA certificate for the +certificate authority that signs OpenFlow controllers' keys (the "controller +CA" certificate). Typically, the same controller CA certificate is installed +on all of the switches within a given administrative unit. There are two +common ways for a switch to obtain the controller CA certificate: + +* Manually copy the certificate to the switch through some secure means, e.g. + using a USB flash drive, or over the network with "scp", or even FTP or HTTP + followed by manual verification. + +* Open vSwitch "bootstrap" mode, in which Open vSwitch accepts and saves the + controller CA certificate that it obtains from the OpenFlow controller on its + first connection. Thereafter the switch will only connect to controllers + signed by the same CA certificate. + +Establishing a Public Key Infrastructure +---------------------------------------- + +Open vSwitch can make use of your existing public key infrastructure. If you +already have a PKI, you may skip forward to the next section. Otherwise, if +you do not have a PKI, the ovs-pki script included with Open vSwitch can help. +To create an initial PKI structure, invoke it as: + +:: + + $ ovs-pki init + +This will create and populate a new PKI directory. The default location for +the PKI directory depends on how the Open vSwitch tree was configured (to see +the configured default, look for the ``--dir`` option description in the output +of ``ovs-pki --help``). + +The pki directory contains two important subdirectories. The `controllerca` +subdirectory contains controller CA files, including the following: + +`cacert.pem` + Root certificate for the controller certificate authority. Each Open vSwitch + must have a copy of this file to allow it to authenticate valid controllers. + +`private/cakey.pem` + Private signing key for the controller certificate authority. This file must + be kept secret. There is no need for switches or controllers to have a copy + of it. + +The `switchca` subdirectory contains switch CA files, analogous to those in the +`controllerca` subdirectory: + +`cacert.pem` + Root certificate for the switch certificate authority. The OpenFlow + controller must have this file to enable it to authenticate valid switches. + +`private/cakey.pem` + Private signing key for the switch certificate authority. This file must be + kept secret. There is no need for switches or controllers to have a copy of + it. + +After you create the initial structure, you can create keys and certificates +for switches and controllers with ovs-pki. Refer to the ovs-pki(8) manage for +complete details. A few examples of its use follow: + +Controller Key Generation +~~~~~~~~~~~~~~~~~~~~~~~~~ + +To create a controller private key and certificate in files named +ctl-privkey.pem and ctl-cert.pem, run the following on the machine that +contains the PKI structure: + +:: + + $ ovs-pki req+sign ctl controller + +ctl-privkey.pem and ctl-cert.pem would need to be copied to the controller for +its use at runtime. If, for testing purposes, you were to use +ovs-testcontroller, the simple OpenFlow controller included with Open vSwitch, +then the --private-key and --certificate options, respectively, would point to +these files. + +It is very important to make sure that no stray copies of ctl-privkey.pem are +created, because they could be used to impersonate the controller. + +Switch Key Generation with Self-Signed Certificates +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you are using self-signed certificates (see "SSL Concepts for OpenFlow"), +this is one way to create an acceptable certificate for your controller to +approve. + +1. Run the following command on the Open vSwitch itself:: + + $ ovs-pki self-sign sc + + .. note:: + This command does not require a copy of any of the PKI files generated by + ``ovs-pki init``, and you should not copy them to the switch because some + of them have contents that must remain secret for security.) + + The ``ovs-pki self-sign`` command has the following output: + + sc-privkey.pem + the switch private key file. For security, the contents of this file must + remain secret. There is ordinarily no need to copy this file off the Open + vSwitch. + + sc-cert.pem + the switch certificate, signed by the switch's own private key. Its + contents are not a secret. + +2. Optionally, copy `controllerca/cacert.pem` from the machine that has the + OpenFlow PKI structure and verify that it is correct. (Otherwise, you will + have to use CA certificate bootstrapping when you configure Open vSwitch in + the next step.) + +3. Configure Open vSwitch to use the keys and certificates (see "Configuring + SSL Support", below). + +Switch Key Generation with a Switch PKI (Easy Method) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you are using a switch PKI (see "SSL Concepts for OpenFlow", above), this +method of switch key generation is a little easier than the alternate method +described below, but it is also a little less secure because it requires +copying a sensitive private key from file from the machine hosting the PKI to +the switch. + +1. Run the following on the machine that contains the PKI structure:: + + $ ovs-pki req+sign sc switch + + This command has the following output: + + sc-privkey.pem + the switch private key file. For security, the contents of this file must + remain secret. + + sc-cert.pem + the switch certificate. Its contents are not a secret. + +2. Copy sc-privkey.pem and sc-cert.pem, plus controllerca/cacert.pem, to the + Open vSwitch. + +3. Delete the copies of sc-privkey.pem and sc-cert.pem on the PKI machine and + any other copies that may have been made in transit. It is very important + to make sure that there are no stray copies of sc-privkey.pem, because they + could be used to impersonate the switch. + + .. warning:: + Don't delete controllerca/cacert.pem! It is not security-sensitive and + you will need it to configure additional switches. + +4. Configure Open vSwitch to use the keys and certificates (see "Configuring + SSL Support", below). + +Switch Key Generation with a Switch PKI (More Secure) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you are using a switch PKI (see "SSL Concepts for OpenFlow", above), then, +compared to the previous method, the method described here takes a little more +work, but it does not involve copying the private key from one machine to +another, so it may also be a little more secure. + +1. Run the following command on the Open vSwitch itself:: + + $ ovs-pki req sc + + .. note:: + This command does not require a copy of any of the PKI files generated by + "ovs-pki init", and you should not copy them to the switch because some of + them have contents that must remain secret for security. + + The "ovs-pki req" command has the following output: + + sc-privkey.pem + the switch private key file. For security, the contents of this file must + remain secret. There is ordinarily no need to copy this file off the Open + vSwitch. + + sc-req.pem + the switch "certificate request", which is essentially the switch's public + key. Its contents are not a secret. + + a fingerprint + this is output on stdout. + +2. Write the fingerprint down on a slip of paper and copy `sc-req.pem` to the + machine that contains the PKI structure. + +3. On the machine that contains the PKI structure, run:: + + $ ovs-pki sign sc switch + + This command will output a fingerprint to stdout and request that you verify + it. Check that it is the same as the fingerprint that you wrote down on the + slip of paper before you answer "yes". + + ``ovs-pki sign`` creates a file named `sc-cert.pem`, which is the switch + certificate. Its contents are not a secret. + +4. Copy the generated `sc-cert.pem`, plus `controllerca/cacert.pem` from the + PKI structure, to the Open vSwitch, and verify that they were copied + correctly. + + You may delete `sc-cert.pem` from the machine that hosts the PKI + structure now, although it is not important that you do so. + + .. warning:: + Don't delete `controllerca/cacert.pem`! It is not security-sensitive and + you will need it to configure additional switches. + +5. Configure Open vSwitch to use the keys and certificates (see "Configuring + SSL Support", below). + +Configuring SSL Support +----------------------- + +SSL configuration requires three additional configuration files. The first two +of these are unique to each Open vSwitch. If you used the instructions above +to build your PKI, then these files will be named `sc-privkey.pem` and +`sc-cert.pem`, respectively: + +- A private key file, which contains the private half of an RSA or DSA key. + + This file can be generated on the Open vSwitch itself, for the greatest + security, or it can be generated elsewhere and copied to the Open vSwitch. + + The contents of the private key file are secret and must not be exposed. + +- A certificate file, which certifies that the private key is that of a + trustworthy Open vSwitch. + + This file has to be generated on a machine that has the private key for the + switch certification authority, which should not be an Open vSwitch; ideally, + it should be a machine that is not networked at all. + + The certificate file itself is not a secret. + +The third configuration file is typically the same across all the switches in a +given administrative unit. If you used the instructions above to build your +PKI, then this file will be named `cacert.pem`: + +- The root certificate for the controller certificate authority. The Open + vSwitch verifies it that is authorized to connect to an OpenFlow controller + by verifying a signature against this CA certificate. + +Once you have these files, configure ovs-vswitchd to use them using the +``ovs-vsctl set-ssl`` command, e.g.:: + + $ ovs-vsctl set-ssl /etc/openvswitch/sc-privkey.pem \ + /etc/openvswitch/sc-cert.pem /etc/openvswitch/cacert.pem + +Substitute the correct file names, of course, if they differ from the ones used +above. You should use absolute file names (ones that begin with ``/``), +because ovs-vswitchd's current directory is unrelated to the one from which you +run ovs-vsctl. + +If you are using self-signed certificates (see "SSL Concepts for OpenFlow") and +you did not copy controllerca/cacert.pem from the PKI machine to the Open +vSwitch, then add the ``--bootstrap`` option, e.g.:: + + $ ovs-vsctl -- --bootstrap set-ssl /etc/openvswitch/sc-privkey.pem \ + /etc/openvswitch/sc-cert.pem /etc/openvswitch/cacert.pem + +After you have added all of these configuration keys, you may specify ``ssl:`` +connection methods elsewhere in the configuration database. ``tcp:`` connection +methods are still allowed even after SSL has been configured, so for security +you should use only ``ssl:`` connections. + +Reporting Bugs +-------------- + +Report problems to bugs@openvswitch.org. diff --git a/Documentation/index.rst b/Documentation/index.rst index a46407a06..be794f169 100644 --- a/Documentation/index.rst +++ b/Documentation/index.rst @@ -45,7 +45,17 @@ The Open vSwitch documentation is organised into multiple sections: First Steps ----------- -**TODO** +Getting started with Open vSwitch (OVS) or Open Virtual Network (OVN) for Open +vSwitch? Start here. + +- **Install:** :doc:`intro/install/general` | + :doc:`intro/install/userspace` | + :doc:`intro/install/netbsd` | + :doc:`intro/install/windows` | + :doc:`intro/install/xenserver` | + :doc:`intro/install/dpdk` + +- **Tutorials:** **TODO** Deeper Dive ----------- diff --git a/Documentation/internals/contributing/documentation-style.rst b/Documentation/internals/contributing/documentation-style.rst index 756f1678d..c32921e70 100644 --- a/Documentation/internals/contributing/documentation-style.rst +++ b/Documentation/internals/contributing/documentation-style.rst @@ -29,8 +29,8 @@ Open vSwitch Documentation Style This file describes the documentation style used in all documentation found in Open vSwitch. Documentation includes any documents found in ``Documentation`` -along with any ``README``, ``INSTALL``, or generally ``rst`` suffixed documents -found in the project tree. +along with any ``README``, ``MAINTAINERS``, or generally ``rst`` suffixed +documents found in the project tree. reStructuredText vs. Sphinx --------------------------- diff --git a/Documentation/intro/install/bash-completion.rst b/Documentation/intro/install/bash-completion.rst new file mode 100644 index 000000000..6df60c65c --- /dev/null +++ b/Documentation/intro/install/bash-completion.rst @@ -0,0 +1,111 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + + Convention for heading levels in Open vSwitch documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + +==================================== +Bash command-line completion scripts +==================================== + +There are two completion scripts available, ovs-appctl-bashcomp.bash and +ovs-vsctl-bashcomp.bash respectively. + +ovs-appctl-bashcomp +------------------- + +ovs-appctl-bashcomp.bash adds bash command-line completion support for +ovs-appctl, ovs-dpctl, ovs-ofctl and ovsdb-tool commands. + +Features +~~~~~~~~ + +- Display available completion or complete on unfinished user input (long + option, subcommand, and argument). + +- Subcommand hints + +- Convert between keywords like ``bridge``, ``port``, ``interface``, or ``dp`` + and the available record in ovsdb. + +Limitations +~~~~~~~~~~~ + +- Only supports a small set of important keywords (``dp``, ``datapath``, ``bridge``, ``switch``, + ``port``, ``interface``, ``iface``). + +- Does not support parsing of nested options. For example: + + + :: + + $ ovsdb-tool create [db [schema]] + +- Does not support expansion on repeated argument. For example: + + :: + + $ ovs-dpctl show [dp...]). + +- Only supports matching on long options, and only in the format ``--option + [arg]``. Do not use ``--option=[arg]``. + +ovs-vsctl-bashcomp +------------------- + +ovs-vsctl-bashcomp.bash adds bash command-line completion support for ovs-vsctl +command. + +Features +~~~~~~~~ + +- Display available completion and complete on user input for global/local + options, command, and argument. + +- Query database and expand keywords like ``table``, ``record``, ``column``, or + ``key``, to available completions. + +- Deal with argument relations like 'one and more', 'zero or one'. + +- Complete multiple ovs-vsctl commands cascaded via ``--``. + +Limitations +~~~~~~~~~~~ + +Completion of very long ``ovs-vsctl`` commands can take up to several seconds. + +Usage +----- + +The bashcomp scripts should be placed at ``/etc/bash_completion.d/`` to be +available for all bash sessions. Running ``make install`` will place the +scripts to ``$(sysconfdir)/bash_completion.d/``, thus, the user should specify +``--sysconfdir=/etc`` at configuration. If OVS is installed from packages, the +scripts will automatically be placed inside ``/etc/bash_completion.d/``. + +If you just want to run the scripts in one bash, you can remove them from +``/etc/bash_completion.d/`` and run the scripts via ``. +ovs-appctl-bashcomp.bash`` or ``. ovs-vsctl-bashcomp.bash``. + +Tests +----- + +Unit tests are added in ``tests/completion.at`` and integrated into autotest +framework. To run the tests, just run ``make check``. diff --git a/Documentation/intro/install/debian.rst b/Documentation/intro/install/debian.rst new file mode 100644 index 000000000..648433144 --- /dev/null +++ b/Documentation/intro/install/debian.rst @@ -0,0 +1,123 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + + Convention for heading levels in Open vSwitch documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + +================================= +Debian Packaging for Open vSwitch +================================= + +This document describes how to build Debian packages for Open vSwitch. To +install Open vSwitch on Debian without building Debian packages, refer to +:doc:`general` instead. + +.. note:: + These instructions should also work on Ubuntu and other Debian derivative + distributions. + +Before You Begin +---------------- + +Before you begin, consider whether you really need to build packages yourself. +Debian "wheezy" and "sid", as well as recent versions of Ubuntu, contain +pre-built Debian packages for Open vSwitch. It is easier to install these than +to build your own. To use packages from your distribution, skip ahead to +"Installing .deb Packages", below. + +Building Open vSwitch Debian packages +------------------------------------- + +You may build from an Open vSwitch distribution tarball or from an Open vSwitch +Git tree with these instructions. + +You do not need to be the superuser to build the Debian packages. + +1. Install the "build-essential" and "fakeroot" packages. For example:: + + $ apt-get install build-essential fakeroot + +2. Obtain and unpack an Open vSwitch source distribution and ``cd`` into its + top level directory. + +3. Install the build dependencies listed under "Build-Depends:" near the top of + ``debian/control``. You can install these any way you like, e.g. with + ``apt-get install``. + +Check your work by running ``dpkg-checkbuilddeps`` in the top level of your ovs +directory. If you've installed all the dependencies properly, +``dpkg-checkbuilddeps`` will exit without printing anything. If you forgot to +install some dependencies, it will tell you which ones. + +4. Build the package:: + + $ fakeroot debian/rules binary + + This will do a serial build that runs the unit tests. This will take + approximately 8 to 10 minutes. If you prefer, you can run a faster parallel + build:: + + $ DEB_BUILD_OPTIONS='parallel=8' fakeroot debian/rules binary + + If you are in a big hurry, you can even skip the unit tests:: + + $ DEB_BUILD_OPTIONS='parallel=8 nocheck' fakeroot debian/rules binary + +.. note:: + + There are a few pitfalls in the Debian packaging building system so that, + occasionally, you may find that in a tree that you have using for a while, + the build command above exits immediately without actually building anything. + To fix the problem, run:: + + $ fakeroot debian/rules clean + + or start over from a fresh copy of the source tree. + +5. The generated .deb files will be in the parent directory of the Open vSwitch + source distribution. + +Installing .deb Packages +------------------------ + +These instructions apply to installing from Debian packages that you built +yourself, as described in the previous section, or from packages provided by +Debian or a Debian derivative distribution such as Ubuntu. In the former case, +use a command such as ``dpkg -i`` to install the .deb files that you build, and +in the latter case use a program such as ``apt-get`` or ``aptitude`` to +download and install the provided packages. + +.. important:: + You must be superuser to install Debian packages. + +1. Start by installing an Open vSwitch kernel module. See + ``debian/openvswitch-switch.README.Debian`` for the available options. + +2. Install the ``openvswitch-switch`` and ``openvswitch-common`` packages. + These packages include the core userspace components of the switch. + +Open vSwitch ``.deb`` packages not mentioned above are rarely useful. Refer to +their individual package descriptions to find out whether any of them are +useful to you. + +Reporting Bugs +-------------- + +Report problems to bugs@openvswitch.org. diff --git a/Documentation/intro/install/dpdk-advanced.rst b/Documentation/intro/install/dpdk-advanced.rst new file mode 100644 index 000000000..44d1cd78c --- /dev/null +++ b/Documentation/intro/install/dpdk-advanced.rst @@ -0,0 +1,938 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + + Convention for heading levels in Open vSwitch documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + +================================= +Open vSwitch with DPDK (Advanced) +================================= + +The Advanced Install Guide explains how to improve OVS performance when using +DPDK datapath. This guide provides information on tuning, system configuration, +troubleshooting, static code analysis and testcases. + +Building as a Shared Library +---------------------------- + +DPDK can be built as a static or a shared library and shall be linked by +applications using DPDK datapath. When building OVS with DPDK, you can link +Open vSwitch against the shared DPDK library. + +.. note:: + Minor performance loss is seen with OVS when using shared DPDK library as + compared to static library. + +To build Open vSwitch using DPDK as a shared library, first refer to +:doc:`/intro/install/dpdk` for download instructions for DPDK and OVS. + +Once DPDK and OVS have been downloaded, you must configure the DPDK library +accordingly. Simply set ``CONFIG_RTE_BUILD_SHARED_LIB=y`` in +``config/common_base``, then build and install DPDK. Once done, DPDK can be +built as usual. For example:: + + $ export DPDK_TARGET=x86_64-native-linuxapp-gcc + $ export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET + $ make install T=$DPDK_TARGET DESTDIR=install + +Once DPDK is built, export the DPDK shared library location and setup OVS as +detailed in :doc:`/intro/install/dpdk`:: + + $ export LD_LIBRARY_PATH=$DPDK_DIR/x86_64-native-linuxapp-gcc/lib + +System Configuration +-------------------- + +To achieve optimal OVS performance, the system can be configured and that +includes BIOS tweaks, Grub cmdline additions, better understanding of NUMA +nodes and apt selection of PCIe slots for NIC placement. + +Recommended BIOS Settings +~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. list-table:: Recommended BIOS Settings + :header-rows: 1 + + * - Setting + - Value + * - C3 Power State + - Disabled + * - C6 Power State + - Disabled + * - MLC Streamer + - Enabled + * - MLC Spacial Prefetcher + - Enabled + * - DCU Data Prefetcher + - Enabled + * - DCA + - Enabled + * - CPU Power and Performance + - Performance + * - Memeory RAS and Performance Config -> NUMA optimized + - Enabled + +PCIe Slot Selection +~~~~~~~~~~~~~~~~~~~ + +The fastpath performance can be affected by factors related to the placement of +the NIC, such as channel speeds between PCIe slot and CPU or the proximity of +PCIe slot to the CPU cores running the DPDK application. Listed below are the +steps to identify right PCIe slot. + +#. Retrieve host details using ``dmidecode``. For example:: + + $ dmidecode -t baseboard | grep "Product Name" + +#. Download the technical specification for product listed, e.g: S2600WT2 + +#. Check the Product Architecture Overview on the Riser slot placement, CPU + sharing info and also PCIe channel speeds + + For example: On S2600WT, CPU1 and CPU2 share Riser Slot 1 with Channel speed + between CPU1 and Riser Slot1 at 32GB/s, CPU2 and Riser Slot1 at 16GB/s. + Running DPDK app on CPU1 cores and NIC inserted in to Riser card Slots will + optimize OVS performance in this case. + +#. Check the Riser Card #1 - Root Port mapping information, on the available + slots and individual bus speeds. In S2600WT slot 1, slot 2 has high bus + speeds and are potential slots for NIC placement. + +Advanced Hugepage Setup +~~~~~~~~~~~~~~~~~~~~~~~ + +Allocate and mount 1 GB hugepages. + +- For persistent allocation of huge pages, add the following options to the + kernel bootline:: + + default_hugepagesz=1GB hugepagesz=1G hugepages=N + + For platforms supporting multiple huge page sizes, add multiple options:: + + default_hugepagesz=<size> hugepagesz=<size> hugepages=N + + where: + + ``N`` + number of huge pages requested + ``size`` + huge page size with an optional suffix ``[kKmMgG]`` + +- For run-time allocation of huge pages:: + + $ echo N > /sys/devices/system/node/nodeX/hugepages/hugepages-1048576kB/nr_hugepages + + where: + + ``N`` + number of huge pages requested + ``X`` + NUMA Node + + .. note:: + For run-time allocation of 1G huge pages, Contiguous Memory Allocator + (``CONFIG_CMA``) has to be supported by kernel, check your Linux distro. + +Now mount the huge pages, if not already done so:: + + $ mount -t hugetlbfs -o pagesize=1G none /dev/hugepages + +Enable HyperThreading +~~~~~~~~~~~~~~~~~~~~~ + +With HyperThreading, or SMT, enabled, a physical core appears as two logical +cores. SMT can be utilized to spawn worker threads on logical cores of the same +physical core there by saving additional cores. + +With DPDK, when pinning pmd threads to logical cores, care must be taken to set +the correct bits of the ``pmd-cpu-mask`` to ensure that the pmd threads are +pinned to SMT siblings. + +Take a sample system configuration, with 2 sockets, 2 * 10 core processors, HT +enabled. This gives us a total of 40 logical cores. To identify the physical +core shared by two logical cores, run:: + + $ cat /sys/devices/system/cpu/cpuN/topology/thread_siblings_list + +where ``N`` is the logical core number. + +In this example, it would show that cores ``1`` and ``21`` share the same +physical core. As cores are counted from 0, the ``pmd-cpu-mask`` can be used +to enable these two pmd threads running on these two logical cores (one +physical core) is:: + + $ ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x200002 + +Isolate Cores +~~~~~~~~~~~~~ + +The ``isolcpus`` option can be used to isolate cores from the Linux scheduler. +The isolated cores can then be used to dedicatedly run HPC applications or +threads. This helps in better application performance due to zero context +switching and minimal cache thrashing. To run platform logic on core 0 and +isolate cores between 1 and 19 from scheduler, add ``isolcpus=1-19`` to GRUB +cmdline. + +.. note:: + It has been verified that core isolation has minimal advantage due to mature + Linux scheduler in some circumstances. + +NUMA/Cluster-on-Die +~~~~~~~~~~~~~~~~~~~ + +Ideally inter-NUMA datapaths should be avoided where possible as packets will +go across QPI and there may be a slight performance penalty when compared with +intra NUMA datapaths. On Intel Xeon Processor E5 v3, Cluster On Die is +introduced on models that have 10 cores or more. This makes it possible to +logically split a socket into two NUMA regions and again it is preferred where +possible to keep critical datapaths within the one cluster. + +It is good practice to ensure that threads that are in the datapath are pinned +to cores in the same NUMA area. e.g. pmd threads and QEMU vCPUs responsible for +forwarding. If DPDK is built with ``CONFIG_RTE_LIBRTE_VHOST_NUMA=y``, vHost +User ports automatically detect the NUMA socket of the QEMU vCPUs and will be +serviced by a PMD from the same node provided a core on this node is enabled in +the ``pmd-cpu-mask``. ``libnuma`` packages are required for this feature. + +Compiler Optimizations +~~~~~~~~~~~~~~~~~~~~~~ + +The default compiler optimization level is ``-O2``. Changing this to more +aggressive compiler optimization such as ``-O3 -march=native`` with +gcc (verified on 5.3.1) can produce performance gains though not siginificant. +``-march=native`` will produce optimized code on local machine and should be +used when software compilation is done on Testbed. + +Performance Tuning +------------------ + +Affinity +~~~~~~~~ + +For superior performance, DPDK pmd threads and Qemu vCPU threads needs to be +affinitized accordingly. + +- PMD thread Affinity + + A poll mode driver (pmd) thread handles the I/O of all DPDK interfaces + assigned to it. A pmd thread shall poll the ports for incoming packets, + switch the packets and send to tx port. pmd thread is CPU bound, and needs + to be affinitized to isolated cores for optimum performance. + + By setting a bit in the mask, a pmd thread is created and pinned to the + corresponding CPU core. e.g. to run a pmd thread on core 2:: + + $ ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x4 + + .. note:: + pmd thread on a NUMA node is only created if there is at least one DPDK + interface from that NUMA node added to OVS. + +- QEMU vCPU thread Affinity + + A VM performing simple packet forwarding or running complex packet pipelines + has to ensure that the vCPU threads performing the work has as much CPU + occupancy as possible. + + For example, on a multicore VM, multiple QEMU vCPU threads shall be spawned. + When the DPDK ``testpmd`` application that does packet forwarding is invoked, + the ``taskset`` command should be used to affinitize the vCPU threads to the + dedicated isolated cores on the host system. + +Multiple Poll-Mode Driver Threads +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +With pmd multi-threading support, OVS creates one pmd thread for each NUMA node +by default. However, in cases where there are multiple ports/rxq's producing +traffic, performance can be improved by creating multiple pmd threads running +on separate cores. These pmd threads can share the workload by each being +responsible for different ports/rxq's. Assignment of ports/rxq's to pmd threads +is done automatically. + +A set bit in the mask means a pmd thread is created and pinned to the +corresponding CPU core. For example, to run pmd threads on core 1 and 2:: + + $ ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x6 + +When using dpdk and dpdkvhostuser ports in a bi-directional VM loopback as +shown below, spreading the workload over 2 or 4 pmd threads shows significant +improvements as there will be more total CPU occupancy available:: + + NIC port0 <-> OVS <-> VM <-> OVS <-> NIC port 1 + +DPDK Physical Port Rx Queues +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + $ ovs-vsctl set Interface <DPDK interface> options:n_rxq=<integer> + +The command above sets the number of rx queues for DPDK physical interface. +The rx queues are assigned to pmd threads on the same NUMA node in a +round-robin fashion. + +DPDK Physical Port Queue Sizes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + $ ovs-vsctl set Interface dpdk0 options:n_rxq_desc=<integer> + $ ovs-vsctl set Interface dpdk0 options:n_txq_desc=<integer> + +The command above sets the number of rx/tx descriptors that the NIC associated +with dpdk0 will be initialised with. + +Different ``n_rxq_desc`` and ``n_txq_desc`` configurations yield different +benefits in terms of throughput and latency for different scenarios. +Generally, smaller queue sizes can have a positive impact for latency at the +expense of throughput. The opposite is often true for larger queue sizes. +Note: increasing the number of rx descriptors eg. to 4096 may have a negative +impact on performance due to the fact that non-vectorised DPDK rx functions may +be used. This is dependant on the driver in use, but is true for the commonly +used i40e and ixgbe DPDK drivers. + +Exact Match Cache +~~~~~~~~~~~~~~~~~ + +Each pmd thread contains one Exact Match Cache (EMC). After initial flow setup +in the datapath, the EMC contains a single table and provides the lowest level +(fastest) switching for DPDK ports. If there is a miss in the EMC then the next +level where switching will occur is the datapath classifier. Missing in the +EMC and looking up in the datapath classifier incurs a significant performance +penalty. If lookup misses occur in the EMC because it is too small to handle +the number of flows, its size can be increased. The EMC size can be modified by +editing the define ``EM_FLOW_HASH_SHIFT`` in ``lib/dpif-netdev.c``. + +As mentioned above, an EMC is per pmd thread. An alternative way of increasing +the aggregate amount of possible flow entries in EMC and avoiding datapath +classifier lookups is to have multiple pmd threads running. + +Rx Mergeable Buffers +~~~~~~~~~~~~~~~~~~~~ + +Rx mergeable buffers is a virtio feature that allows chaining of multiple +virtio descriptors to handle large packet sizes. Large packets are handled by +reserving and chaining multiple free descriptors together. Mergeable buffer +support is negotiated between the virtio driver and virtio device and is +supported by the DPDK vhost library. This behavior is supported and enabled by +default, however in the case where the user knows that rx mergeable buffers are +not needed i.e. jumbo frames are not needed, it can be forced off by adding +``mrg_rxbuf=off`` to the QEMU command line options. By not reserving multiple +chains of descriptors it will make more individual virtio descriptors available +for rx to the guest using dpdkvhost ports and this can improve performance. + +OVS Testcases +------------- + +PHY-VM-PHY (vHost Loopback) +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:doc:`/intro/install/dpdk` details steps for PHY-VM-PHY loopback testcase and +packet forwarding using DPDK testpmd application in the Guest VM. For users +wishing to do packet forwarding using kernel stack below, you need to run the +below commands on the guest:: + + $ ifconfig eth1 1.1.1.2/24 + $ ifconfig eth2 1.1.2.2/24 + $ systemctl stop firewalld.service + $ systemctl stop iptables.service + $ sysctl -w net.ipv4.ip_forward=1 + $ sysctl -w net.ipv4.conf.all.rp_filter=0 + $ sysctl -w net.ipv4.conf.eth1.rp_filter=0 + $ sysctl -w net.ipv4.conf.eth2.rp_filter=0 + $ route add -net 1.1.2.0/24 eth2 + $ route add -net 1.1.1.0/24 eth1 + $ arp -s 1.1.2.99 DE:AD:BE:EF:CA:FE + $ arp -s 1.1.1.99 DE:AD:BE:EF:CA:EE + +PHY-VM-PHY (IVSHMEM) +~~~~~~~~~~~~~~~~~~~~ + +IVSHMEM can also be validated using the PHY-VM-PHY configuration. To begin, +follow the steps described in the :doc:`/intro/install/dpdk` to create and +initialize the database, start ovs-vswitchd and add ``dpdk``-type devices to +bridge ``br0``. Once complete, follow the below steps: + +1. Add DPDK ring port to the bridge:: + + $ ovs-vsctl add-port br0 dpdkr0 -- set Interface dpdkr0 type=dpdkr + +2. Build modified QEMU + + QEMU must be patched to enable IVSHMEM support:: + + $ cd /usr/src/ + $ wget http://wiki.qemu.org/download/qemu-2.2.1.tar.bz2 + $ tar -jxvf qemu-2.2.1.tar.bz2 + $ cd /usr/src/qemu-2.2.1 + $ wget https://raw.githubusercontent.com/netgroup-polito/un-orchestrator/master/orchestrator/compute_controller/plugins/kvm-libvirt/patches/ivshmem-qemu-2.2.1.patch + $ patch -p1 < ivshmem-qemu-2.2.1.patch + $ ./configure --target-list=x86_64-softmmu --enable-debug --extra-cflags='-g' + $ make -j 4 + +3. Generate QEMU commandline:: + + $ mkdir -p /usr/src/cmdline_generator + $ cd /usr/src/cmdline_generator + $ wget https://raw.githubusercontent.com/netgroup-polito/un-orchestrator/master/orchestrator/compute_controller/plugins/kvm-libvirt/cmdline_generator/cmdline_generator.c + $ wget https://raw.githubusercontent.com/netgroup-polito/un-orchestrator/master/orchestrator/compute_controller/plugins/kvm-libvirt/cmdline_generator/Makefile + $ export RTE_SDK=/usr/src/dpdk-16.11 + $ export RTE_TARGET=x86_64-ivshmem-linuxapp-gcc + $ make + $ ./build/cmdline_generator -m -p dpdkr0 XXX + $ cmdline=`cat OVSMEMPOOL` + +4. Start guest VM:: + + $ export VM_NAME=ivshmem-vm + $ export QCOW2_IMAGE=/root/CentOS7_x86_64.qcow2 + $ export QEMU_BIN=/usr/src/qemu-2.2.1/x86_64-softmmu/qemu-system-x86_64 + $ taskset 0x20 $QEMU_BIN -cpu host -smp 2,cores=2 -hda $QCOW2_IMAGE \ + -m 4096 --enable-kvm -name $VM_NAME -nographic -vnc :2 \ + -pidfile /tmp/vm1.pid $cmdline + +5. Build and run the sample ``dpdkr`` app in VM:: + + $ echo 1024 > /proc/sys/vm/nr_hugepages + $ mount -t hugetlbfs nodev /dev/hugepages (if not already mounted) + + # Build the DPDK ring application in the VM + $ export RTE_SDK=/root/dpdk-16.11 + $ export RTE_TARGET=x86_64-ivshmem-linuxapp-gcc + $ make + + # Run dpdkring application + $ ./build/dpdkr -c 1 -n 4 -- -n 0 + # where "-n 0" refers to ring '0' i.e dpdkr0 + +PHY-VM-PHY (vHost Multiqueue) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +vHost Multique functionality can also be validated using the PHY-VM-PHY +configuration. To begin, follow the steps described in +:doc:`/intro/install/dpdk` to create and initialize the database, start +ovs-vswitchd and add ``dpdk``-type devices to bridge ``br0``. Once complete, +follow the below steps: + +1. Configure PMD and RXQs. + + For example, set the number of dpdk port rx queues to at least 2 The number + of rx queues at vhost-user interface gets automatically configured after + virtio device connection and doesn't need manual configuration:: + + $ ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0xC + $ ovs-vsctl set Interface dpdk0 options:n_rxq=2 + $ ovs-vsctl set Interface dpdk1 options:n_rxq=2 + +2. Instantiate Guest VM using QEMU cmdline + + We must configure with appropriate software versions to ensure this feature + is supported. + + .. list-table:: Recommended BIOS Settings + :header-rows: 1 + + * - Setting + - Value + * - QEMU version + - 2.5.0 + * - QEMU thread affinity + - 2 cores (taskset 0x30) + * - Memory + - 4 GB + * - Cores + - 2 + * - Distro + - Fedora 22 + * - Multiqueue + - Enabled + + To do this, instantiate the guest as follows:: + + $ export VM_NAME=vhost-vm + $ export GUEST_MEM=4096M + $ export QCOW2_IMAGE=/root/Fedora22_x86_64.qcow2 + $ export VHOST_SOCK_DIR=/usr/local/var/run/openvswitch + $ taskset 0x30 qemu-system-x86_64 -cpu host -smp 2,cores=2 -m 4096M \ + -drive file=$QCOW2_IMAGE --enable-kvm -name $VM_NAME \ + -nographic -numa node,memdev=mem -mem-prealloc \ + -object memory-backend-file,id=mem,size=$GUEST_MEM,mem-path=/dev/hugepages,share=on \ + -chardev socket,id=char1,path=$VHOST_SOCK_DIR/dpdkvhostuser0 \ + -netdev type=vhost-user,id=mynet1,chardev=char1,vhostforce,queues=2 \ + -device virtio-net-pci,mac=00:00:00:00:00:01,netdev=mynet1,mq=on,vectors=6 \ + -chardev socket,id=char2,path=$VHOST_SOCK_DIR/dpdkvhostuser1 \ + -netdev type=vhost-user,id=mynet2,chardev=char2,vhostforce,queues=2 \ + -device virtio-net-pci,mac=00:00:00:00:00:02,netdev=mynet2,mq=on,vectors=6 + + .. note:: + Queue value above should match the queues configured in OVS, The vector + value should be set to "number of queues x 2 + 2" + +3. Configure the guest interface + + Assuming there are 2 interfaces in the guest named eth0, eth1 check the + channel configuration and set the number of combined channels to 2 for + virtio devices:: + + $ ethtool -l eth0 + $ ethtool -L eth0 combined 2 + $ ethtool -L eth1 combined 2 + + More information can be found in vHost walkthrough section. + +4. Configure kernel packet forwarding + + Configure IP and enable interfaces:: + + $ ifconfig eth0 5.5.5.1/24 up + $ ifconfig eth1 90.90.90.1/24 up + + Configure IP forwarding and add route entries:: + + $ sysctl -w net.ipv4.ip_forward=1 + $ sysctl -w net.ipv4.conf.all.rp_filter=0 + $ sysctl -w net.ipv4.conf.eth0.rp_filter=0 + $ sysctl -w net.ipv4.conf.eth1.rp_filter=0 + $ ip route add 2.1.1.0/24 dev eth1 + $ route add default gw 2.1.1.2 eth1 + $ route add default gw 90.90.90.90 eth1 + $ arp -s 90.90.90.90 DE:AD:BE:EF:CA:FE + $ arp -s 2.1.1.2 DE:AD:BE:EF:CA:FA + + Check traffic on multiple queues:: + + $ cat /proc/interrupts | grep virtio + +vHost Walkthrough +----------------- + +Two types of vHost User ports are available in OVS: + +- vhost-user (``dpdkvhostuser``) + +- vhost-user-client (``dpdkvhostuserclient``) + +vHost User uses a client-server model. The server creates/manages/destroys the +vHost User sockets, and the client connects to the server. Depending on which +port type you use, ``dpdkvhostuser`` or ``dpdkvhostuserclient``, a different +configuration of the client-server model is used. + +For vhost-user ports, Open vSwitch acts as the server and QEMU the client. For +vhost-user-client ports, Open vSwitch acts as the client and QEMU the server. + +vhost-user +~~~~~~~~~~ + +1. Install the prerequisites: + + - QEMU version >= 2.2 + +2. Add vhost-user ports to the switch. + + Unlike DPDK ring ports, DPDK vhost-user ports can have arbitrary names, + except that forward and backward slashes are prohibited in the names. + + For vhost-user, the name of the port type is ``dpdkvhostuser``:: + + $ ovs-vsctl add-port br0 vhost-user-1 -- set Interface vhost-user-1 \ + type=dpdkvhostuser + + This action creates a socket located at + ``/usr/local/var/run/openvswitch/vhost-user-1``, which you must provide to + your VM on the QEMU command line. More instructions on this can be found in + the next section "Adding vhost-user ports to VM" + + .. note:: + If you wish for the vhost-user sockets to be created in a sub-directory of + ``/usr/local/var/run/openvswitch``, you may specify this directory in the + ovsdb like so:: + + $ ovs-vsctl --no-wait \ + set Open_vSwitch . other_config:vhost-sock-dir=subdir` + +3. Add vhost-user ports to VM + + 1. Configure sockets + + Pass the following parameters to QEMU to attach a vhost-user device:: + + -chardev socket,id=char1,path=/usr/local/var/run/openvswitch/vhost-user-1 + -netdev type=vhost-user,id=mynet1,chardev=char1,vhostforce + -device virtio-net-pci,mac=00:00:00:00:00:01,netdev=mynet1 + + where ``vhost-user-1`` is the name of the vhost-user port added to the + switch. + + Repeat the above parameters for multiple devices, changing the chardev + ``path`` and ``id`` as necessary. Note that a separate and different + chardev ``path`` needs to be specified for each vhost-user device. For + example you have a second vhost-user port named ``vhost-user-2``, you + append your QEMU command line with an additional set of parameters:: + + -chardev socket,id=char2,path=/usr/local/var/run/openvswitch/vhost-user-2 + -netdev type=vhost-user,id=mynet2,chardev=char2,vhostforce + -device virtio-net-pci,mac=00:00:00:00:00:02,netdev=mynet2 + + 2. Configure hugepages + + QEMU must allocate the VM's memory on hugetlbfs. vhost-user ports access + a virtio-net device's virtual rings and packet buffers mapping the VM's + physical memory on hugetlbfs. To enable vhost-user ports to map the VM's + memory into their process address space, pass the following parameters + to QEMU:: + + -object memory-backend-file,id=mem,size=4096M,mem-path=/dev/hugepages,share=on + -numa node,memdev=mem -mem-prealloc + + 3. Enable multiqueue support (optional) + + QEMU needs to be configured to use multiqueue:: + + -chardev socket,id=char2,path=/usr/local/var/run/openvswitch/vhost-user-2 + -netdev type=vhost-user,id=mynet2,chardev=char2,vhostforce,queues=$q + -device virtio-net-pci,mac=00:00:00:00:00:02,netdev=mynet2,mq=on,vectors=$v + + where: + + ``$q`` + The number of queues + ``$v`` + The number of vectors, which is ``$q`` * 2 + 2 + + The vhost-user interface will be automatically reconfigured with + required number of rx and tx queues after connection of virtio device. + Manual configuration of ``n_rxq`` is not supported because OVS will work + properly only if ``n_rxq`` will match number of queues configured in + QEMU. + + A least 2 PMDs should be configured for the vswitch when using + multiqueue. Using a single PMD will cause traffic to be enqueued to the + same vhost queue rather than being distributed among different vhost + queues for a vhost-user interface. + + If traffic destined for a VM configured with multiqueue arrives to the + vswitch via a physical DPDK port, then the number of rxqs should also be + set to at least 2 for that physical DPDK port. This is required to + increase the probability that a different PMD will handle the multiqueue + transmission to the guest using a different vhost queue. + + If one wishes to use multiple queues for an interface in the guest, the + driver in the guest operating system must be configured to do so. It is + recommended that the number of queues configured be equal to ``$q``. + + For example, this can be done for the Linux kernel virtio-net driver + with:: + + $ ethtool -L <DEV> combined <$q> + + where: + + ``-L`` + Changes the numbers of channels of the specified network device + ``combined`` + Changes the number of multi-purpose channels. + +Configure the VM using libvirt +++++++++++++++++++++++++++++++ + +You can also build and configure the VM using libvirt rather than QEMU by +itself. + +1. Change the user/group, access control policty and restart libvirtd. + + - In ``/etc/libvirt/qemu.conf`` add/edit the following lines:: + + user = "root" + group = "root" + + - Disable SELinux or set to permissive mode:: + + $ setenforce 0 + + - Restart the libvirtd process, For example, on Fedora:: + + $ systemctl restart libvirtd.service + +2. Instantiate the VM + + - Copy the XML configuration described in :doc:`/intro/install/dpdk` + + - Start the VM:: + + $ virsh create demovm.xml + + - Connect to the guest console:: + + $ virsh console demovm + +3. Configure the VM + + The demovm xml configuration is aimed at achieving out of box performance on + VM. + + - The vcpus are pinned to the cores of the CPU socket 0 using ``vcpupin``. + + - Configure NUMA cell and memory shared using ``memAccess='shared'``. + + - Disable ``mrg_rxbuf='off'`` + +Refer to the `libvirt documentation <http://libvirt.org/formatdomain.html>`__ +for more information. + +vhost-user-client +~~~~~~~~~~~~~~~~~ + +1. Install the prerequisites: + + - QEMU version >= 2.7 + +2. Add vhost-user-client ports to the switch. + + Unlike vhost-user ports, the name given to port does not govern the name of + the socket device. ``vhost-server-path`` reflects the full path of the + socket that has been or will be created by QEMU for the given vHost User + client port. + + For vhost-user-client, the name of the port type is + ``dpdkvhostuserclient``:: + + $ VHOST_USER_SOCKET_PATH=/path/to/socker + $ ovs-vsctl add-port br0 vhost-client-1 \ + -- set Interface vhost-client-1 type=dpdkvhostuserclient \ + options:vhost-server-path=$VHOST_USER_SOCKET_PATH + +3. Add vhost-user-client ports to VM + + 1. Configure sockets + + Pass the following parameters to QEMU to attach a vhost-user device:: + + -chardev socket,id=char1,path=$VHOST_USER_SOCKET_PATH,server + -netdev type=vhost-user,id=mynet1,chardev=char1,vhostforce + -device virtio-net-pci,mac=00:00:00:00:00:01,netdev=mynet1 + + where ``vhost-user-1`` is the name of the vhost-user port added to the + switch. + + If the corresponding dpdkvhostuserclient port has not yet been configured + in OVS with ``vhost-server-path=/path/to/socket``, QEMU will print a log + similar to the following:: + + QEMU waiting for connection on: disconnected:unix:/path/to/socket,server + + QEMU will wait until the port is created sucessfully in OVS to boot the VM. + + One benefit of using this mode is the ability for vHost ports to + 'reconnect' in event of the switch crashing or being brought down. Once + it is brought back up, the vHost ports will reconnect automatically and + normal service will resume. + +DPDK Backend Inside VM +~~~~~~~~~~~~~~~~~~~~~~ + +Additional configuration is required if you want to run ovs-vswitchd with DPDK +backend inside a QEMU virtual machine. Ovs-vswitchd creates separate DPDK TX +queues for each CPU core available. This operation fails inside QEMU virtual +machine because, by default, VirtIO NIC provided to the guest is configured to +support only single TX queue and single RX queue. To change this behavior, you +need to turn on ``mq`` (multiqueue) property of all ``virtio-net-pci`` devices +emulated by QEMU and used by DPDK. You may do it manually (by changing QEMU +command line) or, if you use Libvirt, by adding the following string to +``<interface>`` sections of all network devices used by DPDK:: + + <driver name='vhost' queues='N'/> + +Where: + +``N`` + determines how many queues can be used by the guest. + +This requires QEMU >= 2.2. + +QoS +--- + +Assuming you have a vhost-user port transmitting traffic consisting of packets +of size 64 bytes, the following command would limit the egress transmission +rate of the port to ~1,000,000 packets per second:: + + $ ovs-vsctl set port vhost-user0 qos=@newqos -- \ + --id=@newqos create qos type=egress-policer other-config:cir=46000000 \ + other-config:cbs=2048` + +To examine the QoS configuration of the port, run:: + + $ ovs-appctl -t ovs-vswitchd qos/show vhost-user0 + +To clear the QoS configuration from the port and ovsdb, run:: + + $ ovs-vsctl destroy QoS vhost-user0 -- clear Port vhost-user0 qos + +Refer to vswitch.xml for more details on egress-policer. + +Rate Limiting +-------------- + +Here is an example on Ingress Policing usage. Assuming you have a vhost-user +port receiving traffic consisting of packets of size 64 bytes, the following +command would limit the reception rate of the port to ~1,000,000 packets per +second:: + + $ ovs-vsctl set interface vhost-user0 ingress_policing_rate=368000 \ + ingress_policing_burst=1000` + +To examine the ingress policer configuration of the port:: + + $ ovs-vsctl list interface vhost-user0 + +To clear the ingress policer configuration from the port:: + + $ ovs-vsctl set interface vhost-user0 ingress_policing_rate=0 + +Refer to vswitch.xml for more details on ingress-policer. + +Flow Control +------------ + +Flow control can be enabled only on DPDK physical ports. To enable flow +control support at tx side while adding a port, run:: + + $ ovs-vsctl add-port br0 dpdk0 -- \ + set Interface dpdk0 type=dpdk options:tx-flow-ctrl=true + +Similarly, to enable rx flow control, run:: + + $ ovs-vsctl add-port br0 dpdk0 -- \ + set Interface dpdk0 type=dpdk options:rx-flow-ctrl=true + +To enable flow control auto-negotiation, run:: + + $ ovs-vsctl add-port br0 dpdk0 -- \ + set Interface dpdk0 type=dpdk options:flow-ctrl-autoneg=true + +To turn ON the tx flow control at run time(After the port is being added to +OVS):: + + $ ovs-vsctl set Interface dpdk0 options:tx-flow-ctrl=true + +The flow control parameters can be turned off by setting ``false`` to the +respective parameter. To disable the flow control at tx side, run:: + + $ ovs-vsctl set Interface dpdk0 options:tx-flow-ctrl=false + +pdump +----- + +Pdump allows you to listen on DPDK ports and view the traffic that is passing +on them. To use this utility, one must have libpcap installed on the system. +Furthermore, DPDK must be built with ``CONFIG_RTE_LIBRTE_PDUMP=y`` and +``CONFIG_RTE_LIBRTE_PMD_PCAP=y``. + +.. warning:: + A performance decrease is expected when using a monitoring application like + the DPDK pdump app. + +To use pdump, simply launch OVS as usual. Then, navigate to the ``app/pdump`` +directory in DPDK, ``make`` the application and run like so:: + + $ sudo ./build/app/dpdk-pdump -- \ + --pdump port=0,queue=0,rx-dev=/tmp/pkts.pcap \ + --server-socket-path=/usr/local/var/run/openvswitch + +The above command captures traffic received on queue 0 of port 0 and stores it +in ``/tmp/pkts.pcap``. Other combinations of port numbers, queues numbers and +pcap locations are of course also available to use. For example, to capture all +packets that traverse port 0 in a single pcap file:: + + $ sudo ./build/app/dpdk-pdump -- \ + --pdump 'port=0,queue=*,rx-dev=/tmp/pkts.pcap,tx-dev=/tmp/pkts.pcap' \ + --server-socket-path=/usr/local/var/run/openvswitch + +``server-socket-path`` must be set to the value of ovs_rundir() which typically +resolves to ``/usr/local/var/run/openvswitch``. + +Many tools are available to view the contents of the pcap file. Once example is +tcpdump. Issue the following command to view the contents of ``pkts.pcap``:: + + $ tcpdump -r pkts.pcap + +More information on the pdump app and its usage can be found in the `DPDK docs +<http://dpdk.org/doc/guides/sample_app_ug/pdump.html>`__. + +Jumbo Frames +------------ + +By default, DPDK ports are configured with standard Ethernet MTU (1500B). To +enable Jumbo Frames support for a DPDK port, change the Interface's +``mtu_request`` attribute to a sufficiently large value. For example, to add a +DPDK Phy port with MTU of 9000:: + + $ ovs-vsctl add-port br0 dpdk0 \ + -- set Interface dpdk0 type=dpdk \ + -- set Interface dpdk0 mtu_request=9000` + +Similarly, to change the MTU of an existing port to 6200:: + + $ ovs-vsctl set Interface dpdk0 mtu_request=6200 + +Some additional configuration is needed to take advantage of jumbo frames with +vHost ports: + +1. *mergeable buffers* must be enabled for vHost ports, as demonstrated in the + QEMU command line snippet below:: + + -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \ + -device virtio-net-pci,mac=00:00:00:00:00:01,netdev=mynet1,mrg_rxbuf=on + +2. Where virtio devices are bound to the Linux kernel driver in a guest + environment (i.e. interfaces are not bound to an in-guest DPDK driver), the + MTU of those logical network interfaces must also be increased to a + sufficiently large value. This avoids segmentation of Jumbo Frames received + in the guest. Note that 'MTU' refers to the length of the IP packet only, + and not that of the entire frame. + + To calculate the exact MTU of a standard IPv4 frame, subtract the L2 header + and CRC lengths (i.e. 18B) from the max supported frame size. So, to set + the MTU for a 9018B Jumbo Frame:: + + $ ifconfig eth1 mtu 9000 + +When Jumbo Frames are enabled, the size of a DPDK port's mbuf segments are +increased, such that a full Jumbo Frame of a specific size may be accommodated +within a single mbuf segment. + +Jumbo frame support has been validated against 9728B frames, which is the +largest frame size supported by Fortville NIC using the DPDK i40e driver, but +larger frames and other DPDK NIC drivers may be supported. These cases are +common for use cases involving East-West traffic only. + +vsperf +------ + +The vsperf project aims to develop a vSwitch test framework that can be used to +validate the suitability of different vSwitch implementations in a telco +deployment environment. More information can be found on the `OPNFV wiki +<https://wiki.opnfv.org/display/vsperf/VSperf+Home>`__. + +Bug Reporting +------------- + +Report problems to bugs@openvswitch.org. diff --git a/Documentation/intro/install/dpdk.rst b/Documentation/intro/install/dpdk.rst new file mode 100644 index 000000000..7724c8a9f --- /dev/null +++ b/Documentation/intro/install/dpdk.rst @@ -0,0 +1,600 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + + Convention for heading levels in Open vSwitch documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + +====================== +Open vSwitch with DPDK +====================== + +This document describes how to build and install Open vSwitch using a DPDK +datapath. Open vSwitch can use the DPDK library to operate entirely in +userspace. + +.. warning:: + The DPDK support of Open vSwitch is considered 'experimental'. + +Build requirements +------------------ + +In addition to the requirements described in :doc:`general`, building Open +vSwitch with DPDK will require the following: + +- DPDK 16.11 + +- A `DPDK supported NIC`_ + + Only required when physical ports are in use + +- A suitable kernel + + On Linux Distros running kernel version >= 3.0, only `IOMMU` needs to enabled + via the grub cmdline, assuming you are using **VFIO**. For older kernels, + ensure the kernel is built with ``UIO``, ``HUGETLBFS``, + ``PROC_PAGE_MONITOR``, ``HPET``, ``HPET_MMAP`` support. If these are not + present, it will be necessary to upgrade your kernel or build a custom kernel + with these flags enabled. + +.. TODO(stephenfin): drag the below information in from dpdk-advanced + +Detailed system requirements can be found at `DPDK requirements`_, while more +detailed install information can be found in :doc:`dpdk-advanced`. + +.. _DPDK supported NIC: http://dpdk.org/doc/nics +.. _DPDK requirements: http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html + +Installing +---------- + +DPDK +~~~~ + +1. Download the `DPDK sources`_, extract the file and set ``DPDK_DIR``:: + + $ cd /usr/src/ + $ wget http://fast.dpdk.org/rel/dpdk-16.11.tar.xz + $ tar xf dpdk-16.11.tar.xz + $ export DPDK_DIR=/usr/src/dpdk-16.11 + $ cd $DPDK_DIR + +2. Configure and install DPDK + + Build and install the DPDK library:: + + $ export DPDK_TARGET=x86_64-native-linuxapp-gcc + $ export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET + $ make install T=$DPDK_TARGET DESTDIR=install + + If IVSHMEM support is required, use a different target:: + + $ export DPDK_TARGET=x86_64-ivshmem-linuxapp-gcc + +.. _DPDK sources: http://dpdk.org/rel + +Install OVS +~~~~~~~~~~~ + +OVS can be installed using different methods. For OVS to use DPDK datapath, it +has to be configured with DPDK support (``--with-dpdk``). + +.. note:: + This section focuses on generic recipe that suits most cases. For + distribution specific instructions, refer to one of the more relevant guides. + +.. _OVS sources: http://openvswitch.org/releases/ + +1. Ensure the standard OVS requirements, described in + :ref:`general-build-reqs`, are installed + +2. Bootstrap, if required, as described in :ref:`general-bootstrapping` + +3. Configure the package using the ``--with-dpdk`` flag:: + + $ ./configure --with-dpdk=$DPDK_BUILD + + where ``DPDK_BUILD`` is the path to the built DPDK library. This can be + skipped if DPDK library is installed in its default location + + .. note:: + While ``--with-dpdk`` is required, you can pass any other configuration + option described in :ref:`general-configuring`. + +4. Build and install OVS, as described in :ref:`general-building` + +Additional information can be found in :doc:`general`. + +Setup +----- + +Setup Hugepages +~~~~~~~~~~~~~~~ + +Allocate a number of 2M Huge pages: + +- For persistent allocation of huge pages, write to hugepages.conf file + in `/etc/sysctl.d`:: + + $ echo 'vm.nr_hugepages=2048' > /etc/sysctl.d/hugepages.conf + +- For run-time allocation of huge pages, use the ``sysctl`` utility:: + + $ sysctl -w vm.nr_hugepages=N # where N = No. of 2M huge pages + +To verify hugepage configuration:: + + $ grep HugePages_ /proc/meminfo + +Mount the hugepages, if not already mounted by default:: + + $ mount -t hugetlbfs none /dev/hugepages`` + +.. _dpdk-vfio: + +Setup DPDK devices using VFIO +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +VFIO is prefered to the UIO driver when using recent versions of DPDK. VFIO +support required support from both the kernel and BIOS. For the former, kernel +version > 3.6 must be used. For the latter, you must enable VT-d in the BIOS +and ensure this is configured via grub. To ensure VT-d is enabled via the BIOS, +run:: + + $ dmesg | grep -e DMAR -e IOMMU + +If VT-d is not enabled in the BIOS, enable it now. + +To ensure VT-d is enabled in the kernel, run:: + + $ cat /proc/cmdline | grep iommu=pt + $ cat /proc/cmdline | grep intel_iommu=on + +If VT-d is not enabled in the kernel, enable it now. + +Once VT-d is correctly configured, load the required modules and bind the NIC +to the VFIO driver:: + + $ modprobe vfio-pci + $ /usr/bin/chmod a+x /dev/vfio + $ /usr/bin/chmod 0666 /dev/vfio/* + $ $DPDK_DIR/tools/dpdk-devbind.py --bind=vfio-pci eth1 + $ $DPDK_DIR/tools/dpdk-devbind.py --status + +Setup OVS +~~~~~~~~~ + +Open vSwitch should be started as described in :doc:`general` with the +exception of ovs-vswitchd, which requires some special configuration to enable +DPDK functionality. DPDK configuration arguments can be passed to ovs-vswitchd +via the ``other_config`` column of the ``Open_vSwitch`` table. At a minimum, +the ``dpdk-init`` option must be set to ``true``. For example:: + + $ export DB_SOCK=/usr/local/var/run/openvswitch/db.sock + $ ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true + $ ovs-vswitchd unix:$DB_SOCK --pidfile --detach + +There are many other configuration options, the most important of which are +listed below. Defaults will be provided for all values not explicitly set. + +``dpdk-init`` + Specifies whether OVS should initialize and support DPDK ports. This is a + boolean, and defaults to false. + +``dpdk-lcore-mask`` + Specifies the CPU cores on which dpdk lcore threads should be spawned and + expects hex string (eg '0x123'). + +``dpdk-socket-mem`` + Comma separated list of memory to pre-allocate from hugepages on specific + sockets. + +``dpdk-hugepage-dir`` + Directory where hugetlbfs is mounted + +``vhost-sock-dir`` + Option to set the path to the vhost-user unix socket files. + +If allocating more than one GB hugepage (as for IVSHMEM), you can configure the +amount of memory used from any given NUMA nodes. For example, to use 1GB from +NUMA node 0, run:: + + $ ovs-vsctl --no-wait set Open_vSwitch . \ + other_config:dpdk-socket-mem="1024,0" + +Similarly, if you wish to better scale the workloads across cores, then +multiple pmd threads can be created and pinned to CPU cores by explicity +specifying ``pmd-cpu-mask``. Cores are numbered from 0, so to spawn two pmd +threads and pin them to cores 1,2, run:: + + $ ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x6 + +For details on using ivshmem with DPDK, refer to :doc:`dpdk-advanced`. + +Refer to ovs-vswitchd.conf.db(5) for additional information on configuration +options. + +.. note:: + Changing any of these options requires restarting the ovs-vswitchd + application + +Validating +---------- + +Creating bridges and ports +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can now use ovs-vsctl to set up bridges and other Open vSwitch features. +Bridges should be created with a ``datapath_type=netdev``:: + + $ ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev + +Now you can add DPDK devices. OVS expects DPDK device names to start with +``dpdk`` and end with a portid. ovs-vswitchd should print the number of dpdk +devices found in the log file:: + + $ ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk + $ ovs-vsctl add-port br0 dpdk1 -- set Interface dpdk1 type=dpdk + +After the DPDK ports get added to switch, a polling thread continuously polls +DPDK devices and consumes 100% of the core, as can be checked from 'top' and +'ps' cmds:: + + $ top -H + $ ps -eLo pid,psr,comm | grep pmd + +Creating bonds of DPDK interfaces is slightly different to creating bonds of +system interfaces. For DPDK, the interface type must be explicitly set. For +example:: + + $ ovs-vsctl add-bond br0 dpdkbond dpdk0 dpdk1 \ + -- set Interface dpdk0 type=dpdk \ + -- set Interface dpdk1 type=dpdk + +To stop ovs-vswitchd & delete bridge, run:: + + $ ovs-appctl -t ovs-vswitchd exit + $ ovs-appctl -t ovsdb-server exit + $ ovs-vsctl del-br br0 + +PMD thread statistics +~~~~~~~~~~~~~~~~~~~~~ + +To show current stats:: + + $ ovs-appctl dpif-netdev/pmd-stats-show + +To clear previous stats:: + + $ ovs-appctl dpif-netdev/pmd-stats-clear + +Port/rxq assigment to PMD threads +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To show port/rxq assignment:: + + $ ovs-appctl dpif-netdev/pmd-rxq-show + +To change default rxq assignment to pmd threads, rxqs may be manually pinned to +desired cores using:: + + $ ovs-vsctl set Interface <iface> \ + other_config:pmd-rxq-affinity=<rxq-affinity-list> + +where: + +- ``<rxq-affinity-list>`` ::= ``NULL`` | ``<non-empty-list>`` +- ``<non-empty-list>`` ::= ``<affinity-pair>`` | + ``<affinity-pair>`` , ``<non-empty-list>`` +- ``<affinity-pair>`` ::= ``<queue-id>`` : ``<core-id>`` + +For example:: + + $ ovs-vsctl set interface dpdk0 options:n_rxq=4 \ + other_config:pmd-rxq-affinity="0:3,1:7,3:8" + +This will ensure: + +- Queue #0 pinned to core 3 +- Queue #1 pinned to core 7 +- Queue #2 not pinned +- Queue #3 pinned to core 8 + +After that PMD threads on cores where RX queues was pinned will become +``isolated``. This means that this thread will poll only pinned RX queues. + +.. warning:: + If there are no ``non-isolated`` PMD threads, ``non-pinned`` RX queues will + not be polled. Also, if provided ``core_id`` is not available (ex. this + ``core_id`` not in ``pmd-cpu-mask``), RX queue will not be polled by any PMD + thread. + +.. _dpdk-guest-setup: + +DPDK in the VM +-------------- + +DPDK 'testpmd' application can be run in the Guest VM for high speed packet +forwarding between vhostuser ports. DPDK and testpmd application has to be +compiled on the guest VM. Below are the steps for setting up the testpmd +application in the VM. More information on the vhostuser ports can be found in +:doc:`dpdk-advanced`. + +.. note:: + Support for DPDK in the guest requires QEMU >= 2.2.0. + +To being, instantiate the guest:: + + $ export VM_NAME=Centos-vm export GUEST_MEM=3072M + $ export QCOW2_IMAGE=/root/CentOS7_x86_64.qcow2 + $ export VHOST_SOCK_DIR=/usr/local/var/run/openvswitch + + $ qemu-system-x86_64 -name $VM_NAME -cpu host -enable-kvm \ + -m $GUEST_MEM -drive file=$QCOW2_IMAGE --nographic -snapshot \ + -numa node,memdev=mem -mem-prealloc -smp sockets=1,cores=2 \ + -object memory-backend-file,id=mem,size=$GUEST_MEM,mem-path=/dev/hugepages,share=on \ + -chardev socket,id=char0,path=$VHOST_SOCK_DIR/dpdkvhostuser0 \ + -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \ + -device virtio-net-pci,mac=00:00:00:00:00:01,netdev=mynet1,mrg_rxbuf=off \ + -chardev socket,id=char1,path=$VHOST_SOCK_DIR/dpdkvhostuser1 \ + -netdev type=vhost-user,id=mynet2,chardev=char1,vhostforce \ + -device virtio-net-pci,mac=00:00:00:00:00:02,netdev=mynet2,mrg_rxbuf=off \ + +Download the DPDK sourcs to VM and build DPDK:: + + $ cd /root/dpdk/ + $ wget http://fast.dpdk.org/rel/dpdk-16.11.tar.xz + $ tar xf dpdk-16.11.tar.xz + $ export DPDK_DIR=/root/dpdk/dpdk-16.11 + $ export DPDK_TARGET=x86_64-native-linuxapp-gcc + $ export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET + $ cd $DPDK_DIR + $ make install T=$DPDK_TARGET DESTDIR=install + +Build the test-pmd application:: + + $ cd app/test-pmd + $ export RTE_SDK=$DPDK_DIR + $ export RTE_TARGET=$DPDK_TARGET + $ make + +Setup huge pages and DPDK devices using UIO:: + + $ sysctl vm.nr_hugepages=1024 + $ mkdir -p /dev/hugepages + $ mount -t hugetlbfs hugetlbfs /dev/hugepages # only if not already mounted + $ modprobe uio + $ insmod $DPDK_BUILD/kmod/igb_uio.ko + $ $DPDK_DIR/tools/dpdk-devbind.py --status + $ $DPDK_DIR/tools/dpdk-devbind.py -b igb_uio 00:03.0 00:04.0 + +.. note:: + + vhost ports pci ids can be retrieved using:: + + lspci | grep Ethernet + +Testing +------- + +Below are few testcases and the list of steps to be followed. Before beginning, +ensure a userspace bridge has been created and two DPDK ports added:: + + $ ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev + $ ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk + $ ovs-vsctl add-port br0 dpdk1 -- set Interface dpdk1 type=dpdk + +PHY-PHY +~~~~~~~ + +Add test flows to forward packets betwen DPDK port 0 and port 1:: + + # Clear current flows + $ ovs-ofctl del-flows br0 + + # Add flows between port 1 (dpdk0) to port 2 (dpdk1) + $ ovs-ofctl add-flow br0 in_port=1,action=output:2 + $ ovs-ofctl add-flow br0 in_port=2,action=output:1 + +Transmit traffic into either port. You should see it returned via the other. + +PHY-VM-PHY (vhost loopback) +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Add two ``dpdkvhostuser`` ports to bridge ``br0``:: + + $ ovs-vsctl add-port br0 dpdkvhostuser0 \ + -- set Interface dpdkvhostuser0 type=dpdkvhostuser + $ ovs-vsctl add-port br0 dpdkvhostuser1 \ + -- set Interface dpdkvhostuser1 type=dpdkvhostuser + +Add test flows to forward packets betwen DPDK devices and VM ports:: + + # Clear current flows + $ ovs-ofctl del-flows br0 + + # Add flows + $ ovs-ofctl add-flow br0 in_port=1,action=output:3 + $ ovs-ofctl add-flow br0 in_port=3,action=output:1 + $ ovs-ofctl add-flow br0 in_port=4,action=output:2 + $ ovs-ofctl add-flow br0 in_port=2,action=output:4 + + # Dump flows + $ ovs-ofctl dump-flows br0 + +Create a VM using the following configuration: + ++----------------------+--------+-----------------+ +| configuration | values | comments | ++----------------------+--------+-----------------+ +| qemu version | 2.2.0 | n/a | +| qemu thread affinity | core 5 | taskset 0x20 | +| memory | 4GB | n/a | +| cores | 2 | n/a | +| Qcow2 image | CentOS7| n/a | +| mrg_rxbuf | off | n/a | ++----------------------+--------+-----------------+ + +You can do this directly with QEMU via the ``qemu-system-x86_64`` +application:: + + $ export VM_NAME=vhost-vm + $ export GUEST_MEM=3072M + $ export QCOW2_IMAGE=/root/CentOS7_x86_64.qcow2 + $ export VHOST_SOCK_DIR=/usr/local/var/run/openvswitch + + $ taskset 0x20 qemu-system-x86_64 -name $VM_NAME -cpu host -enable-kvm \ + -m $GUEST_MEM -drive file=$QCOW2_IMAGE --nographic -snapshot \ + -numa node,memdev=mem -mem-prealloc -smp sockets=1,cores=2 \ + -object memory-backend-file,id=mem,size=$GUEST_MEM,mem-path=/dev/hugepages,share=on \ + -chardev socket,id=char0,path=$VHOST_SOCK_DIR/dpdkvhostuser0 \ + -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \ + -device virtio-net-pci,mac=00:00:00:00:00:01,netdev=mynet1,mrg_rxbuf=off \ + -chardev socket,id=char1,path=$VHOST_SOCK_DIR/dpdkvhostuser1 \ + -netdev type=vhost-user,id=mynet2,chardev=char1,vhostforce \ + -device virtio-net-pci,mac=00:00:00:00:00:02,netdev=mynet2,mrg_rxbuf=off + +Alternatively, you can configure the guest using libvirt. Below is an XML +configuration for a 'demovm' guest that can be instantiated using `virsh`:: + + <domain type='kvm'> + <name>demovm</name> + <uuid>4a9b3f53-fa2a-47f3-a757-dd87720d9d1d</uuid> + <memory unit='KiB'>4194304</memory> + <currentMemory unit='KiB'>4194304</currentMemory> + <memoryBacking> + <hugepages> + <page size='2' unit='M' nodeset='0'/> + </hugepages> + </memoryBacking> + <vcpu placement='static'>2</vcpu> + <cputune> + <shares>4096</shares> + <vcpupin vcpu='0' cpuset='4'/> + <vcpupin vcpu='1' cpuset='5'/> + <emulatorpin cpuset='4,5'/> + </cputune> + <os> + <type arch='x86_64' machine='pc'>hvm</type> + <boot dev='hd'/> + </os> + <features> + <acpi/> + <apic/> + </features> + <cpu mode='host-model'> + <model fallback='allow'/> + <topology sockets='2' cores='1' threads='1'/> + <numa> + <cell id='0' cpus='0-1' memory='4194304' unit='KiB' memAccess='shared'/> + </numa> + </cpu> + <on_poweroff>destroy</on_poweroff> + <on_reboot>restart</on_reboot> + <on_crash>destroy</on_crash> + <devices> + <emulator>/usr/bin/qemu-kvm</emulator> + <disk type='file' device='disk'> + <driver name='qemu' type='qcow2' cache='none'/> + <source file='/root/CentOS7_x86_64.qcow2'/> + <target dev='vda' bus='virtio'/> + </disk> + <disk type='dir' device='disk'> + <driver name='qemu' type='fat'/> + <source dir='/usr/src/dpdk-16.11'/> + <target dev='vdb' bus='virtio'/> + <readonly/> + </disk> + <interface type='vhostuser'> + <mac address='00:00:00:00:00:01'/> + <source type='unix' path='/usr/local/var/run/openvswitch/dpdkvhostuser0' mode='client'/> + <model type='virtio'/> + <driver queues='2'> + <host mrg_rxbuf='off'/> + </driver> + </interface> + <interface type='vhostuser'> + <mac address='00:00:00:00:00:02'/> + <source type='unix' path='/usr/local/var/run/openvswitch/dpdkvhostuser1' mode='client'/> + <model type='virtio'/> + <driver queues='2'> + <host mrg_rxbuf='off'/> + </driver> + </interface> + <serial type='pty'> + <target port='0'/> + </serial> + <console type='pty'> + <target type='serial' port='0'/> + </console> + </devices> + </domain> + +Once the guest is configured and booted, configure DPDK packet forwarding +within the guest. To accomplish this, DPDK and testpmd application have to +be first compiled on the VM as described in **Guest Setup**. Once compiled, run +the ``test-pmd`` application:: + + $ cd $DPDK_DIR/app/test-pmd; + $ ./testpmd -c 0x3 -n 4 --socket-mem 1024 -- \ + --burst=64 -i --txqflags=0xf00 --disable-hw-vlan + $ set fwd mac retry + $ start + +When you finish testing, bind the vNICs back to kernel:: + + $ $DPDK_DIR/tools/dpdk-devbind.py --bind=virtio-pci 0000:00:03.0 + $ $DPDK_DIR/tools/dpdk-devbind.py --bind=virtio-pci 0000:00:04.0 + +.. note:: + Appropriate PCI IDs to be passed in above example. The PCI IDs can be + retrieved like so:: + + $ $DPDK_DIR/tools/dpdk-devbind.py --status + +.. note:: + More information on the dpdkvhostuser ports can be found in + :doc:`dpdk-advanced`. + +PHY-VM-PHY (IVSHMEM loopback) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Refer to the :doc:`dpdk-advanced`. + +Limitations +------------ + +- Currently DPDK ports does not use HW offload functionality. +- Network Interface Firmware requirements: Each release of DPDK is + validated against a specific firmware version for a supported Network + Interface. New firmware versions introduce bug fixes, performance + improvements and new functionality that DPDK leverages. The validated + firmware versions are available as part of the release notes for + DPDK. It is recommended that users update Network Interface firmware + to match what has been validated for the DPDK release. + + The latest list of validated firmware versions can be found in the `DPDK + release notes`_. + +.. _DPDK release notes: http://dpdk.org/doc/guides/rel_notes/release_16_11.html + +Reporting Bugs +-------------- + +Report problems to bugs@openvswitch.org. diff --git a/Documentation/intro/install/fedora.rst b/Documentation/intro/install/fedora.rst new file mode 100644 index 000000000..73635386a --- /dev/null +++ b/Documentation/intro/install/fedora.rst @@ -0,0 +1,141 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + + Convention for heading levels in Open vSwitch documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + +=========================================== +Fedora, RHEL 7.x Packaging for Open vSwitch +=========================================== + +This document provides instructions for building and installing Open vSwitch +RPM packages on a Fedora Linux host. Instructions for the installation of Open +vSwitch on a Fedora Linux host without using RPM packages can be found in the +:doc:`general`. + +These instructions have been tested with Fedora 23, and are also applicable for +RHEL 7.x and its derivatives, including CentOS 7.x and Scientific Linux 7.x. + +Build Requirements +------------------ + +To build packages for a Fedora Linux host, you will need the packages described +in the :doc:`general`. Specific packages (by package name) include: + +- rpm-build +- autoconf automake libtool +- systemd-units openssl openssl-devel +- python python-twisted-core python-zope-interface python-six +- desktop-file-utils +- groff graphviz +- procps-ng + +And (optionally): + +- libcap-ng libcap-ng-devel +- dpdk-devel + +Bootstraping +------------ + +Refer to :ref:`general-bootstrapping`. + +Configuring +----------- + +Refer to :ref:`general-configuring`. + +Building +-------- + +User Space RPMs +~~~~~~~~~~~~~~~ + +To build Open vSwitch user-space RPMs, execute the following from the directory +in which `./configure` was executed: + +:: + + $ make rpm-fedora + +This will create the RPMs `openvswitch`, `python-openvswitch`, +`openvswitch-test`, `openvswitch-devel`, `openvswitch-ovn-common`, +`openvswitch-ovn-central`, `openvswitch-ovn-host`, `openvswitch-ovn-vtep`, +`openvswitch-ovn-docker`, and `openvswitch-debuginfo`. + +To enable DPDK support in the openvswitch package, the ``--with dpdk`` option +can be added: + +:: + + $ make rpm-fedora RPMBUILD_OPT="--with dpdk" + +The above commands automatically run the Open vSwitch unit tests, which can +take several minutes. To reduce the build time by disabling the execution of +these tests, the ``--without check`` option can be added: + +:: + + $ make rpm-fedora RPMBUILD_OPT="--without check" + +Kernel OVS Tree Datapath RPM +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To build the Open vSwitch kernel module for the currently running kernel +version, run: + +:: + + $ make rpm-fedora-kmod + +To build the Open vSwitch kernel module for another kernel version, the desired +kernel version can be specified via the `kversion` macro. For example: + +:: + + $ make rpm-fedora-kmod \ + RPMBUILD_OPT='-D "kversion 4.3.4-300.fc23.x86_64"' + +Installing +---------- + +RPM packages can be installed by using the command ``rpm -i``. Package +installation requires superuser privileges. + +The `openvswitch-kmod` RPM should be installed first if the Linux OVS tree +datapath module is to be used. The `openvswitch-kmod` RPM should not be +installed if only the in-tree Linux datapath or user-space datapath is needed. +See the `FAQ document <FAQ.rst>`__ for more information about the various Open +vSwitch datapath options. + +In most cases only the `openvswitch` RPM will need to be installed. The +`python-openvswitch`, `openvswitch-test`, `openvswitch-devel`, and +`openvswitch-debuginfo` RPMs are optional unless required for a specific +purpose. + +The `openvswitch-ovn-*` packages are only needed when using OVN. + +Refer to the `RHEL README <rhel/README.RHEL.rst>`__ for additional usage and +configuration information. + +Reporting Bugs +-------------- + +Report problems to bugs@openvswitch.org. diff --git a/Documentation/intro/install/general.rst b/Documentation/intro/install/general.rst new file mode 100644 index 000000000..631dc9be5 --- /dev/null +++ b/Documentation/intro/install/general.rst @@ -0,0 +1,862 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + + Convention for heading levels in Open vSwitch documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + +========================================= +Open vSwitch on Linux, FreeBSD and NetBSD +========================================= + +This document describes how to build and install Open vSwitch on a generic +Linux, FreeBSD, or NetBSD host. For specifics around installation on a specific +platform, refer to one of the other installation guides listed in :doc:`index`. + +.. _general-build-reqs: + +Build Requirements +------------------ + +To compile the userspace programs in the Open vSwitch distribution, you will +need the following software: + +- GNU make + +- A C compiler, such as: + + - GCC 4.x. + + - Clang. Clang 3.4 and later provide useful static semantic analysis and + thread-safety checks. For Ubuntu, there are nightly built packages + available on clang's website. + + - MSVC 2013. Refer to :doc:`windows` for additional Windows build + instructions. + + While OVS may be compatible with other compilers, optimal support for atomic + operations may be missing, making OVS very slow (see ``lib/ovs-atomic.h``). + +- libssl, from OpenSSL, is optional but recommended if you plan to connect the + Open vSwitch to an OpenFlow controller. libssl is required to establish + confidentiality and authenticity in the connections from an Open vSwitch to + an OpenFlow controller. If libssl is installed, then Open vSwitch will + automatically build with support for it. + +- libcap-ng, written by Steve Grubb, is optional but recommended. It is + required to run OVS daemons as a non-root user with dropped root privileges. + If libcap-ng is installed, then Open vSwitch will automatically build with + support for it. + +- Python 2.7. You must also have the Python ``six`` library. + +On Linux, you may choose to compile the kernel module that comes with the Open +vSwitch distribution or to use the kernel module built into the Linux kernel +(version 3.3 or later). See the `FAQ <FAQ.rst>`__ question "What features +are not available in the Open vSwitch kernel datapath that ships as part of the +upstream Linux kernel?" for more information on this trade-off. You may also +use the userspace-only implementation, at some cost in features and +performance. Refer to :doc:`userspace` for details. + +To compile the kernel module on Linux, you must also install the +following: + +- A supported Linux kernel version. + + For optional support of ingress policing, you must enable kernel + configuration options ``NET_CLS_BASIC``, ``NET_SCH_INGRESS``, and + ``NET_ACT_POLICE``, either built-in or as modules. ``NET_CLS_POLICE`` is + obsolete and not needed.) + + On kernels before 3.11, the ``ip_gre`` module, for GRE tunnels over IP + (``NET_IPGRE``), must not be loaded or compiled in. + + To configure HTB or HFSC quality of service with Open vSwitch, you must + enable the respective configuration options. + + To use Open vSwitch support for TAP devices, you must enable ``CONFIG_TUN``. + +- To build a kernel module, you need the same version of GCC that was used to + build that kernel. + +- A kernel build directory corresponding to the Linux kernel image the module + is to run on. Under Debian and Ubuntu, for example, each linux-image package + containing a kernel binary has a corresponding linux-headers package with + the required build infrastructure. + +If you are working from a Git tree or snapshot (instead of from a distribution +tarball), or if you modify the Open vSwitch build system or the database +schema, you will also need the following software: + +- Autoconf version 2.63 or later. + +- Automake version 1.10 or later. + +- libtool version 2.4 or later. (Older versions might work too.) + +To run the unit tests, you also need: + +- Perl. Version 5.10.1 is known to work. Earlier versions should also + work. + +The datapath tests for userspace and Linux datapaths also rely upon: + +- pyftpdlib. Version 1.2.0 is known to work. Earlier versions should + also work. + +- GNU wget. Version 1.16 is known to work. Earlier versions should also + work. + +The ovs-vswitchd.conf.db(5) manpage will include an E-R diagram, in formats +other than plain text, only if you have the following: + +- dot from graphviz (http://www.graphviz.org/). + +- Perl. Version 5.10.1 is known to work. Earlier versions should also + work. + +If you are going to extensively modify Open vSwitch, consider installing the +following to obtain better warnings: + +- "sparse" version 0.4.4 or later + (http://www.kernel.org/pub/software/devel/sparse/dist/). + +- GNU make. + +- clang, version 3.4 or later + +- flake8, version 2.X, along with the hacking flake8 plugin (for Python code). + The automatic flake8 check that runs against Python code has some warnings + enabled that come from the "hacking" flake8 plugin. If it's not installed, + the warnings just won't occur until it's run on a system with "hacking" + installed. Note that there are problems with flake8 3.0 and the "hacking" + plugin. To ensure you get flake8 2.X, you can use:: + + $ pip install 'flake8<3.0' + +You may find the ovs-dev script found in ``utilities/ovs-dev.py`` useful. + +.. _general-install-reqs: + +Installation Requirements +------------------------- + +The machine you build Open vSwitch on may not be the one you run it on. To +simply install and run Open vSwitch you require the following software: + +- libc compatible with the libc used for build. + +- libssl compatible with the libssl used for build, if OpenSSL was used + for the build. + +- On Linux, the same kernel version configured as part of the build. + +- For optional support of ingress policing on Linux, the "tc" program + from iproute2 (part of all major distributions and available at + http://www.linux-foundation.org/en/Net:Iproute2). + +- Python 2.7. You must also have the Python six library. + +On Linux you should ensure that ``/dev/urandom`` exists. To support TAP +devices, you must also ensure that ``/dev/net/tun`` exists. + +.. _general-bootstrapping: + +Bootstrapping +------------- + +This step is not needed if you have downloaded a released tarball. If +you pulled the sources directly from an Open vSwitch Git tree or got a +Git tree snapshot, then run boot.sh in the top source directory to build +the "configure" script:: + + $ ./boot.sh + +.. _general-configuring: + +Configuring +----------- + +Configure the package by running the configure script. You can usually +invoke configure without any arguments. For example:: + + $ ./configure + +By default all files are installed under ``/usr/local``. Open vSwitch also +expects to find its database in ``/usr/local/etc/openvswitch`` by default. If +you want to install all files into, e.g., ``/usr`` and ``/var`` instead of +``/usr/local`` and ``/usr/local/var`` and expect to use ``/etc/openvswitch`` as +the default database directory, add options as shown here:: + + $ ./configure --prefix=/usr --localstatedir=/var --sysconfdir=/etc + +.. note:: + + Open vSwitch installed with packages like .rpm (e.g. via ``yum install`` or + ``rpm -ivh``) and .deb (e.g. via ``apt-get install`` or ``dpkg -i``) use the + above configure options. + +By default, static libraries are built and linked against. If you want to use +shared libraries instead:: + + $ ./configure --enable-shared + +To use a specific C compiler for compiling Open vSwitch user programs, also +specify it on the configure command line, like so:: + + $ ./configure CC=gcc-4.2 + +To use 'clang' compiler:: + + $ ./configure CC=clang + +To supply special flags to the C compiler, specify them as ``CFLAGS`` on the +configure command line. If you want the default CFLAGS, which include ``-g`` to +build debug symbols and ``-O2`` to enable optimizations, you must include them +yourself. For example, to build with the default CFLAGS plus ``-mssse3``, you +might run configure as follows:: + + $ ./configure CFLAGS="-g -O2 -mssse3" + +For efficient hash computation special flags can be passed to leverage built-in +intrinsics. For example on X86_64 with SSE4.2 instruction set support, CRC32 +intrinsics can be used by passing ``-msse4.2``:: + + $ ./configure CFLAGS="-g -O2 -msse4.2"` + +If you are on a different processor and don't know what flags to choose, it is +recommended to use ``-march=native`` settings:: + + $ ./configure CFLAGS="-g -O2 -march=native" + +With this, GCC will detect the processor and automatically set appropriate +flags for it. This should not be used if you are compiling OVS outside the +target machine. + +.. note:: + CFLAGS are not applied when building the Linux kernel module. Custom CFLAGS + for the kernel module are supplied using the ``EXTRA_CFLAGS`` variable when + running make. For example:: + + $ make EXTRA_CFLAGS="-Wno-error=date-time" + +To build the Linux kernel module, so that you can run the kernel-based switch, +pass the location of the kernel build directory on ``--with-linux``. For +example, to build for a running instance of Linux:: + + $ ./configure --with-linux=/lib/modules/$(uname -r)/build + +.. note:: + If ``--with-linux`` requests building for an unsupported version of Linux, + then ``configure`` will fail with an error message. Refer to the `FAQ + <FAQ.rst>`__ for advice in that case. + +If you wish to build the kernel module for an architecture other than the +architecture of the machine used for the build, you may specify the kernel +architecture string using the KARCH variable when invoking the configure +script. For example, to build for MIPS with Linux:: + + $ ./configure --with-linux=/path/to/linux KARCH=mips + +If you plan to do much Open vSwitch development, you might want to add +``--enable-Werror``, which adds the ``-Werror`` option to the compiler command +line, turning warnings into errors. That makes it impossible to miss warnings +generated by the build. For example:: + + $ ./configure --enable-Werror + +To build with gcov code coverage support, add ``--enable-coverage``:: + + $ ./configure --enable-coverage + +The configure script accepts a number of other options and honors additional +environment variables. For a full list, invoke configure with the ``--help`` +option:: + + $ ./configure --help + +You can also run configure from a separate build directory. This is helpful if +you want to build Open vSwitch in more than one way from a single source +directory, e.g. to try out both GCC and Clang builds, or to build kernel +modules for more than one Linux version. For example:: + + $ mkdir _gcc && (cd _gcc && ./configure CC=gcc) + $ mkdir _clang && (cd _clang && ./configure CC=clang) + +Under certains loads the ovsdb-server and other components perform better when +using the jemalloc memory allocator, instead of the glibc memory allocator. If +you wish to link with jemalloc add it to LIBS:: + + $ ./configure LIBS=-ljemalloc + +.. _general-building: + +Building +-------- + +1. Run GNU make in the build directory, e.g.:: + + $ make + + or if GNU make is installed as "gmake":: + + $ gmake + + If you used a separate build directory, run make or gmake from that + directory, e.g.:: + + $ make -C _gcc + $ make -C _clang + + For improved warnings if you installed ``sparse`` (see "Prerequisites"), add + ``C=1`` to the command line. + + .. note:: + Some versions of Clang and ccache are not completely compatible. If you + see unusual warnings when you use both together, consider disabling + ccache. + +2. Consider running the testsuite. Refer to **Testing** for instructions. + +3. Run ``make install`` to install the executables and manpages into the + running system, by default under ``/usr/local``:: + + $ make install + +5. If you built kernel modules, you may install them, e.g.:: + + $ make modules_install + + It is possible that you already had a Open vSwitch kernel module installed + on your machine that came from upstream Linux (in a different directory). To + make sure that you load the Open vSwitch kernel module you built from this + repository, you should create a ``depmod.d`` file that prefers your newly + installed kernel modules over the kernel modules from upstream Linux. The + following snippet of code achieves the same:: + + $ config_file="/etc/depmod.d/openvswitch.conf" + $ for module in datapath/linux/*.ko; do + modname="$(basename ${module})" + echo "override ${modname%.ko} * extra" >> "$config_file" + echo "override ${modname%.ko} * weak-updates" >> "$config_file" + done + $ depmod -a + + Finally, load the kernel modules that you need. e.g.:: + + $ /sbin/modprobe openvswitch + + To verify that the modules have been loaded, run ``/sbin/lsmod`` and check + that openvswitch is listed:: + + $ /sbin/lsmod | grep openvswitch + + .. note:: + If the ``modprobe`` operation fails, look at the last few kernel log + messages (e.g. with ``dmesg | tail``). Generally, issues like this occur + when Open vSwitch is built for a kernel different from the one into which + you are trying to load it. Run ``modinfo`` on ``openvswitch.ko`` and on a + module built for the running kernel, e.g.:: + + $ /sbin/modinfo openvswitch.ko + $ /sbin/modinfo /lib/modules/$(uname -r)/kernel/net/bridge/bridge.ko + + Compare the "vermagic" lines output by the two commands. If they differ, + then Open vSwitch was built for the wrong kernel. + + If you decide to report a bug or ask a question related to module loading, + include the output from the ``dmesg`` and ``modinfo`` commands mentioned + above. + +.. _general-starting: + +Starting +-------- + +Before starting ovs-vswitchd itself, you need to start its configuration +database, ovsdb-server. Each machine on which Open vSwitch is installed should +run its own copy of ovsdb-server. Before ovsdb-server itself can be started, +configure a database that it can use:: + + $ mkdir -p /usr/local/etc/openvswitch + $ ovsdb-tool create /usr/local/etc/openvswitch/conf.db \ + vswitchd/vswitch.ovsschema + +Configure ovsdb-server to use database created above, to listen on a Unix +domain socket, to connect to any managers specified in the database itself, and +to use the SSL configuration in the database:: + + $ mkdir -p /usr/local/var/run/openvswitch + $ ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock \ + --remote=db:Open_vSwitch,Open_vSwitch,manager_options \ + --private-key=db:Open_vSwitch,SSL,private_key \ + --certificate=db:Open_vSwitch,SSL,certificate \ + --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert \ + --pidfile --detach --log-file + +.. note:: + If you built Open vSwitch without SSL support, then omit ``--private-key``, + ``--certificate``, and ``--bootstrap-ca-cert``.) + +Initialize the database using ovs-vsctl. This is only necessary the first time +after you create the database with ovsdb-tool, though running it at any time is +harmless:: + + $ ovs-vsctl --no-wait init + +Start the main Open vSwitch daemon, telling it to connect to the same Unix +domain socket:: + + $ ovs-vswitchd --pidfile --detach --log-file + +Validating +---------- + +At this point you can use ovs-vsctl to set up bridges and other Open vSwitch +features. For example, to create a bridge named ``br0`` and add ports ``eth0`` +and ``vif1.0`` to it:: + + $ ovs-vsctl add-br br0 + $ ovs-vsctl add-port br0 eth0 + $ ovs-vsctl add-port br0 vif1.0 + +Refer to ovs-vsctl(8) for more details. + +Upgrading +--------- + +When you upgrade Open vSwitch from one version to another you should also +upgrade the database schema: + +1. Stop the Open vSwitch daemons, e.g.:: + + $ kill `cd /usr/local/var/run/openvswitch && cat ovsdb-server.pid ovs-vswitchd.pid` + +2. Install the new Open vSwitch release by using the same configure options as + was used for installing the previous version. If you do not use the same + configure options, you can end up with two different versions of Open + vSwitch executables installed in different locations. + +3. Upgrade the database, in one of the following two ways: + + - If there is no important data in your database, then you may delete the + database file and recreate it with ovsdb-tool, following the instructions + under "Building and Installing Open vSwitch for Linux, FreeBSD or NetBSD". + + - If you want to preserve the contents of your database, back it up first, + then use ``ovsdb-tool convert`` to upgrade it, e.g.:: + + $ ovsdb-tool convert /usr/local/etc/openvswitch/conf.db \ + vswitchd/vswitch.ovsschema + +4. Start the Open vSwitch daemons as described under **Starting** above. + +Hot Upgrading +------------- + +Upgrading Open vSwitch from one version to the next version with minimum +disruption of traffic going through the system that is using that Open vSwitch +needs some considerations: + +1. If the upgrade only involves upgrading the userspace utilities and daemons + of Open vSwitch, make sure that the new userspace version is compatible with + the previously loaded kernel module. + +2. An upgrade of userspace daemons means that they have to be restarted. + Restarting the daemons means that the OpenFlow flows in the ovs-vswitchd + daemon will be lost. One way to restore the flows is to let the controller + re-populate it. Another way is to save the previous flows using a utility + like ovs-ofctl and then re-add them after the restart. Restoring the old + flows is accurate only if the new Open vSwitch interfaces retain the old + 'ofport' values. + +3. When the new userspace daemons get restarted, they automatically flush the + old flows setup in the kernel. This can be expensive if there are hundreds + of new flows that are entering the kernel but userspace daemons are busy + setting up new userspace flows from either the controller or an utility like + ovs-ofctl. Open vSwitch database provides an option to solve this problem + through the ``other_config:flow-restore-wait`` column of the + ``Open_vSwitch`` table. Refer to the ovs-vswitchd.conf.db(5) manpage for + details. + +4. If the upgrade also involves upgrading the kernel module, the old kernel + module needs to be unloaded and the new kernel module should be loaded. This + means that the kernel network devices belonging to Open vSwitch is recreated + and the kernel flows are lost. The downtime of the traffic can be reduced if + the userspace daemons are restarted immediately and the userspace flows are + restored as soon as possible. + +The ovs-ctl utility's ``restart`` function only restarts the userspace daemons, +makes sure that the 'ofport' values remain consistent across restarts, restores +userspace flows using the ovs-ofctl utility and also uses the +``other_config:flow-restore-wait`` column to keep the traffic downtime to the +minimum. The ovs-ctl utility's ``force-reload-kmod`` function does all of the +above, but also replaces the old kernel module with the new one. Open vSwitch +startup scripts for Debian, XenServer and RHEL use ovs-ctl's functions and it +is recommended that these functions be used for other software platforms too. + +.. _general-testing: + +Testing +------- + +This section describe Open vSwitch's built-in support for various test +suites. You must bootstrap, configure and build Open vSwitch (steps are +in "Building and Installing Open vSwitch for Linux, FreeBSD or NetBSD" +above) before you run the tests described here. You do not need to +install Open vSwitch or to build or load the kernel module to run these +test suites. You do not need supervisor privilege to run these test +suites. + +Unit Tests +~~~~~~~~~~ + +Open vSwitch includes a suite of self-tests. Before you submit patches +upstream, we advise that you run the tests and ensure that they pass. If you +add new features to Open vSwitch, then adding tests for those features will +ensure your features don't break as developers modify other areas of Open +vSwitch. + +To run all the unit tests in Open vSwitch, one at a time, run:: + + $ make check + +This takes under 5 minutes on a modern desktop system. + +To run all the unit tests in Open vSwitch in parallel, run:: + + $ make check TESTSUITEFLAGS=-j8 + +You can run up to eight threads. This takes under a minute on a modern 4-core +desktop system. + +To see a list of all the available tests, run: + + $ make check TESTSUITEFLAGS=--list + +To run only a subset of tests, e.g. test 123 and tests 477 through 484, run:: + + $ make check TESTSUITEFLAGS='123 477-484' + +Tests do not have inter-dependencies, so you may run any subset. + +To run tests matching a keyword, e.g. ``ovsdb``, run:: + + $ make check TESTSUITEFLAGS='-k ovsdb' + +To see a complete list of test options, run:: + + $ make check TESTSUITEFLAGS=--help + +The results of a testing run are reported in ``tests/testsuite.log``. Report +report test failures as bugs and include the ``testsuite.log`` in your report. + +.. note:: + Sometimes a few tests may fail on some runs but not others. This is usually a + bug in the testsuite, not a bug in Open vSwitch itself. If you find that a + test fails intermittently, please report it, since the developers may not + have noticed. You can make the testsuite automatically rerun tests that fail, + by adding ``RECHECK=yes`` to the ``make`` command line, e.g.:: + + $ make check TESTSUITEFLAGS=-j8 RECHECK=yes + +Coverage +++++++++ + +If the build was configured with ``--enable-coverage`` and the ``lcov`` utility +is installed, you can run the testsuite and generate a code coverage report by +using the ``check-lcoc`` target:: + + $ make check-lcov + +All the same options are avaiable via TESTSUITEFLAGS. For example:: + + $ make check-lcov TESTSUITEFLAGS=-j8 -k ovn + +Valgrind +++++++++ + +If you have ``valgrind`` installed, you can run the testsuite under +valgrind by using the ``check-valgrind`` target:: + + $ make check-valgrind + +When you do this, the "valgrind" results for test ``<N>`` are reported in files +named ``tests/testsuite.dir/<N>/valgrind.*``. + +All the same options are available via TESTSUITEFLAGS. + +.. hint:: + You may find that the valgrind results are easier to interpret if you put + ``-q`` in ``~/.valgrindrc``, since that reduces the amount of output. + +.. _general-oftest: + +OFTest +~~~~~~ + +OFTest is an OpenFlow protocol testing suite. Open vSwitch includes a Makefile +target to run OFTest with Open vSwitch in "dummy mode". In this mode of +testing, no packets travel across physical or virtual networks. Instead, Unix +domain sockets stand in as simulated networks. This simulation is imperfect, +but it is much easier to set up, does not require extra physical or virtual +hardware, and does not require supervisor privileges. + +To run OFTest with Open vSwitch, first read and follow the instructions under +**Testing** above. Second, obtain a copy of OFTest and install its +prerequisites. You need a copy of OFTest that includes commit 406614846c5 (make +ovs-dummy platform work again). This commit was merged into the OFTest +repository on Feb 1, 2013, so any copy of OFTest more recent than that should +work. Testing OVS in dummy mode does not require root privilege, so you may +ignore that requirement. + +Optionally, add the top-level OFTest directory (containing the ``oft`` program) +to your ``$PATH``. This slightly simplifies running OFTest later. + +To run OFTest in dummy mode, run the following command from your Open vSwitch +build directory:: + + $ make check-oftest OFT=<oft-binary> + +where ``<oft-binary>`` is the absolute path to the ``oft`` program in OFTest. +If you added "oft" to your $PATH, you may omit the OFT variable +assignment + +By default, ``check-oftest`` passes ``oft`` just enough options to enable dummy +mode. You can use ``OFTFLAGS`` to pass additional options. For example, to run +just the ``basic.Echo`` test instead of all tests (the default) and enable +verbose logging, run:: + + $ make check-oftest OFT=<oft-binary> OFTFLAGS='--verbose -T basic.Echo' + +If you use OFTest that does not include commit 4d1f3eb2c792 (oft: change +default port to 6653), merged into the OFTest repository in October 2013, then +you need to add an option to use the IETF-assigned controller port:: + + $ make check-oftest OFT=<oft-binary> OFTFLAGS='--port=6653' + +Interpret OFTest results cautiously. Open vSwitch can fail a given test in +OFTest for many reasons, including bugs in Open vSwitch, bugs in OFTest, bugs +in the "dummy mode" integration, and differing interpretations of the OpenFlow +standard and other standards. + +.. note:: + Open vSwitch has not been validated against OFTest. Report test failures that + you believe to represent bugs in Open vSwitch. Include the precise versions + of Open vSwitch and OFTest in your bug report, plus any other information + needed to reproduce the problem. + +Ryu +~~~ + +Ryu is an OpenFlow controller written in Python that includes an extensive +OpenFlow testsuite. Open vSwitch includes a Makefile target to run Ryu in +"dummy mode". See **OFTest** above for an explanation of dummy mode. + +To run Ryu tests with Open vSwitch, first read and follow the instructions +under **Testing** above. Second, obtain a copy of Ryu, install its +prerequisites, and build it. You do not need to install Ryu (some of the tests +do not get installed, so it does not help). + +To run Ryu tests, run the following command from your Open vSwitch build +directory:: + + $ make check-ryu RYUDIR=<ryu-source-dir>`` + +where ``<ryu-source-dir>`` is the absolute path to the root of the Ryu source +distribution. The default ``<ryu-source-dir>`` is ``$srcdir/../ryu`` +where ``$srcdir`` is your Open vSwitch source directory. If this is correct, +omit ``RYUDIR`` + +.. note:: + Open vSwitch has not been validated against Ryu. Report test failures that + you believe to represent bugs in Open vSwitch. Include the precise versions + of Open vSwitch and Ryu in your bug report, plus any other information + needed to reproduce the problem. + +Datapath testing +~~~~~~~~~~~~~~~~ + +Open vSwitch includes a suite of tests specifically for datapath functionality, +which can be run against the userspace or kernel datapaths. If you are +developing datapath features, it is recommended that you use these tests and +build upon them to verify your implementation. + +The datapath tests make some assumptions about the environment. They must be +run under root privileges on a Linux system with support for network +namespaces. For ease of use, the OVS source tree includes a vagrant box to +invoke these tests. Running the tests inside Vagrant provides kernel isolation, +protecting your development host from kernel panics or configuration conflicts +in the testsuite. If you wish to run the tests without using the vagrant box, +there are further instructions below. + +Vagrant ++++++++ + +.. important:: + + Requires Vagrant (version 1.7.0 or later) and a compatible hypervisor + +.. note:: + You must **Bootstrap** and **Configure** the sources before you run the steps + described here. + +A Vagrantfile is provided allowing to compile and provision the source tree as +found locally in a virtual machine using the following command:: + + $ vagrant up + +This will bring up a Fedora 23 VM by default. If you wish to use a different +box or a vagrant backend not supported by the default box, the ``Vagrantfile`` +can be modified to use a different box as base. + +The VM can be reprovisioned at any time:: + + $ vagrant provision + +OVS out-of-tree compilation environment can be set up with:: + + $ ./boot.sh + $ vagrant provision --provision-with configure_ovs,build_ovs + +This will set up an out-of-tree build environment inside the VM in +``/root/build``. The source code can be found in ``/vagrant``. + +To recompile and reinstall OVS in the VM using RPM:: + + $ ./boot.sh + $ vagrant provision --provision-with configure_ovs,install_rpm + +Two provisioners are included to run system tests with the OVS kernel module or +with a userspace datapath. This tests are different from the self-tests +mentioned above. To run them:: + + $ ./boot.sh + $ vagrant provision --provision-with \ + configure_ovs,test_ovs_kmod,test_ovs_system_userspace + +The results of the testsuite reside in the VM root user's home directory:: + + $ vagrant ssh + $ sudo -s + $ cd /root/build + $ ls tests/system* + +Native +++++++ + +The datapath testsuite as invoked by Vagrant above may also be run manually on +a Linux system with root privileges. These tests may take several minutes to +complete, and cannot be run in parallel. + +Userspace datapath +''''''''''''''''''' + +To invoke the datapath testsuite with the userspace datapath, run:: + + $ make check-system-userspace + +The results of the testsuite are in ``tests/system-userspace-traffic.dir``. + +Kernel datapath +''''''''''''''' + +Make targets are also provided for testing the Linux kernel module. Note that +these tests operate by inserting modules into the running Linux kernel, so if +the tests are able to trigger a bug in the OVS kernel module or in the upstream +kernel then the kernel may panic. + +To run the testsuite against the kernel module which is currently installed on +your system, run:: + + $ make check-kernel + +To install the kernel module from the current build directory and run the +testsuite against that kernel module:: + + $ make check-kmod + +The results of the testsuite are in ``tests/system-kmod-traffic.dir``. + +Continuous Integration with Travis-CI +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A .travis.yml file is provided to automatically build Open vSwitch with various +build configurations and run the testsuite using travis-ci. Builds will be +performed with gcc, sparse and clang with the -Werror compiler flag included, +therefore the build will fail if a new warning has been introduced. + +The CI build is triggered via git push (regardless of the specific branch) or +pull request against any Open vSwitch GitHub repository that is linked to +travis-ci. + +Instructions to setup travis-ci for your GitHub repository: + +1. Go to http://travis-ci.org/ and sign in using your GitHub ID. +2. Go to the "Repositories" tab and enable the ovs repository. You may disable + builds for pushes or pull requests. +3. In order to avoid forks sending build failures to the upstream mailing list, + the notification email recipient is encrypted. If you want to receive email + notification for build failures, replace the the encrypted string: + + 1. Install the travis-ci CLI (Requires ruby >=2.0): gem install travis + 2. In your Open vSwitch repository: travis encrypt mylist@mydomain.org + 3. Add/replace the notifications section in .travis.yml and fill in the + secure string as returned by travis encrypt:: + + notifications: + email: + recipients: + - secure: "....." + + .. note:: + You may remove/omit the notifications section to fall back to default + notification behaviour which is to send an email directly to the author and + committer of the failing commit. Note that the email is only sent if the + author/committer have commit rights for the particular GitHub repository. + +4. Pushing a commit to the repository which breaks the build or the + testsuite will now trigger a email sent to mylist@mydomain.org + +Static Code Analysis +~~~~~~~~~~~~~~~~~~~~ + +Static Analysis is a method of debugging Software by examining code rather than +actually executing it. This can be done through 'scan-build' commandline +utility which internally uses clang (or) gcc to compile the code and also +invokes a static analyzer to do the code analysis. At the end of the build, the +reports are aggregated in to a common folder and can later be analyzed using +'scan-view'. + +Open vSwitch includes a Makefile target to trigger static code analysis:: + + $ ./boot.sh + $ ./configure CC=clang # clang + # or + $ ./configure CC=gcc CFLAGS="-std=gnu99" # gcc + $ make clang-analyze + +You should invoke scan-view to view analysis results. The last line of output +from ``clang-analyze`` will list the command (containing results directory) +that you should invoke to view the results on a browser. + +Reporting Bugs +-------------- + +Report problems to bugs@openvswitch.org. diff --git a/Documentation/intro/install/index.rst b/Documentation/intro/install/index.rst index 35dc67cf5..2366388a2 100644 --- a/Documentation/intro/install/index.rst +++ b/Documentation/intro/install/index.rst @@ -30,5 +30,38 @@ Installing Open vSwitch A collection of guides detailing how to install Open vSwitch in a variety of different environments and using different configurations. +Installation from Source +------------------------ + +.. TODO(stephenfin): The DPDK-ADVANCED doc is mostly usage material. The + install related instructions should be moved to the main doc, while the + rest should be moved to howto and topic docs + +.. TODO(stephenfin): Based on the title alone, the NetBSD doc should probably + be merged into the general install doc + +.. toctree:: + :maxdepth: 2 + + general + netbsd + windows + xenserver + userspace + dpdk + dpdk-advanced + bash-completion + +Installation from Packages +-------------------------- + +Open vSwitch is packaged on a variety of distributions. The tooling required to +build these packages is included in the Open vSwitch tree. The instructions are +provided below. + .. toctree:: :maxdepth: 2 + + debian + fedora + rhel diff --git a/Documentation/intro/install/netbsd.rst b/Documentation/intro/install/netbsd.rst new file mode 100644 index 000000000..2b52eaf08 --- /dev/null +++ b/Documentation/intro/install/netbsd.rst @@ -0,0 +1,61 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + + Convention for heading levels in Open vSwitch documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + +====================== +Open vSwitch on NetBSD +====================== + +On NetBSD, you might want to install requirements from pkgsrc. In that case, +you need at least the following packages. + +- automake +- libtool-base +- gmake +- python27 +- py27-six +- py27-xml +- pkg_alternatives + +Some components have additional requirements. Refer to :doc:`general` for more +information. + +Assuming you are running NetBSD/amd64 6.1.2, you can download and install +pre-built binary packages as the following:: + + $ PKG_PATH=http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/6.1.2/All/ + $ export PKG_PATH + $ pkg_add automake libtool-base gmake python27 py27-six py27-xml \ + pkg_alternatives + +.. note:: + You might get some warnings about minor version mismatch. These can be safely + ignored. + +NetBSD's ``/usr/bin/make`` is not GNU make. GNU make is installed as +``/usr/pkg/bin/gmake`` by the above mentioned ``gmake`` package. + +As all executables installed with pkgsrc are placed in ``/usr/pkg/bin/`` +directory, it might be a good idea to add it to your PATH. + +Open vSwitch on NetBSD is currently "userspace switch" implementation in the +sense described in :doc:`userspace` and the `porting guide`. diff --git a/Documentation/intro/install/rhel.rst b/Documentation/intro/install/rhel.rst new file mode 100644 index 000000000..498fd8f25 --- /dev/null +++ b/Documentation/intro/install/rhel.rst @@ -0,0 +1,216 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + + Convention for heading levels in Open vSwitch documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + +======================================== +RHEL 5.6, 6.x Packaging for Open vSwitch +======================================== + +This document describes how to build and install Open vSwitch on a Red Hat +Enterprise Linux (RHEL) host. If you want to install Open vSwitch on a generic +Linux host, refer to :doc:`general` instead. + +We have tested these instructions with RHEL 5.6 and RHEL 6.0. + +For RHEL 7.x (or derivatives, such as CentOS 7.x), you should follow the +instructions in the :doc:`fedora`. The Fedora spec files are used for RHEL +7.x. + +.. _rhel-prerequisites: + +Prerequisites +------------- + +You may build from an Open vSwitch distribution tarball or from an Open vSwitch +Git tree. + +The default RPM build directory, ``_topdir``, has five directories in the +top-level. + +BUILD/ + where the software is unpacked and built +RPMS/ + where the newly created binary package files are written +SOURCES/ + contains the original sources, patches, and icon files +SPECS/ + contains the spec files for each package to be built +SRPMS/ + where the newly created source package files are written + +Before you begin, note the RPM sources directory on your version of RHEL. The +command ``rpmbuild --showrc`` will show the configuration for each of those +directories. Alternatively, the command ``rpm --eval '%{_topdir}'`` shows the +current configuration for the top level directory and the command ``rpm --eval +'%{_sourcedir}'`` does the same for the sources directory. On RHEL 5, the +default RPM ``_topdir`` is ``/usr/src/redhat`` and the default RPM sources +directory is ``/usr/src/redhat/SOURCES``. On RHEL 6, the default ``_topdir`` is +``$HOME/rpmbuild`` and the default RPM sources directory is +``$HOME/rpmbuild/SOURCES``. + +Build Requirements +------------------ + +To compile the RPMs, you will need to install the packages described in the +:doc:`general` along with some additional packages. These can be installed with +the below command:: + + $ yum install gcc make python-devel openssl-devel kernel-devel graphviz \ + kernel-debug-devel autoconf automake rpm-build redhat-rpm-config \ + libtool + +.. _rhel-bootstrapping: + +Bootstrapping and Configuring +----------------------------- + +If you are building from a distribution tarball, skip to :ref:`rhel-building`. +If not, you must be building from an Open vSwitch Git tree. Determine what +version of Autoconf is installed (e.g. run ``autoconf --version``). If it is +not at least version 2.63, then you must upgrade or use another machine to +build the packages. + +Assuming all requirements have been met, build the tarball by running:: + + $ ./boot.sh + $ ./configure + $ make dist + +You must run this on a machine that has the tools listed in +:ref:`general-build-reqs` as prerequisites for building from a Git tree. +Afterward, proceed with the rest of the instructions using the distribution +tarball. + +Now you have a distribution tarball, named something like +``openvswitch-x.y.z.tar.gz``. Copy this file into the RPM sources directory, +e.g.:: + + $ cp openvswitch-x.y.z.tar.gz $HOME/rpmbuild/SOURCES + +Broken ``build`` symlink +~~~~~~~~~~~~~~~~~~~~~~~~ + +Some versions of the RHEL 6 kernel-devel package contain a broken ``build`` +symlink. If you are using such a version, you must fix the problem before +continuing. + +To find out whether you are affected, run:: + + $ cd /lib/modules/<version> + $ ls -l build/ + +where ``<version>`` is the version number of the RHEL 6 kernel. + +.. note:: + The trailing slash in the final command is important. Be sure to include + it. + +If the ``ls`` command produces a directory listing, your kernel-devel package +is OK. If it produces a ``No such file or directory`` error, your kernel-devel +package is buggy. + +If your kernel-devel package is buggy, then you can fix it with:: + + $ cd /lib/modules/<version> + $ rm build + $ ln -s /usr/src/kernels/<target> build + +where ``<target>`` is the name of an existing directory under +``/usr/src/kernels``, whose name should be similar to ``<version>`` but may +contain some extra parts. Once you have done this, verify the fix with the +same procedure you used above to check for the problem. + +.. _rhel-building: + +Building +-------- + +You should have a distribution tarball named something like +openvswitch-x.y.z.tar.gz. Copy this file into the RPM sources directory:: + + $ cp openvswitch-x.y.z.tar.gz $HOME/rpmbuild/SOURCES + +Make another copy of the distribution tarball in a temporary directory. Then +unpack the tarball and ``cd`` into its root:: + + $ tar xzf openvswitch-x.y.z.tar.gz + $ cd openvswitch-x.y.z + +Userspace +~~~~~~~~~ + +To build Open vSwitch userspace, run:: + + $ rpmbuild -bb rhel/openvswitch.spec + +This produces two RPMs: "openvswitch" and "openvswitch-debuginfo". + +The above command automatically runs the Open vSwitch unit tests. To disable +the unit tests, run:: + + $ rpmbuild -bb --without check rhel/openvswitch.spec + +.. note:: + If the build fails with ``configure: error: source dir + /lib/modules/2.6.32-279.el6.x86_64/build doesn't exist`` or similar, then + the kernel-devel package is missing or buggy. + +Kernel Module +~~~~~~~~~~~~~ + +On RHEL 6, to build the Open vSwitch kernel module, copy +rhel/openvswitch-kmod.files into the RPM sources directory and run:: + + $ rpmbuild -bb rhel/openvswitch-kmod-rhel6.spec + +You might have to specify a kernel version and/or variants, e.g.: + + $ rpmbuild -bb \ + -D "kversion 2.6.32-131.6.1.el6.x86_64" \ + -D "kflavors default debug kdump" \ + rhel/openvswitch-kmod-rhel6.spec + +This produces an "kmod-openvswitch" RPM for each kernel variant, in this +example: "kmod-openvswitch", "kmod-openvswitch-debug", and +"kmod-openvswitch-kdump". + +.. _rhel-script-integrations: + +Red Hat Network Scripts Integration +----------------------------------- + +A RHEL host has default firewall rules that prevent any Open vSwitch tunnel +traffic from passing through. If a user configures Open vSwitch tunnels like +Geneve, GRE, VXLAN, LISP etc., they will either have to manually add iptables +firewall rules to allow the tunnel traffic or add it through a startup script +Refer to the "enable-protocol" command in the ovs-ctl(8) manpage for more +information. + +In addition, simple integration with Red Hat network scripts has been +implemented. Refer to `README.RHEL.rst <rhel/README.RHEL.rst>`__ in the source +tree or /usr/share/doc/openvswitch/README.RHEL.rst in the installed openvswitch +package for details. + +Reporting Bugs +-------------- + +Report problems to bugs@openvswitch.org. diff --git a/Documentation/intro/install/userspace.rst b/Documentation/intro/install/userspace.rst new file mode 100644 index 000000000..036852759 --- /dev/null +++ b/Documentation/intro/install/userspace.rst @@ -0,0 +1,106 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + + Convention for heading levels in Open vSwitch documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + +=================================== +Open vSwitch without Kernel Support +=================================== + +Open vSwitch can operate, at a cost in performance, entirely in userspace, +without assistance from a kernel module. This file explains how to install +Open vSwitch in such a mode. + +This version of Open vSwitch should be built manually with ``configure`` and +``make``. Debian packaging for Open vSwitch is also included, but it has not +been recently tested, and so Debian packages are not a recommended way to use +this version of Open vSwitch. + +.. warning:: + The userspace-only mode of Open vSwitch is considered experimental. It has + not been thoroughly tested. + +Building and Installing +----------------------- + +The requirements and procedure for building, installing, and configuring Open +vSwitch are the same as those given in :doc:`general`. You may omit +configuring, building, and installing the kernel module, and the related +requirements. + +On Linux, the userspace switch additionally requires the kernel TUN/TAP driver +to be available, either built into the kernel or loaded as a module. If you +are not sure, check for a directory named ``/sys/class/misc/tun``. If it does +not exist, then attempt to load the module with ``modprobe tun``. + +The tun device must also exist as ``/dev/net/tun``. If it does not exist, then +create ``/dev/net`` (if necessary) with ``mkdir /dev/net``, then create +``/dev/net/tun`` with ``mknod /dev/net/tun c 10 200``. + +On FreeBSD and NetBSD, the userspace switch additionally requires the kernel +tap(4) driver to be available, either built into the kernel or loaded as a +module. + +Using the Userspace Datapath with ovs-vswitchd +---------------------------------------------- + +To use ovs-vswitchd in userspace mode, create a bridge with +``datapath_type=netdev`` in the configuration database. For example:: + + $ ovs-vsctl add-br br0 + $ ovs-vsctl set bridge br0 datapath_type=netdev + $ ovs-vsctl add-port br0 eth0 + $ ovs-vsctl add-port br0 eth1 + $ ovs-vsctl add-port br0 eth2 + +ovs-vswitchd will create a TAP device as the bridge's local interface, named +the same as the bridge, as well as for each configured internal interface. + +Currently, on FreeBSD, the functionality required for in-band control support +is not implemented. To avoid related errors, you can disable the in-band +support with the following command:: + + $ ovs-vsctl set bridge br0 other_config:disable-in-band=true + +Firewall Rules +-------------- + +On Linux, when a physical interface is in use by the userspace datapath, +packets received on the interface still also pass into the kernel TCP/IP stack. +This can cause surprising and incorrect behavior. You can use "iptables" to +avoid this behavior, by using it to drop received packets. For example, to +drop packets received on eth0:: + + $ iptables -A INPUT -i eth0 -j DROP + $ iptables -A FORWARD -i eth0 -j DROP + +Other Settings +-------------- + +On NetBSD, depending on your network topology and applications, the following +configuration might help. See sysctl(7).:: + + $ sysctl -w net.inet.ip.checkinterface=1 + +Reporting Bugs +-------------- + +Report problems to bugs@openvswitch.org. diff --git a/Documentation/intro/install/windows.rst b/Documentation/intro/install/windows.rst new file mode 100644 index 000000000..eb55720ed --- /dev/null +++ b/Documentation/intro/install/windows.rst @@ -0,0 +1,620 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + + Convention for heading levels in Open vSwitch documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + +======================= +Open vSwitch on Windows +======================= + +.. _windows-build-reqs: + +Build Requirements +------------------ + +Open vSwitch on Linux uses autoconf and automake for generating Makefiles. It +will be useful to maintain the same build system while compiling on Windows +too. One approach is to compile Open vSwitch in a MinGW environment that +contains autoconf and automake utilities and then use Visual C++ as a compiler +and linker. + +The following explains the steps in some detail. + +- Mingw + + Install Mingw on a Windows machine by following the instructions on + `mingw.org <http://www.mingw.org/wiki/Getting_Started>`__. + + This should install mingw at ``C:\Mingw`` and msys at ``C:\Mingw\msys``. Add + ``C:\MinGW\bin`` and ``C:\Mingw\msys\1.0\bin`` to PATH environment variable + of Windows. + + You can either use the MinGW installer or the command line utility + ``mingw-get`` to install both the base packages and additional packages like + automake and autoconf(version 2.68). + + Also make sure that ``/mingw`` mount point exists. If its not, please + add/create the following entry in ``/etc/fstab``:: + + 'C:/MinGW /mingw'. + +- Python + + Install the latest Python 2.x from python.org and verify that its path is + part of Windows' PATH environment variable. + +- Visual Studio + + You will need at least Visual Studio 2013 (update 4) to compile userspace + binaries. In addition to that, if you want to compile the kernel module you + will also need to install Windows Driver Kit (WDK) 8.1 Update. + + It is important to get the Visual Studio related environment variables and to + have the $PATH inside the bash to point to the proper compiler and linker. + One easy way to achieve this for VS2013 is to get into the "VS2013 x86 Native + Tools Command Prompt" (in a default installation of Visual Studio 2013 this + can be found under the following location: ``C:\Program Files (x86)\Microsoft + Visual Studio 12.0\Common7\Tools\Shortcuts``) and through it enter into the + bash shell available from msys by typing ``bash --login``. + + There is support for generating 64 bit binaries too. To compile under x64, + open the "VS2013 x64 Native Tools Command Prompt" (if your current running OS + is 64 bit) or "VS2013 x64 Cross Tools Command Prompt" (if your current + running OS is not 64 bit) instead of opening its x86 variant. This will + point the compiler and the linker to their 64 bit equivalent. + + If after the above step, a ``which link`` inside MSYS's bash says, + ``/bin/link.exe``, rename ``/bin/link.exe`` to something else so that the + Visual studio's linker is used. You should also see a 'which sort' report + ``/bin/sort.exe``. + +- pthreads-win32 + + For pthread support, install the library, dll and includes of pthreads-win32 + project from `sourceware + <ftp://sourceware.org/pub/pthreads-win32/prebuilt-dll-2-9-1-release>`__ to a + directory (e.g.: ``C:/pthread``). You should add the pthread-win32's dll path + (e.g.: ``C:\pthread\dll\x86``) to the Windows' PATH environment variable. + +- OpenSSQL + + To get SSL support for Open vSwitch on Windows, you will need to install + `OpenSSL for Windows <http://www.openssl.org/related/binaries.html>`__ + + Note down the directory where OpenSSL is installed (e.g.: + ``C:/OpenSSL-Win32``) for later use. + +Install Requirements +-------------------- + +* Share network adaptors + + We require that you don't disable the "Allow management operating system to + share this network adapter" under 'Virtual Switch Properties' > 'Connection + type: External network', in the HyperV virtual network switch configuration. + +* Checksum Offloads + + While there is some support for checksum/segmentation offloads in software, + this is still a work in progress. Till the support is complete we recommend + disabling TX/RX offloads for both the VM's as well as the HyperV. + +Bootstrapping +------------- + +This step is not needed if you have downloaded a released tarball. If +you pulled the sources directly from an Open vSwitch Git tree or got a +Git tree snapshot, then run boot.sh in the top source directory to build +the "configure" script:: + + > ./boot.sh + +.. _windows-configuring: + +Configuring +----------- + +Configure the package by running the configure script. You should provide some +configure options to choose the right compiler, linker, libraries, Open vSwitch +component installation directories, etc. For example:: + + > ./configure CC=./build-aux/cccl LD="$(which link)" \ + LIBS="-lws2_32 -liphlpapi" --prefix="C:/openvswitch/usr" \ + --localstatedir="C:/openvswitch/var" \ + --sysconfdir="C:/openvswitch/etc" \ + --with-pthread="C:/pthread" + +.. note:: + By default, the above enables compiler optimization for fast code. For + default compiler optimization, pass the ``--with-debug`` configure option. + +To configure with SSL support, add the requisite additional options:: + + > ./configure CC=./build-aux/cccl LD="`which link`" \ + LIBS="-lws2_32 -liphlpapi" --prefix="C:/openvswitch/usr" \ + --localstatedir="C:/openvswitch/var" + --sysconfdir="C:/openvswitch/etc" \ + --with-pthread="C:/pthread" \ + --enable-ssl --with-openssl="C:/OpenSSL-Win32" + +Finally, to the kernel module also:: + + > ./configure CC=./build-aux/cccl LD="`which link`" \ + LIBS="-lws2_32 -liphlpapi" --prefix="C:/openvswitch/usr" \ + --localstatedir="C:/openvswitch/var" \ + --sysconfdir="C:/openvswitch/etc" \ + --with-pthread="C:/pthread" \ + --enable-ssl --with-openssl="C:/OpenSSL-Win32" \ + --with-vstudiotarget="<target type>" + +Possible values for ``<target type>`` are: ``Debug`` and ``Release`` + +.. note:: + You can directly use the Visual Studio 2013 IDE to compile the kernel + datapath. Open the ovsext.sln file in the IDE and build the solution. + +Refer to :doc:`general` for information on additional configuration options. + +.. _windows-building: + +Building +-------- + +Once correctly configured, building Open vSwitch on Windows is similar to +building on Linux, FreeBSD, or NetBSD. + +#. Run make for the ported executables in the top source directory, e.g.:: + + > make + + For faster compilation, you can pass the ``-j`` argument to make. For + example, to run 4 jobs simultaneously, run ``make -j4``. + + .. note:: + + MSYS 1.0.18 has a bug that causes parallel make to hang. You can overcome + this by downgrading to MSYS 1.0.17. A simple way to downgrade is to exit + all MinGW sessions and then run the below command from MSVC developers + command prompt.:: + + > mingw-get upgrade msys-core-bin=1.0.17-1 + +#. To run all the unit tests in Open vSwitch, one at a time:: + + > make check + + To run all the unit tests in Open vSwitch, up to 8 in parallel:: + + > make check TESTSUITEFLAGS="-j8" + +#. To install all the compiled executables on the local machine, run:: + + > make install + + .. note:: + + This will install the Open vSwitch executables in ``C:/openvswitch``. You + can add ``C:\openvswitch\usr\bin`` and ``C:\openvswitch\usr\sbin`` to + Windows' PATH environment variable for easy access. + +The Kernel Module +~~~~~~~~~~~~~~~~~ + +If you are building the kernel module, you will need to copy the below files to +the target Hyper-V machine. + +- ``./datapath-windows/x64/Win8.1Debug/package/ovsext.inf`` +- ``./datapath-windows/x64/Win8.1Debug/package/OVSExt.sys`` +- ``./datapath-windows/x64/Win8.1Debug/package/ovsext.cat`` +- ``./datapath-windows/misc/install.cmd`` +- ``./datapath-windows/misc/uninstall.cmd`` + +.. note:: + The above path assumes that the kernel module has been built using Windows + DDK 8.1 in Debug mode. Change the path appropriately, if a different WDK + has been used. + +Now run ``./uninstall.cmd`` to remove the old extension. Once complete, run +``./install.cmd`` to insert the new one. For this to work you will have to +turn on ``TESTSIGNING`` boot option or 'Disable Driver Signature +Enforcement' during boot. The following commands can be used:: + + > bcdedit /set LOADOPTIONS DISABLE_INTEGRITY_CHECKS + > bcdedit /set TESTSIGNING ON + > bcdedit /set nointegritychecks ON + +.. note:: + You may have to restart the machine for the settings to take effect. + +In the Virtual Switch Manager configuration you can enable the Open vSwitch +Extension on an existing switch or create a new switch. If you are using an +existing switch, make sure to enable the "Allow Management OS" option for VXLAN +to work (covered later). + +The command to create a new switch named 'OVS-Extended-Switch' using a physical +NIC named 'Ethernet 1' is:: + + PS > New-VMSwitch "OVS-Extended-Switch" -AllowManagementOS $true \ + -NetAdapterName "Ethernet 1" + +.. note:: + You can obtain the list of physical NICs on the host using 'Get-NetAdapter' + command. + +In the properties of any switch, you should should now see "Open vSwitch +Extension" under 'Extensions'. Click the check box to enable the extension. +An alternative way to do the same is to run the following command:: + + PS > Enable-VMSwitchExtension "Open vSwitch Extension" OVS-Extended-Switch + +.. note:: + If you enabled the extension using the command line, a delay of a few seconds + has been observed for the change to be reflected in the UI. This is not a + bug in Open vSwitch. + +Starting +-------- + +.. important:: + The following steps assume that you have installed the Open vSwitch utilities + in the local machine via 'make install'. + +Before starting ovs-vswitchd itself, you need to start its configuration +database, ovsdb-server. Each machine on which Open vSwitch is installed should +run its own copy of ovsdb-server. Before ovsdb-server itself can be started, +configure a database that it can use:: + + > ovsdb-tool create C:\openvswitch\etc\openvswitch\conf.db \ + C:\openvswitch\usr\share\openvswitch\vswitch.ovsschema + +Configure ovsdb-server to use database created above and to listen on a Unix +domain socket:: + + > ovsdb-server -vfile:info --remote=punix:db.sock --log-file \ + --pidfile --detach + +.. note:: + The logfile is created at ``C:/openvswitch/var/log/openvswitch/`` + +Initialize the database using ovs-vsctl. This is only necessary the first time +after you create the database with ovsdb-tool, though running it at any time is +harmless:: + + > ovs-vsctl --no-wait init + +.. tip:: + If you would later like to terminate the started ovsdb-server, run:: + + > ovs-appctl -t ovsdb-server exit + +Start the main Open vSwitch daemon, telling it to connect to the same Unix +domain socket:: + + > ovs-vswitchd -vfile:info --log-file --pidfile --detach + +.. tip:: + If you would like to terminate the started ovs-vswitchd, run:: + + > ovs-appctl exit + +.. note:: + The logfile is created at ``C:/openvswitch/var/log/openvswitch/`` + +Validating +---------- + +At this point you can use ovs-vsctl to set up bridges and other Open vSwitch +features. + +Add bridges +~~~~~~~~~~~ + +Let's start by creating an integration bridge, ``br-int`` and a PIF bridge, +``br-pif``:: + + > ovs-vsctl add-br br-int + > ovs-vsctl add-br br-pif + +.. note:: + There's a known bug that running the ovs-vsctl command does not terminate. + This is generally solved by having ovs-vswitchd running. If you face the + issue despite that, hit Ctrl-C to terminate ovs-vsctl and check the output to + see if your command succeeded. + +Validate that ports are added by dumping from both ovs-dpctl and ovs-vsctl:: + + > ovs-dpctl show + system@ovs-system: + lookups: hit:0 missed:0 lost:0 + flows: 0 + port 2: br-pif (internal) <<< internal port on 'br-pif' bridge + port 1: br-int (internal) <<< internal port on 'br-int' bridge + + > ovs-vsctl show + a56ec7b5-5b1f-49ec-a795-79f6eb63228b + Bridge br-pif + Port br-pif + Interface br-pif + type: internal + Bridge br-int + Port br-int + Interface br-int + type: internal + +.. note:: + There's a known bug that the ports added to OVSDB via ovs-vsctl don't get to + the kernel datapath immediately, ie. they don't show up in the output of + ``ovs-dpctl show`` even though they show up in output of ``ovs-vsctl show``. + In order to workaround this issue, restart ovs-vswitchd. (You can terminate + ovs-vswitchd by running ``ovs-appctl exit``.) + +Add physicals NICs (PIF) +~~~~~~~~~~~~~~~~~~~~~~~~ + +Now, let's add the physical NIC and the internal port to ``br-pif``. In OVS for +Hyper-V, we use the name of the adapter on top of which the Hyper-V virtual +switch was created, as a special name to refer to the physical NICs connected +to the Hyper-V switch, e.g. if we created the Hyper-V virtual switch on top of +the adapter named ``Ethernet0``, then in OVS we use that name (``Ethernet0``) +as a special name to refer to that adapter. + +.. note:: + we assume that the Hyper-V switch on which OVS extension is enabled has a + single physical NIC connected to it. + +An internal port is the virtual adapter created on the Hyper-V switch using the +``AllowManagementOS`` setting. This has already been setup while creating the +switch using the instructions above. In OVS for Hyper-V, we use a the name of +that specific adapter as a special name to refer to that adapter. By default it +is created under the following rule ``vEthernet (<name of the switch>)``. + +As a whole example, if we issue the following in a powershell console:: + + PS C:\package\binaries> Get-NetAdapter | select Name,MacAddress,InterfaceDescription + Name MacAddress InterfaceDescription + ---- ---------- -------------------- + Ethernet1 00-0C-29-94-05-65 Intel(R) PRO/1000 MT Network Connection + vEthernet (external) 00-0C-29-94-05-5B Hyper-V Virtual Ethernet Adapter #2 + Ethernet0 00-0C-29-94-05-5B Intel(R) PRO/1000 MT Network Connection #2 + + PS C:\package\binaries> Get-VMSwitch + Name SwitchType NetAdapterInterfaceDescription + ---- ---------- ------------------------------ + external External Intel(R) PRO/1000 MT Network Connection #2 + +We can see that we have a switch(external) created upon adapter name +'Ethernet0' with an internal port under name ``vEthernet (external)``. Thus +resulting into the following ovs-vsctl commands:: + + > ovs-vsctl add-port br-pif Ethernet0 + > ovs-vsctl add-port br-pif "vEthernet (external)" + +Dumping the ports should show the additional ports that were just added:: + + > ovs-dpctl show + system@ovs-system: + lookups: hit:0 missed:0 lost:0 + flows: 0 + port 4: vEthernet (external) (internal) <<< 'AllowManagementOS' + adapter on + Hyper-V switch + port 2: br-pif (internal) + port 1: br-int (internal) + port 3: Ethernet0 <<< Physical NIC + + > ovs-vsctl show + a56ec7b5-5b1f-49ec-a795-79f6eb63228b + Bridge br-pif + Port "vEthernet (external)" + Interface "vEthernet (external)" + Port br-pif + Interface br-pif + type: internal + Port "Ethernet0" + Interface "Ethernet0" + Bridge br-int + Port br-int + Interface br-int + type: internal + +Add virtual interfaces (VIFs) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Adding VIFs to openvswitch is a two step procedure. The first step is to +assign a 'OVS port name' which is a unique name across all VIFs on this +Hyper-V. The next step is to add the VIF to the ovsdb using its 'OVS port +name' as key. + +First, assign a unique 'OVS port name' to the VIF. The VIF needs to have been +disconnected from the Hyper-V switch before assigning a 'OVS port name' to it. +In the example below, we assign a 'OVS port name' called ``ovs-port-a`` to a +VIF on a VM ``VM1``. By using index 0 for ``$vnic``, the first VIF of the VM +is being addressed. After assigning the name ``ovs-port-a``, the VIF is +connected back to the Hyper-V switch with name ``OVS-HV-Switch``, which is +assumed to be the Hyper-V switch with OVS extension enabled.:: + + PS> import-module .\datapath-windows\misc\OVS.psm1 + PS> $vnic = Get-VMNetworkAdapter <Name of the VM> + PS> Disconnect-VMNetworkAdapter -VMNetworkAdapter $vnic[0] + PS> $vnic[0] | Set-VMNetworkAdapterOVSPort -OVSPortName ovs-port-a + PS> Connect-VMNetworkAdapter -VMNetworkAdapter $vnic[0] \ + -SwitchName OVS-Extended-Switch + +Next, add the VIFs to ``br-int``:: + + > ovs-vsctl add-port br-int ovs-port-a + +Dumping the ports should show the additional ports that were just added:: + + > ovs-dpctl show + system@ovs-system: + lookups: hit:0 missed:0 lost:0 + flows: 0 + port 4: vEthernet (external) (internal) + port 5: ovs-port-a + port 2: br-pif (internal) + port 1: br-int (internal + port 3: Ethernet0 + + > ovs-vsctl show + 4cd86499-74df-48bd-a64d-8d115b12a9f2 + Bridge br-pif + Port "vEthernet (external)" + Interface "vEthernet (external)" + Port "Ethernet0" + Interface "Ethernet0" + Port br-pif + Interface br-pif + type: internal + Bridge br-int + Port br-int + Interface br-int + type: internal + Port "ovs-port-a" + Interface "ovs-port-a" + +Add patch ports and configure VLAN tagging +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The Windows Open vSwitch implementation support VLAN tagging in the switch. +Switch VLAN tagging along with patch ports between ``br-int`` and ``br-pif`` is +used to configure VLAN tagging functionality between two VMs on different +Hyper-Vs. To start, add a patch port from ``br-int`` to ``br-pif``:: + + > ovs-vsctl add-port br-int patch-to-pif + > ovs-vsctl set interface patch-to-pif type=patch \ + options:peer=patch-to-int + +Add a patch port from ``br-pif`` to ``br-int``:: + + > ovs-vsctl add-port br-pif patch-to-int + > ovs-vsctl set interface patch-to-int type=patch \ + options:peer=patch-to-pif + +Re-Add the VIF ports with the VLAN tag:: + + > ovs-vsctl add-port br-int ovs-port-a tag=900 + > ovs-vsctl add-port br-int ovs-port-b tag=900 + +Add tunnels +~~~~~~~~~~~ + +The Windows Open vSwitch implementation support VXLAN and STT tunnels. To add +tunnels. For example, first add the tunnel port between 172.168.201.101 <-> +172.168.201.102:: + + > ovs-vsctl add-port br-int tun-1 + > ovs-vsctl set Interface tun-1 type=<port-type> + > ovs-vsctl set Interface tun-1 options:local_ip=172.168.201.101 + > ovs-vsctl set Interface tun-1 options:remote_ip=172.168.201.102 + > ovs-vsctl set Interface tun-1 options:in_key=flow + > ovs-vsctl set Interface tun-1 options:out_key=flow + +...and the tunnel port between 172.168.201.101 <-> 172.168.201.105:: + + > ovs-vsctl add-port br-int tun-2 + > ovs-vsctl set Interface tun-2 type=<port-type> + > ovs-vsctl set Interface tun-2 options:local_ip=172.168.201.102 + > ovs-vsctl set Interface tun-2 options:remote_ip=172.168.201.105 + > ovs-vsctl set Interface tun-2 options:in_key=flow + > ovs-vsctl set Interface tun-2 options:out_key=flow + +Where ``<port-type>`` is one of: ``stt`` or ``vxlan`` + +.. note:: + Any patch ports created between br-int and br-pif MUST be be deleted prior to + adding tunnels. + +Windows Services +---------------- + +Open vSwitch daemons come with support to run as a Windows service. The +instructions here assume that you have installed the Open vSwitch utilities and +daemons via ``make install``. + +.. note:: + The commands shown here can be run from MSYS bash or Windows command prompt. + +To start, create the database:: + + > ovsdb-tool create C:/openvswitch/etc/openvswitch/conf.db \ + "C:/openvswitch/usr/share/openvswitch/vswitch.ovsschema" + +Create the ovsdb-server service and start it:: + + > sc create ovsdb-server \ + binpath="C:/openvswitch/usr/sbin/ovsdb-server.exe \ + C:/openvswitch/etc/openvswitch/conf.db \ + -vfile:info --log-file --pidfile \ + --remote=punix:db.sock --service --service-monitor" + > sc start ovsdb-server + +.. tip:: + One of the common issues with creating a Windows service is with mungled + paths. You can make sure that the correct path has been registered with the + Windows services manager by running:: + + > sc qc ovsdb-server + +Check that the service is healthy by running:: + + > sc query ovsdb-server + +Initialize the database:: + + > ovs-vsctl --no-wait init + +Create the ovs-vswitchd service and start it:: + + > sc create ovs-vswitchd \ + binpath="C:/openvswitch/usr/sbin/ovs-vswitchd.exe \ + --pidfile -vfile:info --log-file --service --service-monitor" + > sc start ovs-vswitchd + +Check that the service is healthy by running:: + + > sc query ovs-vswitchd + +To stop and delete the services, run:: + + > sc stop ovs-vswitchd + > sc stop ovsdb-server + > sc delete ovs-vswitchd + > sc delete ovsdb-server + +Windows CI Service +------------------ + +`AppVeyor <www.appveyor.com>`__ provides a free Windows autobuild service for +opensource projects. Open vSwitch has integration with AppVeyor for continuous +build. A developer can build test his changes for Windows by logging into +appveyor.com using a github account, creating a new project by linking it to +his development repository in github and triggering a new build. + +TODO +---- + +* Investigate the working of sFlow on Windows and re-enable the unit tests. + +* Investigate and add the feature to provide QoS. + +* Sign the driver & create an MSI for installing the different OpenvSwitch + components on Windows. diff --git a/Documentation/intro/install/xenserver.rst b/Documentation/intro/install/xenserver.rst new file mode 100644 index 000000000..c0f5e3156 --- /dev/null +++ b/Documentation/intro/install/xenserver.rst @@ -0,0 +1,229 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + + Convention for heading levels in Open vSwitch documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + +================================ +Open vSwitch on Citrix XenServer +================================ + +This document describes how to build and install Open vSwitch on a Citrix +XenServer host. If you want to install Open vSwitch on a generic Linux or BSD +host, refer to :doc:`general` instead. + +Open vSwitch should work with XenServer 5.6.100 and later. However, Open +vSwitch requires Python 2.7 or later, so using Open vSwitch with XenServer 6.5 +or earlier requires installing Python 2.7. + +Building +-------- + +You may build from an Open vSwitch distribution tarball or from an Open vSwitch +Git tree. The recommended build environment to build RPMs for Citrix XenServer +is the DDK VM available from Citrix. + +1. If you are building from an Open vSwitch Git tree, then you will need to + first create a distribution tarball by running:: + + $ ./boot.sh + $ ./configure + $ make dist + + You cannot run this in the DDK VM, because it lacks tools that are necessary + to bootstrap the Open vSwitch distribution. Instead, you must run this on a + machine that has the tools listed in :ref:`general-install-reqs` as + prerequisites for building from a Git tree. + +2. Copy the distribution tarball into ``/usr/src/redhat/SOURCES`` inside + the DDK VM. + +3. In the DDK VM, unpack the distribution tarball into a temporary directory + and "cd" into the root of the distribution tarball. + +4. To build Open vSwitch userspace, run:: + + $ rpmbuild -bb xenserver/openvswitch-xen.spec + + This produces three RPMs in ``/usr/src/redhat/RPMS/i386``: + + - ``openvswitch`` + - ``openvswitch-modules-xen`` + - ``openvswitch-debuginfo`` + + The above command automatically runs the Open vSwitch unit tests. To + disable the unit tests, run:: + + $ rpmbuild -bb --without check xenserver/openvswitch-xen.spec + +Build Parameters +---------------- + +``openvswitch-xen.spec`` needs to know a number of pieces of information about +the XenServer kernel. Usually, it can figure these out for itself, but if it +does not do it correctly then you can specify them yourself as parameters to +the build. Thus, the final ``rpmbuild`` step above can be elaborated as:: + + $ VERSION=<Open vSwitch version> + $ KERNEL_NAME=<Xen Kernel name> + $ KERNEL_VERSION=<Xen Kernel version> + $ KERNEL_FLAVOR=<Xen Kernel flavor(suffix)> + $ rpmbuild \ + -D "openvswitch_version $VERSION" \ + -D "kernel_name $KERNEL_NAME" \ + -D "kernel_version $KERNEL_VERSION" \ + -D "kernel_flavor $KERNEL_FLAVOR" \ + -bb xenserver/openvswitch-xen.spec + +where: + +``<openvswitch version>`` + is the version number that appears in the name of the Open vSwitch tarball, + e.g. 0.90.0. + +``<Xen Kernel name>`` + is the name of the XenServer kernel package, e.g. ``kernel-xen`` or + ``kernel-NAME-xen``, without the ``kernel-`` prefix. + +``<Xen Kernel version>`` + is the output of:: + + $ rpm -q --queryformat "%{Version}-%{Release}" <kernel-devel-package>, + + e.g. ``2.6.32.12-0.7.1.xs5.6.100.323.170596``, where + ``<kernel-devel-package>`` is the name of the ``-devel`` package + corresponding to ``<Xen Kernel name>``. + +``<Xen Kernel flavor (suffix)>`` + is either ``xen`` or ``kdump``, where ``xen`` flavor is the main running + kernel flavor and the ``kdump`` flavor is the crashdump kernel flavor. + Commonly, one would specify ``xen`` here. + +For XenServer 6.5 or above, the kernel version naming no longer contains +KERNEL_FLAVOR. In fact, only providing the ``uname -r`` output is enough. So, +the final ``rpmbuild`` step changes to:: + + $ KERNEL_UNAME=<`uname -r` output> + $ rpmbuild \ + -D "kenel_uname $KERNEL_UNAME" \ + -bb xenserver/openvswitch-xen.spec + +Installing Open vSwitch for XenServer +------------------------------------- + +To install Open vSwitch on a XenServer host, or to upgrade to a newer version, +copy the ``openvswitch`` and ``openvswitch-modules-xen`` RPMs to that host with +``scp``, then install them with ``rpm -U``, e.g.:: + + $ scp openvswitch-$VERSION-1.i386.rpm \ + openvswitch-modules-xen-$XEN_KERNEL_VERSION-$VERSION-1.i386.rpm \ + root@<host>: + # Enter <host>'s root password. + $ ssh root@<host> + # Enter <host>'s root password again. + $ rpm -U openvswitch-$VERSION-1.i386.rpm \ + openvswitch-modules-xen-$XEN_KERNEL_VERSION-$VERSION-1.i386.rpm + +To uninstall Open vSwitch from a XenServer host, remove the packages:: + + $ ssh root@<host> + # Enter <host>'s root password again. + $ rpm -e openvswitch openvswitch-modules-xen-$XEN_KERNEL_VERSION + +After installing or uninstalling Open vSwitch, the XenServer should be rebooted +as soon as possible. + +Open vSwitch Boot Sequence on XenServer +--------------------------------------- + +When Open vSwitch is installed on XenServer, its startup script +``/etc/init.d/openvswitch`` runs early in boot. It does roughly the following: + +* Loads the OVS kernel module, openvswitch. + +* Starts ovsdb-server, the OVS configuration database. + +* XenServer expects there to be no bridges configured at startup, but the OVS + configuration database likely still has bridges configured from before + reboot. To match XenServer expectations, the startup script deletes all + configured bridges from the database. + +* Starts ovs-vswitchd, the OVS switching daemon. + +At this point in the boot process, then, there are no Open vSwitch bridges, +even though all of the Open vSwitch daemons are running. Later on in boot, +``/etc/init.d/management-interface`` (part of XenServer, not Open vSwitch) +creates the bridge for the XAPI management interface by invoking +``/opt/xensource/libexec/interface-reconfigure``. Normally this program +consults XAPI's database to obtain information about how to configure the +bridge, but XAPI is not running yet(\*) so it instead consults +``/var/xapi/network.dbcache``, which is a cached copy of the most recent +network configuration. + +(\*) Even if XAPI were running, if this XenServer node is a pool slave then the + query would have to consult the master, which requires network access, + which begs the question of how to configure the management interface. + +XAPI starts later on in the boot process. XAPI can then create other bridges +on demand using ``/opt/xensource/libexec/interface-reconfigure``. Now that +XAPI is running, that program consults XAPI directly instead of reading the +cache. + +As part of its own startup, XAPI invokes the Open vSwitch XAPI plugin script +``/etc/xapi.d/openvswitch-cfg-update`` passing the ``update`` command. The +plugin script does roughly the following: + +* Calls ``/opt/xensource/libexec/interface-reconfigure`` with the ``rewrite`` + command, to ensure that the network cache is up-to-date. + +* Queries the Open vSwitch manager setting (named ``vswitch_controller``) from + the XAPI database for the XenServer pool. + +* If XAPI and OVS are configured for different managers, or if OVS is + configured for a manager but XAPI is not, runs ``ovs-vsctl emer-reset`` to + bring the Open vSwitch configuration to a known state. One effect of + emer-reset is to deconfigure any manager from the OVS database. + +* If XAPI is configured for a manager, configures the OVS manager to match with + ``ovs-vsctl set-manager``. + +Notes +----- + +* The Open vSwitch boot sequence only configures an OVS configuration database + manager. There is no way to directly configure an OpenFlow controller on + XenServer and, as a consequence of the step above that deletes all of the + bridges at boot time, controller configuration only persists until XenServer + reboot. The configuration database manager can, however, configure + controllers for bridges. See the BUGS section of ovs-testcontroller(8) for + more information on this topic. + +* The Open vSwitch startup script automatically adds a firewall rule to allow + GRE traffic. This rule is needed for the XenServer feature called "Cross-Host + Internal Networks" (CHIN) that uses GRE. If a user configures tunnels other + than GRE (ex: Geneve, VXLAN, LISP), they will have to either manually add a + iptables firewall rule to allow the tunnel traffic or add it through a + startup script (Please refer to the "enable-protocol" command in the + ovs-ctl(8) manpage). + +Reporting Bugs +-------------- + +Please report problems to bugs@openvswitch.org. |