.. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Convention for heading levels in Open vSwitch documentation: ======= Heading 0 (reserved for the title in a document) ------- Heading 1 ~~~~~~~ Heading 2 +++++++ Heading 3 ''''''' Heading 4 Avoid deeper levels because they do not render well. ======================================== Connecting VMs Using Tunnels (Userspace) ======================================== This document describes how to use Open vSwitch to allow VMs on two different hosts to communicate over VXLAN tunnels. Unlike :doc:`tunneling`, this configuration works entirely in userspace. .. note:: This guide covers the steps required to configure VXLAN tunneling. The same approach can be used for any of the other tunneling protocols supported by Open vSwitch. .. TODO(stephenfin): Convert this to a (prettier) PNG with same styling as the rest of the document :: +--------------+ | vm0 | 192.168.1.1/24 +--------------+ (vm_port0) | | | +--------------+ | br-int | 192.168.1.2/24 +--------------+ +--------------+ | vxlan0 | | vxlan0 | +--------------+ +--------------+ | | | | | | 172.168.1.1/24 | +--------------+ | | br-phy | 172.168.1.2/24 +--------------+ +---------------+ | dpdk0/eth1 |----------------------------------| eth1 | +--------------+ +---------------+ Host A with OVS. Remote host. Setup ----- This guide assumes the environment is configured as described below. Two Physical Hosts ~~~~~~~~~~~~~~~~~~ The environment assumes the use of two hosts, named `host1` and `host2`. We only detail the configuration of `host1` but a similar configuration can be used for `host2`. Both hosts should be configured with Open vSwitch (with or without DPDK), QEMU/KVM and suitable VM images. Open vSwitch should be running before proceeding. Configuration Steps ------------------- Perform the following configuration on `host1`: #. Create a ``br-int`` bridge:: $ ovs-vsctl --may-exist add-br br-int \ -- set Bridge br-int datapath_type=netdev \ -- br-set-external-id br-int bridge-id br-int \ -- set bridge br-int fail-mode=standalone #. Add a port to this bridge. If using tap ports, first boot a VM and then add the port to the bridge:: $ ovs-vsctl add-port br-int tap0 If using DPDK vhost-user ports, add the port and then boot the VM accordingly, using ``vm_port0`` as the interface name:: $ ovs-vsctl add-port br-int vm_port0 \ -- set Interface vm_port0 type=dpdkvhostuserclient \ options:vhost-server-path=/tmp/vm_port0 #. Configure the IP address of the VM interface *in the VM itself*:: $ ip addr add 192.168.1.1/24 dev eth0 $ ip link set eth0 up #. On `host1`, add a port for the VXLAN tunnel:: $ ovs-vsctl add-port br-int vxlan0 \ -- set interface vxlan0 type=vxlan options:remote_ip=172.168.1.2 .. note:: ``172.168.1.2`` is the remote tunnel end point address. On the remote host this will be ``172.168.1.1`` #. Create a ``br-phy`` bridge:: $ ovs-vsctl --may-exist add-br br-phy \ -- set Bridge br-phy datapath_type=netdev \ -- br-set-external-id br-phy bridge-id br-phy \ -- set bridge br-phy fail-mode=standalone \ other_config:hwaddr= .. note:: This additional bridge is required when running Open vSwitch in userspace rather than kernel-based Open vSwitch. The purpose of this bridge is to allow use of the kernel network stack for routing and ARP resolution. The datapath needs to look-up the routing table and ARP table to prepare the tunnel header and transmit data to the output port. .. note:: ``eth1`` is used rather than ``eth0``. This is to ensure network connectivity is retained. #. Attach ``eth1``/``dpdk0`` to the ``br-phy`` bridge. If the physical port ``eth1`` is operating as a kernel network interface, run:: $ ovs-vsctl --timeout 10 add-port br-phy eth1 $ ip addr add 172.168.1.1/24 dev br-phy $ ip link set br-phy up $ ip addr flush dev eth1 2>/dev/null $ ip link set eth1 up $ iptables -F If instead the interface is a DPDK interface and bound to the ``igb_uio`` or ``vfio`` driver, run:: $ ovs-vsctl --timeout 10 add-port br-phy dpdk0 \ -- set Interface dpdk0 type=dpdk options:dpdk-devargs=0000:06:00.0 $ ip addr add 172.168.1.1/24 dev br-phy $ ip link set br-phy up $ iptables -F The commands are different as DPDK interfaces are not managed by the kernel, thus, the port details are not visible to any ``ip`` commands. .. important:: Attempting to use the kernel network commands for a DPDK interface will result in a loss of connectivity through ``eth1``. Refer to :doc:`/faq/configuration` for more details. Once complete, check the cached routes using ovs-appctl command:: $ ovs-appctl ovs/route/show If the tunnel route is missing, adding it now:: $ ovs-appctl ovs/route/add 172.168.1.1/24 br-phy Repeat these steps if necessary for `host2`, but using the below commands for the VM interface IP address:: $ ip addr add 192.168.1.2/24 dev eth0 $ ip link set eth0 up And the below command for the the `host2` VXLAN tunnel:: $ ovs-vsctl add-port br-int vxlan0 \ -- set interface vxlan0 type=vxlan options:remote_ip=172.168.1.1 Testing ------- With this setup, ping to VXLAN target device (``192.168.1.2``) should work. Traffic will be VXLAN encapsulated and sent over the ``eth1``/``dpdk0`` interface. Tunneling-related Commands -------------------------- Tunnel routing table ~~~~~~~~~~~~~~~~~~~~ To add route:: $ ovs-appctl ovs/route/add / To see all routes configured:: $ ovs-appctl ovs/route/show To delete route:: $ ovs-appctl ovs/route/del / To look up and display the route for a destination:: $ ovs-appctl ovs/route/lookup ARP ~~~ To see arp cache content:: $ ovs-appctl tnl/arp/show To flush arp cache:: $ ovs-appctl tnl/arp/flush To set a specific arp entry:: $ ovs-appctl tnl/arp/set Ports ~~~~~ To check tunnel ports listening in ovs-vswitchd:: $ ovs-appctl tnl/ports/show To set range for VxLan UDP source port:: $ ovs-appctl tnl/egress_port_range To show current range:: $ ovs-appctl tnl/egress_port_range Datapath ~~~~~~~~ To check datapath ports:: $ ovs-appctl dpif/show To check datapath flows:: $ ovs-appctl dpif/dump-flows