path: root/ovn/ovn-architecture.7.xml

<?xml version="1.0" encoding="utf-8"?>
<manpage program="ovn-architecture" section="7" title="OVN Architecture">
  <h1>Name</h1>
  <p>ovn-architecture -- Open Virtual Network architecture</p>

  <h1>Description</h1>

  <p>
    OVN, the Open Virtual Network, is a system to support virtual network
    abstraction.  OVN complements the existing capabilities of OVS to add
    native support for virtual network abstractions, such as virtual L2 and L3
    overlays and security groups.  Services such as DHCP are also desirable
    features.  Just like OVS, OVN's design goal is to have a production-quality
    implementation that can operate at significant scale.
  </p>

  <p>
    An OVN deployment consists of several components:
  </p>

  <ul>
    <li>
      <p>
        A <dfn>Cloud Management System</dfn> (<dfn>CMS</dfn>), which is
        OVN's ultimate client (via its users and administrators).  OVN
        integration requires installing a CMS-specific plugin and
        related software (see below).  OVN initially targets OpenStack
        as CMS.
      </p>

      <p>
        We generally speak of ``the'' CMS, but one can imagine scenarios in
        which multiple CMSes manage different parts of an OVN deployment.
      </p>
    </li>

    <li>
      An OVN Database physical or virtual node (or, eventually, cluster)
      installed in a central location.
    </li>

    <li>
      One or more (usually many) <dfn>hypervisors</dfn>.  Hypervisors must run
      Open vSwitch and implement the interface described in
      <code>IntegrationGuide.rst</code> in the OVS source tree.  Any hypervisor
      platform supported by Open vSwitch is acceptable.
    </li>

    <li>
      <p>
        Zero or more <dfn>gateways</dfn>.  A gateway extends a tunnel-based
        logical network into a physical network by bidirectionally forwarding
        packets between tunnels and a physical Ethernet port.  This allows
        non-virtualized machines to participate in logical networks.  A gateway
        may be a physical host, a virtual machine, or an ASIC-based hardware
        switch that supports the <code>vtep</code>(5) schema.
      </p>

      <p>
        Hypervisors and gateways are together called <dfn>transport node</dfn>
        or <dfn>chassis</dfn>.
      </p>
    </li>
  </ul>

  <p>
    The diagram below shows how the major components of OVN and related
    software interact.  Starting at the top of the diagram, we have:
  </p>

  <ul>
    <li>
      The Cloud Management System, as defined above.
    </li>

    <li>
      <p>
        The <dfn>OVN/CMS Plugin</dfn> is the component of the CMS that
        interfaces to OVN.  In OpenStack, this is a Neutron plugin.
        The plugin's main purpose is to translate the CMS's notion of logical
        network configuration, stored in the CMS's configuration database in a
        CMS-specific format, into an intermediate representation understood by
        OVN.
      </p>

      <p>
        This component is necessarily CMS-specific, so a new plugin needs to be
        developed for each CMS that is integrated with OVN.  All of the
        components below this one in the diagram are CMS-independent.
      </p>
    </li>

    <li>
      <p>
        The <dfn>OVN Northbound Database</dfn> receives the intermediate
        representation of logical network configuration passed down by the
        OVN/CMS Plugin.  The database schema is meant to be ``impedance
        matched'' with the concepts used in a CMS, so that it directly supports
        notions of logical switches, routers, ACLs, and so on.  See
        <code>ovn-nb</code>(5) for details.
      </p>

      <p>
        The OVN Northbound Database has only two clients: the OVN/CMS Plugin
        above it and <code>ovn-northd</code> below it.
      </p>
    </li>

    <li>
      <code>ovn-northd</code>(8) connects to the OVN Northbound Database
      above it and the OVN Southbound Database below it.  It translates the
      logical network configuration in terms of conventional network
      concepts, taken from the OVN Northbound Database, into logical
      datapath flows in the OVN Southbound Database below it.
    </li>

    <li>
      <p>
    The <dfn>OVN Southbound Database</dfn> is the center of the system.
    Its clients are <code>ovn-northd</code>(8) above it and
    <code>ovn-controller</code>(8) on every transport node below it.
      </p>

      <p>
        The OVN Southbound Database contains three kinds of data: <dfn>Physical
        Network</dfn> (PN) tables that specify how to reach hypervisor and
        other nodes, <dfn>Logical Network</dfn> (LN) tables that describe the
        logical network in terms of ``logical datapath flows,'' and
        <dfn>Binding</dfn> tables that link logical network components'
        locations to the physical network.  The hypervisors populate the PN and
        Port_Binding tables, whereas <code>ovn-northd</code>(8) populates the
        LN tables.
      </p>

      <p>
    OVN Southbound Database performance must scale with the number of
    transport nodes.  This will likely require some work on
    <code>ovsdb-server</code>(1) as we encounter bottlenecks.
    Clustering for availability may be needed.
      </p>
    </li>
  </ul>

  <p>
    The remaining components are replicated onto each hypervisor:
  </p>

  <ul>
    <li>
      <code>ovn-controller</code>(8) is OVN's agent on each hypervisor and
      software gateway.  Northbound, it connects to the OVN Southbound
      Database to learn about OVN configuration and status and to
      populate the PN table and the <code>Chassis</code> column in
      <code>Binding</code> table with the hypervisor's status.
      Southbound, it connects to <code>ovs-vswitchd</code>(8) as an
      OpenFlow controller, for control over network traffic, and to the
      local <code>ovsdb-server</code>(1) to allow it to monitor and
      control Open vSwitch configuration.
    </li>

    <li>
      <code>ovs-vswitchd</code>(8) and <code>ovsdb-server</code>(1) are
      conventional components of Open vSwitch.
    </li>
  </ul>

  <pre fixed="yes">
                                  CMS
                                   |
                                   |
                       +-----------|-----------+
                       |           |           |
                       |     OVN/CMS Plugin    |
                       |           |           |
                       |           |           |
                       |   OVN Northbound DB   |
                       |           |           |
                       |           |           |
                       |       ovn-northd      |
                       |           |           |
                       +-----------|-----------+
                                   |
                                   |
                         +-------------------+
                         | OVN Southbound DB |
                         +-------------------+
                                   |
                                   |
                +------------------+------------------+
                |                  |                  |
  HV 1          |                  |    HV n          |
+---------------|---------------+  .  +---------------|---------------+
|               |               |  .  |               |               |
|        ovn-controller         |  .  |        ovn-controller         |
|         |          |          |  .  |         |          |          |
|         |          |          |     |         |          |          |
|  ovs-vswitchd   ovsdb-server  |     |  ovs-vswitchd   ovsdb-server  |
|                               |     |                               |
+-------------------------------+     +-------------------------------+
  </pre>

  <h2>Information Flow in OVN</h2>

  <p>
    Configuration data in OVN flows from north to south.  The CMS, through its
    OVN/CMS plugin, passes the logical network configuration to
    <code>ovn-northd</code> via the northbound database.  In turn,
    <code>ovn-northd</code> compiles the configuration into a lower-level form
    and passes it to all of the chassis via the southbound database.
  </p>

  <p>
    Status information in OVN flows from south to north.  OVN currently
    provides only a few forms of status information.  First,
    <code>ovn-northd</code> populates the <code>up</code> column in the
    northbound <code>Logical_Switch_Port</code> table: if a logical port's
    <code>chassis</code> column in the southbound <code>Port_Binding</code>
    table is nonempty, it sets <code>up</code> to <code>true</code>, otherwise
    to <code>false</code>.  This allows the CMS to detect when a VM's
    networking has come up.
  </p>

  <p>
    Second, OVN provides feedback to the CMS on the realization of its
    configuration, that is, whether the configuration provided by the CMS has
    taken effect.  This feature requires the CMS to participate in a sequence
    number protocol, which works the following way:
  </p>

  <ol>
    <li>
      When the CMS updates the configuration in the northbound database, as
      part of the same transaction, it increments the value of the
      <code>nb_cfg</code> column in the <code>NB_Global</code> table.  (This is
      only necessary if the CMS wants to know when the configuration has been
      realized.)
    </li>

    <li>
      When <code>ovn-northd</code> updates the southbound database based on a
      given snapshot of the northbound database, it copies <code>nb_cfg</code>
      from northbound <code>NB_Global</code> into the southbound database
      <code>SB_Global</code> table, as part of the same transaction.  (Thus, an
      observer monitoring both databases can determine when the southbound
      database is caught up with the northbound.)
    </li>

    <li>
      After <code>ovn-northd</code> receives confirmation from the southbound
      database server that its changes have committed, it updates
      <code>sb_cfg</code> in the northbound <code>NB_Global</code> table to the
      <code>nb_cfg</code> version that was pushed down.  (Thus, the CMS or
      another observer can determine when the southbound database is caught up
      without a connection to the southbound database.)
    </li>

    <li>
      The <code>ovn-controller</code> process on each chassis receives the
      updated southbound database, with the updated <code>nb_cfg</code>.  This
      process in turn updates the physical flows installed in the chassis's
      Open vSwitch instances.  When it receives confirmation from Open vSwitch
      that the physical flows have been updated, it updates <code>nb_cfg</code>
      in its own <code>Chassis</code> record in the southbound database.
    </li>

    <li>
      <code>ovn-northd</code> monitors the <code>nb_cfg</code> column in all of
      the <code>Chassis</code> records in the southbound database.  It keeps
      track of the minimum value among all the records and copies it into the
      <code>hv_cfg</code> column in the northbound <code>NB_Global</code>
      table.  (Thus, the CMS or another observer can determine when all of the
      hypervisors have caught up to the northbound configuration.)
    </li>
  </ol>

  <h2>Chassis Setup</h2>

  <p>
    Each chassis in an OVN deployment must be configured with an Open vSwitch
    bridge dedicated for OVN's use, called the <dfn>integration bridge</dfn>.
    System startup scripts may create this bridge prior to starting
    <code>ovn-controller</code> if desired.  If this bridge does not exist when
    ovn-controller starts, it will be created automatically with the default
    configuration suggested below.  The ports on the integration bridge include:
  </p>

  <ul>
    <li>
      On any chassis, tunnel ports that OVN uses to maintain logical network
      connectivity.  <code>ovn-controller</code> adds, updates, and removes
      these tunnel ports.
    </li>

    <li>
      On a hypervisor, any VIFs that are to be attached to logical networks.
      The hypervisor itself, or the integration between Open vSwitch and the
      hypervisor (described in <code>IntegrationGuide.rst</code>) takes care of
      this.  (This is not part of OVN or new to OVN; this is pre-existing
      integration work that has already been done on hypervisors that support
      OVS.)
    </li>

    <li>
      On a gateway, the physical port used for logical network connectivity.
      System startup scripts add this port to the bridge prior to starting
      <code>ovn-controller</code>.  This can be a patch port to another bridge,
      instead of a physical port, in more sophisticated setups.
    </li>
  </ul>

  <p>
    Other ports should not be attached to the integration bridge.  In
    particular, physical ports attached to the underlay network (as opposed to
    gateway ports, which are physical ports attached to logical networks) must
    not be attached to the integration bridge.  Underlay physical ports should
    instead be attached to a separate Open vSwitch bridge (they need not be
    attached to any bridge at all, in fact).
  </p>

  <p>
    The integration bridge should be configured as described below.
    The effect of each of these settings is documented in
    <code>ovs-vswitchd.conf.db</code>(5):
  </p>

  <!-- Keep the following in sync with create_br_int() in
       ovn/controller/ovn-controller.c. -->
  <dl>
    <dt><code>fail-mode=secure</code></dt>
    <dd>
      Avoids switching packets between isolated logical networks before
      <code>ovn-controller</code> starts up.  See <code>Controller Failure
      Settings</code> in <code>ovs-vsctl</code>(8) for more information.
    </dd>

    <dt><code>other-config:disable-in-band=true</code></dt>
    <dd>
      Suppresses in-band control flows for the integration bridge.  It would be
      unusual for such flows to show up anyway, because OVN uses a local
      controller (over a Unix domain socket) instead of a remote controller.
      It's possible, however, for some other bridge in the same system to have
      an in-band remote controller, and in that case this suppresses the flows
      that in-band control would ordinarily set up.  Refer to the documentation
      for more information.
    </dd>
  </dl>

  <p>
    The customary name for the integration bridge is <code>br-int</code>, but
    another name may be used.
  </p>

  <h2>Logical Networks</h2>

  <p>
    A <dfn>logical network</dfn> implements the same concepts as physical
    networks, but they are insulated from the physical network with tunnels or
    other encapsulations.  This allows logical networks to have separate IP and
    other address spaces that overlap, without conflicting, with those used for
    physical networks.  Logical network topologies can be arranged without
    regard for the topologies of the physical networks on which they run.
  </p>

  <p>
    Logical network concepts in OVN include:
  </p>

  <ul>
    <li>
      <dfn>Logical switches</dfn>, the logical version of Ethernet switches.
    </li>

    <li>
      <dfn>Logical routers</dfn>, the logical version of IP routers.  Logical
      switches and routers can be connected into sophisticated topologies.
    </li>

    <li>
      <dfn>Logical datapaths</dfn> are the logical version of an OpenFlow
      switch.  Logical switches and routers are both implemented as logical
      datapaths.
    </li>

    <li>
      <p>
        <dfn>Logical ports</dfn> represent the points of connectivity in and
        out of logical switches and logical routers.  Some common types of
        logical ports are:
      </p>

      <ul>
        <li>
          Logical ports representing VIFs.
        </li>

        <li>
          <dfn>Localnet ports</dfn> represent the points of connectivity
          between logical switches and the physical network.  They are
          implemented as OVS patch ports between the integration bridge
          and the separate Open vSwitch bridge that underlay physical
          ports attach to.
        </li>

        <li>
          <dfn>Logical patch ports</dfn> represent the points of
          connectivity between logical switches and logical routers, and
          in some cases between peer logical routers.  There is a pair of
          logical patch ports at each such point of connectivity, one on
          each side.
        </li>
        <li>
          <dfn>Localport ports</dfn> represent the points of local
          connectivity between logical switches and VIFs. These ports are
          present in every chassis (not bound to any particular one) and
          traffic from them will never go through a tunnel. A
          <code>localport</code> is expected to only generate traffic destined
          for a local destination, typically in response to a request it
          received.
          One use case is how OpenStack Neutron uses a <code>localport</code>
          port for serving metadata to VM's residing on every hypervisor. A
          metadata proxy process is attached to this port on every host and all
          VM's within the same network will reach it at the same IP/MAC address
          without any traffic being sent over a tunnel. Further details can be
          seen at https://docs.openstack.org/developer/networking-ovn/design/metadata_api.html.
        </li>
      </ul>
    </li>
  </ul>

  <h2>Life Cycle of a VIF</h2>

  <p>
    Tables and their schemas presented in isolation are difficult to
    understand.  Here's an example.
  </p>

  <p>
    A VIF on a hypervisor is a virtual network interface attached either
    to a VM or a container running directly on that hypervisor (This is
    different from the interface of a container running inside a VM).
  </p>

  <p>
    The steps in this example refer often to details of the OVN and OVN
    Northbound database schemas.  Please see <code>ovn-sb</code>(5) and
    <code>ovn-nb</code>(5), respectively, for the full story on these
    databases.
  </p>

  <ol>
    <li>
      A VIF's life cycle begins when a CMS administrator creates a new VIF
      using the CMS user interface or API and adds it to a switch (one
      implemented by OVN as a logical switch).  The CMS updates its own
      configuration.  This includes associating unique, persistent identifier
      <var>vif-id</var> and Ethernet address <var>mac</var> with the VIF.
    </li>

    <li>
      The CMS plugin updates the OVN Northbound database to include the new
      VIF, by adding a row to the <code>Logical_Switch_Port</code>
      table.  In the new row, <code>name</code> is <var>vif-id</var>,
      <code>mac</code> is <var>mac</var>, <code>switch</code> points to
      the OVN logical switch's Logical_Switch record, and other columns
      are initialized appropriately.
    </li>

    <li>
      <code>ovn-northd</code> receives the OVN Northbound database update.  In
      turn, it makes the corresponding updates to the OVN Southbound database,
      by adding rows to the OVN Southbound database <code>Logical_Flow</code>
      table to reflect the new port, e.g. add a flow to recognize that packets
      destined to the new port's MAC address should be delivered to it, and
      update the flow that delivers broadcast and multicast packets to include
      the new port.  It also creates a record in the <code>Binding</code> table
      and populates all its columns except the column that identifies the
      <code>chassis</code>.
    </li>

    <li>
      On every hypervisor, <code>ovn-controller</code> receives the
      <code>Logical_Flow</code> table updates that <code>ovn-northd</code> made
      in the previous step.  As long as the VM that owns the VIF is powered
      off, <code>ovn-controller</code> cannot do much; it cannot, for example,
      arrange to send packets to or receive packets from the VIF, because the
      VIF does not actually exist anywhere.
    </li>

    <li>
      Eventually, a user powers on the VM that owns the VIF.  On the hypervisor
      where the VM is powered on, the integration between the hypervisor and
      Open vSwitch (described in <code>IntegrationGuide.rst</code>) adds the VIF
      to the OVN integration bridge and stores <var>vif-id</var> in
      <code>external_ids</code>:<code>iface-id</code> to indicate that the
      interface is an instantiation of the new VIF.  (None of this code is new
      in OVN; this is pre-existing integration work that has already been done
      on hypervisors that support OVS.)
    </li>

    <li>
      On the hypervisor where the VM is powered on, <code>ovn-controller</code>
      notices <code>external_ids</code>:<code>iface-id</code> in the new
      Interface. In response, in the OVN Southbound DB, it updates the
      <code>Binding</code> table's <code>chassis</code> column for the
      row that links the logical port from <code>external_ids</code>:<code>
      iface-id</code> to the hypervisor. Afterward, <code>ovn-controller</code>
      updates the local hypervisor's OpenFlow tables so that packets to and from
      the VIF are properly handled.
    </li>

    <li>
      Some CMS systems, including OpenStack, fully start a VM only when its
      networking is ready.  To support this, <code>ovn-northd</code> notices
      the <code>chassis</code> column updated for the row in
      <code>Binding</code> table and pushes this upward by updating the
      <ref column="up" table="Logical_Switch_Port" db="OVN_NB"/> column
      in the OVN Northbound database's <ref table="Logical_Switch_Port"
      db="OVN_NB"/> table to indicate that the VIF is now up.  The CMS,
      if it uses this feature, can then react by allowing the VM's
      execution to proceed.
    </li>

    <li>
      On every hypervisor but the one where the VIF resides,
      <code>ovn-controller</code> notices the completely populated row in the
      <code>Binding</code> table.  This provides <code>ovn-controller</code>
      the physical location of the logical port, so each instance updates the
      OpenFlow tables of its switch (based on logical datapath flows in the OVN
      DB <code>Logical_Flow</code> table) so that packets to and from the VIF
      can be properly handled via tunnels.
    </li>

    <li>
      Eventually, a user powers off the VM that owns the VIF.  On the
      hypervisor where the VM was powered off, the VIF is deleted from the OVN
      integration bridge.
    </li>

    <li>
      On the hypervisor where the VM was powered off,
      <code>ovn-controller</code> notices that the VIF was deleted.  In
      response, it removes the <code>Chassis</code> column content in the
      <code>Binding</code> table for the logical port.
    </li>

    <li>
      On every hypervisor, <code>ovn-controller</code> notices the empty
      <code>Chassis</code> column in the <code>Binding</code> table's row
      for the logical port.  This means that <code>ovn-controller</code> no
      longer knows the physical location of the logical port, so each instance
      updates its OpenFlow table to reflect that.
    </li>

    <li>
      Eventually, when the VIF (or its entire VM) is no longer needed by
      anyone, an administrator deletes the VIF using the CMS user interface or
      API.  The CMS updates its own configuration.
    </li>

    <li>
      The CMS plugin removes the VIF from the OVN Northbound database,
      by deleting its row in the <code>Logical_Switch_Port</code> table.
    </li>

    <li>
      <code>ovn-northd</code> receives the OVN Northbound update and in turn
      updates the OVN Southbound database accordingly, by removing or updating
      the rows from the OVN Southbound database <code>Logical_Flow</code> table
      and <code>Binding</code> table that were related to the now-destroyed
      VIF.
    </li>

    <li>
      On every hypervisor, <code>ovn-controller</code> receives the
      <code>Logical_Flow</code> table updates that <code>ovn-northd</code> made
      in the previous step.  <code>ovn-controller</code> updates OpenFlow
      tables to reflect the update, although there may not be much to do, since
      the VIF had already become unreachable when it was removed from the
      <code>Binding</code> table in a previous step.
    </li>
  </ol>

  <h2>Life Cycle of a Container Interface Inside a VM</h2>

  <p>
    OVN provides virtual network abstractions by converting information
    written in OVN_NB database to OpenFlow flows in each hypervisor.  Secure
    virtual networking for multi-tenants can only be provided if OVN controller
    is the only entity that can modify flows in Open vSwitch.  When the
    Open vSwitch integration bridge resides in the hypervisor, it is a
    fair assumption to make that tenant workloads running inside VMs cannot
    make any changes to Open vSwitch flows.
  </p>

  <p>
    If the infrastructure provider trusts the applications inside the
    containers not to break out and modify the Open vSwitch flows, then
    containers can be run in hypervisors.  This is also the case when
    containers are run inside the VMs and Open vSwitch integration bridge
    with flows added by OVN controller resides in the same VM.  For both
    the above cases, the workflow is the same as explained with an example
    in the previous section ("Life Cycle of a VIF").
  </p>

  <p>
    This section talks about the life cycle of a container interface (CIF)
    when containers are created in the VMs and the Open vSwitch integration
    bridge resides inside the hypervisor.  In this case, even if a container
    application breaks out, other tenants are not affected because the
    containers running inside the VMs cannot modify the flows in the
    Open vSwitch integration bridge.
  </p>

  <p>
    When multiple containers are created inside a VM, there are multiple
    CIFs associated with them.  The network traffic associated with these
    CIFs need to reach the Open vSwitch integration bridge running in the
    hypervisor for OVN to support virtual network abstractions.  OVN should
    also be able to distinguish network traffic coming from different CIFs.
    There are two ways to distinguish network traffic of CIFs.
  </p>

  <p>
    One way is to provide one VIF for every CIF (1:1 model).  This means that
    there could be a lot of network devices in the hypervisor.  This would slow
    down OVS because of all the additional CPU cycles needed for the management
    of all the VIFs.  It would also mean that the entity creating the
    containers in a VM should also be able to create the corresponding VIFs in
    the hypervisor.
  </p>

  <p>
    The second way is to provide a single VIF for all the CIFs (1:many model).
    OVN could then distinguish network traffic coming from different CIFs via
    a tag written in every packet.  OVN uses this mechanism and uses VLAN as
    the tagging mechanism.
  </p>

  <ol>
    <li>
      A CIF's life cycle begins when a container is spawned inside a VM by
      the either the same CMS that created the VM or a tenant that owns that VM
      or even a container Orchestration System that is different than the CMS
      that initially created the VM.  Whoever the entity is, it will need to
      know the <var>vif-id</var> that is associated with the network interface
      of the VM through which the container interface's network traffic is
      expected to go through.  The entity that creates the container interface
      will also need to choose an unused VLAN inside that VM.
    </li>

    <li>
      The container spawning entity (either directly or through the CMS that
      manages the underlying infrastructure) updates the OVN Northbound
      database to include the new CIF, by adding a row to the
      <code>Logical_Switch_Port</code> table.  In the new row,
      <code>name</code> is any unique identifier,
      <code>parent_name</code> is the <var>vif-id</var> of the VM
      through which the CIF's network traffic is expected to go through
      and the <code>tag</code> is the VLAN tag that identifies the
      network traffic of that CIF.
    </li>

    <li>
      <code>ovn-northd</code> receives the OVN Northbound database update.  In
      turn, it makes the corresponding updates to the OVN Southbound database,
      by adding rows to the OVN Southbound database's <code>Logical_Flow</code>
      table to reflect the new port and also by creating a new row in the
      <code>Binding</code> table and populating all its columns except the
      column that identifies the <code>chassis</code>.
    </li>

    <li>
      On every hypervisor, <code>ovn-controller</code> subscribes to the
      changes in the <code>Binding</code> table.  When a new row is created
      by <code>ovn-northd</code> that includes a value in
      <code>parent_port</code> column of <code>Binding</code> table, the
      <code>ovn-controller</code> in the hypervisor whose OVN integration bridge
      has that same value in <var>vif-id</var> in
      <code>external_ids</code>:<code>iface-id</code>
      updates the local hypervisor's OpenFlow tables so that packets to and
      from the VIF with the particular VLAN <code>tag</code> are properly
      handled.  Afterward it updates the <code>chassis</code> column of
      the <code>Binding</code> to reflect the physical location.
    </li>

    <li>
      One can only start the application inside the container after the
      underlying network is ready.  To support this, <code>ovn-northd</code>
      notices the updated <code>chassis</code> column in <code>Binding</code>
      table and updates the <ref column="up" table="Logical_Switch_Port"
      db="OVN_NB"/> column in the OVN Northbound database's
      <ref table="Logical_Switch_Port" db="OVN_NB"/> table to indicate that the
      CIF is now up.  The entity responsible to start the container application
      queries this value and starts the application.
    </li>

    <li>
      Eventually the entity that created and started the container, stops it.
      The entity, through the CMS (or directly) deletes its row in the
      <code>Logical_Switch_Port</code> table.
    </li>

    <li>
      <code>ovn-northd</code> receives the OVN Northbound update and in turn
      updates the OVN Southbound database accordingly, by removing or updating
      the rows from the OVN Southbound database <code>Logical_Flow</code> table
      that were related to the now-destroyed CIF.  It also deletes the row in
      the <code>Binding</code> table for that CIF.
    </li>

    <li>
      On every hypervisor, <code>ovn-controller</code> receives the
      <code>Logical_Flow</code> table updates that <code>ovn-northd</code> made
      in the previous step.  <code>ovn-controller</code> updates OpenFlow
      tables to reflect the update.
    </li>
  </ol>

  <h2>Architectural Physical Life Cycle of a Packet</h2>

  <p>
    This section describes how a packet travels from one virtual machine or
    container to another through OVN.  This description focuses on the physical
    treatment of a packet; for a description of the logical life cycle of a
    packet, please refer to the <code>Logical_Flow</code> table in
    <code>ovn-sb</code>(5).
  </p>

  <p>
    This section mentions several data and metadata fields, for clarity
    summarized here:
  </p>

  <dl>
    <dt>tunnel key</dt>
    <dd>
      When OVN encapsulates a packet in Geneve or another tunnel, it attaches
      extra data to it to allow the receiving OVN instance to process it
      correctly.  This takes different forms depending on the particular
      encapsulation, but in each case we refer to it here as the ``tunnel
      key.''  See <code>Tunnel Encapsulations</code>, below, for details.
    </dd>

    <dt>logical datapath field</dt>
    <dd>
      A field that denotes the logical datapath through which a packet is being
      processed.
      <!-- Keep the following in sync with MFF_LOG_DATAPATH in
           ovn/lib/logical-fields.h. -->
      OVN uses the field that OpenFlow 1.1+ simply (and confusingly) calls
      ``metadata'' to store the logical datapath.  (This field is passed across
      tunnels as part of the tunnel key.)
    </dd>

    <dt>logical input port field</dt>
    <dd>
      <p>
        A field that denotes the logical port from which the packet
        entered the logical datapath.
        <!-- Keep the following in sync with MFF_LOG_INPORT in
             ovn/lib/logical-fields.h. -->
        OVN stores this in Open vSwitch extension register number 14.
      </p>

      <p>
        Geneve and STT tunnels pass this field as part of the tunnel key.
        Although VXLAN tunnels do not explicitly carry a logical input port,
        OVN only uses VXLAN to communicate with gateways that from OVN's
        perspective consist of only a single logical port, so that OVN can set
        the logical input port field to this one on ingress to the OVN logical
        pipeline.
      </p>
    </dd>

    <dt>logical output port field</dt>
    <dd>
      <p>
        A field that denotes the logical port from which the packet will
        leave the logical datapath.  This is initialized to 0 at the
        beginning of the logical ingress pipeline.
        <!-- Keep the following in sync with MFF_LOG_OUTPORT in
             ovn/lib/logical-fields.h. -->
        OVN stores this in Open vSwitch extension register number 15.
      </p>

      <p>
        Geneve and STT tunnels pass this field as part of the tunnel key.
        VXLAN tunnels do not transmit the logical output port field.
        Since VXLAN tunnels do not carry a logical output port field in
        the tunnel key, when a packet is received from VXLAN tunnel by
        an OVN hypervisor, the packet is resubmitted to table 8 to
        determine the output port(s);  when the packet reaches table 32,
        these packets are resubmitted to table 33 for local delivery by
        checking a MLF_RCV_FROM_VXLAN flag, which is set when the packet
        arrives from a VXLAN tunnel.
      </p>
    </dd>

    <dt>conntrack zone field for logical ports</dt>
    <dd>
      A field that denotes the connection tracking zone for logical ports.
      The value only has local significance and is not meaningful between
      chassis.  This is initialized to 0 at the beginning of the logical
        <!-- Keep the following in sync with MFF_LOG_CT_ZONE in
             ovn/lib/logical-fields.h. -->
      ingress pipeline.  OVN stores this in Open vSwitch extension register
      number 13.
    </dd>

    <dt>conntrack zone fields for routers</dt>
    <dd>
      Fields that denote the connection tracking zones for routers.  These
      values only have local significance and are not meaningful between
      chassis.  OVN stores the zone information for DNATting in Open vSwitch
        <!-- Keep the following in sync with MFF_LOG_DNAT_ZONE and
        MFF_LOG_SNAT_ZONE in ovn/lib/logical-fields.h. -->
      extension register number 11 and zone information for SNATing in
      Open vSwitch extension register number 12.
    </dd>

    <dt>logical flow flags</dt>
    <dd>
      The logical flags are intended to handle keeping context between
      tables in order to decide which rules in subsequent tables are
      matched.  These values only have local significance and are not
      meaningful between chassis.  OVN stores the logical flags in
        <!-- Keep the following in sync with MFF_LOG_FLAGS in
             ovn/lib/logical-fields.h. -->
      Open vSwitch extension register number 10.
    </dd>

    <dt>VLAN ID</dt>
    <dd>
      The VLAN ID is used as an interface between OVN and containers nested
      inside a VM (see <code>Life Cycle of a container interface inside a
      VM</code>, above, for more information).
    </dd>
  </dl>

  <p>
    Initially, a VM or container on the ingress hypervisor sends a packet on a
    port attached to the OVN integration bridge.  Then:
  </p>

  <ol>
    <li>
      <p>
        OpenFlow table 0 performs physical-to-logical translation.  It matches
        the packet's ingress port.  Its actions annotate the packet with
        logical metadata, by setting the logical datapath field to identify the
        logical datapath that the packet is traversing and the logical input
        port field to identify the ingress port.  Then it resubmits to table 8
        to enter the logical ingress pipeline.
      </p>

      <p>
        Packets that originate from a container nested within a VM are treated
        in a slightly different way.  The originating container can be
        distinguished based on the VIF-specific VLAN ID, so the
        physical-to-logical translation flows additionally match on VLAN ID and
        the actions strip the VLAN header.  Following this step, OVN treats
        packets from containers just like any other packets.
      </p>

      <p>
        Table 0 also processes packets that arrive from other chassis.  It
        distinguishes them from other packets by ingress port, which is a
        tunnel.  As with packets just entering the OVN pipeline, the actions
        annotate these packets with logical datapath and logical ingress port
        metadata.  In addition, the actions set the logical output port field,
        which is available because in OVN tunneling occurs after the logical
        output port is known.  These three pieces of information are obtained
        from the tunnel encapsulation metadata (see <code>Tunnel
        Encapsulations</code> for encoding details).  Then the actions resubmit
        to table 33 to enter the logical egress pipeline.
      </p>
    </li>

    <li>
      <p>
        OpenFlow tables 8 through 31 execute the logical ingress pipeline from
        the <code>Logical_Flow</code> table in the OVN Southbound database.
        These tables are expressed entirely in terms of logical concepts like
        logical ports and logical datapaths.  A big part of
        <code>ovn-controller</code>'s job is to translate them into equivalent
        OpenFlow (in particular it translates the table numbers:
        <code>Logical_Flow</code> tables 0 through 23 become OpenFlow tables 8
        through 31).
      </p>

      <p>
        Each logical flow maps to one or more OpenFlow flows.  An actual packet
        ordinarily matches only one of these, although in some cases it can
        match more than one of these flows (which is not a problem because all
        of them have the same actions).  <code>ovn-controller</code> uses the
        first 32 bits of the logical flow's UUID as the cookie for its OpenFlow
        flow or flows.  (This is not necessarily unique, since the first 32
        bits of a logical flow's UUID is not necessarily unique.)
      </p>

      <p>
        Some logical flows can map to the Open vSwitch ``conjunctive match''
        extension (see <code>ovs-fields</code>(7)).  Flows with a
        <code>conjunction</code> action use an OpenFlow cookie of 0, because
        they can correspond to multiple logical flows.  The OpenFlow flow for a
        conjunctive match includes a match on <code>conj_id</code>.
      </p>

      <p>
        Some logical flows may not be represented in the OpenFlow tables on a
        given hypervisor, if they could not be used on that hypervisor.  For
        example, if no VIF in a logical switch resides on a given hypervisor,
        and the logical switch is not otherwise reachable on that hypervisor
        (e.g. over a series of hops through logical switches and routers
        starting from a VIF on the hypervisor), then the logical flow may not
        be represented there.
      </p>

      <p>
        Most OVN actions have fairly obvious implementations in OpenFlow (with
        OVS extensions), e.g. <code>next;</code> is implemented as
        <code>resubmit</code>, <code><var>field</var> =
        <var>constant</var>;</code> as <code>set_field</code>.  A few are worth
        describing in more detail:
      </p>

      <dl>
        <dt><code>output:</code></dt>
        <dd>
          Implemented by resubmitting the packet to table 32.  If the pipeline
          executes more than one <code>output</code> action, then each one is
          separately resubmitted to table 32.  This can be used to send
          multiple copies of the packet to multiple ports.  (If the packet was
          not modified between the <code>output</code> actions, and some of the
          copies are destined to the same hypervisor, then using a logical
          multicast output port would save bandwidth between hypervisors.)
        </dd>

        <dt><code>get_arp(<var>P</var>, <var>A</var>);</code></dt>
        <dt><code>get_nd(<var>P</var>, <var>A</var>);</code></dt>
        <dd>
          <p>
            Implemented by storing arguments into OpenFlow fields, then
            resubmitting to table 66, which <code>ovn-controller</code>
            populates with flows generated from the <code>MAC_Binding</code>
            table in the OVN Southbound database.  If there is a match in table
            66, then its actions store the bound MAC in the Ethernet
            destination address field.
          </p>

          <p>
            (The OpenFlow actions save and restore the OpenFlow fields used for
            the arguments, so that the OVN actions do not have to be aware of
            this temporary use.)
          </p>
        </dd>

        <dt><code>put_arp(<var>P</var>, <var>A</var>, <var>E</var>);</code></dt>
        <dt><code>put_nd(<var>P</var>, <var>A</var>, <var>E</var>);</code></dt>
        <dd>
          <p>
            Implemented by storing the arguments into OpenFlow fields, then
            outputting a packet to <code>ovn-controller</code>, which updates
            the <code>MAC_Binding</code> table.
          </p>

          <p>
            (The OpenFlow actions save and restore the OpenFlow fields used for
            the arguments, so that the OVN actions do not have to be aware of
            this temporary use.)
          </p>
        </dd>
      </dl>
    </li>

    <li>
      <p>
        OpenFlow tables 32 through 47 implement the <code>output</code> action
        in the logical ingress pipeline.  Specifically, table 32 handles
        packets to remote hypervisors, table 33 handles packets to the local
        hypervisor, and table 34 checks whether packets whose logical ingress
        and egress port are the same should be discarded.
      </p>

      <p>
        Logical patch ports are a special case.  Logical patch ports do not
        have a physical location and effectively reside on every hypervisor.
        Thus, flow table 33, for output to ports on the local hypervisor,
        naturally implements output to unicast logical patch ports too.
        However, applying the same logic to a logical patch port that is part
        of a logical multicast group yields packet duplication, because each
        hypervisor that contains a logical port in the multicast group will
        also output the packet to the logical patch port.  Thus, multicast
        groups implement output to logical patch ports in table 32.
      </p>

      <p>
        Each flow in table 32 matches on a logical output port for unicast or
        multicast logical ports that include a logical port on a remote
        hypervisor.  Each flow's actions implement sending a packet to the port
        it matches.  For unicast logical output ports on remote hypervisors,
        the actions set the tunnel key to the correct value, then send the
        packet on the tunnel port to the correct hypervisor.  (When the remote
        hypervisor receives the packet, table 0 there will recognize it as a
        tunneled packet and pass it along to table 33.)  For multicast logical
        output ports, the actions send one copy of the packet to each remote
        hypervisor, in the same way as for unicast destinations.  If a
        multicast group includes a logical port or ports on the local
        hypervisor, then its actions also resubmit to table 33.  Table 32 also
        includes:
      </p>

      <ul>
        <li>
          A higher-priority rule to match packets received from VXLAN tunnels,
          based on flag MLF_RCV_FROM_VXLAN, and resubmit these packets to table
          33 for local delivery.  Packets received from VXLAN tunnels reach
          here because of a lack of logical output port field in the tunnel key
          and thus these packets needed to be submitted to table 8 to
          determine the output port.
        </li>
        <li>
          A higher-priority rule to match packets received from ports of type
          <code>localport</code>, based on the logical input port, and resubmit
          these packets to table 33 for local delivery.  Ports of type
          <code>localport</code> exist on every hypervisor and by definition
          their traffic should never go out through a tunnel.
        </li>
        <li>
          A higher-priority rule to match packets that have the MLF_LOCAL_ONLY
          logical flow flag set, and whose destination is a multicast address.
          This flag indicates that the packet should not be delivered to remote
          hypervisors, even if the multicast destination includes ports on
          remote hypervisors. This flag is used when
          <code>ovn-controller</code> is the originator of the multicast packet.
          Since each <code>ovn-controller</code> instance is originating these
          packets, the packets only need to be delivered to local ports.
        </li>
        <li>
          A fallback flow that resubmits to table 33 if there is no other
          match.
        </li>
      </ul>

      <p>
        Flows in table 33 resemble those in table 32 but for logical ports that
        reside locally rather than remotely.  For unicast logical output ports
        on the local hypervisor, the actions just resubmit to table 34.  For
        multicast output ports that include one or more logical ports on the
        local hypervisor, for each such logical port <var>P</var>, the actions
        change the logical output port to <var>P</var>, then resubmit to table
        34.
      </p>

      <p>
        A special case is that when a localnet port exists on the datapath,
        remote port is connected by switching to the localnet port. In this
        case, instead of adding a flow in table 32 to reach the remote port, a
        flow is added in table 33 to switch the logical outport to the localnet
        port, and resubmit to table 33 as if it were unicasted to a logical
        port on the local hypervisor.
      </p>

      <p>
        Table 34 matches and drops packets for which the logical input and
        output ports are the same and the MLF_ALLOW_LOOPBACK flag is not
        set.  It resubmits other packets to table 40.
      </p>
    </li>

    <li>
      <p>
        OpenFlow tables 40 through 63 execute the logical egress pipeline from
        the <code>Logical_Flow</code> table in the OVN Southbound database.
        The egress pipeline can perform a final stage of validation before
        packet delivery.  Eventually, it may execute an <code>output</code>
        action, which <code>ovn-controller</code> implements by resubmitting to
        table 64.  A packet for which the pipeline never executes
        <code>output</code> is effectively dropped (although it may have been
        transmitted through a tunnel across a physical network).
      </p>

      <p>
        The egress pipeline cannot change the logical output port or cause
        further tunneling.
      </p>
    </li>

    <li>
     <p>
       Table 64 bypasses OpenFlow loopback when MLF_ALLOW_LOOPBACK is set.
       Logical loopback was handled in table 34, but OpenFlow by default also
       prevents loopback to the OpenFlow ingress port.  Thus, when
       MLF_ALLOW_LOOPBACK is set, OpenFlow table 64 saves the OpenFlow ingress
       port, sets it to zero, resubmits to table 65 for logical-to-physical
       transformation, and then restores the OpenFlow ingress port,
       effectively disabling OpenFlow loopback prevents.  When
       MLF_ALLOW_LOOPBACK is unset, table 64 flow simply resubmits to table
       65.
     </p>
    </li>

    <li>
      <p>
        OpenFlow table 65 performs logical-to-physical translation, the
        opposite of table 0.  It matches the packet's logical egress port.  Its
        actions output the packet to the port attached to the OVN integration
        bridge that represents that logical port.  If the logical egress port
        is a container nested with a VM, then before sending the packet the
        actions push on a VLAN header with an appropriate VLAN ID.
      </p>
    </li>
  </ol>

  <h2>Logical Routers and Logical Patch Ports</h2>

  <p>
    Typically logical routers and logical patch ports do not have a
    physical location and effectively reside on every hypervisor.  This is
    the case for logical patch ports between logical routers and logical
    switches behind those logical routers, to which VMs (and VIFs) attach.
  </p>

  <p>
    Consider a packet sent from one virtual machine or container to another
    VM or container that resides on a different subnet.  The packet will
    traverse tables 0 to 65 as described in the previous section
    <code>Architectural Physical Life Cycle of a Packet</code>, using the
    logical datapath representing the logical switch that the sender is
    attached to.  At table 32, the packet will use the fallback flow that
    resubmits locally to table 33 on the same hypervisor.  In this case,
    all of the processing from table 0 to table 65 occurs on the hypervisor
    where the sender resides.
  </p>

  <p>
    When the packet reaches table 65, the logical egress port is a logical
    patch port.  The implementation in table 65 differs depending on the OVS
    version, although the observed behavior is meant to be the same:
  </p>

  <ul>
    <li>
      In OVS versions 2.6 and earlier, table 65 outputs to an OVS patch
      port that represents the logical patch port.  The packet re-enters
      the OpenFlow flow table from the OVS patch port's peer in table 0,
      which identifies the logical datapath and logical input port based
      on the OVS patch port's OpenFlow port number.
    </li>

    <li>
      In OVS versions 2.7 and later, the packet is cloned and resubmitted
      directly to the first OpenFlow flow table in the ingress pipeline,
      setting the logical ingress port to the peer logical patch port, and
      using the peer logical patch port's logical datapath (that
      represents the logical router).
    </li>
  </ul>

  <p>
    The packet re-enters the ingress pipeline in order to traverse tables
    8 to 65 again, this time using the logical datapath representing the
    logical router.  The processing continues as described in the previous
    section <code>Architectural Physical Life Cycle of a Packet</code>.
    When the packet reachs table 65, the logical egress port will once
    again be a logical patch port.  In the same manner as described above,
    this logical patch port will cause the packet to be resubmitted to
    OpenFlow tables 8 to 65, this time using the logical datapath
    representing the logical switch that the destination VM or container
    is attached to.
  </p>

  <p>
    The packet traverses tables 8 to 65 a third and final time.  If the
    destination VM or container resides on a remote hypervisor, then table
    32 will send the packet on a tunnel port from the sender's hypervisor
    to the remote hypervisor.  Finally table 65 will output the packet
    directly to the destination VM or container.
  </p>

  <p>
    The following sections describe two exceptions, where logical routers
    and/or logical patch ports are associated with a physical location.
  </p>

  <h3>Gateway Routers</h3>

  <p>
    A <dfn>gateway router</dfn> is a logical router that is bound to a
    physical location.  This includes all of the logical patch ports of
    the logical router, as well as all of the peer logical patch ports on
    logical switches.  In the OVN Southbound database, the
    <code>Port_Binding</code> entries for these logical patch ports use
    the type <code>l3gateway</code> rather than <code>patch</code>, in
    order to distinguish that these logical patch ports are bound to a
    chassis.
  </p>

  <p>
    When a hypervisor processes a packet on a logical datapath
    representing a logical switch, and the logical egress port is a
    <code>l3gateway</code> port representing connectivity to a gateway
    router, the packet will match a flow in table 32 that sends the
    packet on a tunnel port to the chassis where the gateway router
    resides.  This processing in table 32 is done in the same manner as
    for VIFs.
  </p>

  <p>
    Gateway routers are typically used in between distributed logical
    routers and physical networks.  The distributed logical router and
    the logical switches behind it, to which VMs and containers attach,
    effectively reside on each hypervisor.  The distributed router and
    the gateway router are connected by another logical switch, sometimes
    referred to as a <code>join</code> logical switch.  On the other
    side, the gateway router connects to another logical switch that has
    a localnet port connecting to the physical network.
  </p>

  <p>
    When using gateway routers, DNAT and SNAT rules are associated with
    the gateway router, which provides a central location that can handle
    one-to-many SNAT (aka IP masquerading).
  </p>

  <h3>Distributed Gateway Ports</h3>

  <p>
    <dfn>Distributed gateway ports</dfn> are logical router patch ports
    that directly connect distributed logical routers to logical
    switches with localnet ports.
  </p>

  <p>
    The primary design goal of distributed gateway ports is to allow as
    much traffic as possible to be handled locally on the hypervisor
    where a VM or container resides.  Whenever possible, packets from
    the VM or container to the outside world should be processed
    completely on that VM's or container's hypervisor, eventually
    traversing a localnet port instance on that hypervisor to the
    physical network.  Whenever possible, packets from the outside
    world to a VM or container should be directed through the physical
    network directly to the VM's or container's hypervisor, where the
    packet will enter the integration bridge through a localnet port.
  </p>

  <p>
    In order to allow for the distributed processing of packets
    described in the paragraph above, distributed gateway ports need to
    be logical patch ports that effectively reside on every hypervisor,
    rather than <code>l3gateway</code> ports that are bound to a
    particular chassis.  However, the flows associated with distributed
    gateway ports often need to be associated with physical locations,
    for the following reasons:
  </p>

  <ul>
    <li>
      <p>
        The physical network that the localnet port is attached to
        typically uses L2 learning.  Any Ethernet address used over the
        distributed gateway port must be restricted to a single physical
        location so that upstream L2 learning is not confused.  Traffic
        sent out the distributed gateway port towards the localnet port
        with a specific Ethernet address must be sent out one specific
        instance of the distributed gateway port on one specific
        chassis.  Traffic received from the localnet port (or from a VIF
        on the same logical switch as the localnet port) with a specific
        Ethernet address must be directed to the logical switch's patch
        port instance on that specific chassis.
      </p>

      <p>
        Due to the implications of L2 learning, the Ethernet address and
        IP address of the distributed gateway port need to be restricted
        to a single physical location.  For this reason, the user must
        specify one chassis associated with the distributed gateway
        port.  Note that traffic traversing the distributed gateway port
        using other Ethernet addresses and IP addresses (e.g. one-to-one
        NAT) is not restricted to this chassis.
      </p>

      <p>
        Replies to ARP and ND requests must be restricted to a single
        physical location, where the Ethernet address in the reply
        resides.  This includes ARP and ND replies for the IP address
        of the distributed gateway port, which are restricted to the
        chassis that the user associated with the distributed gateway
        port.
      </p>
    </li>

    <li>
      In order to support one-to-many SNAT (aka IP masquerading), where
      multiple logical IP addresses spread across multiple chassis are
      mapped to a single external IP address, it will be necessary to
      handle some of the logical router processing on a specific chassis
      in a centralized manner.  Since the SNAT external IP address is
      typically the distributed gateway port IP address, and for
      simplicity, the same chassis associated with the distributed
      gateway port is used.
    </li>
  </ul>

  <p>
    The details of flow restrictions to specific chassis are described
    in the <code>ovn-northd</code> documentation.
  </p>

  <p>
    While most of the physical location dependent aspects of distributed
    gateway ports can be handled by restricting some flows to specific
    chassis, one additional mechanism is required.  When a packet
    leaves the ingress pipeline and the logical egress port is the
    distributed gateway port, one of two different sets of actions is
    required at table 32:
  </p>

  <ul>
    <li>
      If the packet can be handled locally on the sender's hypervisor
      (e.g. one-to-one NAT traffic), then the packet should just be
      resubmitted locally to table 33, in the normal manner for
      distributed logical patch ports.
    </li>

    <li>
      However, if the packet needs to be handled on the chassis
      associated with the distributed gateway port (e.g. one-to-many
      SNAT traffic or non-NAT traffic), then table 32 must send the
      packet on a tunnel port to that chassis.
    </li>
  </ul>

  <p>
    In order to trigger the second set of actions, the
    <code>chassisredirect</code> type of southbound
    <code>Port_Binding</code> has been added.  Setting the logical
    egress port to the type <code>chassisredirect</code> logical port is
    simply a way to indicate that although the packet is destined for
    the distributed gateway port, it needs to be redirected to a
    different chassis.  At table 32, packets with this logical egress
    port are sent to a specific chassis, in the same way that table 32
    directs packets whose logical egress port is a VIF or a type
    <code>l3gateway</code> port to different chassis.  Once the packet
    arrives at that chassis, table 33 resets the logical egress port to
    the value representing the distributed gateway port.  For each
    distributed gateway port, there is one type
    <code>chassisredirect</code> port, in addition to the distributed
    logical patch port representing the distributed gateway port.
  </p>

  <h3>High Availability for Distributed Gateway Ports</h3>

  <p>
    OVN allows you to specify a prioritized list of chassis for a distributed
    gateway port.  This is done by associating multiple
    <code>Gateway_Chassis</code> rows with a <code>Logical_Router_Port</code>
    in the <code>OVN_Northbound</code> database.
  </p>

  <p>
    When multiple chassis have been specified for a gateway, all chassis that
    may send packets to that gateway will enable BFD on tunnels to all
    configured gateway chassis.  The current master chassis for the gateway
    is the highest priority gateway chassis that is currently viewed as
    active based on BFD status.
  </p>

  <p>
    For more information on L3 gateway high availability, please refer to
    http://docs.openvswitch.org/en/latest/topics/high-availability.
  </p>

  <h2>Multiple localnet logical switches connected to a Logical Router</h2>

  <p>
    It is possible to have multiple logical switches each with a localnet port
    (representing physical networks) connected to a logical router, in which
    one localnet logical switch may provide the external connectivity via a
    distributed gateway port and rest of the localnet logical switches use
    VLAN tagging in the physical network. It is expected that
    <code>ovn-bridge-mappings</code> is configured appropriately on the
    chassis for all these localnet networks.
  </p>

  <h3>East West routing</h3>
  <p>
    East-West routing between these localnet VLAN tagged logical switches
    work almost the same way as normal logical switches. When the VM sends
    such a packet, then:
  </p>
  <ol>
    <li>
      It first enters the ingress pipeline, and then egress pipeline of the
      source localnet logical switch datapath. It then enters the ingress
      pipeline of the logical router datapath via the logical router port in
      the source chassis.
    </li>

    <li>
      Routing decision is taken.
    </li>

    <li>
      <p>
        From the router datapath, packet enters the ingress pipeline and then
        egress pipeline of the destination localnet logical switch datapath
        and goes out of the integration bridge to the provider bridge (
        belonging to the destination logical switch) via the localnet port.
        While sending the packet to provider bridge, we also replace router
        port MAC as source MAC with a chassis unique MAC.
      </p>

      <p>
        This chassis unique MAC is configured as global ovs config on each
        chassis (eg. via "<code>ovs-vsctl set open . external-ids:
        ovn-chassis-mac-mappings="phys:aa:bb:cc:dd:ee:$i$i"</code>").  For more
        details, see <code>ovn-controller</code>(8).
      </p>

      <p>
        If the above is not configured, then source MAC would be the router
        port MAC.  This could create problem if we have more than one chassis.
        This is because, since the router port is distributed, the same
        (MAC,VLAN) tuple will seen by physical network from other chassis as
        well, which could cause these issues:
      </p>
      
      <ul>
        <li>
          Continuous MAC moves in top-of-rack switch (ToR).
        </li>
        <li>
          ToR dropping the traffic, which is causing continuous MAC moves.
        </li>
        <li>
          ToR blocking the ports from which MAC moves are happening.
        </li>
      </ul>
    </li>

    <li>
      The destination chassis receives the packet via the localnet port and
      sends it to the integration bridge. The packet enters the
      ingress pipeline and then egress pipeline of the destination localnet
      logical switch and finally gets delivered to the destination VM port.
    </li>
  </ol>

  <h3>External traffic</h3>

  <p>
    The following happens when a VM sends an external traffic (which requires
    NATting) and the chassis hosting the VM doesn't have a distributed gateway
    port.
  </p>

  <ol>
    <li>
      The packet first enters the ingress pipeline, and then egress pipeline of
      the source localnet logical switch datapath. It then enters the ingress
      pipeline of the logical router datapath via the logical router port in
      the source chassis.
    </li>

    <li>
      Routing decision is taken. Since the gateway router or the distributed
      gateway port doesn't reside in the source chassis, the traffic is
      redirected to the gateway chassis via the tunnel port.
    </li>

    <li>
      The gateway chassis receives the packet via the tunnel port and the
      packet enters the egress pipeline of the logical router datapath. NAT
      rules are applied here. The packet then enters the ingress pipeline and
      then egress pipeline of the localnet logical switch datapath which
      provides external connectivity and finally goes out via the localnet
      port of the logical switch which provides external connectivity.
    </li>
  </ol>

  <p>
    Although this works, the VM traffic is tunnelled when sent from the compute
    chassis to the gateway chassis. In order for it to work properly, the MTU
    of the localnet logical switches must be lowered to account for the tunnel
    encapsulation.
  </p>

  <h2>
    Centralized routing for localnet VLAN tagged logical switches connected
    to a Logical Router
  </h2>

  <p>
    To overcome the tunnel encapsulation problem described in the previous
    section, <code>OVN</code> supports the option of enabling centralized
    routing for localnet VLAN tagged logical switches. CMS can configure the
    option <ref column="options:reside-on-redirect-chassis"
    table="Logical_Router_Port" db="OVN_NB"/> to <code>true</code> for each
    <ref table="Logical_Router_Port" db="OVN_NB"/> which connects to the
    localnet VLAN tagged logical switches. This causes the gateway
    chassis (hosting the distributed gateway port) to handle all the
    routing for these networks, making it centralized. It will reply to
    the ARP requests for the logical router port IPs.
  </p>

  <p>
    If the logical router doesn't have a distributed gateway port connecting
    to the localnet logical switch which provides external connectivity,
    then this option is ignored by <code>OVN</code>.
  </p>

  <p>
    The following happens when a VM sends an east-west traffic which needs to
    be routed:
  </p>

  <ol>
    <li>
      The packet first enters the ingress pipeline, and then egress pipeline of
      the source localnet logical switch datapath and is sent out via the
      localnet port of the source localnet logical switch (instead of sending
      it to router pipeline).
    </li>

    <li>
      The gateway chassis receives the packet via the localnet port of the
      source localnet logical switch and sends it to the integration bridge.
      The packet then enters the ingress pipeline, and then egress pipeline of
      the source localnet logical switch datapath and enters the ingress
      pipeline of the logical router datapath.
    </li>

    <li>
      Routing decision is taken.
    </li>

    <li>
      From the router datapath, packet enters the ingress pipeline and then
      egress pipeline of the destination localnet logical switch datapath.
      It then goes out of the integration bridge to the provider bridge (
      belonging to the destination logical switch) via the localnet port.
    </li>

    <li>
      The destination chassis receives the packet via the localnet port and
      sends it to the integration bridge. The packet enters the
      ingress pipeline and then egress pipeline of the destination localnet
      logical switch and finally delivered to the destination VM port.
    </li>
  </ol>

  <p>
    The following happens when a VM sends an external traffic which requires
    NATting:
  </p>

  <ol>
    <li>
      The packet first enters the ingress pipeline, and then egress pipeline of
      the source localnet logical switch datapath and is sent out via the
      localnet port of the source localnet logical switch (instead of sending
      it to router pipeline).
    </li>

    <li>
      The gateway chassis receives the packet via the localnet port of the
      source localnet logical switch and sends it to the integration bridge.
      The packet then enters the ingress pipeline, and then egress pipeline of
      the source localnet logical switch datapath and enters the ingress
      pipeline of the logical router datapath.
    </li>

    <li>
      Routing decision is taken and NAT rules are applied.
    </li>

    <li>
      From the router datapath, packet enters the ingress pipeline and then
      egress pipeline of the localnet logical switch datapath which provides
      external connectivity. It then goes out of the integration bridge to the
      provider bridge (belonging to the logical switch which provides external
      connectivity) via the localnet port.
    </li>
  </ol>

  <p>
    The following happens for the reverse external traffic.
  </p>

  <ol>
    <li>
      The gateway chassis receives the packet from the localnet port of
      the logical switch which provides external connectivity. The packet then
      enters the ingress pipeline and then egress pipeline of the localnet
      logical switch (which provides external connectivity). The packet then
      enters the ingress pipeline of the logical router datapath.
    </li>

    <li>
      The ingress pipeline of the logical router datapath applies the unNATting
      rules. The packet then enters the ingress pipeline and then egress
      pipeline of the source localnet logical switch. Since the source VM
      doesn't reside in the gateway chassis, the packet is sent out via the
      localnet port of the source logical switch.
    </li>

    <li>
      The source chassis receives the packet via the localnet port and
      sends it to the integration bridge. The packet enters the
      ingress pipeline and then egress pipeline of the source localnet
      logical switch and finally gets delivered to the source VM port.
    </li>
  </ol>

  <h2>Life Cycle of a VTEP gateway</h2>

  <p>
    A gateway is a chassis that forwards traffic between the OVN-managed
    part of a logical network and a physical VLAN,  extending a
    tunnel-based logical network into a physical network.
  </p>

  <p>
    The steps below refer often to details of the OVN and VTEP database
    schemas.  Please see <code>ovn-sb</code>(5), <code>ovn-nb</code>(5)
    and <code>vtep</code>(5), respectively, for the full story on these
    databases.
  </p>

  <ol>
    <li>
      A VTEP gateway's life cycle begins with the administrator registering
      the VTEP gateway as a <code>Physical_Switch</code> table entry in the
      <code>VTEP</code> database.  The <code>ovn-controller-vtep</code>
      connected to this VTEP database, will recognize the new VTEP gateway
      and create a new <code>Chassis</code> table entry for it in the
      <code>OVN_Southbound</code> database.
    </li>

    <li>
      The administrator can then create a new <code>Logical_Switch</code>
      table entry, and bind a particular vlan on a VTEP gateway's port to
      any VTEP logical switch.  Once a VTEP logical switch is bound to
      a VTEP gateway, the <code>ovn-controller-vtep</code> will detect
      it and add its name to the <var>vtep_logical_switches</var>
      column of the <code>Chassis</code> table in the <code>
      OVN_Southbound</code> database.  Note, the <var>tunnel_key</var>
      column of VTEP logical switch is not filled at creation.  The
      <code>ovn-controller-vtep</code> will set the column when the
      correponding vtep logical switch is bound to an OVN logical network.
    </li>

    <li>
      Now, the administrator can use the CMS to add a VTEP logical switch
      to the OVN logical network.  To do that, the CMS must first create a
      new <code>Logical_Switch_Port</code> table entry in the <code>
      OVN_Northbound</code> database.  Then, the <var>type</var> column
      of this entry must be set to "vtep".  Next, the <var>
      vtep-logical-switch</var> and <var>vtep-physical-switch</var> keys
      in the <var>options</var> column must also be specified, since
      multiple VTEP gateways can attach to the same VTEP logical switch.
    </li>

    <li>
      The newly created logical port in the <code>OVN_Northbound</code>
      database and its configuration will be passed down to the <code>
      OVN_Southbound</code> database as a new <code>Port_Binding</code>
      table entry.  The <code>ovn-controller-vtep</code> will recognize the
      change and bind the logical port to the corresponding VTEP gateway
      chassis.  Configuration of binding the same VTEP logical switch to
      a different OVN logical networks is not allowed and a warning will be
      generated in the log.
    </li>

    <li>
      Beside binding to the VTEP gateway chassis, the <code>
      ovn-controller-vtep</code> will update the <var>tunnel_key</var>
      column of the VTEP logical switch to the corresponding <code>
      Datapath_Binding</code> table entry's <var>tunnel_key</var> for the
      bound OVN logical network.
    </li>

    <li>
      Next, the <code>ovn-controller-vtep</code> will keep reacting to the
      configuration change in the <code>Port_Binding</code> in the
      <code>OVN_Northbound</code> database, and updating the
      <code>Ucast_Macs_Remote</code> table in the <code>VTEP</code> database.
      This allows the VTEP gateway to understand where to forward the unicast
      traffic coming from the extended external network.
    </li>

    <li>
      Eventually, the VTEP gateway's life cycle ends when the administrator
      unregisters the VTEP gateway from the <code>VTEP</code> database.
      The <code>ovn-controller-vtep</code> will recognize the event and
      remove all related configurations (<code>Chassis</code> table entry
      and port bindings) in the <code>OVN_Southbound</code> database.
    </li>

    <li>
      When the <code>ovn-controller-vtep</code> is terminated, all related
      configurations in the <code>OVN_Southbound</code> database and
      the <code>VTEP</code> database will be cleaned, including
      <code>Chassis</code> table entries for all registered VTEP gateways
      and their port bindings, and all <code>Ucast_Macs_Remote</code> table
      entries and the <code>Logical_Switch</code> tunnel keys.
    </li>
  </ol>

  <h2>Native OVN services for external logical ports</h2>

  <p>
    To support OVN native services (like DHCP/IPv6 RA/DNS lookup) to the
    cloud resources which are external, OVN supports <code>external</code>
    logical ports.
  </p>

  <p>
    Below are some of the use cases where <code>external</code> ports can be
    used.
  </p>

  <ul>
    <li>
      VMs connected to SR-IOV nics - Traffic from these VMs by passes the
      kernel stack and local <code>ovn-controller</code> do not bind these
      ports and cannot serve the native services.
    </li>
    <li>
      When CMS supports provisioning baremetal servers.
    </li>
  </ul>

  <p>
    OVN will provide the native services if CMS has done the below
    configuration in the <dfn>OVN Northbound Database</dfn>.
  </p>

  <ul>
    <li>
      A row is created in <code>Logical_Switch_Port</code>, configuring the
      <ref column="addresses" table="Logical_Switch_Port" db="OVN_NB"/> column
      and setting the <ref column="type" table="Logical_Switch_Port"
      db="OVN_NB"/> to <code>external</code>.
    </li>

    <li>
      <ref column="ha_chassis_group" table="Logical_Switch_Port"
      db="OVN_NB"/> column is configured.
    </li>

    <li>
      The HA chassis which belongs to the HA chassis group has the
      <code>ovn-bridge-mappings</code> configured and has proper L2
      connectivity so that it can receive the DHCP and other related request
      packets from these external resources.
    </li>

    <li>
      The Logical_Switch of this port has a <code>localnet</code> port.
    </li>

    <li>
      Native OVN services are enabled by configuring the DHCP and other
      options like the way it is done for the normal logical ports.
    </li>
  </ul>

  <p>
    It is recommended to use the same HA chassis group for all the external
    ports of a logical switch. Otherwise, the physical switch might see MAC
    flap issue when different chassis provide the native services. For
    example when supporting native DHCPv4 service, DHCPv4 server mac
    (configured in <ref column="options:server_mac" table="DHCP_Options"
    db="OVN_NB"/> column in table <ref table="DHCP_Options"/>) originating
    from different ports can cause MAC flap issue.
    The MAC of the logical router IP(s) can also flap if the same HA chassis
    group is not set for all the external ports of a logical switch.
  </p>

  <h1>Security</h1>

  <h2>Role-Based Access Controls for the Soutbound DB</h2>
  <p>
    In order to provide additional security against the possibility of an OVN
    chassis becoming compromised in such a way as to allow rogue software to
    make arbitrary modifications to the southbound database state and thus
    disrupt the OVN network, role-based access controls (see
    <code>ovsdb-server(1)</code> for additional details) are provided for the
    southbound database.
  </p>

  <p>
    The implementation of role-based access controls (RBAC) requires the
    addition of two tables to an OVSDB schema: the <code>RBAC_Role</code>
    table, which is indexed by role name and maps the the names of the various
    tables that may be modifiable for a given role to individual rows in a
    permissions table containing detailed permission information for that role,
    and the permission table itself which consists of rows containing the
    following information:
  </p>
  <dl>
    <dt><code>Table Name</code></dt>
    <dd>
      The name of the associated table. This column exists primarily as an
      aid for humans reading the contents of this table.
    </dd>

    <dt><code>Auth Criteria</code></dt>
    <dd>
      A set of strings containing the names of columns (or column:key pairs
      for columns containing string:string maps).  The contents of at least
      one of the columns or column:key values in a row to be modified,
      inserted, or deleted must be equal to the ID of the client attempting
      to act on the row in order for the authorization check to pass. If the
      authorization criteria is empty, authorization checking is disabled and
      all clients for the role will be treated as authorized.
    </dd>

    <dt><code>Insert/Delete</code></dt>
    <dd>
       Row insertion/deletion permission; boolean value indicating whether
       insertion and deletion of rows is allowed for the associated table.
       If true, insertion and deletion of rows is allowed for authorized
       clients.
    </dd>

    <dt><code>Updatable Columns</code></dt>
    <dd>
      A set of strings containing the names of columns or column:key pairs
      that may be updated or mutated by authorized clients. Modifications to
      columns within a row are only permitted when the authorization check
      for the client passes and all columns to be modified are included in
      this set of modifiable columns.
    </dd>
  </dl>

  <p>
    RBAC configuration for the OVN southbound database is maintained by
    ovn-northd. With RBAC enabled, modifications are only permitted for the
    <code>Chassis</code>, <code>Encap</code>, <code>Port_Binding</code>, and
    <code>MAC_Binding</code> tables, and are resstricted as follows:
  </p>
  <dl>
    <dt><code>Chassis</code></dt>
    <dd>
      <p>
       <code>Authorization</code>: client ID must match the chassis name.
      </p>
      <p>
        <code>Insert/Delete</code>: authorized row insertion and deletion
        are permitted.
      </p>
      <p>
        <code>Update</code>: The columns <code>nb_cfg</code>,
        <code>external_ids</code>, <code>encaps</code>, and
        <code>vtep_logical_switches</code> may be modified when authorized.
      </p>
    </dd>

    <dt><code>Encap</code></dt>
    <dd>
      <p>
        <code>Authorization</code>: client ID must match the chassis name.
      </p>
      <p>
        <code>Insert/Delete</code>: row insertion and row deletion
        are permitted.
      </p>
      <p>
        <code>Update</code>: The columns <code>type</code>,
        <code>options</code>, and <code>ip</code> can be modified.
      </p>
    </dd>

    <dt><code>Port_Binding</code></dt>
    <dd>
      <p>
        <code>Authorization</code>: disabled (all clients are considered
        authorized. A future enhancement may add columns (or keys to
        <code>external_ids</code>) in order to control which chassis are
        allowed to bind each port.
      </p>
      <p>
        <code>Insert/Delete</code>: row insertion/deletion are not permitted
        (ovn-northd maintains rows in this table.
      </p>
      <p>
        <code>Update</code>: Only modifications to the <code>chassis</code>
        column are permitted.
      </p>
    </dd>

    <dt><code>MAC_Binding</code></dt>
    <dd>
      <p>
        <code>Authorization</code>: disabled (all clients are considered
        to be authorized).
      </p>
      <p>
        <code>Insert/Delete</code>: row insertion/deletion are permitted.
      </p>
      <p>
        <code>Update</code>: The columns <code>logical_port</code>,
        <code>ip</code>, <code>mac</code>, and <code>datapath</code> may be
        modified by ovn-controller.
      </p>
    </dd>
  </dl>

  <p>
    Enabling RBAC for ovn-controller connections to the southbound database
    requires the following steps:
  </p>

  <ol>
    <li>
      Creating SSL certificates for each chassis with the certificate CN field
      set to the chassis name (e.g. for a chassis with
      <code>external-ids:system-id=chassis-1</code>, via the command
      "<code>ovs-pki -u req+sign chassis-1 switch</code>").
    </li>
    <li>
      Configuring each ovn-controller to use SSL when connecting to the
      southbound database (e.g. via "<code>ovs-vsctl set open .
      external-ids:ovn-remote=ssl:x.x.x.x:6642</code>").
    </li>
    <li>
      Configuring a southbound database SSL remote with "ovn-controller" role
      (e.g. via "<code>ovn-sbctl set-connection role=ovn-controller
      pssl:6642</code>").
    </li>
  </ol>

  <h2>Encrypt Tunnel Traffic with IPsec</h2>
  <p>
    OVN tunnel traffic goes through physical routers and switches. These
    physical devices could be untrusted (devices in public network) or might be
    compromised. Enabling encryption to the tunnel traffic can prevent the
    traffic data from being monitored and manipulated.
  </p>
  <p>
    The tunnel traffic is encrypted with IPsec. The CMS sets the
    <code>ipsec</code> column in the northbound <code>NB_Global</code> table to
    enable or disable IPsec encrytion. If <code>ipsec</code> is true, all OVN
    tunnels will be encrypted. If <code>ipsec</code> is false, no OVN tunnels
    will be encrypted.
  </p>
  <p>
    When CMS updates the <code>ipsec</code> column in the northbound
    <code>NB_Global</code> table, <code>ovn-northd</code> copies the value to
    the <code>ipsec</code> column in the southbound <code>SB_Global</code>
    table. <code>ovn-controller</code> in each chassis monitors the southbound
    database and sets the options of the OVS tunnel interface accordingly. OVS
    tunnel interface options are monitored by the
    <code>ovs-monitor-ipsec</code> daemon which configures IKE daemon to set up
    IPsec connections.
  </p>
  <p>
    Chassis authenticates each other by using certificate. The authentication
    succeeds if the other end in tunnel presents a certificate signed by a
    trusted CA and the common name (CN) matches the expected chassis name.  The
    SSL certificates used in role-based access controls (RBAC) can be used in
    IPsec. Or use <code>ovs-pki</code> to create different certificates. The
    certificate is required to be x.509 version 3, and with CN field and
    subjectAltName field being set to the chassis name.
  </p>
  <p>
    The CA certificate, chassis certificate and private key are required to be
    installed in each chassis before enabling IPsec. Please see
    <code>ovs-vswitchd.conf.db</code>(5) for setting up CA based IPsec
    authentication.
  </p>
  <h1>Design Decisions</h1>

  <h2>Tunnel Encapsulations</h2>

  <p>
    OVN annotates logical network packets that it sends from one hypervisor to
    another with the following three pieces of metadata, which are encoded in
    an encapsulation-specific fashion:
  </p>

  <ul>
    <li>
      24-bit logical datapath identifier, from the <code>tunnel_key</code>
      column in the OVN Southbound <code>Datapath_Binding</code> table.
    </li>

    <li>
      15-bit logical ingress port identifier.  ID 0 is reserved for internal
      use within OVN.  IDs 1 through 32767, inclusive, may be assigned to
      logical ports (see the <code>tunnel_key</code> column in the OVN
      Southbound <code>Port_Binding</code> table).
    </li>

    <li>
      16-bit logical egress port identifier.  IDs 0 through 32767 have the same
      meaning as for logical ingress ports.  IDs 32768 through 65535,
      inclusive, may be assigned to logical multicast groups (see the
      <code>tunnel_key</code> column in the OVN Southbound
      <code>Multicast_Group</code> table).
    </li>
  </ul>

  <p>
    For hypervisor-to-hypervisor traffic, OVN supports only Geneve and STT
    encapsulations, for the following reasons:
  </p>

  <ul>
    <li>
      Only STT and Geneve support the large amounts of metadata (over 32 bits
      per packet) that OVN uses (as described above).
    </li>

    <li>
      STT and Geneve use randomized UDP or TCP source ports that allows
      efficient distribution among multiple paths in environments that use ECMP
      in their underlay.
    </li>

    <li>
      NICs are available to offload STT and Geneve encapsulation and
      decapsulation.
    </li>
  </ul>

  <p>
    Due to its flexibility, the preferred encapsulation between hypervisors is
    Geneve.  For Geneve encapsulation, OVN transmits the logical datapath
    identifier in the Geneve VNI.

    <!-- Keep the following in sync with ovn/controller/physical.h. -->
    OVN transmits the logical ingress and logical egress ports in a TLV with
    class 0x0102, type 0x80, and a 32-bit value encoded as follows, from MSB to
    LSB:
  </p>

  <diagram>
    <header name="">
      <bits name="rsv" above="1" below="0" width=".25"/>
      <bits name="ingress port" above="15" width=".75"/>
      <bits name="egress port" above="16" width=".75"/>
    </header>
  </diagram>

  <p>
    Environments whose NICs lack Geneve offload may prefer STT encapsulation
    for performance reasons.  For STT encapsulation, OVN encodes all three
    pieces of logical metadata in the STT 64-bit tunnel ID as follows, from MSB
    to LSB:
  </p>

  <diagram>
    <header name="">
      <bits name="reserved" above="9" below="0" width=".5"/>
      <bits name="ingress port" above="15" width=".75"/>
      <bits name="egress port" above="16" width=".75"/>
      <bits name="datapath" above="24" width="1.25"/>
    </header>
  </diagram>

  <p>
    For connecting to gateways, in addition to Geneve and STT, OVN supports
    VXLAN, because only VXLAN support is common on top-of-rack (ToR) switches.
    Currently, gateways have a feature set that matches the capabilities as
    defined by the VTEP schema, so fewer bits of metadata are necessary.  In
    the future, gateways that do not support encapsulations with large amounts
    of metadata may continue to have a reduced feature set.
  </p>
</manpage>