Improve error handling for TCP connections

In the abstract DNSService's _dns_handle_tcp method, error handling
is broken in a way that stops the main loop for handling TCP
connections.

Because socket.timeout is a subclass of socket.error, the error
handling block for socket.timeout is never reached.

Because of this, error handling of a TCP timeout is sent to the
socket.error block.  Due to the way eventlet hijacks these errors,
the errorcode is not available and a KeyError is raised.  This
KeyError interferes with the main loop because it is not caught.

Further improvement may include ensuring that these main loops
can never die due to unexpected exceptions.

Many thanks to Erik Andersson for pointing out the issue, which
was seemingly innocuous but ended up being the cause of our
problems.

Closes-bug: 1549980
Change-Id: I47e1260a0818cc42cbd56e4d296e083f8fcbbae5

1 files changed, 8 insertions, 5 deletions

diff --git a/designate/service.py b/designate/service.py
index 8ac4e73e..2a50b126 100644
--- a/designate/service.py
+++ b/designate/service.py
@@ -290,17 +290,20 @@ class DNSService(object):
                         break
                     payload += data
 
+            # NOTE: Any uncaught exceptions will result in the main loop
+            # ending unexpectedly. Ensure proper ordering of blocks, and
+            # ensure no exceptions are generated from within.
+            except socket.timeout:
+                client.close()
+                LOG.warn(_LW("TCP Timeout from: %(host)s:%(port)d") %
+                         {'host': addr[0], 'port': addr[1]})
+
             except socket.error as e:
                 client.close()
                 errname = errno.errorcode[e.args[0]]
                 LOG.warn(_LW("Socket error %(err)s from: %(host)s:%(port)d") %
                          {'host': addr[0], 'port': addr[1], 'err': errname})
 
-            except socket.timeout:
-                client.close()
-                LOG.warn(_LW("TCP Timeout from: %(host)s:%(port)d") %
-                         {'host': addr[0], 'port': addr[1]})
-
             except struct.error:
                 client.close()
                 LOG.warn(_LW("Invalid packet from: %(host)s:%(port)d") %