wait_for: treat broken connections as "unready" (#28839)

* wait_for: treat broken connections as "unready"

We have observed the following condition while waiting for hosts:

```
Traceback (most recent call last):
  File "/var/folders/f8/23xp00654plcv2b2tcc028680000gn/T/ansible_8hxm4_/ansible_module_wait_for.py", line 585, in <module>
    main()
  File "/var/folders/f8/23xp00654plcv2b2tcc028680000gn/T/ansible_8hxm4_/ansible_module_wait_for.py", line 535, in main
    s.shutdown(socket.SHUT_RDWR)
  File "/usr/local/opt/python/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 57] Socket is not connected
```

This appears to happen while the host is still starting; we believe something is
accepting our connection but immediately resetting it. In these cases, we'd
prefer to continue waiting instead of immediately failing the play.

This patch has been applied locally for some time, and we have seen no adverse
effects.

* wait_for: fixup change

We were missing an import and a space after the `#`

(cherry picked from commit 402b095841d901b012f20e9473bdee6b0f43a398)

2 files changed, 18 insertions, 4 deletions

diff --git a/CHANGELOG.md b/CHANGELOG.md
index c1cbb913fc..e46c688212 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -54,6 +54,7 @@ Ansible Changes By Release
 * Use safe_load instead on load to read a yaml document
 * Fix for win_file to respect check mode when deleting directories
 * Include_role now complains about invalid arguments
+* Added socket conditions to ignore for wait_for, no need to error for closing already closed connection
 
 
 <a id="2.3.2"></a>
diff --git a/lib/ansible/modules/utilities/logic/wait_for.py b/lib/ansible/modules/utilities/logic/wait_for.py
index c226c0b6bc..0b93f31091 100644
--- a/lib/ansible/modules/utilities/logic/wait_for.py
+++ b/lib/ansible/modules/utilities/logic/wait_for.py
@@ -160,6 +160,7 @@ EXAMPLES = '''
 
 import binascii
 import datetime
+import errno
 import math
 import os
 import re
@@ -549,15 +550,27 @@ def main():
                                 break
 
                         # Shutdown the client socket
-                        s.shutdown(socket.SHUT_RDWR)
-                        s.close()
+                        try:
+                            s.shutdown(socket.SHUT_RDWR)
+                        except socket.error as e:
+                            if e.errno != errno.ENOTCONN:
+                                raise
+                        # else, the server broke the connection on its end, assume it's not ready
+                        else:
+                            s.close()
                         if matched:
                             # Found our string, success!
                             break
                     else:
                         # Connection established, success!
-                        s.shutdown(socket.SHUT_RDWR)
-                        s.close()
+                        try:
+                            s.shutdown(socket.SHUT_RDWR)
+                        except socket.error as e:
+                            if e.errno != errno.ENOTCONN:
+                                raise
+                        # else, the server broke the connection on its end, assume it's not ready
+                        else:
+                            s.close()
                         break
 
             # Conditions not yet met, wait and try again