summaryrefslogtreecommitdiff
path: root/python
Commit message (Collapse)AuthorAgeFilesLines
* python: Replace pyOpenSSL with ssl.Timothy Redaelli2021-11-032-40/+57
| | | | | | | | | | | | | | | | | | | | | | | | Currently, pyOpenSSL is half-deprecated upstream and so it's removed on some distributions (for example on CentOS Stream 9, https://issues.redhat.com/browse/CS-336), but since OVS only supports Python 3 it's possible to replace pyOpenSSL with "import ssl" included in base Python 3. Stream recv and send had to be splitted as _recv and _send, since SSLError is a subclass of socket.error and so it was not possible to except for SSLWantReadError and SSLWantWriteError in recv and send of SSLStream. TCPstream._open cannot be used in SSLStream, since Python ssl module requires the SSL socket to be created before connecting it, so SSLStream._open needs to create the socket, create SSL socket and then connect the SSL socket. Reported-by: Timothy Redaelli <tredaelli@redhat.com> Reported-at: https://bugzilla.redhat.com/1988429 Signed-off-by: Timothy Redaelli <tredaelli@redhat.com> Acked-by: Terry Wilson <twilson@redhat.com> Tested-by: Terry Wilson <twilson@redhat.com> Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
* python: socket-util: Split inet_open_active function and use connect_ex.Timothy Redaelli2021-11-031-14/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | In an upcoming patch, PyOpenSSL will be replaced with Python ssl module, but in order to do an async connection with Python ssl module the ssl socket must be created when the socket is created, but before the socket is connected. So, inet_open_active function is splitted in 3 parts: - inet_create_socket_active: creates the socket and returns the family and the socket, or (error, None) if some error needs to be returned. - inet_connect_active: connect the socket and returns the errno (it returns 0 if errno is EINPROGRESS or EWOULDBLOCK). connect is replaced by connect_ex, since Python suggest to use it for asynchronous connects and it's also cleaner since inet_connect_active returns errno that connect_ex already returns, moreover due to a Python limitation connect cannot not be used with ssl module. inet_open_active function is changed in order to use the new functions inet_create_socket_active and inet_connect_active. Signed-off-by: Timothy Redaelli <tredaelli@redhat.com> Acked-by: Terry Wilson <twilson@redhat.com> Tested-by: Terry Wilson <twilson@redhat.com> Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
* python: idl: Avoid sending transactions when the DB is not synced up.Terry Wilson2021-10-121-0/+5
| | | | | | | | | | | | | | This ports the C IDL change f50714b to the Python IDL: Until now the code here would happily try to send transactions to the database server even if the database connection was not in the correct state. In some cases this could lead to strange behavior, such as sending a database transaction for a database that the IDL had just learned did not exist on the server. Signed-off-by: Terry Wilson <twilson@redhat.com> Acked-by: Dumitru Ceara <dceara@redhat.com> Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
* ovsdb-data: Deduplicate string atoms.Ilya Maximets2021-09-242-21/+45
| | | | | | | | | | | | | | | | | | | | | | | | | ovsdb-server spends a lot of time cloning atoms for various reasons, e.g. to create a diff of two rows or to clone a row to the transaction. All atoms, except for strings, contains a simple value that could be copied in efficient way, but duplicating strings every time has a significant performance impact. Introducing a new reference-counted structure 'ovsdb_atom_string' that allows to not copy strings every time, but just increase a reference counter. This change allows to increase transaction throughput in benchmarks up to 2x for standalone databases and 3x for clustered databases, i.e. number of transactions that ovsdb-server can handle per second. It also noticeably reduces memory consumption of ovsdb-server. Next step will be to consolidate this structure with json strings, so we will not need to duplicate strings while converting database objects to json and back. Signed-off-by: Ilya Maximets <i.maximets@ovn.org> Acked-by: Dumitru Ceara <dceara@redhat.com> Acked-by: Mark D. Gray <mark.d.gray@redhat.com>
* ovsdb-cs: Perform forced reconnects without a backoff.Dumitru Ceara2021-07-232-0/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The ovsdb-cs layer triggers a forced reconnect in various cases: - when an inconsistency is detected in the data received from the remote server. - when the remote server is running in clustered mode and transitioned to "follower", if the client is configured in "leader-only" mode. - when explicitly requested by upper layers (e.g., by the user application, through the IDL layer). In such cases it's desirable that reconnection should happen as fast as possible, without the current exponential backoff maintained by the underlying reconnect object. Furthermore, since 3c2d6274bcee ("raft: Transfer leadership before creating snapshots."), leadership changes inside the clustered database happen more often and, therefore, "leader-only" clients need to reconnect more often too. Forced reconnects call jsonrpc_session_force_reconnect() which will not reset backoff. To make sure clients reconnect as fast as possible in the aforementioned scenarios we first call the new API, jsonrpc_session_reset_backoff(), in ovsdb-cs, for sessions that are in state CS_S_MONITORING (i.e., the remote is likely still alive and functioning fine). jsonrpc_session_reset_backoff() resets the number of backoff-free reconnect retries to the number of remotes configured for the session, ensuring that all remotes are retried exactly once with backoff 0. This commit also updates the Python IDL and jsonrpc implementations. The Python IDL wasn't tracking the IDL_S_MONITORING state explicitly, we now do that too. Tests were also added to make sure the IDL forced reconnects happen without backoff. Reported-at: https://bugzilla.redhat.com/1977264 Suggested-by: Ilya Maximets <i.maximets@ovn.org> Signed-off-by: Dumitru Ceara <dceara@redhat.com> Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
* python: Add cooperative_yield() API method to Idl.Terry Wilson2021-07-161-0/+11
| | | | | | | | | | | | When using eventlet monkey_patch()'d code, greenthreads can be blocked on connection for several seconds while the database contents are parsed. Eventlet recommends adding a sleep(0) call to cooperatively yield in cpu-bound code. asyncio code has asyncio.sleep(0). This patch adds an API method that defaults to doing nothing, but can be overridden to yield as needed. Signed-off-by: Terry Wilson <twilson@redhat.com> Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
* python: Update bundled sortedcontainers to 2.4.0.Timothy Redaelli2021-07-165-1856/+2494
| | | | | | | | This is needed since the current bundled version doesn't work on Python 3.10+. Signed-off-by: Timothy Redaelli <tredaelli@redhat.com> Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
* ovsdb: Make clients aware of relay service model.Ilya Maximets2021-07-151-0/+16
| | | | | | | | | | | Clients needs to re-connect from the relay that has no connection with the database source. Also, relay acts similarly to the follower from a clustered model from the consistency point of view, so it's not suitable for leader-only connections. Acked-by: Mark D. Gray <mark.d.gray@redhat.com> Acked-by: Dumitru Ceara <dceara@redhat.com> Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
* python: Fix Idl.run change_seqno update.Bodo Petermann2021-07-071-5/+6
| | | | | | | | | | | | | | | | | Fix an issue where Idl.run() returned False even if there was a change. If Idl.run() reads multiple messages from the database server, some may constitute changes and some may not. Changed the way change_seqno is reset: if a message is not a change, reset change_seqno only to the value before reading this message, not to the value before reading the first message. This will fix the return value in a scenario where some message was a change and the last one wasn't. The new change_seqno will now be the value after handling the message with the last change. Fixes: c39751e44539 ("python: Monitor Database table to manage lifecycle of IDL client.") Signed-off-by: Bodo Petermann <b.petermann@syseleven.de> Acked-by: Alin Gabriel Serdean <aserdean@ovn.org> Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
* Remove Python 2 leftovers.Rosemarie O'Riorden2021-06-227-25/+7
| | | | | | | Fixes: 1ca0323e7c29 ("Require Python 3 and remove support for Python 2.") Reported at: https://bugzilla.redhat.com/show_bug.cgi?id=1949875 Signed-off-by: Rosemarie O'Riorden <roriorde@redhat.com> Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
* python: idl: Allow retry even when using a single remote.Ilya Maximets2021-06-111-2/+1
| | | | | | | | | | | | As described in commit [1], it's possible that remote IP is backed by a load-balancer and re-connection to this same IP will lead to connection to a different server. This case is supported for C version of IDL and should be supported in a same way for python implementation. [1] ca367fa5f8bb ("ovsdb-idl.c: Allows retry even when using a single remote.") Signed-off-by: Ilya Maximets <i.maximets@ovn.org> Acked-by: Dumitru Ceara <dceara@redhat.com>
* xml2nroff: Fix formatting of action headers in ovs-actions(7) manpage.Ben Pfaff2021-05-061-4/+3
| | | | | | | | | | | | | | | | The action headings were coming out all smashed together, like "Theoutputaction". This fixes them so that they appear correctly, like "The output action". The previous code stripped starting and ending spaces on a per-node basis, so that "The ", "<code>output</code>", and " action" each got stripped down to "The", "output", "action" after processing. This commit changes it so that stripping happens after concatenation, fixing the problem. Signed-off-by: Ben Pfaff <blp@ovn.org> Acked-By: Timothy Redaelli <tredaelli@redhat.com> Tested-By: Timothy Redaelli <tredaelli@redhat.com>
* python: Send notifications after the transaction ends.Terry Wilson2021-03-151-15/+24
| | | | | | | | | | | | | | | | | | | | | | | | | The Python IDL notification mechanism was sending a notification for each processed update in a transaction as it was processed. This causes issues with multi-row changes that contain references to each other. For example, if a Logical_Router_Port is created along with a Gateway_Chassis, and the LRP.gateway_chassis set to that GC, then when the notify() passes the CREATE event for the LRP, the GC will not yet have been processed, so __getattr__ when _uuid_to_row fails to find the GC, will return the default value for LRP.gateway_chassis which is []. This patch has the process_update methods return the notifications that would be produced when a row changes, so they can be queued and sent after all rows have been processed. Fixes: d7d417fcddf9 ("Allow subclasses of Idl to define a notification hook") Signed-off-by: Terry Wilson <twilson@redhat.com> Acked-by: Brian Haley <haleyb.dev@gmail.com> Acked-by: Dumitru Ceara <dceara@redhat.com> Tested-by: Flavio Fernandes <flavio@flaviof.com> Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
* reconnect: Add Python implementation of received_attempt(), and test.Ben Pfaff2021-01-072-3/+23
| | | | | | | | | | This follows up on commit 4241d652e465 ("jsonrpc: Avoid disconnecting prematurely due to long poll intervals."), which implemented the same thing in C. Signed-off-by: Ben Pfaff <blp@ovn.org> Requested-by: Ilya Maximets <i.maximets@ovn.org> Acked-by: Ilya Maximets <i.maximets@ovn.org>
* python: Update build system to ensure dirs.py is created.Mark Gray2020-11-263-13/+22
| | | | | | | | | | Update build system to ensure dirs.py is created when it is a dependency for a build target. Also, update setup.py to check for that dependency. Fixes: 943c4a325045 ("python: set ovs.dirs variables with build system values") Signed-off-by: Mark Gray <mark.d.gray@redhat.com> Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
* python: Don't raise an Exception on failure to connect via SSL.Terry Wilson2020-11-161-2/+13
| | | | | | | | | | | | | | | | | | | | With other socket types, trying to connect and failing will return an error code, but if an SSL Stream is used, then when check_connection_completion(sock) is called, SSL will raise an exception that doesn't derive from socket.error which is handled. This adds handling for SSL.SysCallError which has the same arguments as socket.error (errno, string). A future enhancement could be to go through SSLStream class and implement error checking for all of the possible exceptions similar to how lib/stream-ssl.c's interpret_ssl_error() works across the various methods that are implemented. Fixes: d90ed7d65ba8 ("python: Add SSL support to the python ovs client library") Signed-off-by: Terry Wilson <twilson@redhat.com> Acked-by: Thomas Neuman <thomas.neuman@nutanix.com> Acked-by: Mark Michelson <mmichels@redhat.com> Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
* python: set ovs.dirs variables with build system valuesMark Gray2020-11-163-37/+8
| | | | | | | | | | | | ovs/dirs.py should be auto-generated using the template ovs/dirs.py.template at build time. This will set the ovs.dirs python variables with a value specified by the environment or, if the environment variable is not set, from the build system. Signed-off-by: Mark Gray <mark.d.gray@redhat.com> Acked-By: Timothy Redaelli <tredaelli@redhat.com> Signed-off-by: Ian Stokes <ian.stokes@intel.com>
* python: Fixup python shebangs to python3.Greg Rose2020-08-261-2/+2
| | | | | | | | | | | | | | | | | | Builds on RHEL 8.2 systems are failing due to this issue. See [1] as to why this is necessary. I used the following command to identify files that need this fix: find . -type f -executable | /usr/lib/rpm/redhat/brp-mangle-shebangs I also updated the copyright notices as needed. 1. https://fedoraproject.org/wiki/Changes/Make_ambiguous_python_shebangs_error Fixes: 1ca0323e7c29 ("Require Python 3 and remove support for Python 2.") Signed-off-by: Greg Rose <gvrose8192@gmail.com> Acked-by: Aaron Conole <aconole@redhat.com> Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
* python: Fix plural forms of OVSDB types.Ben Pfaff2020-07-071-1/+10
| | | | | | | | | | | | Fixes two problems. First, the plural of chassis is also chassis. Second, for linguistic analysis we need to consider plain words, not words that have (e.g.) \fB and \fR pasted into them for nroff output. This makes the OVN manpage for ovn-sb(5) talk about "set of Chassis" not "set of Chassiss". Acked-by: Numan Siddique <numans@ovn.org> Signed-off-by: Ben Pfaff <blp@ovn.org>
* Handle refTable values with setkey()Terry Wilson2020-03-201-2/+1
| | | | | | | | | | | | For columns like QoS.queues where we have a map containing refTable values, assigning w/ __setattr__ e.g. qos.queues={1: $queue_row} works, but using using qos.setkey('queues', 1, $queue_row) results in an Exception. The opdat argument can essentially just be the JSON representation of the map column instead of trying to build it. Signed-off-by: Terry Wilson <twilson@redhat.com> Signed-off-by: Ben Pfaff <blp@ovn.org>
* docs: handle multi line headers for nroffIhar Hrachyshka2020-02-131-1/+8
| | | | | | | | | | | Before the fix, headers split into multiple lines were producing bogus quote characters in nroff output and failed to indent headers properly. Specifically, it fixes a header and its indentation in ovn-architecture(7). Signed-off-by: Ihar Hrachyshka <ihrachys@redhat.com> Signed-off-by: Ben Pfaff <blp@ovn.org>
* Remove dependency on python3-sixTimothy Redaelli2019-12-2016-154/+107
| | | | | | | | | | | Since Python 2 support was removed in 1ca0323e7c29 ("Require Python 3 and remove support for Python 2."), python3-six is not needed anymore. Moreover python3-six is not available on RHEL/CentOS7 without using EPEL and so this patch is needed in order to release OVS 2.13 on RHEL7. Signed-off-by: Timothy Redaelli <tredaelli@redhat.com> Signed-off-by: Ben Pfaff <blp@ovn.org>
* Require Python 3 and remove support for Python 2.Ben Pfaff2019-09-271-7/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Python 2 reaches end-of-life on January 1, 2020, which is only a few months away. This means that OVS needs to stop depending on in the next release that should occur roughly that same time. Therefore, this commit removes all support for Python 2. It also makes Python 3 a mandatory build dependency. Some of the interesting consequences: - HAVE_PYTHON, HAVE_PYTHON2, and HAVE_PYTHON3 conditionals have been removed, since we now know that Python3 is available. - $PYTHON and $PYTHON2 are removed, and $PYTHON3 is always available. - Many tests for Python 2 support have been removed, and the ones that depended on Python 3 now run unconditionally. This allowed several macros in the testsuite to be removed, making the code clearer. This does make some of the changes to the testsuite files large due to indentation level changes. - #! lines for Python now use /usr/bin/python3 instead of /usr/bin/python. - Packaging depends on Python 3 packages. Acked-by: Numan Siddique <nusiddiq@redhat.com> Tested-by: Numan Siddique <nusiddiq@redhat.com> Signed-off-by: Ben Pfaff <blp@ovn.org>
* Add a __str__ method to idl.RowTerry Wilson2019-09-251-0/+6
| | | | | | | | | | It's sometimes handy to log an entire Row object, so this just adds a string representation of the object as: Tablename(col1=val1, col2=val2, ..., coln=valn) Signed-off-by: Terry Wilson <twilson@redhat.com> Signed-off-by: Ben Pfaff <blp@ovn.org>
* Shutdown SSL connection before closing socketTerry Wilson2019-07-121-0/+8
| | | | | | | | | | | | | | | Without shutting down the SSL connection, log messages like: stream_ssl|WARN|SSL_read: unexpected SSL connection close jsonrpc|WARN|ssl:127.0.0.1:47052: receive error: Protocol error reconnect|WARN|ssl:127.0.0.1:47052: connection dropped (Protocol error) would occur whenever the socket is closed. This just adds an SSLStream.close() that calls shutdown() and ignores SSL errors, the same way that lib/stream-ssl.c does in ssl_close(). Signed-off-by: Terry Wilson <twilson@redhat.com> Signed-off-by: Ben Pfaff <blp@ovn.org>
* python: Monitor Database table to manage lifecycle of IDL client.Ted Elhourani2019-03-222-15/+207
| | | | | | | | | | | | | | | | | | | | | | | | The Python IDL implementation supports ovsdb cluster connections. This patch is a follow up to commit 31e434fc98, it adds the option of connecting to the leader (the default) in the Raft-based cluster. It mimics the exisiting C IDL support for clusters introduced in commit 1b1d2e6daa. The _Server database schema is first requested, then a monitor of the Database table in the _Server Database. Method __check_server_db verifies the eligibility of the server. If the attempt to obtain a monitor of the _Server database fails and a cluster id was not provided this implementation proceeds to request the data monitor. If a cluster id was provided via the set_cluster_id method then the connection is aborted and a connection to a different node is instead attempted, until a valid cluster node is found. Thus, when supplied, cluster id is interpreted as the intention to only allow connections to a clustered database. If not supplied, connections to standalone nodes, or nodes that do not have the _Server database are allowed. change_seqno is not incremented in the case of Database table updates. Acked-by: Numan Siddique <nusiddiq@redhat.com> Signed-off-by: Ted Elhourani <ted.elhourani@nutanix.com> Signed-off-by: Ben Pfaff <blp@ovn.org>
* python: Fix package requirements with old setuptoolsTimothy Redaelli2019-03-221-2/+2
| | | | | | | | | | | | | | | | | | | | Commit 00fcc832d598 ("Update Python package requirements") added a PEP 508 environment marker to install pywin32 on Windows systems. This requires a new setuptools version (>= 20.5), but (at least) RHEL/CentOS7 and Debian Jessie are using an older version of setuptools and so python extension failed to build. This commit adds "extras_require" instead of the PEP 508 environment markers in order to have the conditional dependency of pywin32, but by remaining compatible with the old setuptools versions. CC: Alin Gabriel Serdean <aserdean@cloudbasesolutions.com> CC: Lucian Petrut <lpetrut@cloudbasesolutions.com> Fixes: 00fcc832d598 ("Update Python package requirements") Signed-off-by: Timothy Redaelli <tredaelli@redhat.com> Acked-by: Alin Gabriel Serdean <aserdean@ovn.org> Signed-off-by: Alin Gabriel Serdean <aserdean@ovn.org>
* vlog: Better handle syslog handler exceptions.Ilya Maximets2019-02-251-11/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 'set_levels_from_string' doesn't check for exceptions that could happen while opening syslog files or connecting to syslog sockets. For example, if rsyslog stopped on a system: $ test-unixctl.py -vFACILITY:daemon --detach Traceback (most recent call last): File "../../../../tests/test-unixctl.py", line 90, in <module> main() File "../../../../tests/test-unixctl.py", line 61, in main ovs.vlog.handle_args(args) File "python/ovs/vlog.py", line 463, in handle_args msg = Vlog.set_levels_from_string(verbose) File "python/ovs/vlog.py", line 345, in set_levels_from_string Vlog.add_syslog_handler(words[1]) File "python/ovs/vlog.py", line 321, in add_syslog_handler facility=syslog_facility) File "/python2.7/logging/handlers.py", line 759, in __init__ self._connect_unixsocket(address) File "/python2.7/logging/handlers.py", line 787, in _connect_unixsocket self.socket.connect(address) File "/python2.7/socket.py", line 224, in meth return getattr(self._sock,name)(*args) socket.error: [Errno 111] Connection refused In this case "/dev/log" file exists, so the check inside 'add_syslog_handler' doesn't help. We need to catch the exceptions in 'set_levels_from_string' same way as it done in 'init' function. Also, we don't really need to check for '/dev/log' existence, because exception will be catched on the upper layer and properly handled by disabling the corresponding logger. Fixes: d69d61c7c175 ("vlog: Ability to override the default log facility.") Signed-off-by: Ilya Maximets <i.maximets@samsung.com> Signed-off-by: Ben Pfaff <blp@ovn.org>
* Update Python package requirementsLucian Petrut2019-02-251-1/+2
| | | | | | | | The Python ovs package relies on pywin32 for Windows support. For this reason, pywin32 should be included in the requirements list. Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
* python: Fix E117 over-indented.Ilya Maximets2019-02-123-3/+3
| | | | | | | | | | | | | | | | New check was added to recent pycodestyle-2.5.0 and flake8 complains while building on Travis: ../utilities/bugtool/ovs-bugtool.in:767:17: E117 over-indented ../utilities/bugtool/ovs-bugtool.in:771:17: E117 over-indented ../utilities/bugtool/ovs-bugtool.in:774:17: E117 over-indented ../utilities/bugtool/ovs-bugtool.in:778:17: E117 over-indented ../python/ovs/db/error.py:33:17: E117 over-indented ../python/ovs/poller.py:118:21: E117 over-indented ../python/ovs/reconnect.py:244:17: E117 over-indented Signed-off-by: Ilya Maximets <i.maximets@samsung.com> Signed-off-by: Ben Pfaff <blp@ovn.org>
* Un-revert Work around Python/C JSON unicode differencesTerry Wilson2019-01-152-6/+13
| | | | | | | | | | | | | | | | | This fix was reverted because it depended on a small bit of code in a patch that was reverted that changed some python/ovs testing and build. The fix is still necessary. The OVS C-based JSON parser operates on bytes, so the parser_feed function returns the number of bytes that are processed. The pure Python JSON parser currently operates on unicode, so it expects that Parser.feed() returns a number of characters. This difference leads to parsing errors when unicode characters are passed to the C JSON parser from Python. Acked-by: Lucas Alvares Gomes <lucasagomes@gmail.com> Signed-off-by: Terry Wilson <twilson@redhat.com> Signed-off-by: Ben Pfaff <blp@ovn.org>
* python: Escape backslashes while formatting logs.Ilya Maximets2019-01-151-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Since python version 3.7 (and some 3.6+ versions) regexp engine changed to treat the wrong escape sequences as errors. Previously, if the replace string had something like '\u0000', '\u' was qualified as a bad escape sequence and treated just as a sequence of characters '\' and 'u'. But know this triggers an error: Traceback (most recent call last): File "/usr/lib/python3.7/sre_parse.py", line 1021, in parse_template this = chr(ESCAPES[this][1]) KeyError: '\\u' From the documentation [1]: Unknown escapes consisting of '\' and an ASCII letter in replacement templates for re.sub() were deprecated in Python 3.5, and will now cause an error. [1] https://docs.python.org/3/whatsnew/3.7.html#api-and-feature-removals We need to escape the backslash by another one to keep regexp engine from errors. In case of '\\u000', '\\' is a valid escape sequence and the 'u' is a simple character. To be 100% safe we need to use 're.escape(replace)', but it escapes too many characters making the logs hard to read. This change fixes Python 3 tests on systems with python 3.7. Should be backward compatible. Reported-by: Ben Pfaff <blp@ovn.org> Signed-off-by: Ilya Maximets <i.maximets@samsung.com> Signed-off-by: Ben Pfaff <blp@ovn.org>
* python: Avoid flake8 warning for unused variables.Ben Pfaff2019-01-112-2/+2
| | | | | | Acked-by: Numan Siddique <nusiddiq@redhat.com> Tested-by: Numan Siddique <nusiddiq@redhat.com> Signed-off-by: Ben Pfaff <blp@ovn.org>
* python: Fix invalid escape sequences.Ben Pfaff2019-01-116-8/+8
| | | | | | | | | | It appears that Python silently treats invalid escape sequences in strings as literals, e.g. "\." is the same as "\\.". Newer versions of checkpatch complain, and it does seem reasonable to me to fix these. Acked-by: Numan Siddique <nusiddiq@redhat.com> Tested-by: Numan Siddique <nusiddiq@redhat.com> Signed-off-by: Ben Pfaff <blp@ovn.org>
* nroff: Fix fonts for h2, h3, h4.Ben Pfaff2019-01-101-5/+5
| | | | | | | | Without this change, the fonts are wrong if a title contains formatting like <code> or <var>. Acked-by: Mark Michelson <mmichels@redhat.com> Signed-off-by: Ben Pfaff <blp@ovn.org>
* nroff: Increase width for .IP used for ordered lists.Ben Pfaff2019-01-101-1/+1
| | | | | | | | | | | | | | | | | | | The ordered lists that a .25in width produced looked OK in PostScript or PDF output, but in text output every list item spanned two lines, like this: 1. First list item. 2. Second list item. With this change, they appear normally: 1. First list item. 2. Second list item. Acked-by: Mark Michelson <mmichels@redhat.com> Signed-off-by: Ben Pfaff <blp@ovn.org>
* stream: Allow timeout configuration for open_block.Ilya Maximets2019-01-101-3/+13
| | | | | | | | | | | | | | | | | | | | On some systems in case where remote is not responding, socket could remain in SYN_SENT state for a really long time without errors waiting for connection. This leads to situations where open_blok() hangs for a few minutes waiting for connection to the DOWN remote. For example, our "multiple remotes" idl tests hangs waiting for connection to the WRONG_PORT on FreeBSD in CirrusCI environment. This leads to test failures because Alarm signal arrives much faster than ETIMEDOUT from the socket. This patch allowes to specify timeout value for 'open_block' function. If the connection takes more time, socket will be closed with ETIMEDOUT error code. Negative value or None in python could be used to wait infinitely. Signed-off-by: Ilya Maximets <i.maximets@samsung.com> Signed-off-by: Ben Pfaff <blp@ovn.org>
* python: jsonrpc: Pick new remote on disconnect.Ilya Maximets2018-12-271-3/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If attempt to open non-blocking connection results with EINPROGRESS, further polling will trigger DISCONNECT action in case of failures. While handling this action, jsonrpc python library closes the connection but does not change the current remote. This leads to subsequent connection to the same remote. And the story starts from the beginning producing infinite attempts to connect to a single remote regardless of existense of others. Like this: reconnect | DBG | tcp:127.0.0.1:45932: entering BACKOFF reconnect | INFO | tcp:127.0.0.1:45932: connecting... reconnect | DBG | tcp:127.0.0.1:45932: entering CONNECTING poller | DBG | 999-ms timeout reconnect | INFO | tcp:127.0.0.1:45932: connection attempt timed out reconnect | DBG | tcp:127.0.0.1:45932: entering BACKOFF poller | DBG | 0-ms timeout reconnect | INFO | tcp:127.0.0.1:45932: connecting... <...> reconnect | DBG | tcp:127.0.0.1:45932: entering CONNECTING poller | DBG | 1999-ms timeout reconnect | INFO | tcp:127.0.0.1:45932: connection attempt timed out reconnect | INFO | tcp:127.0.0.1:45932: waiting 4 seconds before reconnect reconnect | DBG | tcp:127.0.0.1:45932: entering BACKOFF <...> Fix that by always picking the new remote on disconnect. This mimics the behaviour of jsonrpc C library. Fixes "multiple remotes" tests on FreeBSD. CC: Numan Siddique <nusiddiq@redhat.com> Fixes: 31e434fc985c ("python jsonrpc: Allow jsonrpc_session to have more than one remote.") Signed-off-by: Ilya Maximets <i.maximets@samsung.com> Signed-off-by: Ben Pfaff <blp@ovn.org>
* socket-util: Report POLLHUP as an error while connection completion checking.Ilya Maximets2018-12-241-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Otherwise failed non-blocking connection could be reported as connected. This causes errors in all following operations with the socket. At least this is true on FreeBSD, where POLLHUP could be set without POLLERR. For example, stream_open_block() tests fails with the following error reporting successful connection to the 'WRONG_PORT': ./ovsdb-idl.at:1817: $PYTHON2 $srcdir/test-stream.py tcp:127.0.0.1:$WRONG_PORT stdout: ./ovsdb-idl.at:1817: exit code was 0, expected 1 2399. ovsdb-idl.at:1817: FAILED (ovsdb-idl.at:1817) Also added new tests to track this issue in C library: 'Check Stream open block - C - tcp' 'Check Stream open block - C - tcp6' CC: Numan Siddique <nusiddiq@redhat.com> Fixes: c1aa16d191d2 ("ovs python: ovs.stream.open_block() returns success even if the remote is unreachable") Fixes: d6cedfd9d29d ("socket-util: Avoid using SO_ERROR.") Signed-off-by: Ilya Maximets <i.maximets@samsung.com> Signed-off-by: Ben Pfaff <blp@ovn.org>
* python: Catch setsockopt exceptions for TCP stream.Ilya Maximets2018-12-201-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | 'sock.setsockopt' could throw exceptions. For example, if non-blocking connection failed before the call: Traceback (most recent call last): File "../.././test-ovsdb.py", line 896, in <module> main(sys.argv) File "../.././test-ovsdb.py", line 891, in main func(*args) File "../.././test-ovsdb.py", line 604, in do_idl ovs.stream.Stream.open(r)) File "/root/git_/ovs/python/ovs/stream.py", line 190, in open error, sock = cls._open(suffix, dscp) File "/root/git_/ovs/python/ovs/stream.py", line 744, in _open sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) File "/usr/local/lib/python2.7/socket.py", line 228, in meth return getattr(self._sock,name)(*args) socket.error: [Errno 54] Connection reset by peer This fixes tests on FreeBSD. Signed-off-by: Ilya Maximets <i.maximets@samsung.com> Signed-off-by: Ben Pfaff <blp@ovn.org>
* python-c-ext: Fix memory leak in Parser_finishEric Lapointe2018-10-311-0/+1
| | | | | | | The memory returned by json_parser_finish needs to be freed by the caller. Signed-off-by: Eric Lapointe <elapointe@corsa.com> Signed-off-by: Ben Pfaff <blp@ovn.org>
* Revert "Test the Python C JSON extension"Ilya Maximets2018-10-153-24/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This reverts commit a7be68a4d77791bbe02c37f7ad8ae60b02e5679e and a subsequent commit 4617d1f6bd24c543f533f6485b42ebca6b0a8371. There are too many issues with these patches. It's better to revert them for now and make a separate fixed versions later if needed. List of issues (maybe not full): 1. 'make clean' removes entire 'python' directory. 2. Fully broken Travis-CI testsuite build: building 'ovs._json' extension creating build/temp.linux-x86_64-2.7 error: could not create 'build/temp.linux-x86_64-2.7': \ Permission denied https://travis-ci.org/openvswitch/ovs/jobs/440693765 3. Broken local testsuite build on Ubuntu 18.04: running build_ext building 'ovs._json' extension creating build/temp.linux-x86_64-3.6 creating build/temp.linux-x86_64-3.6/ovs <...> /usr/bin/ld: .libs/libopenvswitch.a(util.o): \ relocation R_X86_64_TPOFF32 against `var.7749' can not be \ used when making a shared object; recompile with -fPIC <...> collect2: error: ld returned 1 exit status 4. Fedora build failure because of 'setuptools' ('distutils') hard dependency on 'redhat-rpm-config' package: building 'ovs._json' extension <...> gcc: error: <...>/redhat-hardened-cc1: No such file or directory 5. Looks like 'setuptools' also could download and install unwanted python modules during package build. Signed-off-by: Ilya Maximets <i.maximets@samsung.com> Signed-off-by: Ben Pfaff <blp@ovn.org>
* Work around Python/C JSON unicode differencesTerry Wilson2018-10-111-2/+7
| | | | | | | | | | | | | The OVS C-based JSON parser operates on bytes, so the parser_feed function returns the number of bytes that are processed. The pure Python JSON parser currently operates on unicode, so it expects that Parser.feed() returns a number of characters. This difference leads to parsing errors when unicode characters are passed to the C JSON parser from Python. Signed-off-by: Terry Wilson <twilson@redhat.com> Signed-off-by: Ben Pfaff <blp@ovn.org> Acked-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
* Test the Python C JSON extensionTerry Wilson2018-10-112-4/+17
| | | | | | | | | | | | | | | | | | The C JSON parser was added quite a while ago, but unless you configure with --enable-shared and have the Python 2/3 development libraries installed, and the resulting python-ovs module installed, 'make check' won't actually test it. This patch changes Python-based tests to run from the $builddir/python directory and makes the tests configurable to use both JSON backends. There are some unicode failures in the C JSON extension that I left unfixed in this patch to make it easy to show run the new tests on broken code. The next patch in this set works around the issue. Signed-off-by: Terry Wilson <twilson@redhat.com> Signed-off-by: Ben Pfaff <blp@ovn.org> Acked-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
* Python: Make Row's __getattr__ less error proneLucas Alvares Gomes2018-10-051-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Calling getattr() on a Row object after invoking delkey() with a value that does not exist in the object will cause getattr() to fail with a KeyError error. For example: Oct 05 14:59:28 neutron-server[28435]: File "/usr/local/lib/python2.7/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run Oct 05 14:59:28 neutron-server[28435]: txn.results.put(txn.do_commit()) Oct 05 14:59:28 neutron-server[28435]: File "/usr/local/lib/python2.7/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 86, in do_commit Oct 05 14:59:28 neutron-server[28435]: command.run_idl(txn) Oct 05 14:59:28 neutron-server[28435]: File "/usr/local/lib/python2.7/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 299, in run_idl Oct 05 14:59:28 neutron-server[28435]: if isinstance(getattr(record, self.column), dict): Oct 05 14:59:28 neutron-server[28435]: File "/usr/local/lib/python2.7/dist-packages/ovs/db/idl.py", line 843, in __getattr__ Oct 05 14:59:28 neutron-server[28435]: del dmap[key] Oct 05 14:59:28 neutron-server[28435]: KeyError: 'bogusvalue' This patch is replacing the "del dmap[key]" instruction with a "dmap.pop(key, None)" instruction instead because a pop() (with a default value) will not raise an exception in case the key does not exist in the object in the first place, it will just ignore it. Signed-off-by: Lucas Alvares Gomes <lucasagomes@gmail.com> Signed-off-by: Ben Pfaff <blp@ovn.org>
* tests: Use environment variable for default timeout.Ilya Maximets2018-08-151-1/+7
| | | | | | | | | | | Introduce new 'OVS_CTL_TIMEOUT' environment variable that, if set, will be used as a default timeout for OVS control utilities. Setting it in 'atlocal.in' will cover all the hangs inside the testsuite, even when utils called in a subshell. Signed-off-by: Ilya Maximets <i.maximets@samsung.com> Signed-off-by: Ben Pfaff <blp@ovn.org>
* python jsonrpc: Allow jsonrpc_session to have more than one remote.Numan Siddique2018-08-142-9/+49
| | | | | | | | | | | | | | Python IDL implementation doesn't have the support to connect to the cluster dbs. This patch adds this support. We are still missing the support in python idl class to connect to the cluster master. That support will be added in an upcoming patch. This patch is similar to the commit 8cf6bbb184 which added multiple remote support in the C jsonrpc implementation. Acked-by: Mark Michelson <mmichels@redhat.com> Signed-off-by: Numan Siddique <nusiddiq@redhat.com> Signed-off-by: Ben Pfaff <blp@ovn.org>
* ovs python: ovs.stream.open_block() returns success even if the remote is ↵Numan Siddique2018-08-143-6/+45
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | unreachable The python function ovs.socket_util.check_connection_completion() uses select() (provided by python) to monitor the socket file descriptor. The select() returns 1 when the file descriptor becomes ready. For error cases like - 111 (Connection refused) and 113 (No route to host) (POLLERR), ovs.poller._SelectSelect.poll() expects the exceptfds list to be set by select(). But that is not the case. As per the select() man page, writefds list will be set for POLLERR. Please see "Correspondence between select() and poll() notifications" section of select(2) man page. Because of this behavior, ovs.socket_util.check_connection_completion() returns success even if the remote is unreachable or not listening on the port. This patch fixes this issue by using poll() to check the connection status similar to the C implementation of check_connection_completion(). A new function 'get_system_poll() is added in ovs/poller.py which returns the select.poll() object. If select.poll is monkey patched by eventlet/gevent, it gets the original select.poll() and returns it. The test cases added in this patch fails without the fix. Suggested-by: Ben Pfaff <blp@ovn.org> Signed-off-by: Numan Siddique <nusiddiq@redhat.com> Signed-off-by: Ben Pfaff <blp@ovn.org> Acked-by: Mark Michelson <mmichels@redhat.com>
* Add ovs.compat module to python packageTerry Wilson2018-08-101-1/+2
| | | | | | Signed-off-by: Terry Wilson <twilson@redhat.com> Signed-off-by: Ben Pfaff <blp@ovn.org> Acked-by: Timothy Redaelli <tredaelli@redhat.com>
* tests: Don't log to syslog during tests.Ben Pfaff2018-08-091-3/+5
| | | | | | | | Until now, "make check" generated a huge amount of output to syslog. This commit suppresses it. Acked-by: Ilya Maximets <i.maximets@samsung.com> Signed-off-by: Ben Pfaff <blp@ovn.org>
View Lorry Controller Status