| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We already expose functions to handle the instantiation
of classes such as RPCServer and RPCTransport but the
same was never done for RPCClient so the API is
inconsistent in its enforcement.
This adds a get_rpc_client function that should be used
instead of instatiating the RPCClient class directly to
be more consistent.
This also allows to handle more logic inside the function
in the future such as if implementations for an async client
is implemented, as investigation in [1] has shown.
[1] https://review.opendev.org/c/openstack/oslo.messaging/+/858936
Change-Id: Ia4d1f0497b9e2728bde02f4ff05fdc175ddffe66
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Introduced changes:
- pre-commit config and rules
- Add pre-commit to pep8 gate, Flake8 is covered in the pre-commit hooks.
- Applying fixes for pre-commit compliance in all code.
Also commit hash will be used instead of version tags in pre-commit to
prevend arbitrary code from running in developer's machines.
pre-commit will be used to:
- trailing whitespace;
- Replaces or checks mixed line ending (mixed-line-ending);
- Forbid files which have a UTF-8 byte-order marker (check-byte-order-marker);
- Checks that non-binary executables have a proper
shebang (check-executables-have-shebangs);
- Check for files that contain merge conflict strings (check-merge-conflict);
- Check for debugger imports and py37+ breakpoint()
calls in python source (debug-statements);
- Attempts to load all yaml files to verify syntax (check-yaml);
- Run flake8 checks (flake8) (local)
For further details about tests please refer to:
https://github.com/pre-commit/pre-commit-hooks
Change-Id: Ibd0c3d64fdc5c293d9d676d33eab828d9fde971f
Co-authored-by: Moisés Guimarães de Medeiros <moguimar@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The purpose of this patch is to add an endpoint directly in RPC
dispatcher, so this endpoint will always be available, in a cross
project manner, without the need for projects to manage it by themself.
This endpoint stay disabled by default, so this change is harmless
without a specific configuration option.
To enable this ping endpoint, an operator will just have to add a new
parameter in the [DEFAULT] section, alongside with rpc_response_timeout
[DEFAULT]
rpc_ping_enabled=true # default is false
The purpose of this new endpoint is to help operators do a RPC call (a
ping) toward a specific RPC callback (e.g. a nova-compute, or a
neutron-agent).
This is helping a lot for monitoring agents (for example, if agents are
deployed in a kubernetes pod).
The endpoint is named oslo_rpc_server_ping.
Change-Id: I51cf67e060f240e6eb82260e70a057fe599f9063
Signed-off-by: Arnaud Morin <arnaud.morin@corp.ovh.com>
|
|
|
|
|
|
|
|
|
| |
Remove six, the python 2/3 compatibility library. It's not needed
anymore since the repo is python3 only.
Remove a now unneeded hacking test.
Change-Id: I40522c4accb4aaf8115d11fee8b081e2d991cb4d
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The repo is Python 3 now, so update hacking to version 3.0 which
supports Python 3.
Update local hacking check for new flake8 version.
Blacklist:
W504 line break after binary operator
Fix:
E741 ambiguous variable name
E117 over-indented
E305 expected 2 blank lines after class or function definition, found 1
F841 local variable 'e' is assigned to but never used
W605 invalid escape sequence '\.'
Change-Id: I99d574ca6569f1f177d2c5ce1011f269f4343619
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The blocking executor has been deprecated in Pike and marked for removal
in Rocky, but some user like Mistral asked us to wait before. We decided
to remove this executor for Train or next cycle, now we are in the
Ussuri and after some researchs on usage I think we can go ahead.
This patch drop the deprecation warnings, related unit tests and
set the server with the threading executor is the default executor.
Change-Id: If07bab61ee2b148658b88be98b12f8539f274efe
Closes-Bug: #1715141
|
|
|
|
|
|
|
|
| |
Waiting on a threading.Event with eventlet can cause busy looping via
epoll_wait, see related bug for more details.
Change-Id: I007613058a2d21d1712c02fa6d1602b63705c1ab
Related-bug: #1518430
|
|
|
|
|
|
|
|
|
|
| |
This broke non-rabbitmq message bus drivers. See:
https://bugs.launchpad.net/oslo.messaging/+bug/1855775
This reverts commit b104f254ab43d5e2589e845eff84b1bd096cb42d.
Change-Id: I17a448a768b544482b375b0076889db989e03e8c
|
|
|
|
|
|
|
|
|
| |
Typically a simple log will not narrow down the
performance, but give us more information about
the service status.
Change-Id: I51c8f2743dd39cccd3d1d021d3c50dc09f70cd97
Closes-Bug: #1847747
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
With this feature, it is possible to specialize the parameters to send.
`options = oslo_messaging.TransportOptions(at_least_once=True)`
TransportOptions is used in every single driver,
for example in RabbitMQ driver is used to handle the mandatory flag.
Notes:
- The idea of creating a new class TransportOptions is because I'd like
to have an abstract class not related only to the RPCClient
- at_least_once is the first parameter, when needed we can add the
others.
Implements: blueprint transport-options (second point)
The blueprint link is [1]
To test it you can use [2]
1- https://blueprints.launchpad.net/oslo.messaging/+spec/transport-options
2- https://github.com/Gsantomaggio/rabbitmq-utils/
tree/master/openstack/mandatory_test
Change-Id: I1858e4a990507d3c2bac2ef7fbef75d8c2dbfce2
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
With this new parameter is possible to pass other parameters
from the client to the drivers.
So it is possible to tune the driver behavior.
For example can be used to send the mandatory flag in RabbitMQ
Note:
- The transport_options parameter is not actually used (yet).
- This part of blueprint transport-options (first part)
Implements: blueprint transport-options
The blueprint link is
https://blueprints.launchpad.net/oslo.messaging/+spec/transport-options
Change-Id: Iff23a9344c2c87259cf36b0d14c0a9fc075a2a72
|
|
|
|
|
| |
Change-Id: Iec04c18ac3565a3610377d94caf128c6704a89eb
Closes-Bug: #1816816
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This adds an optional call_monitor_timeout parameter to the RPC client,
which if specified, will enable heartbeating of long-running calls by
the server. This enables the user to increase the regular timeout to
a much larger value, allowing calls to take a very long time, but
with heartbeating to indicate that they are still running on the server
side. If the server stops heartbeating, then the call_monitor_timeout
takes over and we fail with the usual MessagingTimeout instead of waiting
for the longer overall timeout to expire.
Change-Id: I60334aaf019f177a984583528b71d00859d31f84
|
|
|
|
|
| |
Change-Id: I4d30ff269b2e34f4409e8d1ce822eb93918cf5a2
Closes-Bug: #1712393
|
|\ |
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Since RPC and notifications can have different backends, it is
useful to warn users if they use a notification transport in RPC
and vice versa. This patch introduces RPCTransport and
NotificationTransport subclasses of Transport, so it's easier to
add different behavior for them if need be.
Related-Bug: #1680192
Change-Id: Iab60544d69053c8e74c28a2d5c84665be749013f
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The FakeExchangeManager uses an instance-level storage for FakeExchanges
mapping[1]. When a client--server pair is created, each keeps their own
instance of FakeDriver -> FakeExchangeManager -> FakeExchange, each of
which has their own (instance-level) copy of e.g _server_queues[2], making
it impossible for them to communicate.
This patch makes the _exchanges mapping a class-level attribute in order
to keep the registered exchanges shared between all Manager instances,
allowing client and server communication (within a single process).
The test_server unit-tests had to be refactored to explicitly pass an exchange
name when building a target. This is required for an exchange name change to
have any effect during a test case run time when compared to passing the
exchange name through the URL. This issue was revealed with this patch.
[1] https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_fake.py#L145,#L148
[2] https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_fake.py#L88,#L92
Change-Id: I8dff66f4cafeb1f4c57dbfbfaba5d49e50f55fee
Closes-Bug: #1714055
|
|\ |
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Using blocking executor is not recommended for application. But it looks
like some use it just because it's the default and are not aware their
should change it despite of documentation and logging message.
Choosing the application threading model is an important step
of an application.
This change deprecates it, in the future we will just make executor
mandatory. This will ensure that application make a choice.
Also this will reduce headache of oslo.messaging developers to
make the driver code working in a sync and async.
And to finish test coverage of blocking executor is 0%...
This rework some tests to remove logging.captureWarnings() that can
catch unwanted warning of other tests. Tests mocks warning instead.
Related-bug: #694728
Change-Id: Ic67164d12e7a9bed76d6e64ca2ced12e3984ff5f
|
|/
|
|
|
|
|
|
| |
The serializer tests occasionally fail since the order of a dict() is
not guaranteed. Explicitly compute the serializer instead of
pre-computing it.
Change-Id: I74f8fa227e6508491b5982fe7e2841683724540c
|
|
|
|
|
|
|
|
|
| |
The get_rpc_transport wraps get_transport to unify the API in
anticipation of comprehensive separation of RPC and Notification
messaging backends.
Related-Bug: 1680192
Change-Id: Ic6af07b98ff43806c2af38a3ba129991f1e0ec86
|
|
|
|
|
|
|
|
|
| |
OpenStack projects should be using mock rather than mox.
oslo.messaging was using mox's stubs via oslotest rather than
mock.patch.object. The code is converted to use mock.patch.object
via fixtures.MockPatchObject.
Change-Id: I19490b4e8211c35b237ebfd38bf2f8b8b44cbf61
|
|
|
|
|
|
|
|
|
|
|
| |
dict doesn't seem always keep order on Python. we have two separate part
to prepare test data[1], calling 'for k, v in self.args.items()' and
'for arg in self.args:' may get data in different order.
This commit makes we collect test data in same order.
Change-Id: I49f0d1b9aed110ad1e59140fd70760682e944d74
Related-Bug: #1677151
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
debtcollector's decorator updated_kwarg_default_value is used to indicate
default value will be changed in future version. We introduced parameter
access_policy and add itfor constructor method of RPCDispatcher in
d3a8f280ebd6fd12865fd20c4d772774e39aefa2. Other services usually call
get_rpc_server method instead of calling constructor method of
RPCDispatcher. get_rpc_server method constructs RPCDispatcher instance
in [1], then FutureWarning will be generated even we set parameter
access_policy explicitly. Need move the decorator updated_kwarg_default_value
to method get_rec_server to avoid the case.
Closes-Bug: #1661572
[1] https://github.com/openstack/oslo.messaging/blob/f1efaf1b6ed360b280d48e0ab41d1dd73c1553ef/oslo_messaging/rpc/server.py#L200.
Change-Id: I3214045856d0bbd05a0000eeafb769e4697cd34c
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The mock module has been added to Python 3.3 as unittest.mock.
The third party mock module doesn't seem to be maintained anymore.
This is follow up of 72c501454ea33b603918de35565a66dda8abd66a and
add hacking rule to avoid regression issue.
oslotest prepares mock for six in oslotest/__init__.py as follow:
six.add_move(six.MovedModule('mock', 'mock', 'unittest.mock')) and
oslo_messaging/tests/__init__.py imports oslotest before importing
test submodules to setup six.moves for mock, then "from six.moves
import mock" works well.
Note: unittest.mock also detects wrong usage of method
assert_called_once_with.
Change-Id: I3d09733789cfa2550cf47c24f2553357d36bcc0d
|
|
|
|
|
|
|
|
|
| |
Replace mox with mock in some files for testing.
See:
https://blueprints.launchpad.net/nova/+spec/remove-mox-ocata
Change-Id: Ic2a3c760d03d19c5a07d71796f24580393102163
|
|
|
|
|
|
| |
assertGreater provides a nicer error message if it fails.
Change-Id: I17d6393ec151a7881786a119dcfa3925c936a0f6
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Implements access_policy for dispatcher to restrict endpoint methods.
Implements the following access policies:
* LegacyRPCAccessPolicy
* DefaultRPCAccessPolicy
* ExplicitRPCAccessPolicy
* Implement decorator @rpc.expose for use with the
ExplicitRPCAccessPolicy
* Modify get_rpc_server to allow optional access_policy argument
* Set default access_policy to LegacyRPCAccessPolicy (Nova exposes
_associate_floating_ip in tempest tests). Added debtcollector
notification.
* Add test cases for access_policy=None
* Clarify documentation
Change-Id: I42239e6c8a8be158ddf5c3b1773463b7dc93e881
Closes-Bug: 1194279
Closes-Bug: 1555845
|
|
|
|
|
|
|
|
|
|
|
|
| |
Many assertEqual sentences don't follow assertEqual(expected, actual),
These misplaces have 2 impacts:
1, giving confusing messages when some tests failed.
2, mislead other developers, new test modules may follow these wrong pattern.
This patch fix all of them.
Change-Id: Icc6e404dd02fe47b76c7366d31d91c8d8e464f54
Closes-Bug: #1604213
|
|
|
|
|
|
|
|
|
|
|
| |
This patch removes log_failure argument from the function
serialize_remote_exception and from driver implementations
using it (because it is never used and always defaults to True)
and prevents error logging in this function (because these errors
are already logged by servers while processing incoming messages).
Change-Id: Ic01bb11d6c4f018a17f3219cdbd07ef4d30fa434
Closes-Bug: 1580352
|
|
|
|
|
|
|
|
|
| |
This patch fixes the bug in the RPCClient class when a client's
version_cap is set, but target.version is unset. The code does not
check this case, which results in unhandled exceptions.
Change-Id: I623c14b74b9101bb4ab199dff6609fab44388c4a
Closes-Bug: 1574615
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Current Listener interface has poll() method which return messages
To use it we need have poller thread which is located in MessageHandlerServer
But my investigations of existing driver's code shows that some implemetations have
its own thread inside for processing connection event loop. This event loop received
messages and store in queue object. And then our poller's thread reads this queue
This situation can be improved. we can remove poller's thread, remove queue object
and just call on_message server's callback from connection eventloop thread
This path provide posibility to do this for one of drivers and leave as is other drivers
Change-Id: I3e3d4369d8fdadcecf079d10af58b1e4f5616047
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Dispatcher should be responsible for routing message to the
callback method of endpoint object and returning result back to the
server only. But now it is also responsible for sending reply,
ack/reque messages etc.
Also this patch makes small improvements:
1) Notification dispatcher now requeue message if endpoint raises exception
2) unstable behaviour of test_mask_passwords test is fixed
Change-Id: I5f23e23644e90919cb67f81fc306ee85c5e09974
|
|
|
|
|
| |
Change-Id: I1afadaabe908df4d66c94d36539c7380e424c186
Closes-Bug: #1554996
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Now we have situation when openstack projects like Mistral needs
extra oslo.messaging functionality.
But it is too complicated now to to implement something new and
integrate it with current code because there is a little bit mess.
1) Executor should be responsible for how to run jobs
(but now also has code with server logic)
2) Dispatcher should be responsible for routing message to the
target endpoint for processing (but it also has serialisation, sending replies,
executing some executor's callbacks etc)
3) Server should do all server specific logic, we need to have different
implementation of servers for RPC and notification, not different implementations
of dispatchers
This patch fixes 1-st point
Change-Id: Ib6408f408889bb7b7056722be636a5547b1a780d
|
|
|
|
|
|
|
|
|
| |
-- "cls.scenarios = filter(f, cls.scenarios)"
It's better to set a list to cls.scenarios but filter()/map() returns
an iterator on Python 3. This patch we replaced the filter()/map() with
a list comprehension in test_server.py.
Change-Id: I16640e564d41c6a1c29e229049467ae788bb1437
|
|
|
|
|
|
|
| |
related to Id3af696193af2ccf5e5f3a1ae1d22f4f80860606. we need to make
override_pool_size accessible for external use.
Change-Id: I686c8c341f117dbc0b02443306395d5fd011c2f1
|
|
|
|
|
|
|
|
|
| |
Somehow we sneaked in a change to ignore pep8 on tests/ directory.
Let's cleanup the code and re-enable pep8
Closes-Bug: #1540981
Change-Id: Ic002134a0d0c498991b062466e00068d6ac1f1ab
|
|\ |
|
| |
| |
| |
| |
| |
| |
| |
| | |
Python 3 deprecated the logger.warn method, please see:
https://docs.python.org/3/library/logging.html#logging.warning,
so we prefer to use warning to avoid DeprecationWarning.
Change-Id: I55148140bb2e85039356726057ae1d552e69b663
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fixed pep8 errors:
* E127 continuation line over-indented for visual indent
* E226 missing whitespace around arithmetic operator
* E231 missing whitespace after ','
* E241 multiple spaces after ','
* E261 at least two spaces before inline comment
* E301 expected 1 blank line, found 0
* E302 expected 2 blank lines, found 1
* E303 too many blank lines (2)
* E501 line too long (80 > 79 characters)
Fixed pep8 warnings:
* W292 no newline at end of file
* W391 blank line at end of file
Change-Id: I1736f8a42c3a335a17a0b9b6e64782487ed3a495
|
|\ |
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
1. Use translation marker functions, their argument must just be a string
2. Any message with more than one variable should use named
interpolation instead of positional to allow translators
to move the variables around in the string to account for
differences in grammar and writing direction.
3. String interpolation should be delayed to be handled by the logging
code, rather than being done at the point of the logging call.
For more details, please refert to oslo.i18n guideline [1]
Note: this commit doesn't touch test code.
[1] http://docs.openstack.org/developer/oslo.i18n/guidelines.html
Change-Id: I5f013d65b20396bbe0e5a2cceaed2a33fad1af23
|
|\ \ |
|
| |/
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Each config option has limitation for type and value. If we use
method CONF.set_override without parameter enforce_type=True, we
may pass wrong type to config option.
This commit makes sure calling method CONF.set_override with
enforce_type=True.
Change-Id: I1430c93777d6e5d98086a8700663ab9f58c1e5ab
Closes-Bug: #1517839
|
|/
|
|
|
|
|
|
| |
The usage of assertEqual(True/False, ***) should be changed
to a meaningful format of assertTrue/False(***).
Change-Id: Idda812a34be8aa53654d6791eac47028339d3cb7
Closes-Bug:#1512207
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Gnocchi performs better if measurements are write in batch
When Ceilometer is used with Gnocchi, this is not possible.
This change introduce a new notification listener that allows that.
On the driver side, a default batch implementation is provided.
It's just call the legacy poll method many times.
Driver can override it to provide a better implementation.
For example, kafka handles batch natively and take benefit of this.
Change-Id: I16184da24b8661aff7f4fba6196ecf33165f1a77
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This change formalises locking in MessageHandlingServer, which closes
several bugs:
* It adds locking for internal state when using the blocking executor,
which closes a number of races.
* It does not hold a lock while executing server functions,
which removes a potential cause of deadlock if the server does its
own locking.
* It fixes a regression introduced in change
gI3cfbe1bf02d451e379b1dcc23dacb0139c03be76. If multiple threads
called wait() simultaneously, only 1 of them would wait and the
others would return immediately, despite message handling not having
completed. With this change only 1 will call the underlying wait,
but all will wait on its completion.
Additionally, it introduces some new functionality:
* It allows the user to make calls in any order and it will ensure,
with locking, that these will be reordered appropriately.
* The caller can pass a `timeout` argument to any server method, which
will cause it to raise an exception if it waits too long.
* The caller can pass a `log_after` argument to any server method,
which will cause it to raise a log message if it waits too long. It
can also be used to disable logging when waiting is intentional.
We remove DummyCondition as it no longer has any users.
This change was originally committed as change
I9d516b208446963dcd80b75e2d5a2cecb1187efa, but was reverted as it
caused a hang in a Nova test. This was caused by the locking behaviour
for handling restarting a previously stopped server. The original
patch caused the state to 'wrap' immediately after the user called
wait(). This caused a hang in tests which redundantly called stop()
and wait() multiple times. This new patch only wraps when the user
calls start() again. Callers who do not restart a server will
therefore not be affected by the wrapping behaviour. Callers who do
restart a server will be no worse than before. We add a deprecation
warning on restart, as this operation is inherently racy with this api
and there is a simple, safe alternative.
This new version has been successfully tested against the unit and
functional tests of nova, cinder, glance, and ceilometer.
Change-Id: Ic79f87e7b069c1f62d6121486fd6cafd732fdde7
|
|
|
|
|
|
|
|
|
|
| |
This reverts commit d700c382791b6352bb80a0dc455589085881669f.
This commit is causing a timeout/lock wait condition when using the in
memory rpc bus. It exposed in the Nova unit / functional tests which use
this extensively.
Change-Id: I9610a5533383955f926dbbb78ab679f45cd7bcdb
Closes-Bug: #1514876
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This change formalises locking in MessageHandlingServer. It allows the
user to make calls in any order and it will ensure, with locking, that
these will be reordered appropriately. It also adds locking for
internal state when using the blocking executor, which closes a number
of races.
It fixes a regression introduced in change
gI3cfbe1bf02d451e379b1dcc23dacb0139c03be76. If multiple threads called
wait() simultaneously, only 1 of them would wait and the others would
return immediately, despite message handling not having completed.
With this change only 1 will call the underlying wait, but all will
wait on its completion.
We add a common logging mechanism when waiting too long. Specifically,
we now log a single message when waiting on any lock for longer than
30 seconds.
We remove DummyCondition as it no longer has any users.
Change-Id: I9d516b208446963dcd80b75e2d5a2cecb1187efa
|