summaryrefslogtreecommitdiff
path: root/oslo_messaging/_drivers/impl_rabbit.py
Commit message (Collapse)AuthorAgeFilesLines
* Fix the help info formatjinxingfang2016-08-091-1/+1
| | | | | | The tail of 'kombu_compression' help info miss a whitespace. Change-Id: I2499b8cbaa10c044cfa315ec9638dd4e82023846
* fix a typo in impl_rabbit.pymaoshuai2016-07-291-1/+1
| | | | Change-Id: I75f99d7e3a6b193e30d8d9baad6a939fbdd0ca6d
* Merge "Introduce TTL for idle connections"Jenkins2016-07-271-1/+6
|\
| * Introduce TTL for idle connectionsKirill Bespalov2016-07-221-1/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We can reduce a workload of rabbitmq through implementation of expiration mechanism for idle connections in the pool with next properties: conn_pool_ttl (default 20 min) conn_pool_min_size: the pool size limit for expire() (default 2) The problem is timeless idle connections in the pool, which can be created via some single huge workload of RPCServer. One SEND connection is heartbeat thread + some network activity every n second. So, we can reduce it. Here is two ways to implement an expiration: [1] Create a separated thread for checking expire date of connections [2] Make call expire() on pool.get() or pool.put() The [1] has some threading overhead, but probably insignificant because the thread can sleep 99% time and wake up every 20 mins (by default). Anyway current implementation is [2]. Change-Id: Ie8781d10549a044656824ceb78b2fe2e4f7f8b43
* | Delete fanout queues on gracefully shutdownKirill Bespalov2016-07-261-1/+5
|/ | | | | | | | | | No reasons to kept fanout queues in case then a rpc server is gracefully shutdown. The expiration time of the fanout queue is too long (30 mins), so for large scales it can accumulate a lot of messages before it be removed Closes-Bug: 1606213 Change-Id: Ieaa35c454df542042f3a5424d70f87d486693024
* Remove rabbitmq max_retriesMehdi Abaakouk2016-06-291-0/+1
| | | | | | | | | | | | It was never clear was the purpose of this option. For the RPC API point of view raising MessageDeliveryFailure is kind of unexpected. For the Notification API, this is weird that when application put retry=-1, the lib stop to retry only with rabbit. So this change deprecates this option for removal. Change-Id: I4ac3a062426418276c96e1745d1c96a8525c36da
* Improve the impl_rabbit loggingKirill Bespalov2016-06-071-21/+45
| | | | | | | | | It would be better if we could log a client port and some additional information about a connection, to understand where is problems occurs. The client port especially helpful for associating events in a rabbitmq.log with oslo.messaging logs. Change-Id: I0dae619f5a1688fb2f64fd90623fde2444609f04
* [impl_rabbit] Remove deprecated get_expiration methodKirill Bespalov2016-05-311-15/+1
| | | | | | | | | | This method is called on each message publishing and as shows profiling (CProfile), the method is time-consuming, because it uses pkg_resources module. As we uses kombu>=3.0.25 since Mitaka, we no longer need in it. Change-Id: I00723a3de81249946a8220ad47129f059d6d544f
* Merge "Fix consuming from missing queues"Jenkins2016-05-201-4/+24
|\
| * Fix consuming from missing queuesKirill Bespalov2016-05-181-4/+24
| | | | | | | | | | | | | | | | | | Redeclare queues on 'Basic.consume: (404) no queue' exception and enable by default consumer declaration with nowait=False in order to wait for a broker response. Change-Id: I99f2bc858dbc7c18a2f328ee26f39105ed17cee3 Closes-Bug: #1581148
* | rabbit: Deprecates host, port, auth optionsMehdi Abaakouk2016-05-181-0/+12
| | | | | | | | | | | | | | | | | | | | We keep backward compatibily of this options since the begin of oslo.messaging while other driver get theses options trough the TransportURL. This changes mark them as deprecated. Change-Id: I8009c0512f3d7864d2699abf8335bcded259fa02
* | Merge "Use single producer and to avoid an exchange redeclaration"Jenkins2016-05-171-19/+24
|\ \ | |/ |/|
| * Use single producer and to avoid an exchange redeclarationKirill Bespalov2016-05-051-19/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Current implementation of the Connection class on each call of _publish method creates its own channel-bounded producer and if auto_declared is set to True then will be send exchange declaration request to the broker despite the fact that the exchange might be created by previous calls. It leads to the significant ~11-30% overhead. I got the following results with notification client & server in the simulator: - http://s32.postimg.org/jzgkbs1qt/notify_client.png I suggest to retain the sets of already declared exchanges and queues per each connection instance and to use a single channel-bounded producer. No needs to create a producer on each call, it is just wrapper over channel. Change-Id: I89564dd1aa079476c68c50979b6b79f8889db5d9
* | Rabbit driver: failure of rpc-calls with float timeoutGevorg Davoian2016-05-101-1/+7
|/ | | | | | | | This patch fixes the bug discovered in the rabbbit driver and related to invoking rpc-calls with floating-point timeouts. Change-Id: Ic4742e08ecaa892382c6a2d4810f6cf7296ee8e8 Closes-Bug: 1579029
* Gracefully handle missing TCP_USER_TIMEOUTBrian Elliott2016-04-201-3/+11
| | | | | | | | On older kernels (<2.6.37) the TCP_USER_TIMEOUT socket option is not available. Handle this case gracefully. Change-Id: Ie74ea715d40ad886f5ed8fdee0469232ed62c996 Closes-Bug: 1572649
* Fix oslo.messaging for Mac OS XDmitry Mescheryakov2016-04-061-1/+2
| | | | | | | | TCP_USER_TIMEOUT is not defined on Mac OS X, so we should not set it on that platform. Change-Id: If745d4981dea24e4267c95473059ff71176ae7d9 Closes-Bug: #1566862
* Fix typos in Oslo.messaging filesNguyen Hung Phuong2016-03-291-2/+2
| | | | | | TrivialFix Change-Id: If093e39ce8b163a541dd835ceb1ac0b3ab89e5a1
* Fix Break in Windows platformsDavanum Srinivas2016-03-281-2/+5
| | | | | | | | In Ibf6139ac2c22d9eeda7030fb87b7f1139d92332e, we added a TCP timeout that does not work on Windows (with python 2.7.9) Closes-Bug: #1562706 Change-Id: I2819225c2d3d3b418c977c03868db8d0c8022d5a
* Always set all socket timeoutsMehdi Abaakouk2016-03-211-26/+31
| | | | | | | | | | | | | | | When exchange/queue are declared the timeout is the default (None) when it should be the no more than heartbeat timeout. Also we have to set TCP_USER_TIMEOUT to ensure our timeouts are respected. socket.settimeout() of python relies on select() that can be blocked because kernel doesn't return during a recv() because of TCP connection breakage. This change fixes that by always setting TCP_USER_TIMEOUT and python socket timeout each time the connection is establish. Change-Id: Ibf6139ac2c22d9eeda7030fb87b7f1139d92332e
* Bump rabbit_transient_queues_ttl to 30 minsDavanum Srinivas2016-03-141-1/+1
| | | | | | | | | | | | | | In I83a8d09dc0cdae24c12d7043ec810529a9ce57ab, we made reply and fanout queues expire instead of auto-delete and set the rabbit_transient_queues_ttl to 10 mins. In grenade, we seem to see the queues expiring just around the time the new side is coming up which causes havoc. There is no reason the rabbit_transient_queues_ttl should not be higher. So let's bump up the expiry to 30 mins. Closes-Bug: #1545002 Change-Id: I70a29b762198129fe0a3e904d9f2a7d4242b322c
* Fail quickly if there on bad passwordDavanum Srinivas2016-02-291-0/+4
| | | | | | | | | | | | | | Inspired by: https://bugs.launchpad.net/oslo.messaging/+bug/1508512/comments/1 Add authentication_failure_close capability on the client to tell the server that it can send us connection.close command indicating ACCESS_REFUSED as the reasona. If this capability is absent then authentication failures are reported in the legacy fashion: by abruptly closing the network connection. Partial-Bug: #1508512 Change-Id: Icf6108678256e6f922620837c71f8827fe8e8c52
* Merge "Documents the mirror queue policy of RabbitMQ 3.0"Jenkins2016-02-241-5/+15
|\
| * Documents the mirror queue policy of RabbitMQ 3.0Jian Wen2016-02-191-5/+15
| | | | | | | | | | | | | | | | | | | | | | | | DocImpact In RabbitMQ 3.0, queue mirroring is no longer controlled by the x-ha-policy argument when declaring a queue. If you just want to make sure that all queues (except those with auto-generated names) are mirrored across all nodes, run: rabbitmqctl set_policy HA '^(?!amq\.).*' '{"ha-mode": "all"}' Change-Id: I905b42c9b853c686f50f28752b4ec09b006d2420 Closes-bug: #1547306
* | Reduce number of rabbitmq consumer tag usedMehdi Abaakouk2016-02-191-11/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The rabbit driver creates a rabbit consumer tag per AMQP consumer (in oslo.messsaging meaning) while we should create only one consumer tag per queue name. For RPC, this doesn't change anything because each AMQP consumer (in oslo.messaging meaning) have its own queue. But for notification API, this is a huge improvement. For ceilometer this reduces the number of rabbit consumer to declare from 15 to 2 on rabbit side. Change-Id: I181df12439485950e944c280c8920476976447d7
* | Fix kombu accept different TTL since version 3.0.25ChangBo Guo(gcb)2016-02-181-1/+15
|/ | | | | | | | | | | | | kombu accept TTL as seconds instead of millisecond since version 3.0.25, We remove TTL conversion in commit d49ddc3b9828f097d5bb39bca0381386a9de7762, which is valid only when kombu >=3.0.25, so need do conversion according to kombu version. Note: Remove this workaround when all supported branches with requirement kombu >=3.0.25 Closes-Bug: #1531148 Change-Id: I762265f23084a95f2d24cb434bad7d9556d4edd5
* rabbit: improvements to QoSJohn Eckersberg2016-02-121-4/+11
| | | | | | | | | | | | | | | | - Don't call _set_qos from both ensure_connection and on_reconnection, instead consolidate and only call from _set_current_channel - Only set QoS on PURPOSE_LISTEN connections. It's a waste of a roundtrip to call it on PURPOSE_SEND connections and slows things down unnecessarily - Guard against rabbit_qos_prefetch_count being set to a negative value - Tests, because we love them Change-Id: I365414c541d895dcd49ebcd32c3a456a92c392d6
* Merge "Correctly set socket timeout for publishing"Jenkins2016-02-111-1/+11
|\
| * Correctly set socket timeout for publishingDmitry Mescheryakov2016-02-111-1/+11
| | | | | | | | | | | | | | Previously a wrong attribute was taken which does not work with current versions of kombu and amqp we have in gate. Change-Id: I9dc2453be3047f993930c2b5c1e74720c8c3b26c
* | [kombu] Implement experimental message compressionYulia Portnova2016-02-101-2/+7
|/ | | | | | | | | | | | | | | | | | Add two new options disabled by default to set the kombu serializer and compression algorithm: http://docs.celeryproject.org/projects/kombu/en/latest/reference/kombu.html#message-producer During the spec discussion, it was apparent that drivers were doing things differently (ZMQ and pickling for example). There was concerns around the CPU vs message size tradeoff among other things: https://review.openstack.org/#/c/247668/ So in this review, we leave the defaults as it was before, but add EXPERIMENTAL options, to see how these holds up in a large stress test and will help test these in the CI as well. Change-Id: If240774657c2175874c45243db7854ba372982bd
* Fix misspellingsvenkatamahesh2016-02-041-1/+1
| | | | Change-Id: Ic2f98e3fbf7fd1591838a9bc440b20ddf2833e6b
* Merge "Python 3 deprecated the logger.warn method in favor of warning"Jenkins2016-01-261-12/+13
|\
| * Python 3 deprecated the logger.warn method in favor of warningChangBo Guo(gcb)2016-01-211-12/+13
| | | | | | | | | | | | | | | | Python 3 deprecated the logger.warn method, please see: https://docs.python.org/3/library/logging.html#logging.warning, so we prefer to use warning to avoid DeprecationWarning. Change-Id: I55148140bb2e85039356726057ae1d552e69b663
* | Merge "rabbit: Add option to configure QoS prefetch count"Jenkins2016-01-261-0/+17
|\ \ | |/ |/|
| * rabbit: Add option to configure QoS prefetch countJohn Eckersberg2016-01-191-0/+17
| | | | | | | | | | Change-Id: I585cd4636e62cdd0c9f1595a6c029cb56f845be9 Closes-Bug: #1531222
* | Merge "Fix URL in warning message"Jenkins2016-01-211-1/+1
|\ \ | |/ |/|
| * Fix URL in warning messageFlavio Percoco2016-01-201-1/+1
| | | | | | | | | | | | | | Doc's url uses the package name. This was probably replaced by mistake when namespaces were removed. Change-Id: I165bcc605aeef219fcd3cb45e284b32c9909da5c
* | Merge "Kombu: make reply and fanout queues expire instead of auto-delete"Jenkins2016-01-191-13/+47
|\ \
| * | Kombu: make reply and fanout queues expire instead of auto-deleteJohn Eckersberg2016-01-131-13/+47
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Right now fanout and reply queues are unconditionally created with auto-delete flag which causes a number of problems listed in bug 1495568. Replacing auto-delete with queue expiration with some sane timeout should fix all these issues at once. Another problem being fixed is that auto-delete flag does not causes the queue to be deleted if it never had consumers. An orphaned fanout queue might appear that way and it will grow indefinitely until somebody manually removes it. See bug 1515278 for details. A new rabbit_transient_queues_ttl config parameter is introduced which configures the TTL for reply and fanout queues. It is a positive integer representing timeout in seconds. By default it is set to 10 minutes. That should be enough for application to reconnect or for server to send reply to client which already died. At the same time, it seems that not so many messages could be accumulated in fanout queues during that time. DocImpact With this change RabbitMQ driver defines reply and fanout queues differently comparing with the previous release: now they are defined with queue TTL (https://www.rabbitmq.com/ttl.html#queue-ttl) instead of auto-delete flag. That helps avoid a number of issues, see commit description for details. A new rabbit_transient_queues_ttl parameter is defined which controls the TTL value. It is set to 10 minutes by default. The change does not affect upgrade in any way. Closes-bug: #1495568 Closes-bug: #1515278 Co-Authored-by: Dmitry Mescheryakov <dmescheryakov@mirantis.com> Change-Id: I83a8d09dc0cdae24c12d7043ec810529a9ce57ab
* | | rabbit: making interval_max configurableJaveme2016-01-191-5/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The parameter interval_max in rabbitmq Connection already exists[1] but is not used, it should be passed to kombu and configurable. This patch makes interval_max take effect: * Pass interval_max to kombu (just has been done in this patch: [2]). * Allow interval_max to be read from configuration file. [1]: https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/ _drivers/impl_rabbit.py#L396 [2]: https://review.openstack.org/#/c/265660/ Change-Id: I62bb7630fa29e82e10aadbc5795f45d3749fe8d8
* | | Merge "rabbit: set interval max for auto retry"Jenkins2016-01-181-0/+1
|\ \ \ | |/ / |/| |
| * | rabbit: set interval max for auto retryShahar Lev2016-01-101-0/+1
| |/ | | | | | | | | | | interval_max value is now correctly passed on to autoretry (was unused before) Change-Id: If98f7c81778b2eb93acae1d215657bf8ba21c1bb
* | Merge "rabbit: Missing to pass parameter timeout to next"Jenkins2016-01-111-1/+2
|\ \ | |/ |/|
| * rabbit: Missing to pass parameter timeout to nextJaveme2016-01-051-1/+2
| | | | | | | | | | | | | | | | | | Missing to pass parameter 'timeout' to the method '_ensure_publishing' when publishing a message. That will cause RPCClient.call unable to pass timeout to the underlying. Change-Id: I1abc7e9c6f5204fa336f29a94114c18d8b982fcb
* | Merge "Improvement of logging acorrding to oslo.i18n guideline"Jenkins2016-01-081-8/+8
|\ \
| * | Improvement of logging acorrding to oslo.i18n guidelineChangBo Guo(gcb)2016-01-071-8/+8
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1. Use translation marker functions, their argument must just be a string 2. Any message with more than one variable should use named interpolation instead of positional to allow translators to move the variables around in the string to account for differences in grammar and writing direction. 3. String interpolation should be delayed to be handled by the logging code, rather than being done at the point of the logging call. For more details, please refert to oslo.i18n guideline [1] Note: this commit doesn't touch test code. [1] http://docs.openstack.org/developer/oslo.i18n/guidelines.html Change-Id: I5f013d65b20396bbe0e5a2cceaed2a33fad1af23
* | Merge "Optimize sending of a reply in RPC server"Jenkins2016-01-081-0/+1
|\ \
| * | Optimize sending of a reply in RPC serverKonstantin Kalin2015-12-281-0/+1
| |/ | | | | | | | | | | | | | | | | | | RPC server uses direct_send method to send replies. The method sets "passive" flag on an exchange. Thus auto_declare doesn't bring any good there, but it costs an additional "exchange.declare" call to RabbitMQ. simulator.py RPC tests shows that a perfomance gain is ~7% Change-Id: Ia9f5f9f52c6cb324eca4ca6a63e80d604ddefa23
* | rabbit: fix unit conversion error of expirationJaveme2016-01-061-7/+1
|/ | | | | | | | kombu has already made a conversion from seconds to milliseconds, so we do not need to do it again. Closes-Bug: #1531148 Change-Id: Ia4478f15f1980a93a53f7e1e6570f68aeec75c69
* Merge "batch notification listener"Jenkins2015-12-111-1/+2
|\
| * batch notification listenerMehdi Abaakouk2015-12-081-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Gnocchi performs better if measurements are write in batch When Ceilometer is used with Gnocchi, this is not possible. This change introduce a new notification listener that allows that. On the driver side, a default batch implementation is provided. It's just call the legacy poll method many times. Driver can override it to provide a better implementation. For example, kafka handles batch natively and take benefit of this. Change-Id: I16184da24b8661aff7f4fba6196ecf33165f1a77
View Lorry Controller Status