summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* rollback transactions on queue terminationbug21368Matthias Radestock2009-10-112-80/+59
| | | | | | | | | | That way we don't leave garbage - transactionally published, but uncommitted messages - in the message store. Also, we we can get rid of the pending_commits state wart in disk_queue. That is possible because both tx commits and queue deletions are issued by the queue process and tx commits are synchronous, so there is never a chance of there being a pending commit when doing a deletion.
* fixing erroneous use of thunk(_) in spec to fun()Matthew Sackman2009-10-081-1/+1
|
* cosmeticMatthias Radestock2009-10-081-8/+5
|
* make msg_store responsible for sync'ingMatthias Radestock2009-10-082-130/+148
| | | | | | | | | | | | | | | | The API to the msg_store has changed: now instead of asking whether a sync is needed for a set of msg ids, and subsequently requesting a sync, we request a sync for a set of msg ids and supply a callback that is invoked when that sync is done. That way the msg_store can make its own decisions on when to sync, and less logic is required by callers. During queue deletion we must remove *all* queue messages from the store, including those that are part of committed transactions for which the disk_queue has not yet received the sync callback. To do that we keep a record of these messages in a dict in the state. The dict also ensures that we do not act on a sync callback involving a queue which has since been deleted and perhaps recreated.
* merge default into bug21368Matthias Radestock2009-10-0722-861/+4369
|\
| * cosmetic refactoringMatthew Sackman2009-09-301-10/+10
| |
| * merging in from defaultMatthew Sackman2009-09-3018-83/+128
| |\
| * | turn the msg_store into a separate processMatthias Radestock2009-09-252-181/+200
| | |
| * | merge default into bug21368Matthias Radestock2009-09-2523-151/+354
| |\ \
| * | | move constants to where they belongMatthias Radestock2009-09-252-14/+10
| | | |
| * | | get rid of message attriburs in msg_store and msg_fileMatthias Radestock2009-09-243-59/+40
| | | |
| * | | replace msg_store:attrs with containsMatthias Radestock2009-09-235-68/+65
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a step on the path to getting rid of message attributes in msg_store. msg_store:attrs was only being used in disk_queue:prune, to detect when the store contained a non-persistent message and remove that message from the store and the rabbit_disk_queue table. Now rabbit_disk_queue records contain an IsPersistent flag. By making the msg count delta generator pay attention to that flag we trim non-persistent messages from the store during its initialisation, disk_queue:prune no longer needs to remove messages from the store, it just needs to remove all messages from the rabbit_disk_queue table which are no longer referenced by the store - hence the new msg_store:contains function. Keeping the IsPersistent flag in the rabbit_disk_queue table is sub-optimal since it means we store it once per message reference rather than just once per message. That's a small price to pay though for the cleaner interaction between the disk_queue and msg_store, and the opportunity to remove the notion of message attributes from msg_store altogether. Populating the new field in rabbit_disk_queue is straightforward in most places except disk_queue:tx_commit. That used to just be given {MsgId, IsDelivered} tuples, so I had to change the API to {MsgId, IsDelivered, IsPersistent} tuples.
| * | | merge headsMatthew Sackman2009-09-231-1/+1
| |\ \ \
| | * \ \ merge headsMatthew Sackman2009-09-231-19/+26
| | |\ \ \
| | * | | | english is betterMatthew Sackman2009-09-231-1/+1
| | | | | |
| * | | | | compact (well, delete, actually) files with no valid messagesMatthias Radestock2009-09-231-1/+1
| | |/ / / | |/| | |
| * | | | on startup, compact files from which we removed unreferenced messagesMatthias Radestock2009-09-231-19/+26
| |/ / /
| * | | inverting previous changeset...Matthew Sackman2009-09-231-4/+3
| | | |
| * | | overload open_file so it can cope with filenumbers and names, plus ↵Matthew Sackman2009-09-231-3/+4
| | | | | | | | | | | | | | | | associated minor refactorings
| * | | Made sure the magic marker message cannot be produced by a client by forcing ↵Matthew Sackman2009-09-231-3/+3
| | | | | | | | | | | | | | | | the guid to be the empty binary
| * | | rename load_messages to build_indexMatthias Radestock2009-09-231-16/+15
| | | | | | | | | | | | | | | | | | | | since that's what it does plus some minor cosmetic changes
| * | | remove verify_messages_referenced checkMatthias Radestock2009-09-231-16/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This can actually fail, e.g. when a message got ack'ed but the corresponding mnesia delete in dq hasn't been flushed yet. load_messages and the pruning in dq take care of this situation.
| * | | get rid of message cache limitMatthias Radestock2009-09-231-10/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | limiting the size of the message cache is pointless. Firstly, the ets memory info does not include binaries. Secondly, the cache is only holding onto messages which current queue processes are holding onto, so we are not actually leaking any memory, and the only cost is the cost of the cache entries themselves, which should be small.
| * | | refactoring: get rid of MnesiaDelete flag to remove_messagesMatthias Radestock2009-09-221-23/+14
| | | |
| * | | do not call msg_store:attrs/2 from disk_queue:{phantom_fetch,prefetch}Matthias Radestock2009-09-222-20/+11
| | | | | | | | | | | | | | | | | | | | | | | | It turns out that we don't actually need the 'persistent' attribute. So this saves us a potentially expensive interaction with the msg_store.
| * | | fix typoMatthias Radestock2009-09-221-1/+1
| | | |
| * | | drop disk_queue/msg_store disk_only modeMatthias Radestock2009-09-103-457/+119
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | and all the mode switching and memory management logic that goes with it. The 2G limitition of dets make the disk_only mode not worthwhile. In the process I refactored the msg_location access in msg_store s.t. it shouldn't be much effort to plug in a different index store in the future. Also some minor tweaks and tidying up here and there.
| * | | initialise msg_store ref counts via a generatorMatthias Radestock2009-09-102-57/+80
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is more flexible than the previous ref_count function, allowing the ref counts to be obtained w/o consuming any memory at the supplying end in a variety of scenarios. We use the dq_msg_loc ets/dets table to store the ref counts. That table is later updated with the full details of the messages (their file and position, etc). At the end we prune any entries that have a ref count but no associated file - i.e. the referenced message couldn't be found on disk. This change should also fix the "All replicas on diskfull nodes are not active yet" error observed in bug 21530 since we no longer need the indices on the rabbit_disk_queue mnesia table which we identified as the most likely cause of that error.
| * | | refer to files by their number not their nameMatthias Radestock2009-09-081-88/+84
| | | | | | | | | | | | | | | | | | | | This makes matching faster and keeps record sizes smaller. It also means we can get rid of one bit of state.
| * | | minor tidying up: use symbolic record indices instead of numeric onesMatthias Radestock2009-09-081-7/+8
| | | |
| * | | api tweak: pass durable queues in list rather than setMatthias Radestock2009-09-082-5/+4
| | | | | | | | | | | | | | | | | | | | lists are more pleasant than sets in APIs, plus in our use we have a list to start with.
| * | | change guid to a binary, using the md5 of term_to_binaryMatthias Radestock2009-09-085-39/+44
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The main motivation is to reduce the memory and on-disk footprint of the guid from ~34 bytes to 16. But it turns out that this actually results in a speed improvement of a few percent as well, even for non-persistent messaging, presumably due to the memory management effects and the fact that 16 byte binaries are easier to copy between processes than the deep(ish) original guid structure.
| * | | refactoring: move dets/ets match into sort_msg_locations_by_offsetMatthias Radestock2009-09-071-18/+9
| | | |
| * | | rename prune_mnesia to pruneMatthias Radestock2009-09-071-10/+10
| | | | | | | | | | | | | | | | since we prune more than just mnesia
| * | | shut up dialyzerMatthias Radestock2009-09-071-3/+4
| | | |
| * | | refactoring of delete_empty_filesMatthias Radestock2009-09-071-23/+26
| | | | | | | | | | | | | | | | | | | | giving it a name that describes what it does, and extracting recursion vestiges into caller.
| * | | generalise persistent flag to message attributes in rabbit_msg_storeMatthias Radestock2009-09-072-21/+19
| | | | | | | | | | | | | | | | | | | | This is just a renaming exercise, but it turns rabbit_msg_store into a general purpose message store.
| * | | generalise persistent flag to message attributes in rabbit_msg_fileMatthias Radestock2009-09-071-50/+43
| | | |
| * | | resolve msg_id type confusionMatthias Radestock2009-09-075-9/+14
| | | | | | | | | | | | | | | | rabbit_mixed_queue and rabbit_disk_queue see guids, not non_neg_integers.
| * | | perform term/binary conversion of msg body in rabbit_msg_fileMatthias Radestock2009-09-072-21/+21
| | | | | | | | | | | | | | | | thus further generalising rabbit_msg_file
| * | | do not return body size from rabbit_msg_file:readMatthias Radestock2009-09-072-12/+15
| | | | | | | | | | | | | | | | | | | | | | | | it's not used anywhere and was cluttering the api Also, make type sigs more meaningful and do not include rabbit.hrl, thus underlining the general nature of this module.
| * | | big refactoring: extract msg_store from disk_queueMatthias Radestock2009-09-062-1157/+1315
| | | | | | | | | | | | | | | | The msg_store knows nothing about queues, or message structure.
| * | | rename #file_summary_entry to #file_summaryMatthias Radestock2009-09-031-30/+30
| | | | | | | | | | | | | | | | to match what we call the containing table
| * | | rename #message_store_entry to #msg_locationMatthias Radestock2009-09-031-38/+38
| | | | | | | | | | | | | | | | to match what we call the containing table
| * | | Switched to using a file to hold the disk_only data. Also found a bug where ↵Matthew Sackman2009-09-031-22/+27
| | | | | | | | | | | | | | | | vaporise was wiping out the disk_only data (both as a file, and when it was in mnesia). The result was that if the dq was in disk_only mode before being vaporised, it would refuse to start up again. Thus vaporise now pushes the queue back to ram_disk mode if necessary, after wiping out the contents of the mnesia table. Finally, all tests pass again.
| * | | the disk queue sometimes needs much longer to shut down safely if it's ↵Matthew Sackman2009-09-031-1/+1
| | | | | | | | | | | | | | | | heavily loaded - up the limit to 5 seconds. However, I suspect something like 60 seconds is more likely to be realistic value
| * | | use a record in file summaryMatthew Sackman2009-09-031-46/+66
| | | |
| * | | ensure that we fsync after copying messages across during compactionMatthew Sackman2009-09-031-1/+1
| | | |
| * | | commented on purpose of cacheMatthew Sackman2009-09-031-0/+11
| | | |
| * | | sort files for compaction numericallyMatthew Sackman2009-09-031-1/+1
| | | |
View Lorry Controller Status