1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
|
Design for Hybrid SQL/CLFS-Based Store in Qpid
==============================================
CLFS (Common Log File System) is a new facility in recent Windows versions.
CLFS is an ARIES-compliant log intended to support high performance and
transactional applications. CLFS is available in Windows Server 2003R2 and
higher, as well as Windows Vista and Windows 7.
There is currently an all-SQL store in Qpid. The new hybrid SQL-CLFS store
moves the message, messages-mapping to queues, and transaction aspects
of the SQL store into CLFS logs. Records of queues, exchanges, bindings,
and configurations will remain in SQL. The main goal of this change is
to yield higher performance on the time-critical messaging operations.
CLFS and, therefore, the new hybrid store, is not available on Windows XP
and Windows Server prior to 2003R2; these platforms will need to run the
all-SQL store.
Note for future consideration: it is possible to maintain all durable
objects in CLFS, which would remove the need for SQL completely. It would
require added log handling as well as the logic to ensure referential
integrity between exchanges and queues via bindings as SQL does today.
Also, the CLFS store counts on the SQL-stored queue records being correct
when recovering messages; if a message operation in the log refers to a queue
ID that's unknown, the CLFS store assumes the queue was deleted in the
previous broker session and the log wasn't updated. That sort of assumption
would need to be revisited if all content moves to a log.
CLFS Capabilities
-----------------
This section explains some of the key CLFS concepts that are important
in order to understand the designed use of CLFS for the store. It is
not a complete explanation and is not feature-complete. Please see the
CLFS documentation at MSDN for complete details
(http://msdn.microsoft.com/en-us/library/bb986747%28v=VS.85%29.aspx).
CLFS provides logs; each log can be dedicated or multiplexed. A multiplexed
log has multiple streams of independent log records; a dedicated log has
only one stream. Each log uses containers to hold the actual data; a log
requires a minimum of two containers, each of which must be at least 512KB.
Thus, the smallest log possible is 1MB. They can, of course, be larger, but
with 1 MB as minimum size for a log, they shouldn't be used willy-nilly.
The maximum number of streams per log is approximately 100.
As records are written to the log CLFS assigns Log Sequence Numbers (LSNs).
The first valid LSN in a log stream is called the Base, or Tail. CLFS
can automatically reclaim and reuse container space for the log as the
base LSN is moved when records are no longer needed. When a log is multiplexed,
a stream which doesn't move its tail can prevent CLFS from reclaiming space
and cause the log to grow indefinitely. Thus, mixing streams which don't
update (and, thus, move their tails) with streams that are very dynamic in
a single log will probably cause the log to continue to expand even though
much of the space will be unused.
CLFS provides three LSN types that are used to chain records together:
- Next: This is a forward sequence maintained by CLFS itself by the order
records are put into the stream.
- Undo-next, Undo-prev: These are backward-looking chains that are used
to link a new record to some previous record(s) in the same stream.
Also note that although log files are simply located in the file system,
easily locatable, streams within a log are not easily known or listable
outside of some application-specific recording of the stream names somewhere.
Log Usage
---------
There are two logs in use.
- Message: Each message will be represented by a chain of log records. All
messages will be intermixed in the same dedicated stream. Each portion of
a message content (sometimes they are written in multiple chunks) as well
as each operation involving a message (enqueue, dequeue, etc.) will be
in a log record chained to the others related to the same message.
- Transaction: Each transaction, local and distributed, will be represented
by a chain of log records. The record content will denote the transaction
as local or distributed.
Both transaction and message logs use the LSN of the first record for a
given object (message or transaction) as the persistence ID for that object.
The LSN is a CLFS-maintained, always-increasing value that is 64 bits long,
the same as a persistence ID.
Log records that relate to a transaction or message previously logged use the
log record undo-prev LSN to indicate which transaction/message the record
relates to.
Message Log Records
-------------------
Message log records will be one of the following types:
- Message-Start: the first (and possibly only) section of message content
- Message-Chunk: second and succeeding message content chunks
- Message-Delete: marks the end of the message's lifetime
- Message-Enqueue: records the message's placement on a queue
- Message-Dequeue: records the message's removal from a queue
The LSN of the Message-Start record is the persistence ID for the message.
The log record undo-prev LSN is used to link each subsequent record for that
message to the Message-Start record.
A message's sequence of log records is extended for each operation on that
message, until the message is deleted whereupon a Message-Delete record is
written. When the Message-Delete is written, the log's base LSN can be moved
up to the next earliest message if the deleted one opens up a set of
records at the tail of the log that are no longer needed. To help maintain
the order and know when the base can be moved, the store keeps message
information in a STL map whose key is the message ID (Message-Start LSN).
Thus, the first entry in the map is the earliest ID/LSN in use.
During recovery, messages still residing in the log can be ignored when the
record sequence for the message ends with Message-Delete. Similarly, there
may be log records for messages that are deleted; in this case the previous
LSN won't be one that's still within the log and, therefore, there won't have
been a Message Start record recovered and the record can be ignored.
Transaction Log Records
-----------------------
Transaction log records will be one of the following types:
- Dtx-Start: Start of a distributed transaction
- Tx-Start: Start of a local transaction
- End: End of the transaction
- Rollback: Marks that the transaction is rolled back
- Prepare: Marks the dtx as prepared
- Commit: Marks the transaction as committed
- Delete: Notes that the transaction is no longer valid
Transactions are also identified by the LSN of the start (Dtx-Start or
Tx-Start) record. Successive records associated with the same transaction
are linked backwards using the undo-prev LSN.
The association between messages and transactions is maintained in the
message log; if the message enqueue/dequeue operation is part of a transaction,
the operation includes a transaction ID. The transaction log maintains the
state of the transaction itself. Thus, each operation (enqueue, dequeue,
prepare, rollback, commit) is a single log record.
A few notes:
- The transactions need to be recovered and sorted out prior to recovering
the messages. The message recovery needs to know if a enqueue/dequeue
associated with a transaction can be discarded or should be acted on.
- Transaction IDs need to remain valid as long as any messages exist that
refer to them. This prevents the problem of trying to recover a message
with a transaction ID that doesn't exist - was it finalized? was it aborted?
Reference to a missing transaction ID can be ignored with assurance that
the message was deleted further along or the transaction would still be there.
- Transaction IDs needing to be valid requires that a refcount be kept on each
transaction at run time. As messages are deleted, the transaction set can
be notified that the message is gone. To enforce this, Message objects have
a boost::shared_ptr to each Transaction they're associated with. When the
Message is destroyed, refs to Transactions go down too. When Transaction is
destroyed, it's done so write its delete to the log.
In-Memory Objects
-----------------
The store holds the message and transaction relationships in memory. CLFS is
a backing store for that information so it can be reliably reconstructed in
the event of a failure. This is a change from the SQL-only store where all
of the information is maintained in SQL and none is kept in memory. The
CLFS-using store is designed for high-throughput operation where it is assumed
that messages will transit the broker (and, therefore, the store) quickly.
- Message list: this is a map of persistence ID (message LSN) to a list of
queues where the message is located and an indication that there is
(or isn't) a transaction involved and in which direction (enqueue/dequeue)
so a dequeued message doesn't get deleted while a transacted enqueue is
pending.
- Transaction list: also probably a map of id/LSN to a transaction object.
The transaction object needs to keep a list of messages/queues that are
impacted as well as the transaction state and Xid (for dtx).
- Right now log records are written as need with no preallocation or
reservation. It may be better to pre-reserve records in some cases, such
as a transaction prepare where the space for commit or rollback may be
reserved at the same time. This may be the only case where losing a
record may be an issue - needs some more thought.
Recovery
--------
During recovery, need to verify recovered messages' queues exist; if there's a
failure after a queue's deletion is final but before the messages are recorded
as dequeued (and possibly deleted) the remainder of those dequeues (and
possibly deleting the message) needs to be handled during recovery by not
restoring them for the broker, and also logging their deletion. Could also
skip the logging of deletion and let the normal tail-maintenance eventually
move up over the old message entries. Since the invalid messages won't be
kept in the message map, their IDs won't be taken into account when maintaining
the tail - the tail will move up over them as soon as enough messages come
and go.
Plugin Options
--------------
The command-line options added by the CLFS plugin are;
--connect The SQL connect string for the SQL parts; same as the
SQL plugin.
--catalog The SQL database (catalog) name; same as the SQL plugin.
--store-dir The directory to store the logs in. Defaults to the
broker --data-dir value. If --no-data-dir specified,
--store-dir must be.
--container-size The size of each container in the log, in bytes. The
minimum size is 512K (smaller sizes will be rounded up).
Additionally, the size will be rounded up to a multiple
of the sector size on the disk holding the log. Once
the log is created, each newly added container will
be the same size as the initial container(s). Default
is 1MB.
--initial-containers The number of containers to populate a new log with
if a new log is created. Ignored if the log exists.
Default is 2.
--max-write-buffers The maximum number of write buffers that the plugin can
use before CLFS automatically flushes the log to disk.
Lower values flush more often; higher values have
higher performance. Default is 10.
Maybe need an option to hold messages of a certain size in memory? I think
maybe the broker proper holds the message content, so the store need not.
Testing
-------
More tests will need to be written to stress the log container extension
capability and ensure that moving the base LSN works properly and the store
doesn't continually grow the log without bounds.
Note that running "qpid-perftest --durable yes" stresses the log extension
and tail maintenance. It doesn't get run as a normal regression test but should
be run when playing with the container/tail maintenance logic to ensure it's
not broken.
|
