more info10.5-circular-buffer

2 files changed, 290 insertions, 0 deletions

diff --git a/email b/email
new file mode 100644
index 00000000000..d09fa3c4440
--- /dev/null
+++ b/email
@@ -0,0 +1,149 @@
+Hi Andrei , Sujatha
+
+I want to discuss two things first the actual implementation and the second thing will be
+the interface of circular queue with actual events processing and relay log.
+So in the circular buffer granularity of reading and writing will be either one event or
+one transaction.To achieve this will need to have transaction length in GTID event
+itself.
+>>Opinion needed:- How to store transaction length in the gtid_log_event, There are two
+options
+1. Store it in constant 8byte length.Issue with this is it will be 8bytes even if size
+is 1 byte so lots of wasted storage.
+2. Use variable memory_length to store the transaction length(net_store_length) mysql
+does the same Although there are some issues with this approach
+  1. get_data_size() will be accurate only after gtid_log_event::write_event is called.
+circular queue member variables.
+  2. We have already exhausted flags2(1 byte) in gtid_log_event so in future if we need to
+store flags we have to allocate space after storing transaction length. Since length is
+not fixed calculating flag will be more complex.
+Athough 2nd issue can be averted by storing one(or maybe 2) byte for flag in
+gtid_event_length.
+|--Old Gtid_event --|1 byte flag| 4byte thread id| variable transaction size|
+
+Circular queue Implementation
+
+Members:
+uchar* buffer;
+uchar* buffer_end;
+uint64 size;
+uint elements;
+/*
+ Some time we can have empty space in end if transaction/event is big to fit
+ in continues space.
+*/
+uchar* buffer_usable_ptr;
+uchar* write_head;
+uchar* read_head;
+uchar* flush_head;
+std::mutex read_lock;
+std::mutex write_lock;
+uchar* buffer_end;
+uint64 size;
+uint elements;
+
+so as we can see there are there are total three type of pointers pointers for the
+for the queue read write operation.
+
+write head :- this pointer in the buffer will be pointing to the current write
+head all the new inserts will be done after the write head. we will be guaranteeing
+that the granularity is  satisfied. and transaction / events will be written in
+continuous memory so in a case of when we have write head close to buffer end and
+we cannot have any space for whole transaction or event then we will write from
+starting of buffer given if it is empty in starting.
+
+read head:- reading operation will be simpler. user can read in 2 granularity
+either one whole transaction or one event. reading and writing granularity should be same. 
+
+flush head:- this will be the pointer which will be actually freeing the space
+on buffer.
+Some Important buffer method Implementation:-
+
+Empty space for this case
+S= Buffer_start
+F= Flush Pointer
+W= write pointer
+E= buffer end
+U= Buffer usable_ptr
+
+if this case
+S--F----W-----E
+E - W + F -S
+
+if this case
+S--W-- F --- U--E
+F-W
+
+Write code
+TS = Transaction size
+
+if this case
+S--F----W----E
+if W > F
+  if E-W > TS
+    then write data and W+= TS
+
+  if E-W < TS
+    buffer_usable_ptr= write_head
+    write_head= 0
+    It will become the second case
+
+
+if this case
+S--W----F--U--E
+if W < F
+  if TS < F -W
+    write
+    W+= TS
+  else
+    give error that buffer is full
+    and write into file
+
+Read from queue
+
+lock the mutex
+return_addr= R
+if R < W
+  S--R---W--E
+  R+= TS/ES
+  unlock the mutex
+  return return_addr
+
+if R > W
+  S--W---R--U--E
+  R += TS/ES
+  if R == U
+  R= 0
+  unlock the mutex
+  return return_addr
+(in rest of the cases)
+return NULL
+
+>>Opiniion needed:- I still have to figure out how I will be able to get a flush pointer.
+for example let's assume we have a one binlog group commit with 10 transactions.
+and our granularity is transaction level. and for simplicity let's assume we have
+10 worker threads so each thread will execute one transaction and it will increment
+read pointer to next transaction. so after 10 worker committing the 10 transaction
+, circular queue read pointer is advanced with 10 transaction, but now the issue
+is how we advance the flush pointer ? how we call back to queue that please advance
+flush pointer.
+
+>>Opinion needed:-
+second issue which I am facing is this how to differentiate whether circular buffer
+is full or empty when flush point and buffer pointer point to same location on buffer.
+one way is using a counter of elements second way is never have flush pointer And write
+pointer pointing to same location we can have something like this that flush. pointer
+is is always one less than right pointer. using elements approach is more intuitive but
+it will be hard to maintain number of elements when we are not sure about how to have
+flush logic.
+
+>>Opinion needed:-
+Handling in case of full buffer:- so when circular queue is full , we will be writing
+the events from io thread to temp file.
+so after this we have two options
+first option we can wait until buffer is empty and then copy from the temp file to buffer.
+this will be done by io thread. we will be temporary stop the thread from reading the events
+from network instead it will be copying events from file to buffer.
+second option which is also easier to implement when we have half buffer empty we will 
+send a broadcast condition that  buffer is half empty so the IO thread which will be
+listening to this condition will copy the events from temporary file to the circular
+buffer.
diff --git a/note b/note
new file mode 100644
index 00000000000..3b9458be16a
--- /dev/null
+++ b/note
@@ -0,0 +1,141 @@
+write_transaction or stmt
+one way get the cache length and then gtid length and then last event (mostly
+query log event) and add them up and save into gtid packet 
+second way no calculation relie on actual binlog event writen in cache and then
+write it into gtid event (fuck it wont work , what about fucking fsync and 
+i have to call write agian really bad idea)
+so what the fuck to do
+plan so far
+we will be giing wia first method since secind  meyhod will not work
+there are three type if  events one with comit one wi rollback and xid log event
+so query log  dvent and xid xheexkpoint eveent
+it shhould  be eastt yo get datta size from xid log but for qury we n3d to 
+do some  shit.
+one  option mite bee to use fjxed length since  there are onlyb2 options
+for  the  time this  is the the  only solutiin --- asshole this wont work
+
+So the plan right now
+write_transaction_or_stmt
+1. Rememember the current posiotion in cache (thread trx/stmt cache) (in case of
+rollback and commit only , Xid event will have fixed lenght), why the not genneralize 
+it ?
+2. write the event in cache(IO_CACHE) (how the fuck I will be doing i t, no clue)
+3.read the whole cache length , trx and stmt
+4 Pass it to write_gtid_event
+
+::write()
+No fucking clue
+here babe
+  {
+   5984     Log_event_writer writer(&cache_data->cache_log, cache_data);
+   5985
+   5986     /*
+   5987     ┆ Write pending event to the cache.
+   5988     */
+   5989     DBUG_EXECUTE_IF("simulate_disk_full_at_flush_pending",
+   5990     ┆   ┆   ┆   ┆   {DBUG_SET("+d,simulate_file_write_error");});
+   5991     if (writer.write(pending))
+   5992 }
+
+DDL
+fuck me
+
+So now for the writing in gtid_log_event
+We will have the size of transaction - gtid_event_size
+On the time of write , We need to have one flag
+But how ? I mean we just have
+bro i just explained that in #replication channel 
+see there
+in shorrt extendable one byte  flag every time when we over shoot
+how to store thread id 4bytes
+how we will be storing transaction size maybe use net_write_packet or 8 solid bytes
+
+FOR THE TIME LETS HAVE 8 +4 no flexible , we will change it later
+
+
+
+
+
+
+
+
+Circular buffer
+
+Memeber variables
+  uchar* buffer;
+  uchar* buffer_end;
+  uint64 size;
+  uint elements;
+  /*
+   Some time we can have empty space in end if transaction/event is big to fit
+   in continues space.
+  */
+  uchar* buffer_usable_ptr;
+  // Actual free space
+  uint64 usable_free_space;
+  uchar* write_head;
+  uchar* read_head;
+  uchar* flush_head;
+  std::mutex read_lock;
+  std::mutex write_lock;
+
+Empty space for this case
+S= Buffer_start
+F= Flush Pointer
+W= write pointer
+E= buffer end
+U= Buffer usable_ptr
+
+if this case
+S--F----W-----E
+E - W + F -S
+
+if this case
+S--W-- F --- U--E
+F-W
+
+
+Write code
+TS = Transaction size
+
+if this case
+S--F----W----E
+if E-W > TS
+then write data and W+= TS
+
+if E-W < TS
+buffer_usable_ptr= write_head
+write_head= 0
+It will become the second case
+
+
+if this case
+S--W----F--U--E
+if TS < F -W
+write
+W+= TS
+else
+  give error that buffer is full
+and write into file
+
+Read from queue
+ R < W
+S--R---W--E
+
+lock the mutex
+R+= TS/ES
+unlock the mutex
+return the old read ptr
+
+R > W
+S--W---R--U--E
+lock the mutex
+R += TS/ES
+if R == U
+R= 0
+unlock the mutex
+return the old ptr
+
+
+
+Bharthi no 02024407100 vandna mam