WL#3984 (Revise locking of mysql.general_log and mysql.slow_log)

Bug#25422 (Hang with log tables)
Bug 17876 (Truncating mysql.slow_log in a SP after using cursor locks the
          thread)
Bug 23044 (Warnings on flush of a log table)
Bug 29129 (Resetting general_log while the GLOBAL READ LOCK is set causes
           a deadlock)

Prior to this fix, the server would hang when performing concurrent
ALTER TABLE or TRUNCATE TABLE statements against the LOG TABLES,
which are mysql.general_log and mysql.slow_log.

The root cause traces to the following code:
in sql_base.cc, open_table()
  if (table->in_use != thd)
  {
    /* wait_for_condition will unlock LOCK_open for us */
    wait_for_condition(thd, &LOCK_open, &COND_refresh);
  }
The problem with this code is that the current implementation of the
LOGGER creates 'fake' THD objects, like
- Log_to_csv_event_handler::general_log_thd
- Log_to_csv_event_handler::slow_log_thd
which are not associated to a real thread running in the server,
so that waiting for these non-existing threads to release table locks
cause the dead lock.

In general, the design of Log_to_csv_event_handler does not fit into the
general architecture of the server, so that the concept of general_log_thd
and slow_log_thd has to be abandoned:
- this implementation does not work with table locking
- it will not work with commands like SHOW PROCESSLIST
- having the log tables always opened does not integrate well with DDL
operations / FLUSH TABLES / SET GLOBAL READ_ONLY

With this patch, the fundamental design of the LOGGER has been changed to:
- always open and close a log table when writing a log
- remove totally the usage of fake THD objects
- clarify how locking of log tables is implemented in general.

See WL#3984 for details related to the new locking design.

Additional changes (misc bugs exposed and fixed):

1)

mysqldump which would ignore some tables in dump_all_tables_in_db(),
 but forget to ignore the same in dump_all_views_in_db().

2)

mysqldump would also issue an empty "LOCK TABLE" command when all the tables
to lock are to be ignored (numrows == 0), instead of not issuing the query.

3)

Internal errors handlers could intercept errors but not warnings
(see sql_error.cc).

4)

Implementing a nested call to open tables, for the performance schema tables,
exposed an existing bug in remove_table_from_cache(), which would perform:
  in_use->some_tables_deleted=1;
against another thread, without any consideration about thread locking.
This call inside remove_table_from_cache() was not required anyway,
since calling mysql_lock_abort() takes care of aborting -- cleanly -- threads
that might hold a lock on a table.
This line (in_use->some_tables_deleted=1) has been removed.


sql/handler.cc:
  Moved logic for system / log tables in the SQL layer.
sql/handler.h:
  Moved logic for system / log tables in the SQL layer.
sql/lock.cc:
  Revised locking of log tables
sql/log.cc:
  Major cleanup: changed how log tables are locked / written to.
sql/log.h:
  Major cleanup: changed how log tables are locked / written to.
sql/mysql_priv.h:
  performance schema helpers
sql/slave.cc:
  open_ltable() lock flags
sql/sp.cc:
  open_ltable() lock flags
sql/sql_acl.cc:
  open_ltable() lock flags
sql/sql_class.h:
  performance schema helpers
sql/sql_delete.cc:
  log tables cleanup in TRUNCATE
sql/sql_error.cc:
  Internal handlers can also intercept warnings
sql/sql_insert.cc:
  open_ltable() lock flags
sql/sql_parse.cc:
  performance schema helpers
sql/sql_plugin.cc:
  open_ltable() lock flags
sql/sql_rename.cc:
  log tables cleanup in RENAME
sql/sql_servers.cc:
  open_ltable() lock flags
sql/sql_show.cc:
  Move INFORMATION_SCHEMA_NAME to table.cc
sql/sql_table.cc:
  log tables cleanup (admin operations, ALTER TABLE)
sql/sql_udf.cc:
  open_ltable() lock flags
sql/table.cc:
  Implemented TABLE_CATEGORY.
sql/share/errmsg.txt:
  Changed the wording and name of ER_CANT_READ_LOCK_LOG_TABLE
sql/table.h:
  Implemented TABLE_CATEGORY.
storage/csv/ha_tina.cc:
  Moved logic for system / log tables in the SQL layer.
storage/csv/ha_tina.h:
  Moved logic for system / log tables in the SQL layer.
storage/myisam/ha_myisam.cc:
  Moved logic for system / log tables in the SQL layer.
storage/myisam/ha_myisam.h:
  Moved logic for system / log tables in the SQL layer.
client/mysqldump.c:
  Don't lock tables in the ignore list.
  Don't issue empty LOCK TABLES queries.
sql/sql_base.cc:
  log tables cleanup
  performance schema helpers
mysql-test/r/ps.result:
  Adjust test results
mysql-test/r/show_check.result:
  Adjust test results
mysql-test/r/status.result:
  Adjust test results
mysql-test/t/log_state.test:
  Added tests for Bug#29129
mysql-test/t/ps.test:
  Make the test output deterministic
mysql-test/t/show_check.test:
  Make the test output deterministic
mysql-test/r/log_state.result:
  Changed the default location of the log output to LOG_FILE,
  for backward compatibility with MySQL 5.0
  ---
  Adjust test results
mysql-test/r/log_tables.result:
  cleanup for -ps-protocol
mysql-test/t/log_tables.test:
  cleanup for -ps-protocol
sql/set_var.cc:
  Changed the default location of the log output to LOG_FILE,
  for backward compatibility with MySQL 5.0
  ---
  log tables cleanup

1 files changed, 9 insertions, 40 deletions

diff --git a/sql/sql_table.cc b/sql/sql_table.cc
index dc3e72554fa..502600a673f 100644
--- a/sql/sql_table.cc
+++ b/sql/sql_table.cc
@@ -1513,7 +1513,7 @@ int mysql_rm_table_part2(THD *thd, TABLE_LIST *tables, bool if_exists,
       table->db_type= share->db_type();
 
     /* Disable drop of enabled log tables */
-    if (share && share->log_table &&
+    if (share && (share->table_category == TABLE_CATEGORY_PERFORMANCE) &&
         check_if_log_table(table->db_length, table->db,
                            table->table_name_length, table->table_name, 1))
     {
@@ -3966,7 +3966,7 @@ static bool mysql_admin_table(THD* thd, TABLE_LIST* tables,
   Item *item;
   Protocol *protocol= thd->protocol;
   LEX *lex= thd->lex;
-  int result_code, disable_logs= 0;
+  int result_code;
   DBUG_ENTER("mysql_admin_table");
 
   if (end_active_trans(thd))
@@ -4014,22 +4014,6 @@ static bool mysql_admin_table(THD* thd, TABLE_LIST* tables,
       if (view_operator_func == NULL)
         table->required_type=FRMTYPE_TABLE;
 
-      /*
-        If we want to perform an admin operation on the log table
-        (E.g. rename) and lock_type >= TL_READ_NO_INSERT disable
-        log tables
-      */
-
-      if (check_if_log_table(table->db_length, table->db,
-                             table->table_name_length,
-                             table->table_name, 1) &&
-          lock_type >= TL_READ_NO_INSERT)
-      {
-        disable_logs= 1;
-        logger.lock();
-        logger.tmp_close_log_tables(thd);
-      }
-
       open_and_lock_tables(thd, table);
       thd->no_warnings_for_error= 0;
       table->next_global= save_next_global;
@@ -4099,8 +4083,7 @@ static bool mysql_admin_table(THD* thd, TABLE_LIST* tables,
     }
 
     /* Close all instances of the table to allow repair to rename files */
-    if (lock_type == TL_WRITE && table->table->s->version &&
-        !table->table->s->log_table)
+    if (lock_type == TL_WRITE && table->table->s->version)
     {
       pthread_mutex_lock(&LOCK_open);
       const char *old_message=thd->enter_cond(&COND_refresh, &LOCK_open,
@@ -4258,7 +4241,7 @@ send_result_message:
       close_thread_tables(thd);
       if (!result_code) // recreation went ok
       {
-        if ((table->table= open_ltable(thd, table, lock_type)) &&
+        if ((table->table= open_ltable(thd, table, lock_type, 0)) &&
             ((result_code= table->table->file->analyze(thd, check_opt)) > 0))
           result_code= 0; // analyze went ok
       }
@@ -4324,10 +4307,9 @@ send_result_message:
     }
     if (table->table)
     {
-      /* in the below check we do not refresh the log tables */
       if (fatal_error)
         table->table->s->version=0;               // Force close of table
-      else if (open_for_modify && !table->table->s->log_table)
+      else if (open_for_modify)
       {
         if (table->table->s->tmp_table)
           table->table->file->info(HA_STATUS_CONST);
@@ -4350,24 +4332,11 @@ send_result_message:
   }
 
   send_eof(thd);
-  if (disable_logs)
-  {
-    if (logger.reopen_log_tables())
-      my_error(ER_CANT_ACTIVATE_LOG, MYF(0));
-    logger.unlock();
-  }
   DBUG_RETURN(FALSE);
 
  err:
   ha_autocommit_or_rollback(thd, 1);
   close_thread_tables(thd);			// Shouldn't be needed
-  /* enable logging back if needed */
-  if (disable_logs)
-  {
-    if (logger.reopen_log_tables())
-      my_error(ER_CANT_ACTIVATE_LOG, MYF(0));
-    logger.unlock();
-  }
   if (table)
     table->table=0;
   DBUG_RETURN(TRUE);
@@ -4812,7 +4781,7 @@ mysql_discard_or_import_tablespace(THD *thd,
    not complain when we lock the table
  */
   thd->tablespace_op= TRUE;
-  if (!(table=open_ltable(thd,table_list,TL_WRITE)))
+  if (!(table=open_ltable(thd, table_list, TL_WRITE, 0)))
   {
     thd->tablespace_op=FALSE;
     DBUG_RETURN(-1);
@@ -5728,7 +5697,7 @@ bool mysql_alter_table(THD *thd,char *new_db, char *new_name,
 #ifdef WITH_PARTITION_STORAGE_ENGINE
       if (alter_info->flags & ALTER_PARTITION)
       {
-	my_error(ER_WRONG_USAGE, MYF(0), "PARTITION", "log table");
+        my_error(ER_WRONG_USAGE, MYF(0), "PARTITION", "log table");
         DBUG_RETURN(TRUE);
       }
 #endif
@@ -5817,7 +5786,7 @@ view_err:
     start_waiting_global_read_lock(thd);
     DBUG_RETURN(error);
   }
-  if (!(table=open_ltable(thd,table_list,TL_WRITE_ALLOW_READ)))
+  if (!(table=open_ltable(thd, table_list, TL_WRITE_ALLOW_READ, 0)))
     DBUG_RETURN(TRUE);
   table->use_all_columns();
 
@@ -6986,7 +6955,7 @@ bool mysql_checksum_table(THD *thd, TABLE_LIST *tables,
 
     strxmov(table_name, table->db ,".", table->table_name, NullS);
 
-    t= table->table= open_ltable(thd, table, TL_READ);
+    t= table->table= open_ltable(thd, table, TL_READ, 0);
     thd->clear_error();			// these errors shouldn't get client
 
     protocol->prepare_for_resend();