MDEV-28757 S3 tables can hang in SELECT

If we got a read error from S3, we did not signal threads waiting
to read blocks in the read-range. This caused these threads to
hang forever.

There is still one issue left that the S3 error will be logged as an
'table is crashed' error instead of the IO error. This will be fixed
by a larger patch in 10.6 that improves error reporting from Aria.

There is no test case for this as it is very hard to repeat.
I tested this with a patch that causes random read failures in S3
used perl multi-threaded test with 8 threads to simulate reads.
This patch fixes all found hangs.

2 files changed, 10 insertions, 14 deletions

diff --git a/storage/maria/ma_locking.c b/storage/maria/ma_locking.c
index f895e66d9e0..e9a94b05c18 100644
--- a/storage/maria/ma_locking.c
+++ b/storage/maria/ma_locking.c
@@ -553,7 +553,6 @@ void _ma_mark_file_crashed(MARIA_SHARE *share)
 {
   uchar buff[2];
   DBUG_ENTER("_ma_mark_file_crashed");
-  CRASH_IF_S3_TABLE(share);
 
   share->state.changed|= STATE_CRASHED;
   if (share->no_status_updates)
@@ -561,7 +560,6 @@ void _ma_mark_file_crashed(MARIA_SHARE *share)
 
   mi_int2store(buff, share->state.changed);
 
-
   /*
     We can ignore the errors, as if the mark failed, there isn't anything
     else we can do;  The user should already have got an error that the
diff --git a/storage/maria/ma_pagecache.c b/storage/maria/ma_pagecache.c
index 00ae13b948a..a2802d909c0 100644
--- a/storage/maria/ma_pagecache.c
+++ b/storage/maria/ma_pagecache.c
@@ -2893,10 +2893,14 @@ static void read_big_block(PAGECACHE *pagecache,
   if (pagecache->big_block_read(pagecache, &args, &block->hash_link->file,
                                 &data))
   {
+    pagecache->big_block_free(&data);
     pagecache_pthread_mutex_lock(&pagecache->cache_lock);
     block_to_read->status|= PCBLOCK_ERROR;
     block_to_read->error= (int16) my_errno;
-    pagecache->big_block_free(&data);
+
+    /* Handle the block that we originally wanted with read */
+    block->status|= PCBLOCK_ERROR;
+    block->error= block_to_read->error;
     goto error;
   }
 
@@ -2980,6 +2984,7 @@ end:
   block_to_read->status&= ~PCBLOCK_BIG_READ;
   if (block_to_read != block)
   {
+    /* Unlock the 'first block' in the big read */
     remove_reader(block_to_read);
     unreg_request(pagecache, block_to_read, 1);
   }
@@ -2993,18 +2998,11 @@ error:
     Read failed. Mark all readers waiting for the a block covered by the
     big block that the read failed
   */
-  for (offset= pagecache->block_size, page= page_to_read + 1;
-       offset < data.length;
-       offset+= pagecache->block_size, page++)
+  for (offset= 0, page= page_to_read + 1;
+       offset < big_block_size_in_pages;
+       offset++)
   {
-    DBUG_ASSERT(offset + pagecache->block_size <= data.length);
-    if (page == our_page)
-    {
-      DBUG_ASSERT(!(block->status & PCBLOCK_READ));
-      block->status|= PCBLOCK_ERROR;
-      block->error= (int16) my_errno;
-    }
-    else
+    if (page != our_page)
     {
       PAGECACHE_BLOCK_LINK *bl;
       bl= find_block(pagecache,  &block->hash_link->file, page, 1,