WT-3314 clarify WT_PANIC error handling (#3418)

A correctly-written WiredTiger application will have an error handler
for WT_PANIC, and that isn't explicit in the documentation.

4 files changed, 32 insertions, 13 deletions

diff --git a/dist/api_err.py b/dist/api_err.py
index 06887476dbc..bfa4459d438 100644
--- a/dist/api_err.py
+++ b/dist/api_err.py
@@ -41,10 +41,10 @@ errors = [
         WT_CURSOR::update or WT_CURSOR::remove.'''),
     Error('WT_PANIC', -31804,
         'WiredTiger library panic', '''
-        This error indicates an underlying problem that requires the
-        application exit and restart. The application can exit
-        immediately when \c WT_PANIC is returned from a WiredTiger
-        interface, no further WiredTiger calls are required.'''),
+        This error indicates an underlying problem that requires a database
+        restart. The application may exit immediately, no further WiredTiger
+        calls are required (and further calls will themselves immediately
+        fail).'''),
     Error('WT_RESTART', -31805,
         'restart the operation (internal)', undoc=True),
     Error('WT_RUN_RECOVERY', -31806,
diff --git a/examples/c/ex_event_handler.c b/examples/c/ex_event_handler.c
index 153b44c35bf..acd9d9beecc 100644
--- a/examples/c/ex_event_handler.c
+++ b/examples/c/ex_event_handler.c
@@ -70,6 +70,10 @@ handle_wiredtiger_error(WT_EVENT_HANDLER *handler,
 	    "app_id %s, thread context %p, error %d, message %s\n",
 	    custom_handler->app_id, (void *)session, error, message);
 
+	/* Exit if the database has a fatal error. */
+	if (error == WT_PANIC)
+		exit (1);
+
 	return (0);
 }
 
diff --git a/src/docs/error-handling.dox b/src/docs/error-handling.dox
index 62be498fc15..eb9ca6bb82a 100644
--- a/src/docs/error-handling.dox
+++ b/src/docs/error-handling.dox
@@ -17,13 +17,18 @@ are thrown as \c WiredTigerException, which may be caught by the
 application.
 
 The \c WiredTigerRollbackException is a specific type of \c WiredTigerException,
-it is thrown when there is a conflict between concurrent operations.
+thrown when there is a conflict between concurrent operations.
 An application that catches this exception should call rollback() on
 the relevant transaction, and retry as necessary.
 
 The \c WiredTigerPanicException is a specific type of \c WiredTigerException,
-it is thrown when there is an underlying problem that requires the
-application to exit and restart.
+thrown when there is a fatal error requiring database restart. Applications
+will normally handle \c WiredTigerPanicException as a special case. A
+correctly-written WiredTiger application will likely catch
+\c WiredTigerPanicException and immediately exit or otherwise handle fatal
+errors. Note that no further WiredTiger calls are required after
+\c WiredTigerPanicException is caught (and further calls will themselves
+immediately fail).
 
 The following is a complete list of possible WiredTiger-specific
 return values, all constants defined in the com.wiredtiger.db.wiredtiger class:
@@ -47,7 +52,7 @@ This error is returned when an error is not covered by a specific error return.
 This error indicates an operation did not find a value to return. This includes cursor search and other operations where no record matched the cursor's search key such as WT_CURSOR::update or WT_CURSOR::remove.
 
 @par <code>WT_PANIC</code>
-This error indicates an underlying problem that requires the application exit and restart. The application can exit immediately when \c WT_PANIC is returned from a WiredTiger interface, no further WiredTiger calls are required.
+This error indicates an underlying problem that requires a database restart. The application may exit immediately, no further WiredTiger calls are required (and further calls will themselves immediately fail).
 
 @par <code>WT_RUN_RECOVERY</code>
 This error is generated when wiredtiger_open is configured to return an error if recovery is required to use the database.
@@ -73,7 +78,7 @@ Note that ::wiredtiger_strerror is not thread-safe.
 @m_if{c}
 @section error_handling_event Error handling using the WT_EVENT_HANDLER
 
-More complex error handling can be configured by passing an implementation
+Specific error handling can be configured by passing an implementation
 of WT_EVENT_HANDLER to ::wiredtiger_open or WT_CONNECTION::open_session.
 
 For example, both informational and error messages might be passed to an
@@ -81,6 +86,17 @@ application-specific logging function that added a timestamp and logged
 the message to a file, and error messages might additionally be output to
 the \c stderr file stream.
 
+Additionally, applications will normally handle \c WT_PANIC as a special
+case.  WiredTiger will always call the error handler callback with
+\c WT_PANIC in the case of a fatal error requiring database restart,
+however, WiredTiger cannot guarantee applications will see an application
+thread return \c WT_PANIC from a WiredTiger API call. For this reason, a
+correctly-written WiredTiger application will likely specify at least an
+error handler which will immediately exit or otherwise handle fatal errors.
+Note that no further WiredTiger calls are required after an error handler
+is called with \c WT_PANIC (and further calls will themselves immediately
+fail).
+
 @snippet ex_event_handler.c Function event_handler
 @snippet ex_event_handler.c Configure event_handler
 
diff --git a/src/include/wiredtiger.in b/src/include/wiredtiger.in
index e38c41baccd..848448a453d 100644
--- a/src/include/wiredtiger.in
+++ b/src/include/wiredtiger.in
@@ -3143,10 +3143,9 @@ const char *wiredtiger_version(int *majorp, int *minorp, int *patchp)
 #define	WT_NOTFOUND	(-31803)
 /*!
  * WiredTiger library panic.
- * This error indicates an underlying problem that requires the application exit
- * and restart. The application can exit immediately when \c WT_PANIC is
- * returned from a WiredTiger interface, no further WiredTiger calls are
- * required.
+ * This error indicates an underlying problem that requires a database restart.
+ * The application may exit immediately, no further WiredTiger calls are
+ * required (and further calls will themselves immediately fail).
  */
 #define	WT_PANIC	(-31804)
 /*! @cond internal */