Library Version 11.2.5.3

Library Version 12.1.6.1

- + - +

Architecting Transactional Data Store applications			Architecting Transactional Data + Store applications
Prev	Chapter 11. - Berkeley DB Transactional Data Store Applications -	Chapter 11. Berkeley DB Transactional Data Store Applications	Next

@@ -34,350 +33,473 @@

Architecting Transactional Data Store applications

Architecting Transactional Data + Store applications

+ When building Transactional Data Store applications, the + architecture decisions involve application startup (running + recovery) and handling system or application failure. For + details on performing recovery, see the Recovery procedures. +

+ Recovery in a database environment is a single-threaded + procedure, that is, one thread of control or process must + complete database environment recovery before any other thread + of control or process operates in the Berkeley DB environment. +

+ Performing recovery first marks any existing database + environment as "failed" and then removes it, causing threads + of control running in the database environment to fail and + return to the application. This feature allows applications to + recover environments without concern for threads of control + that might still be running in the removed environment. The + subsequent re-creation of the database environment is + serialized, so multiple threads of control attempting to + create a database environment will serialize behind a single + creating thread. +

+ One consideration in removing (as part of recovering) a + database environment which may be in use by another thread, is + the type of mutex being used by the Berkeley DB library. In + the case of database environment failure when using + test-and-set mutexes, threads of control waiting on a mutex + when the environment is marked "failed" will quickly notice + the failure and will return an error from the Berkeley DB API. + In the case of environment failure when using blocking + mutexes, where the underlying system mutex implementation does + not unblock mutex waiters after the thread of control holding + the mutex dies, threads waiting on a mutex when an environment + is recovered might hang forever. Applications blocked on + events (for example, an application blocked on a network + socket, or a GUI event) may also fail to notice environment + recovery within a reasonable amount of time. Systems with such + mutex implementations are rare, but do exist; applications on + such systems should use an application architecture where the + thread recovering the database environment can explicitly + terminate any process using the failed environment, or + configure Berkeley DB for test-and-set mutexes, or incorporate + some form of long-running timer or watchdog process to wake or + kill blocked processes should they block for too long. +

- When building Transactional Data Store applications, the architecture - decisions involve application startup (running recovery) and handling - system or application failure. For details on performing recovery, see - the Recovery procedures. -

+ Regardless, it makes little sense for multiple threads of + control to simultaneously attempt recovery of a database + environment, since the last one to run will remove all + database environments created by the threads of control that + ran before it. However, for some applications, it may make + sense for applications to have a single thread of control that + performs recovery and then removes the database environment, + after which the application launches a number of processes, + any of which will create the database environment and continue + forward. +

- Recovery in a database environment is a single-threaded procedure, that - is, one thread of control or process must complete database environment - recovery before any other thread of control or process operates in the - Berkeley DB environment. -

- Performing recovery first marks any existing database environment as - "failed" and then removes it, causing threads of control running in the - database environment to fail and return to the application. This - feature allows applications to recover environments without concern for - threads of control that might still be running in the removed - environment. The subsequent re-creation of the database environment is - serialized, so multiple threads of control attempting to create a - database environment will serialize behind a single creating thread. -

- One consideration in removing (as part of recovering) a database - environment which may be in use by another thread, is the type of mutex - being used by the Berkeley DB library. In the case of database - environment failure when using test-and-set mutexes, threads of control - waiting on a mutex when the environment is marked "failed" will quickly - notice the failure and will return an error from the Berkeley DB API. - In the case of environment failure when using blocking mutexes, where - the underlying system mutex implementation does not unblock mutex - waiters after the thread of control holding the mutex dies, threads - waiting on a mutex when an environment is recovered might hang forever. - Applications blocked on events (for example, an application blocked on - a network socket, or a GUI event) may also fail to notice environment - recovery within a reasonable amount of time. Systems with such mutex - implementations are rare, but do exist; applications on such systems - should use an application architecture where the thread recovering the - database environment can explicitly terminate any process using the - failed environment, or configure Berkeley DB for test-and-set mutexes, - or incorporate some form of long-running timer or watchdog process to - wake or kill blocked processes should they block for too long. -

- Regardless, it makes little sense for multiple threads of control to - simultaneously attempt recovery of a database environment, since the - last one to run will remove all database environments created by the - threads of control that ran before it. However, for some applications, - it may make sense for applications to have a single thread of control - that performs recovery and then removes the database environment, after - which the application launches a number of processes, any of which will - create the database environment and continue forward. -

- There are three common ways to architect Berkeley DB Transactional Data - Store applications. The one chosen is usually based on whether or not - the application is comprised of a single process or group of processes - descended from a single process (for example, a server started when the - system first boots), or if the application is comprised of unrelated - processes (for example, processes started by web connections or users - logged into the system). -

+ There are four ways to architect Berkeley DB + Transactional Data Store applications. The one chosen is + usually based on whether or not the application is comprised + of a single process or group of processes descended from a + single process (for example, a server started when the system + first boots), or if the application is comprised of unrelated + processes (for example, processes started by web connections + or users logged into the system). +

- The first way to architect Transactional Data Store - applications is as a single process (the process may or may not - be multithreaded.) -
+ The first way to architect Transactional Data Store + applications is as a single process (the process may + or may not be multithreaded.) +
+
+ When this process starts, it runs recovery on the + database environment and then opens its databases. The + application can subsequently create new threads as it + chooses. Those threads can either share already open + Berkeley DB DB_ENV and DB handles, or create their + own. In this architecture, databases are rarely opened + or closed when more than a single thread of control is + running; that is, they are opened when only a single + thread is running, and closed after all threads but + one have exited. The last thread of control to exit + closes the databases and the database environment. +
+
+ This architecture is simplest to implement because + thread serialization is easy and failure detection + does not require monitoring multiple processes. +

- When this process starts, it runs recovery on the database - environment and then opens its databases. The application can - subsequently create new threads as it chooses. Those threads - can either share already open Berkeley DB DB_ENV and DB - handles, or create their own. In this architecture, databases - are rarely opened or closed when more than a single thread of - control is running; that is, they are opened when only a single - thread is running, and closed after all threads but one have - exited. The last thread of control to exit closes the - databases and the database environment. -
+ If the application's thread model allows processes + to continue after thread failure, the DB_ENV->failchk() + method can be used to determine if the database + environment is usable after thread failure. If the + application does not call DB_ENV->failchk(), or + DB_ENV->failchk() returns DB_RUNRECOVERY, + the application must + behave as if there has been a system failure, + performing recovery and re-creating the database + environment. Once these actions have been taken, other + threads of control can continue (as long as all + existing Berkeley DB handles are first discarded). +

- This architecture is simplest to implement because thread - serialization is easy and failure detection does not require - monitoring multiple processes. -
-
- If the application's thread model allows processes to continue - after thread failure, the DB_ENV->failchk() method can be used to - determine if the database environment is usable after thread - failure. If the application does not call DB_ENV->failchk(), or - DB_ENV->failchk() returns - DB_RUNRECOVERY, - the application must - behave as if there has been a system failure, performing - recovery and re-creating the database environment. Once these - actions have been taken, other threads of control can continue - (as long as all existing Berkeley DB handles are first - discarded). -
+ Note that by default DB_ENV->failchk() will only notify the + calling thread that the database environment is unusable. + However, you can optionally cause DB_ENV->failchk() to broadcast + this to other threads of control by using the + --enable-failchk_broadcast flag when you + compile your Berkeley DB library. If this option is turned + on, then all threads of control using the database + environment will return + DB_RUNRECOVERY + when they attempt to obtain a mutex lock. In this + situation, a DB_EVENT_FAILCHK_PANIC or + DB_EVENT_MUTEX_DIED event will also be + raised. (You use DB_ENV->set_event_notify() to examine events). +
+
+ The second way to architect Transactional Data + Store applications is as a group of related processes + (the processes may or may not be multithreaded). +

- The second way to architect Transactional Data Store - applications is as a group of related processes (the processes - may or may not be multithreaded). -
-
- This architecture requires the order in which threads of control are - created be controlled to serialize database environment recovery. -
-
- In addition, this architecture requires that threads of control - be monitored. If any thread of control exits with open - Berkeley DB handles, the application may call the DB_ENV->failchk() - method to detect lost mutexes and locks and determine if the - application can continue. If the application does not call - DB_ENV->failchk(), or DB_ENV->failchk() returns that the database - environment can no longer be used, the application must behave - as if there has been a system failure, performing recovery and - creating a new database environment. Once these actions have - been taken, other threads of control can be continued (as long - as all existing Berkeley DB handles are first discarded), or - -
+ This architecture requires the order in which + threads of control are created be controlled to + serialize database environment recovery. +

- The easiest way to structure groups of related processes is to - first create a single "watcher" process (often a script) that - starts when the system first boots, runs recovery on the - database environment and then creates the processes or threads - that will actually perform work. The initial thread has no - further responsibilities other than to wait on the threads of - control it has started, to ensure none of them unexpectedly - exit. If a thread of control exits, the watcher process - optionally calls the DB_ENV->failchk() method. If the application - does not call DB_ENV->failchk() or if DB_ENV->failchk() returns that the - environment can no longer be used, the watcher kills all of the - threads of control using the failed environment, runs recovery, - and starts new threads of control to perform work. -
+ In addition, this architecture requires that + threads of control be monitored. If any thread of + control exits with open Berkeley DB handles, the + application may call the DB_ENV->failchk() method to detect + lost mutexes and locks and determine if the + application can continue. If the application does not + call DB_ENV->failchk(), or DB_ENV->failchk() returns that the + database environment can no longer be used, the + application must behave as if there has been a system + failure, performing recovery and creating a new + database environment. Once these actions have been + taken, other threads of control can be continued (as + long as all existing Berkeley DB handles are first + discarded). +
+
+ The easiest way to structure groups of related + processes is to first create a single "watcher" + process (often a script) that starts when the system + first boots, runs recovery on the database environment + and then creates the processes or threads that will + actually perform work. The initial thread has no + further responsibilities other than to wait on the + threads of control it has started, to ensure none of + them unexpectedly exit. If a thread of control exits, + the watcher process optionally calls the DB_ENV->failchk() + method. If the application does not call DB_ENV->failchk() + or if DB_ENV->failchk() returns that the environment can no + longer be used, the watcher kills all of the threads + of control using the failed environment, runs + recovery, and starts new threads of control to perform + work. +
- The third way to architect Transactional Data Store - applications is as a group of unrelated processes (the - processes may or may not be multithreaded). This is the most - difficult architecture to implement because of the level of - difficulty in some systems of finding and monitoring unrelated - processes. There are several possible techniques to implement - this architecture. -
+ The third way to architect Transactional Data Store + applications is as a group of related processes that rely + on DB_ENV->failchk() broadcasting to inform other threads and + processes that recovery is required. DB_ENV->failchk() + broadcasting is not enabled by default for the DB + library, but using broadcasting means that a watcher + process is not required. Instead, if DB_ENV->failchk() fails + then all other threads and processes operating in that + environment will also be notified of that failure so that + they can know to run recovery. +

- One solution is to log a thread of control ID when a new - Berkeley DB handle is opened. For example, an initial - "watcher" process could run recovery on the database - environment and then create a sentinel file. Any "worker" - process wanting to use the environment would check for the - sentinel file. If the sentinel file does not exist, the worker - would fail or wait for the sentinel file to be created. Once - the sentinel file exists, the worker would register its process - ID with the watcher (via shared memory, IPC or some other - registry mechanism), and then the worker would open its - DB_ENV handles and proceed. When the worker finishes - using the environment, it would unregister its process ID with - the watcher. The watcher periodically checks to ensure that no - worker has failed while using the environment. If a worker - fails while using the environment, the watcher removes the - sentinel file, kills all of the workers currently using the - environment, runs recovery on the environment, and finally - creates a new sentinel file. -
+ To enable DB_ENV->failchk() broadcasting use the + --enable-failchk_broadcast flag when you + configure the library. On Windows, use + HAVE_FAILCHK_BROADCAST when you compile + the library. +

- The weakness of this approach is that, on some systems, it is - difficult to determine if an unrelated process is still - running. For example, POSIX systems generally disallow sending - signals to unrelated processes. The trick to monitoring - unrelated processes is to find a system resource held by the - process that will be modified if the process dies. On POSIX - systems, flock- or fcntl-style locking will work, as will - LockFile on Windows systems. Other systems may have to use - other process-related information such as file reference counts - or modification times. In the worst case, threads of control - can be required to periodically re-register with the watcher - process: if the watcher has not heard from a thread of control - in a specified period of time, the watcher will take action, - recovering the environment. -
-
- The Berkeley DB library includes one built-in implementation of this approach, - the DB_ENV->open() method's DB_REGISTER flag: -
-
- If the DB_REGISTER flag is set, each process opening the - database environment first checks to see if recovery needs to - be performed. If recovery needs to be performed for any reason - (including the initial creation of the database environment), - and DB_RECOVER is also specified, recovery will be performed - and then the open will proceed normally. If recovery needs to - be performed and DB_RECOVER is not specified, - DB_RUNRECOVERY - will be returned. If recovery does not need to be performed, - DB_RECOVER will be ignored. -
-
- Prior to the actual recovery beginning, the DB_EVENT_REG_PANIC - event is set for the environment. Processes in the application using - the DB_ENV->set_event_notify() method will be notified when they do their next - operations in the environment. Processes receiving this event should - exit the environment. Also, the DB_EVENT_REG_ALIVE event will be - triggered if there are other processes currently attached to the - environment. Only the process doing the recovery will receive this - event notification. It will receive this notification once for each - process still attached to the environment. The parameter of the - DB_ENV->set_event_notify() callback will contain the process identifier of the - process still attached. The process doing the recovery can then - signal the attached process or perform some other operation prior to - recovery (i.e. kill the attached process). -
-
- The DB_ENV->set_timeout() method's DB_SET_REG_TIMEOUT flag can be set to - establish a wait period before starting recovery. This creates a - window of time for other processes to receive the DB_EVENT_REG_PANIC - event and exit the environment. -
-
- There are three additional requirements for the DB_REGISTER - architecture to work: -
+ If DB_ENV->failchk() broadcasting is enabled for your library + and a thread of control encounters a failure when + DB_ENV->failchk() is run, then all other threads and processes + operating in that environment will be notified. If a + failure is broadcast, then threads and processes will + receive + DB_RUNRECOVERY + when they attempt to preform any one of a range of + activities, including: +
- - First, all applications using the database environment must - specify the DB_REGISTER flag when opening the environment. - However, there is no additional requirement if the application - chooses a single process to recover the environment, as the - first process to open the database environment will know to - perform recovery. -
  + When entering a DB API. +
- - Second, there can only be a single DB_ENV handle per database - environment in each process. As the DB_REGISTER locking is - per-process, not per-thread, multiple DB_ENV handles in a single - environment could race with each other, potentially causing - data corruption. -
  + When locking a mutex. +
- - Third, the DB_REGISTER implementation does not explicitly - terminate processes using the database environment which is - being recovered. Instead, it relies on the processes - themselves noticing the database environment has been discarded - from underneath them. For this reason, the DB_REGISTER flag - should be used with a mutex implementation that does not block - in the operating system, as that risks a thread of control - blocking forever on a mutex which will never be granted. Using - any test-and-set mutex implementation ensures this cannot - happen, and for that reason the DB_REGISTER flag is generally - used with a test-and-set mutex implementation. -
  + When performing disk or network I/O. +
- A second solution for groups of unrelated processes is also - based on a "watcher process". This solution is intended for - systems where it is not practical to monitor the processes - sharing a database environment, but it is possible to monitor - the environment to detect if a thread of control has failed - holding open Berkeley DB handles. This would be done by having - a "watcher" process periodically call the DB_ENV->failchk() method. - If DB_ENV->failchk() returns that the environment can no longer be - used, the watcher would then take action, recovering the - environment. -
+ Threads and processes that are + monitoring events will also receive + DB_EVENT_FAILCHK_PANIC or + DB_EVENT_MUTEX_DIED. You use + DB_ENV->set_event_notify() to examine events. +
+
+
+ The fourth way to architect Transactional Data Store + applications is as a group of unrelated processes (the + processes may or may not be multithreaded). This is + the most difficult architecture to implement because + of the level of difficulty in some systems of finding + and monitoring unrelated processes. There are several + possible techniques to implement this architecture. +
+
+ One solution is to log a thread of control ID when + a new Berkeley DB handle is opened. For example, an + initial "watcher" process could run recovery on the + database environment and then create a sentinel file. + Any "worker" process wanting to use the environment + would check for the sentinel file. If the sentinel + file does not exist, the worker would fail or wait for + the sentinel file to be created. Once the sentinel + file exists, the worker would register its process ID + with the watcher (via shared memory, IPC or some other + registry mechanism), and then the worker would open + its DB_ENV handles and proceed. When the worker + finishes using the environment, it would unregister + its process ID with the watcher. The watcher + periodically checks to ensure that no worker has + failed while using the environment. If a worker fails + while using the environment, the watcher removes the + sentinel file, kills all of the workers currently + using the environment, runs recovery on the + environment, and finally creates a new sentinel file. +
+
+ The weakness of this approach is that, on some + systems, it is difficult to determine if an unrelated + process is still running. For example, POSIX systems + generally disallow sending signals to unrelated + processes. The trick to monitoring unrelated processes + is to find a system resource held by the process that + will be modified if the process dies. On POSIX + systems, flock- or fcntl-style locking will work, as + will LockFile on Windows systems. Other systems may + have to use other process-related information such as + file reference counts or modification times. In the + worst case, threads of control can be required to + periodically re-register with the watcher process: if + the watcher has not heard from a thread of control in + a specified period of time, the watcher will take + action, recovering the environment. +
+
+ The Berkeley DB library includes one built-in + implementation of this approach, the DB_ENV->open() + method's DB_REGISTER flag: +

- The weakness of this approach is that all threads of control - using the environment must specify an "ID" function and an - "is-alive" function using the DB_ENV->set_thread_id() method. (In - other words, the Berkeley DB library must be able to assign a - unique ID to each thread of control, and additionally determine - if the thread of control is still running. It can be difficult - to portably provide that information in applications using a - variety of different programming languages and running on a - variety of different platforms.) -
+ If the DB_REGISTER flag is set, each process + opening the database environment first checks to see + if recovery needs to be performed. If recovery needs + to be performed for any reason (including the initial + creation of the database environment), and + DB_RECOVER is also specified, recovery will be + performed and then the open will proceed normally. If + recovery needs to be performed and DB_RECOVER is not + specified, DB_RUNRECOVERY + will be returned. If recovery does not need to be performed, DB_RECOVER + will be ignored. +

- A third solution for groups of unrelated processes is a hybrid of the two - above. Along with implementing the built-in sentinel approach with the - the DB_ENV->open() methods DB_REGISTER flag, the DB_FAILCHK flag can be specified. - When using both flags, each process opening the database environment first - checks to see if recocvery needs to be performed. If recovery needs to be - performed for any reason, it will first determine if a thread of control - exited while holding database read locks, and release those. Then it will - abort any unresolved transactions. If these steps are successful, the process - opening the environment will continue without the need for any - additional recocvery. If these steps are unsuccessful, then additional - recovery will be performed if DB_RECOVER is specified and if DB_RECOVER is not - specified, DB_RUNRECOVERYwill be returned. -
+ Prior to the actual recovery beginning, the + DB_EVENT_REG_PANIC event is set for the environment. + Processes in the application using the + DB_ENV->set_event_notify() method will be notified when they do + their next operations in the environment. Processes + receiving this event should exit the environment. + Also, the DB_EVENT_REG_ALIVE event will be triggered + if there are other processes currently attached to the + environment. Only the process doing the recovery will + receive this event notification. It will receive this + notification once for each process still attached to + the environment. The parameter of the + DB_ENV->set_event_notify() callback will contain the process + identifier of the process still attached. The process + doing the recovery can then signal the attached + process or perform some other operation prior to + recovery (i.e. kill the attached process). +

- Since this solution is hybrid of the first two, all of the requirements of both - of them must be implemented (will need "ID" function, "is-alive" function, - single DB_ENV handle per database, etc.) -
+ The DB_ENV->set_timeout() method's DB_SET_REG_TIMEOUT + flag can be set to establish a wait period before + starting recovery. This creates a window of time for + other processes to receive the DB_EVENT_REG_PANIC + event and exit the environment. +

- The described approaches are different, and should not be - combined. Applications might use either the DB_REGISTER - approach, the DB_ENV->failchk() or the hybrid approach, but not together in - the same application. For example, a POSIX application written - as a library underneath a wide variety of interfaces and - differing APIs might choose the DB_REGISTER approach for a - few reasons: first, it does not require making periodic calls - to the DB_ENV->failchk() method; second, when implementing in a - variety of languages, is may be more difficult to specify - unique IDs for each thread of control; third, it may be more - difficult determine if a thread of control is still running, as - any particular thread of control is likely to lack sufficient - permissions to signal other processes. Alternatively, an - application with a dedicated watcher process, running with - appropriate permissions, might choose the DB_ENV->failchk() approach - as supporting higher overall throughput and reliability, as - that approach allows the application to abort unresolved - transactions and continue forward without having to recover the - database environment. The hybrid approach is useful in situations - where running a dedicated watcher process is not practical but getting the - equivalent of DB_ENV->failchk() on the DB_ENV->open() is important. -
+ There are three additional requirements for the + DB_REGISTER architecture to work: +
+
+
- +
  + First, all applications using the database + environment must specify the DB_REGISTER + flag when opening the environment. However, + there is no additional requirement if the + application chooses a single process to + recover the environment, as the first process + to open the database environment will know to + perform recovery. +
  +
- +
  + Second, there can only be a single DB_ENV + handle per database environment in each + process. As the DB_REGISTER locking is + per-process, not per-thread, multiple DB_ENV + handles in a single environment could race + with each other, potentially causing data + corruption. +
  +
- +
  + Third, the DB_REGISTER implementation + does not explicitly terminate processes using + the database environment which is being + recovered. Instead, it relies on the processes + themselves noticing the database environment + has been discarded from underneath them. For + this reason, the DB_REGISTER flag should be + used with a mutex implementation that does not + block in the operating system, as that risks a + thread of control blocking forever on a mutex + which will never be granted. Using any + test-and-set mutex implementation ensures this + cannot happen, and for that reason the + DB_REGISTER flag is generally used with a + test-and-set mutex implementation. +
  +
+
+
+ A second solution for groups of unrelated processes + is also based on a "watcher process". This solution is + intended for systems where it is not practical to + monitor the processes sharing a database environment, + but it is possible to monitor the environment to + detect if a thread of control has failed holding open + Berkeley DB handles. This would be done by having a + "watcher" process periodically call the DB_ENV->failchk() + method. If DB_ENV->failchk() returns that the environment + can no longer be used, the watcher would then take + action, recovering the environment. +
+
+ The weakness of this approach is that all threads + of control using the environment must specify an "ID" + function and an "is-alive" function using the + DB_ENV->set_thread_id() method. (In other words, the + Berkeley DB library must be able to assign a unique ID + to each thread of control, and additionally determine + if the thread of control is still running. It can be + difficult to portably provide that information in + applications using a variety of different programming + languages and running on a variety of different + platforms.) +
+
+ A third solution for groups of unrelated processes + is a hybrid of the two above. Along with implementing + the built-in sentinel approach with the the DB_ENV->open() + methods DB_REGISTER flag, the DB_FAILCHK flag can + be specified. When using both flags, each process + opening the database environment first checks to see + if recovery needs to be performed. If recovery needs + to be performed for any reason, it will first + determine if a thread of control exited while holding + database read locks, and release those. Then it will + abort any unresolved transactions. If these steps are + successful, the process opening the environment will + continue without the need for any additional recovery. + If these steps are unsuccessful, then additional + recovery will be performed if DB_RECOVER is + specified and if DB_RECOVER is not specified, DB_RUNRECOVERY + will be returned. +
+
+ Since this solution is hybrid of the first two, all + of the requirements of both of them must be + implemented (will need "ID" function, "is-alive" + function, single DB_ENV handle per database, etc.) +
+
+ The described approaches are different, and should + not be combined. Applications might use either the + DB_REGISTER approach, the DB_ENV->failchk() or the hybrid + approach, but not together in the same application. + For example, a POSIX application written as a library + underneath a wide variety of interfaces and differing + APIs might choose the DB_REGISTER approach for a few + reasons: first, it does not require making periodic + calls to the DB_ENV->failchk() method; second, when + implementing in a variety of languages, is may be more + difficult to specify unique IDs for each thread of + control; third, it may be more difficult determine if + a thread of control is still running, as any + particular thread of control is likely to lack + sufficient permissions to signal other processes. + Alternatively, an application with a dedicated watcher + process, running with appropriate permissions, might + choose the DB_ENV->failchk() approach as supporting higher + overall throughput and reliability, as that approach + allows the application to abort unresolved + transactions and continue forward without having to + recover the database environment. The hybrid approach + is useful in situations where running a dedicated + watcher process is not practical but getting the + equivalent of DB_ENV->failchk() on the DB_ENV->open() is + important. +

- Obviously, when implementing a process to monitor other threads of - control, it is important the watcher process' code be as simple and - well-tested as possible, because the application may hang if it fails. -

+ Obviously, when implementing a process to monitor other + threads of control, it is important the watcher process' code + be as simple and well-tested as possible, because the + application may hang if it fails. +