Fix Bug 699794 -- device subclass open_device call must return child code

Even with changes to detect and clean up from errors in setpagedevice (b5)
and .bigstring, the segfault was still possible because the error return
code from the child was being ignored, and the device is_open was set true
when the child device was NOT open. Attempt to 'fillpage' on a clist device
that is not open is what caused the SEGV.

1 files changed, 14 insertions, 6 deletions

diff --git a/base/gdevsclass.c b/base/gdevsclass.c
index 5a8c22487..20c7f529e 100644
--- a/base/gdevsclass.c
+++ b/base/gdevsclass.c
@@ -99,13 +99,21 @@
  */
 int default_subclass_open_device(gx_device *dev)
 {
-    if (dev->child) {
-        dev_proc(dev->child, open_device)(dev->child);
-        dev->child->is_open = true;
-        gx_update_from_subclass(dev);
+    int code = 0;
+
+    /* observed with Bug 699794, don't set is_open = true if the open_device failed */
+    /* and make sure to propagate the return code from the child device to caller.  */
+    /* Only open the child if it was closed  and if child open is OK, return 1.     */
+    /* (see gs_opendevice) */
+    if (dev->child && dev->child->is_open == 0) {
+        code = dev_proc(dev->child, open_device)(dev->child);
+        if (code >= 0) {
+            dev->child->is_open = true;
+            code = 1;	/* device had been closed, but now is open */
+        }
+        gx_update_from_subclass(dev);	/* this is probably safe to do even if the open failed */
     }
-
-    return 0;
+    return code;
 }
 
 void default_subclass_get_initial_matrix(gx_device *dev, gs_matrix *pmat)