Stage 22: allow builtin token concatenation outside $@.

* m4/m4module.h (m4_is_arg_composite): New prototype.
(m4_symbol_value_copy): Change return type.
(m4_arg_text): Add parameter.
(M4ARG): Adjust callers.
* m4/m4private.h: Adjust comments.
* m4/symtab.c (m4_symbol_value_copy): Detect when builtins are
flattened.
* m4/input.c (init_builtin_token): Add parameter, and allow
concatenating builtins.
(m4__next_token): Adjust caller.
* m4/macro.c (m4_is_arg_composite): New function.
(expand_argument): Allow builtin concatenation.
(m4_arg_text): Add parameter.
(m4__arg_adjust_refcount, m4__arg_print): Adjust callers.
(m4_arg_equal): Fix comparison of builtin tokens.
* modules/m4.c (define, pushdef): Warn when flattening builtins.
* doc/m4.texinfo (Define): Remove dead comment.
(Defn): Update to reflect code changes.
* tests/builtins.at (defn): Remove xfail.
* NEWS: Document this change.

Signed-off-by: Eric Blake <ebb9@byu.net>

10 files changed, 207 insertions, 152 deletions

diff --git a/ChangeLog b/ChangeLog
index cd1f9276..3470af11 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,32 @@
+2008-05-05  Eric Blake  <ebb9@byu.net>
+
+	Stage 22: allow builtin token concatenation outside $@.
+	Adjust the input and argument parsing engines to append builtins
+	alongside text.  Make define warn when builtins must be
+	flattened.
+	Memory impact: slight penalty, with fewer builtins flattened.
+	Speed impact: slight penalty, from more bookkeeping.
+	* m4/m4module.h (m4_is_arg_composite): New prototype.
+	(m4_symbol_value_copy): Change return type.
+	(m4_arg_text): Add parameter.
+	(M4ARG): Adjust callers.
+	* m4/m4private.h: Adjust comments.
+	* m4/symtab.c (m4_symbol_value_copy): Detect when builtins are
+	flattened.
+	* m4/input.c (init_builtin_token): Add parameter, and allow
+	concatenating builtins.
+	(m4__next_token): Adjust caller.
+	* m4/macro.c (m4_is_arg_composite): New function.
+	(expand_argument): Allow builtin concatenation.
+	(m4_arg_text): Add parameter.
+	(m4__arg_adjust_refcount, m4__arg_print): Adjust callers.
+	(m4_arg_equal): Fix comparison of builtin tokens.
+	* modules/m4.c (define, pushdef): Warn when flattening builtins.
+	* doc/m4.texinfo (Define): Remove dead comment.
+	(Defn): Update to reflect code changes.
+	* tests/builtins.at (defn): Remove xfail.
+	* NEWS: Document this change.
+
 2008-05-03  Eric Blake  <ebb9@byu.net>
 
 	Document define_blind.
diff --git a/NEWS b/NEWS
index 205c6511..35440ee0 100644
--- a/NEWS
+++ b/NEWS
@@ -216,6 +216,10 @@ promoted to 2.0.
    using `builtin' or `indir' to perform nested `shift' calls triggered an
    assertion failure.
 
+** Fix regression introduced in 1.4.10b (but not present in 1.4.11) where
+   the command-line option -dV, as well as the builtin `debugmode(V)',
+   failed to enable `t' and `c' debug options.
+
 ** Fix the `m4wrap' builtin to accumulate wrapped text in FIFO order, as
    required by POSIX.  The manual mentions a way to restore the LIFO order
    present in earlier GNU M4 versions.  NOTE: this change exposes a bug
@@ -236,9 +240,19 @@ promoted to 2.0.
    then apply this patch:
      http://git.sv.gnu.org/gitweb/?p=autoconf.git;a=commitdiff;h=56d42fa71
 
+** The `defn' builtin now warns when operating on an undefined macro name.
+   To simulate 1.4.x behavior, use:
+     pushdef(`defn', `ifdef(`$1', `builtin(`defn', `$1')')')
+
 ** Enhance the `ifdef', `ifelse', and `shift' builtins, as well as all
    user macros, to transparently handle builtin tokens generated by `defn'.
 
+** Allow the concatenation of builtin macros with arbitrary text in
+   several contexts, via the `defn' builtin or argument expansion, rather
+   than warning and converting the builtin token to an empty string.
+   However, it is still not possible to use a concatenated builtin when
+   defining a macro.
+
 ** Enhance the `defn', `dumpdef', `ifdef', `popdef', `traceon', `traceoff',
    and `undefine' macros to warn when encountering a builtin token in the
    context of a macro name, rather than acting on the empty string.  This
diff --git a/doc/m4.texinfo b/doc/m4.texinfo
index 75d7fc84..e4467659 100644
--- a/doc/m4.texinfo
+++ b/doc/m4.texinfo
@@ -1798,6 +1798,23 @@ Defines @var{name} to expand to @var{expansion}.  If
 The expansion of @code{define} is void.
 The macro @code{define} is recognized only with parameters.
 @end deffn
+@comment Other implementations, such as Solaris, can define a macro
+@comment with a builtin token attached to text:
+@comment  define(foo, a`'defn(`divnum')b)
+@comment  defn(`foo') => ab
+@comment  dumpdef(`foo') => foo: a<divnum>b
+@comment  len(defn(`foo')) => 3
+@comment  index(defn(`foo'), defn(`divnum')) => 1
+@comment  foo => a0b
+@comment It may be worth making some changes to support this behavior,
+@comment or something similar to it.
+@comment
+@comment But be sure it has sane semantics, with potentially deferred
+@comment expansion of builtins.  For example, this should not warn
+@comment about trying to access the definition of an undefined macro:
+@comment  define(`foo', `ifdef(`$1', 'defn(`defn')`)')foo(`oops')
+@comment Also, think how to handle conflicting argument counts:
+@comment  define(`bar', defn(`dnl', `len'))
 
 The following example defines the macro @var{foo} to expand to the text
 @samp{Hello World.}.
@@ -1834,13 +1851,6 @@ definition of a macro if it has several definitions from @code{pushdef}
 (@pxref{Pushdef}).  Some other implementations of @code{m4} replace all
 definitions of a macro with @code{define}.  @xref{Incompatibilities},
 for more details.
-@comment FIXME - See Austin group XCU ERN 118; this is considered
-@comment ambiguous in the current version of POSIX.  The best thing to
-@comment do here would probably be keep GNU semantics of popdef/pushdef
-@comment in the m4 module unconditionally, then have a shadow builtin in
-@comment the traditional module that does the undefine/pushdef
-@comment semantics, rather than our current keying off of
-@comment POSIXLY_CORRECT within the m4 module.
 
 As a @acronym{GNU} extension, the first argument to @code{define} does
 not have to be a simple word.
@@ -2175,19 +2185,9 @@ empty and triggers a warning.
 If @var{name} is a user-defined macro, the quoted definition is simply
 the quoted expansion text.  If, instead, @var{name} is a builtin, the
 expansion is a special token, which points to the builtin's internal
-definition.  This token is only meaningful as the second argument to
+definition.  This token meaningful primarily as the second argument to
 @code{define} (and @code{pushdef}), and is silently converted to an
-empty string in most other contexts.
-@comment FIXME - Other implementations, such as Solaris, can pass a
-@comment builtin token around to other macros, flattening it only on output:
-@comment  define(foo, a`'defn(`divnum')b)
-@comment  defn(`foo') => ab
-@comment  dumpdef(`foo') => foo: a<divnum>b
-@comment  len(defn(`foo')) => 3
-@comment  index(defn(`foo'), defn(`divnum')) => 1
-@comment  foo => a0b
-@comment It may be worth making some changes to support this behavior,
-@comment or something similar to it.
+empty string in many other contexts.
 
 The macro @code{defn} is recognized only with parameters.
 @end deffn
@@ -2349,28 +2349,49 @@ bar
 @result{}0
 @end example
 
-A warning is issued if @var{name} is undefined.  Also, at present,
-concatenating a builtin token with anything else is not supported as a
-macro definition, and a warning is issued.
-@comment FIXME - handle defining macros with mixed text and builtins.
+A warning is issued if @var{name} is undefined.  Also note that as of M4
+1.6, @code{defn} with multiple arguments can join text with builtin
+tokens.  However, when defining a macro via @code{define} or
+@code{pushdef}, a warning is issued and the builtin token ignored if the
+builtin token does not occur in isolation.  A future version of
+@acronym{GNU} M4 may lift this restriction.
 
-@comment xfail
 @example
+$ @kbd{m4 -d}
 defn(`foo')
 @error{}m4:stdin:1: Warning: defn: undefined macro `foo'
 @result{}
-define(`echo', `$@@')
+define(`a', `A')define(`AA', `b')
 @result{}
-define(`foo', `a')
+traceon(`defn', `define')
 @result{}
-define(`bar', defn(`foo', `divnum'))
+defn(`a', `divnum', `a')
+@error{}m4trace: -1- defn(`a', `divnum', `a') -> ``A'<divnum>`A''
+@result{}AA
+define(`mydivnum', defn(`divnum', `divnum'))mydivnum
+@error{}m4trace: -2- defn(`divnum', `divnum') -> `<divnum><divnum>'
+@error{}m4:stdin:5: Warning: define: cannot concatenate builtins
+@error{}m4trace: -1- define(`mydivnum', `<divnum><divnum>') -> `'
 @result{}
-define(`blah', echo(defn(`divnum', `foo')))
+traceoff(`defn', `define')dumpdef(`mydivnum')
+@error{}mydivnum:@tabchar{}`'
 @result{}
-bar
-@result{}a0
-blah
-@result{}0a
+define(`mydivnum', defn(`divnum')defn(`divnum'))mydivnum
+@error{}m4:stdin:7: Warning: define: cannot concatenate builtins
+@result{}
+define(`mydivnum', defn(`divnum')`a')mydivnum
+@error{}m4:stdin:8: Warning: define: cannot concatenate builtins
+@result{}A
+define(`mydivnum', `a'defn(`divnum'))mydivnum
+@error{}m4:stdin:9: Warning: define: cannot concatenate builtins
+@result{}A
+define(`q', ``$@@'')
+@result{}
+define(`foo', q(`a', defn(`divnum')))foo
+@error{}m4:stdin:11: Warning: define: cannot concatenate builtins
+@result{}a,
+ifdef(`foo', `yes', `no')
+@result{}yes
 @end example
 
 @node Pushdef
diff --git a/m4/input.c b/m4/input.c
index 0f48768e..1f89916c 100644
--- a/m4/input.c
+++ b/m4/input.c
@@ -114,7 +114,8 @@ static	int	eof_read		(m4_input_block *, m4 *, bool, bool,
 					 bool);
 static	void	eof_unget		(m4_input_block *, int);
 
-static	void	init_builtin_token	(m4 *, m4_symbol_value *);
+static	void	init_builtin_token	(m4 *, m4_obstack *,
+					 m4_symbol_value *);
 static	void	append_quote_token	(m4 *, m4_obstack *,
 					 m4_symbol_value *);
 static	bool	match_input		(m4 *, const char *, bool);
@@ -449,18 +450,18 @@ m4_push_string_init (m4 *context)
    rather than copying everything consecutively onto the input stack.
    Must be called between push_string_init and push_string_finish.
 
-   If VALUE contains text, then convert the current input block into a
-   chain if it is not one already, and add the contents of VALUE as a
-   new link in the chain.  LEVEL describes the current expansion
-   level, or SIZE_MAX if VALUE is composite, its contents reside
-   entirely on the current_input stack, and VALUE lives in temporary
-   storage.  If VALUE is a simple string, then it belongs to the
-   current macro expansion.  If VALUE is composite, then each text
-   link has a level of SIZE_MAX if it belongs to the current macro
-   expansion, otherwise it is a back-reference where level tracks
-   which stack it came from.  The resulting input block chain contains
-   links with a level of SIZE_MAX if the text belongs to the input
-   stack, otherwise the level where the back-reference comes from.
+   Convert the current input block into a chain if it is not one
+   already, and add the contents of VALUE as a new link in the chain.
+   LEVEL describes the current expansion level, or SIZE_MAX if VALUE
+   is composite, its contents reside entirely on the current_input
+   stack, and VALUE lives in temporary storage.  If VALUE is a simple
+   string, then it belongs to the current macro expansion.  If VALUE
+   is composite, then each text link has a level of SIZE_MAX if it
+   belongs to the current macro expansion, otherwise it is a
+   back-reference where level tracks which stack it came from.  The
+   resulting input block chain contains links with a level of SIZE_MAX
+   if the text belongs to the input stack, otherwise the level where
+   the back-reference comes from.
 
    Return true only if a reference was created to the contents of
    VALUE, in which case, LEVEL is less than SIZE_MAX and the lifetime
@@ -1122,18 +1123,36 @@ m4_pop_wrapup (m4 *context)
 }
 
 /* Populate TOKEN with the builtin token at the top of the input
-   stack, then consume the input.  If TOKEN is NULL, discard the
-   builtin token instead.  */
+   stack, then consume the input.  If OBS, TOKEN will be converted to
+   a composite token using storage from OBS as necessary; otherwise,
+   if TOKEN is NULL, the builtin token is discarded.  */
 static void
-init_builtin_token (m4 *context, m4_symbol_value *token)
+init_builtin_token (m4 *context, m4_obstack *obs, m4_symbol_value *token)
 {
   m4__symbol_chain *chain;
   assert (isp->funcs == &composite_funcs);
   chain = isp->u.u_c.chain;
   assert (!chain->quote_age && chain->type == M4__CHAIN_FUNC
 	  && chain->u.builtin);
-  if (token)
-    m4__set_symbol_value_builtin (token, chain->u.builtin);
+  if (obs)
+    {
+      assert (token);
+      if (token->type == M4_SYMBOL_VOID)
+	{
+	  token->type = M4_SYMBOL_COMP;
+	  token->u.u_c.chain = token->u.u_c.end = NULL;
+	  token->u.u_c.wrapper = false;
+	  token->u.u_c.has_func = false;
+	}
+      assert (token->type == M4_SYMBOL_COMP);
+      m4__append_builtin (obs, chain->u.builtin, &token->u.u_c.chain,
+			  &token->u.u_c.end);
+    }
+  else if (token)
+    {
+      assert (token->type == M4_SYMBOL_VOID);
+      m4__set_symbol_value_builtin (token, chain->u.builtin);
+    }
   chain->u.builtin = NULL;
 }
 
@@ -1535,7 +1554,7 @@ m4__next_token (m4 *context, m4_symbol_value *token, int *line,
 
     if (ch == CHAR_BUILTIN)		/* BUILTIN TOKEN */
       {
-	init_builtin_token (context, token);
+	init_builtin_token (context, obs, token);
 #ifdef DEBUG_INPUT
 	m4_print_token (context, "next_token", M4_TOKEN_MACDEF, token);
 #endif
@@ -1590,34 +1609,8 @@ m4__next_token (m4 *context, m4_symbol_value *token, int *line,
 	      m4_error_at_line (context, EXIT_FAILURE, 0, file, *line, caller,
 				_("end of file in string"));
 	    if (ch == CHAR_BUILTIN)
-	      {
-		/* TODO support concatenation of builtins.  */
-		if (obstack_object_size (obs_safe) == 0
-		    && token->type == M4_SYMBOL_VOID)
-		  {
-		    /* Strip quotes if they surround a lone builtin
-		       token.  */
-		    assert (quote_level == 1);
-		    init_builtin_token (context, token);
-		    ch = peek_char (context, false);
-		    if (m4_has_syntax (M4SYNTAX, ch, M4_SYNTAX_RQUOTE))
-		      {
-			ch = next_char (context, false, false, false);
-#ifdef DEBUG_INPUT
-			m4_print_token (context, "next_token", M4_TOKEN_MACDEF,
-					token);
-#endif
-			return M4_TOKEN_MACDEF;
-		      }
-		    token->type = M4_SYMBOL_VOID;
-		  }
-		else
-		  init_builtin_token (context, NULL);
-		m4_warn_at_line (context, 0, file, *line, caller,
-				 _("cannot quote builtin"));
-		continue;
-	      }
-	    if (ch == CHAR_QUOTE)
+	      init_builtin_token (context, obs, obs ? token : NULL);
+	    else if (ch == CHAR_QUOTE)
 	      append_quote_token (context, obs, token);
 	    else if (m4_has_syntax (M4SYNTAX, ch, M4_SYNTAX_RQUOTE))
 	      {
@@ -1649,36 +1642,8 @@ m4__next_token (m4 *context, m4_symbol_value *token, int *line,
 	      m4_error_at_line (context, EXIT_FAILURE, 0, file, *line, caller,
 				_("end of file in string"));
 	    if (ch == CHAR_BUILTIN)
-	      {
-		/* TODO support concatenation of builtins.  */
-		if (obstack_object_size (obs_safe) == 0
-		    && token->type == M4_SYMBOL_VOID)
-		  {
-		    /* Strip quotes if they surround a lone builtin
-		       token.  */
-		    assert (quote_level == 1);
-		    init_builtin_token (context, token);
-		    ch = peek_char (context, false);
-		    if (MATCH (context, ch, context->syntax->quote.str2,
-			       false))
-		      {
-			ch = next_char (context, false, false, false);
-			MATCH (context, ch, context->syntax->quote.str2, true);
-#ifdef DEBUG_INPUT
-			m4_print_token (context, "next_token", M4_TOKEN_MACDEF,
-					token);
-#endif
-			return M4_TOKEN_MACDEF;
-		      }
-		    token->type = M4_SYMBOL_VOID;
-		  }
-		else
-		  init_builtin_token (context, NULL);
-		m4_warn_at_line (context, 0, file, *line, caller,
-				 _("cannot quote builtin"));
-		continue;
-	      }
-	    if (MATCH (context, ch, context->syntax->quote.str2, true))
+	      init_builtin_token (context, obs, obs ? token : NULL);
+	    else if (MATCH (context, ch, context->syntax->quote.str2, true))
 	      {
 		if (--quote_level == 0)
 		  break;
@@ -1708,10 +1673,7 @@ m4__next_token (m4 *context, m4_symbol_value *token, int *line,
 				_("end of file in comment"));
 	    if (ch == CHAR_BUILTIN)
 	      {
-		/* TODO support concatenation of builtins.  */
-		m4_warn_at_line (context, 0, file, *line, caller,
-				 _("cannot comment builtin"));
-		init_builtin_token (context, NULL);
+		init_builtin_token (context, NULL, NULL);
 		continue;
 	      }
 	    if (m4_has_syntax (M4SYNTAX, ch, M4_SYNTAX_ECOMM))
@@ -1740,10 +1702,7 @@ m4__next_token (m4 *context, m4_symbol_value *token, int *line,
 				_("end of file in comment"));
 	    if (ch == CHAR_BUILTIN)
 	      {
-		/* TODO support concatenation of builtins.  */
-		m4_warn_at_line (context, 0, file, *line, caller,
-				 _("cannot comment builtin"));
-		init_builtin_token (context, NULL);
+		init_builtin_token (context, NULL, NULL);
 		continue;
 	      }
 	    if (MATCH (context, ch, context->syntax->comm.str2, true))
diff --git a/m4/m4module.h b/m4/m4module.h
index 5b5e01b2..21a83399 100644
--- a/m4/m4module.h
+++ b/m4/m4module.h
@@ -153,7 +153,7 @@ struct m4_string_pair
 /* Grab the text contents of argument I, or abort if the argument is
    not text.  Assumes that `m4 *context' and `m4_macro_args *argv' are
    in scope.  */
-#define M4ARG(i) m4_arg_text (context, argv, i)
+#define M4ARG(i) m4_arg_text (context, argv, i, false)
 
 /* Grab the length of the text contents of argument I, or abort if the
    argument is not text.  Assumes that `m4 *context' and
@@ -312,7 +312,7 @@ extern bool	m4_symbol_value_flatten_args (m4_symbol_value *);
 
 extern m4_symbol_value *m4_symbol_value_create	  (void);
 extern void		m4_symbol_value_delete	  (m4_symbol_value *);
-extern void		m4_symbol_value_copy	  (m4 *, m4_symbol_value *,
+extern bool		m4_symbol_value_copy	  (m4 *, m4_symbol_value *,
 						   m4_symbol_value *);
 extern bool		m4_is_symbol_value_text   (m4_symbol_value *);
 extern bool		m4_is_symbol_value_func   (m4_symbol_value *);
@@ -352,7 +352,8 @@ extern size_t	m4_arg_argc		(m4_macro_args *);
 extern m4_symbol_value *m4_arg_symbol	(m4_macro_args *, size_t);
 extern bool	m4_is_arg_text		(m4_macro_args *, size_t);
 extern bool	m4_is_arg_func		(m4_macro_args *, size_t);
-extern const char *m4_arg_text		(m4 *, m4_macro_args *, size_t);
+extern bool	m4_is_arg_composite	(m4_macro_args *, size_t);
+extern const char *m4_arg_text		(m4 *, m4_macro_args *, size_t, bool);
 extern bool	m4_arg_equal		(m4 *, m4_macro_args *, size_t,
 					 size_t);
 extern bool	m4_arg_empty		(m4_macro_args *, size_t);
diff --git a/m4/m4private.h b/m4/m4private.h
index 48a00755..7e1f7a8c 100644
--- a/m4/m4private.h
+++ b/m4/m4private.h
@@ -532,7 +532,7 @@ typedef enum {
   M4_TOKEN_COMMA,	/* Argument separator, M4_SYMBOL_TEXT.  */
   M4_TOKEN_CLOSE,	/* Argument list end, M4_SYMBOL_TEXT.  */
   M4_TOKEN_SIMPLE,	/* Single character, M4_SYMBOL_TEXT.  */
-  M4_TOKEN_MACDEF,	/* Macro's definition (see "defn"), M4_SYMBOL_FUNC.  */
+  M4_TOKEN_MACDEF,	/* Builtin token, M4_SYMBOL_FUNC or M4_SYMBOL_COMP.  */
   M4_TOKEN_ARGV		/* A series of parameters, M4_SYMBOL_COMP.  */
 } m4__token_type;
 
diff --git a/m4/macro.c b/m4/macro.c
index e58e657a..f5bec181 100644
--- a/m4/macro.c
+++ b/m4/macro.c
@@ -327,15 +327,10 @@ expand_argument (m4 *context, m4_obstack *obs, m4_symbol_value *argp,
 	case M4_TOKEN_CLOSE:
 	  if (paren_level == 0)
 	    {
-	      /* FIXME - For now, we match the behavior of the branch,
-		 except we don't issue warnings.  But in the future,
-		 we want to allow concatenation of builtins and
-		 text.  */
-	      len = obstack_object_size (obs);
-	      if (argp->type == M4_SYMBOL_FUNC && !len)
-		return type == M4_TOKEN_COMMA;
+	      assert (argp->type != M4_SYMBOL_FUNC);
 	      if (argp->type != M4_SYMBOL_COMP)
 		{
+		  len = obstack_object_size (obs);
 		  VALUE_MODULE (argp) = NULL;
 		  if (len)
 		    {
@@ -347,7 +342,16 @@ expand_argument (m4 *context, m4_obstack *obs, m4_symbol_value *argp,
 		    m4_set_symbol_value_text (argp, "", len, 0);
 		}
 	      else
-		m4__make_text_link (obs, NULL, &argp->u.u_c.end);
+		{
+		  m4__make_text_link (obs, NULL, &argp->u.u_c.end);
+		  if (argp->u.u_c.chain == argp->u.u_c.end
+		      && argp->u.u_c.chain->type == M4__CHAIN_FUNC)
+		    {
+		      const m4__builtin *func = argp->u.u_c.chain->u.builtin;
+		      argp->type = M4_SYMBOL_FUNC;
+		      argp->u.builtin = func;
+		    }
+		}
 	      return type == M4_TOKEN_COMMA;
 	    }
 	  /* fallthru */
@@ -369,6 +373,7 @@ expand_argument (m4 *context, m4_obstack *obs, m4_symbol_value *argp,
 	case M4_TOKEN_WORD:
 	case M4_TOKEN_SPACE:
 	case M4_TOKEN_STRING:
+	case M4_TOKEN_MACDEF:
 	  if (!expand_token (context, obs, type, &token, line, first))
 	    age = 0;
 	  if (token.type == M4_SYMBOL_COMP)
@@ -390,13 +395,6 @@ expand_argument (m4 *context, m4_obstack *obs, m4_symbol_value *argp,
 	    }
 	  break;
 
-	case M4_TOKEN_MACDEF:
-	  if (argp->type == M4_SYMBOL_VOID && obstack_object_size (obs) == 0)
-	    m4_symbol_value_copy (context, argp, &token);
-	  else
-	    argp->type = M4_SYMBOL_TEXT;
-	  break;
-
 	case M4_TOKEN_ARGV:
 	  assert (paren_level == 0 && argp->type == M4_SYMBOL_VOID
 		  && obstack_object_size (obs) == 0
@@ -1025,6 +1023,8 @@ m4__arg_adjust_refcount (m4 *context, m4_macro_args *argv, bool increase)
 		    m4__adjust_refcount (context, chain->u.u_s.level,
 					 increase);
 		  break;
+		case M4__CHAIN_FUNC:
+		  break;
 		case M4__CHAIN_ARGV:
 		  assert (chain->u.u_a.argv->inuse);
 		  m4__arg_adjust_refcount (context, chain->u.u_a.argv,
@@ -1219,7 +1219,6 @@ m4_is_arg_text (m4_macro_args *argv, size_t arg)
   return false;
 }
 
-/* TODO - add m4_is_arg_comp to distinguish concatenation of builtins.  */
 /* Given ARGV, return true if argument ARG is a single builtin
    function.  Only non-zero indices less than argc can return
    true.  */
@@ -1231,12 +1230,28 @@ m4_is_arg_func (m4_macro_args *argv, size_t arg)
   return m4_is_symbol_value_func (m4_arg_symbol (argv, arg));
 }
 
+/* Given ARGV, return true if argument ARG contains a builtin token
+   concatenated with anything else.  Only non-zero indices less than
+   argc can return true.  */
+bool
+m4_is_arg_composite (m4_macro_args *argv, size_t arg)
+{
+  m4_symbol_value *value;
+  if (arg == 0 || argv->argc <= arg || argv->flatten || !argv->has_func)
+    return false;
+  value = m4_arg_symbol (argv, arg);
+  if (value->type == M4_SYMBOL_COMP && value->u.u_c.has_func)
+    return true;
+  return false;
+}
+
 /* Given ARGV, return the text at argument ARG.  Abort if the argument
    is not text.  Arg 0 is always text, and indices beyond argc return
-   the empty string.  The result is always NUL-terminated, even if it
-   includes embedded NUL characters.  */
+   the empty string.  If FLATTEN, builtins are ignored.  The result is
+   always NUL-terminated, even if it includes embedded NUL
+   characters.  */
 const char *
-m4_arg_text (m4 *context, m4_macro_args *argv, size_t arg)
+m4_arg_text (m4 *context, m4_macro_args *argv, size_t arg, bool flatten)
 {
   m4_symbol_value *value;
   m4__symbol_chain *chain;
@@ -1246,7 +1261,7 @@ m4_arg_text (m4 *context, m4_macro_args *argv, size_t arg)
     return argv->argv0;
   if (argv->argc <= arg)
     return "";
-  value = m4_arg_symbol (argv, arg);
+  value = arg_symbol (argv, arg, NULL, flatten);
   if (m4_is_symbol_value_text (value))
     return m4_get_symbol_value_text (value);
   assert (value->type == M4_SYMBOL_COMP);
@@ -1259,12 +1274,18 @@ m4_arg_text (m4 *context, m4_macro_args *argv, size_t arg)
 	case M4__CHAIN_STR:
 	  obstack_grow (obs, chain->u.u_s.str, chain->u.u_s.len);
 	  break;
+	case M4__CHAIN_FUNC:
+	  if (flatten)
+	    break;
+	  assert (!"m4_arg_text");
+	  abort ();
 	case M4__CHAIN_ARGV:
+	  assert (!chain->u.u_a.has_func || flatten || argv->flatten);
 	  m4__arg_print (context, obs, chain->u.u_a.argv, chain->u.u_a.index,
 			 m4__quote_cache (M4SYNTAX, NULL, chain->quote_age,
 					  chain->u.u_a.quotes),
-			 argv->flatten || chain->u.u_a.flatten, NULL, NULL,
-			 NULL, false, false);
+			 flatten || argv->flatten || chain->u.u_a.flatten,
+			 NULL, NULL, NULL, false, false);
 	  break;
 	default:
 	  assert (!"m4_arg_text");
@@ -1368,6 +1389,9 @@ m4_arg_equal (m4 *context, m4_macro_args *argv, size_t indexa, size_t indexb)
 	{
 	  tmpb.next = NULL;
 	  tmpb.type = M4__CHAIN_STR;
+	  tmpb.u.u_s.str = NULL;
+	  tmpb.u.u_s.len = 0;
+	  chain = &tmpb;
 	  m4__arg_print (context, obs, cb->u.u_a.argv, cb->u.u_a.index,
 			 m4__quote_cache (M4SYNTAX, NULL, cb->quote_age,
 					  cb->u.u_a.quotes),
@@ -1526,6 +1550,7 @@ m4__arg_print (m4 *context, m4_obstack *obs, m4_macro_args *argv, size_t arg,
   size_t sep_len;
   size_t *plen = quote_each ? NULL : &len;
 
+  flatten |= argv->flatten;
   if (chainp)
     assert (!max_len && *chainp);
   if (!sep)
diff --git a/m4/symtab.c b/m4/symtab.c
index 69f2200a..76ff1cba 100644
--- a/m4/symtab.c
+++ b/m4/symtab.c
@@ -405,10 +405,13 @@ arg_destroy_CB (m4_hash *hash, const void *name, void *arg, void *ignored)
   return NULL;
 }
 
-void
+/* Copy the symbol SRC into DEST.  Return true if builtin tokens were
+   flattened.  */
+bool
 m4_symbol_value_copy (m4 *context, m4_symbol_value *dest, m4_symbol_value *src)
 {
   m4_symbol_value *next;
+  bool result = false;
 
   assert (dest);
   assert (src);
@@ -455,7 +458,7 @@ m4_symbol_value_copy (m4 *context, m4_symbol_value *dest, m4_symbol_value *src)
       }
       break;
     case M4_SYMBOL_FUNC:
-      /* Nothing further to do.  */
+      m4__set_symbol_value_builtin (dest, src->u.builtin);
       break;
     case M4_SYMBOL_PLACEHOLDER:
       m4_set_symbol_value_placeholder (dest,
@@ -476,9 +479,14 @@ m4_symbol_value_copy (m4 *context, m4_symbol_value *dest, m4_symbol_value *src)
 	      case M4__CHAIN_STR:
 		obstack_grow (obs, chain->u.u_s.str, chain->u.u_s.len);
 		break;
+	      case M4__CHAIN_FUNC:
+		result = true;
+		break;
 	      case M4__CHAIN_ARGV:
 		quotes = m4__quote_cache (M4SYNTAX, NULL, chain->quote_age,
 					  chain->u.u_a.quotes);
+		if (chain->u.u_a.has_func && !chain->u.u_a.flatten)
+		  result = true;
 		m4__arg_print (context, obs, chain->u.u_a.argv,
 			       chain->u.u_a.index, quotes, true, NULL, NULL,
 			       NULL, false, false);
@@ -503,6 +511,7 @@ m4_symbol_value_copy (m4 *context, m4_symbol_value *dest, m4_symbol_value *src)
   if (VALUE_ARG_SIGNATURE (src))
     VALUE_ARG_SIGNATURE (dest) = m4_hash_dup (VALUE_ARG_SIGNATURE (src),
 					      arg_copy_CB);
+  return result;
 }
 
 static void *
diff --git a/modules/m4.c b/modules/m4.c
index 0b714efa..5cb6d114 100644
--- a/modules/m4.c
+++ b/modules/m4.c
@@ -157,7 +157,8 @@ M4BUILTIN_HANDLER (define)
     {
       m4_symbol_value *value = m4_symbol_value_create ();
 
-      m4_symbol_value_copy (context, value, m4_arg_symbol (argv, 2));
+      if (m4_symbol_value_copy (context, value, m4_arg_symbol (argv, 2)))
+	m4_warn (context, 0, M4ARG (0), _("cannot concatenate builtins"));
       m4_symbol_define (M4SYMTAB, M4ARG (1), value);
     }
   else
@@ -179,7 +180,8 @@ M4BUILTIN_HANDLER (pushdef)
     {
       m4_symbol_value *value = m4_symbol_value_create ();
 
-      m4_symbol_value_copy (context, value, m4_arg_symbol (argv, 2));
+      if (m4_symbol_value_copy (context, value, m4_arg_symbol (argv, 2)))
+	m4_warn (context, 0, M4ARG (0), _("cannot concatenate builtins"));
       m4_symbol_pushdef (M4SYMTAB, M4ARG (1), value);
     }
   else
diff --git a/tests/builtins.at b/tests/builtins.at
index b059e7b5..3f67c2cc 100644
--- a/tests/builtins.at
+++ b/tests/builtins.at
@@ -230,11 +230,6 @@ AT_CLEANUP
 
 AT_SETUP([defn])
 
-dnl This test is a reminder that defn needs to be fixed to handle
-dnl concatenation of builtin tokens with text, and user macros need
-dnl to handle builtin tokens without flattening.
-AT_XFAIL_IF([:])
-
 AT_DATA([[in.m4]],
 [[define(`e', `$@')define(`q', ``$@'')define(`u', `$*')
 define(`cmp', `ifelse($1, $2, `yes', `no')')define(`d', defn(`defn'))