tao_idl: docs/annotations.md

2 files changed, 488 insertions, 28 deletions

diff --git a/TAO/TAO_IDL/docs/README.md b/TAO/TAO_IDL/docs/README.md
index d8cfeac28ed..f9a0bacbda0 100644
--- a/TAO/TAO_IDL/docs/README.md
+++ b/TAO/TAO_IDL/docs/README.md
@@ -3,34 +3,38 @@
 ## Topics
 
 - [Historical Document Describing How to Write a Backend](WRITING_A_BE).
+- [Creating Annotations for IDLv4](annotations.md)
 - [Less Relevant Historical Documents from Sun](historical)
 
-## TAO\_IDL File Layout
+## TAO\_IDL Layout
 
 TAO\_IDL is broken up into a few different modules, these can be generalized
 as:
 
-- AST
+- `AST`
   - "Abstract Syntax Tree"
   - Nodes representing the Abstract Syntax Tree
-- FE
+- `FE`
   - "Front End"
   - Parsing functionality that generates the Abstract Syntax Tree.
   - Among other things, contains the Bison grammar `fe/idl.ypp` and Flex
     token `fe/idl.ll` files.
-- BE
+- `BE`
   - "Back End"
   - Transformed Abstract Syntax Tree
-- UTL
+- `UTL`
   - "Util"
   - Utility Classes and Functions.
-  - Contains `idl_global` class which is made up of `include/idl_global.h` and
-    `utl/utl_global.cpp`.
-- DRV
+  - Contains class for the `idl_global` singleton which is made up of
+    `include/idl_global.h` and `utl/utl_global.cpp`.
+- `DRV`
   - "Driver"
   - The `tao_idl` program it self which drives the other modules.
 
-These are broken into these places on the file system:
+Lacking namespaces for the most part, in TAO\_IDL function and class names
+generally start with these modules, using the "C" way of emulating namespaces.
+
+The modules are broken into these places on the file system:
 
 - `include`
   - Contains the header files for AST, FE, and UTL modules.
diff --git a/TAO/TAO_IDL/docs/annotations.md b/TAO/TAO_IDL/docs/annotations.md
index 0da490d92be..29975088b96 100644
--- a/TAO/TAO_IDL/docs/annotations.md
+++ b/TAO/TAO_IDL/docs/annotations.md
@@ -3,44 +3,500 @@
 **How to use the internal API to implement built-in annotations in `tao_idl` or a
 compiler that uses `tao_idl`.**
 
-## Annotations
+**Table of Contents:**
+
+<!-- vim-markdown-toc GFM -->
+
+* [IDL Annotations](#idl-annotations)
+  * [Special Cases of Annotations](#special-cases-of-annotations)
+    * [Unions Discriminators](#unions-discriminators)
+    * [Base Types in Sequences](#base-types-in-sequences)
+    * [Base Types in Arrays](#base-types-in-arrays)
+* [Defining Annotations](#defining-annotations)
+  * [`@document` Example](#document-example)
+  * [What Can Go in Annotations](#what-can-go-in-annotations)
+* [Reading Annotations in the AST](#reading-annotations-in-the-ast)
+  * [Reading `@document` Annotations](#reading-document-annotations)
+    * [Reading Annotations Manually](#reading-annotations-manually)
+  * [Reading Special Cases of Annotations](#reading-special-cases-of-annotations)
+    * [Unions Discriminators](#unions-discriminators-1)
+    * [Base Types in Sequences](#base-types-in-sequences-1)
+    * [Base Types in Arrays](#base-types-in-arrays-1)
+* [Limitations](#limitations)
+
+<!-- vim-markdown-toc -->
+
+## IDL Annotations
 
 Annotations are a feature of IDLv4 that allows IDL authors to pass hints to the
 IDL compiler that can change compiler behavior and code generation. They are
-similar to `#pragmas`, but more powerful because they are integrated with IDL
-and are more expressive. In the latest IDL specification as of writing, version
-4.2, they are described in section 7.4.15.1.
+similar to some uses of `#pragma`, but are more powerful because they are
+integrated with IDL and are more expressive. In the latest IDL specification as
+of writing, version 4.2, they are described in section 7.4.15.1.
 
-Annotations exist in other languages like, Python, Java, and C#, although in
-the former they are called attributes and have a significantly different
-syntax. Like Python and Java, annotations can appear front of declarations and
-look like function calls preceded by an `@` symbol. Unlike Python and Java,
-though they "may be applied to any IDL constructs or sub-constructs", but the
-OMG has not clarified this as of IDL 4.2. What this means though is that
-annotations can be applied more places than the other languages allow.
+Annotations exist in other languages like Java, Python (as decorators), and C#
+(as attributes). Like Java and Python, annotations can appear in front of
+declarations, have `@` at the beginning, and can look like function call.
+
+Here is what an example of IDL using some OMG standard annotations might look
+like:
 
-In `tao_idl`, here is an example of how annotations may be used:
 ```
-enum Urgency {
+enum Urgency_t {
   SUPPLEMENTARY,
   @default_literal INFORMATIVE,
   CRITICAL
 };
 
-@range(min=0,max=24)
-typedef Hours short;
+@unit("Hour(s)")
+@range(min=0,max=23)
+typedef short Hours_t;
+
+@unit("Day(s)")
+typedef unsigned long Days_t;
+
+struct Time_t {
+  Hours_t hours;
+  Days_t days;
+};
 
-@extensibility(FINAL)
+@mutable
 struct Report {
   @key
   unsigned long index;
 
-  @default(12)
-  Hours expiration;
+  @optional
+  Expiration_t expiration;
 
-  Urgency urgency;
+  @optional
+  Urgency_t urgency;
 
   string message;
 };
 ```
 
+### Special Cases of Annotations
+
+Annotations "may be applied to any IDL constructs or sub-constructs", as
+defined by the OMG. This is very vague and the OMG has not clarified this as of
+IDL 4.2. [(Also see Limitation #1)](#limitation1). What can be said about it
+though is that other than before normal declarations, like before structs,
+typedefs, and constants, in TAO\_IDL as of writing, annotations can also be
+applied in the following cases. See ["Reading Special Cases of
+Annotations"](#reading-special-cases-of-annotations) for how to have the
+compiler use these kinds of annotations.
+
+#### Unions Discriminators
+
+```
+enum GradeType {
+  PASS_FAIL,
+  PASS_70,
+  PASS_80
+};
+union Grade (@key GradeType) {
+case PASS_FAIL:
+  boolean pass;
+case PASS_70:
+case PASS_80:
+default:
+  short grade;
+};
+```
+
+#### Base Types in Sequences
+
+```
+struct Event {
+  short data;
+};
+typedef sequence<@external Report, 12> Dozen_Events;
+```
+
+#### Base Types in Arrays
+
+```
+struct Event {
+  short data;
+};
+typedef Dozen_Events Event @external [12];
+```
+
+## Defining Annotations
+
+Annotations should be defined after the AST is initialized and ready to be
+used, but before any user defined IDL is processed. The recommended place for
+this is `BE_post_init ()` which is located in `be/be_init.cpp` in `tao_idl`.
+Annotations are nodes in the AST and could be defined by hand, simulating what
+happens in `fe/idl.yy`. However a string parsing utility has been added just
+for this purpose, `idl_global->eval (const char* idl)`. `eval ()` processes IDL
+as if it were in an IDL file so annotations can be defined using the IDL
+annotation notation.
+
+### `@document` Example
+
+As a simple example, If we wanted to make a annotation that inserted comments
+into the product files for documentation purposes, we could design an
+annotation like this this:
+
+```
+@annotation document {
+  enum API_Type {
+    INTERNAL_API,
+    USER_API,
+    LEGACY_API
+  };
+  string comment;
+  API_Type api_type default INTERNAL_API;
+  boolean deprecated default FALSE;
+};
+```
+
+To use it without defining it in every IDL file, we need to embed it into
+`BE_post_init()`.
+
+```C++
+void BE_post_init (char *[], long)
+{
+  // ...
+  if (idl_global->idl_version_ > IDL_VERSION_3)
+    {
+      idl_global->eval (
+        "@annotation document {\n"
+        "  enum API_Type {\n"
+        "    INTERNAL_API,\n"
+        "    USER_API,\n"
+        "    LEGACY_API\n"
+        "  };\n"
+        "  string comment;\n"
+        "  API_Type api_type default INTERNAL_API;\n"
+        "  boolean deprecated default FALSE;\n"
+        "};\n"
+      );
+    }
+  // ...
+}
+```
+
+The new lines aren't strictly necessary but might help if a syntax error occurs
+because it will refer to the line number of this string as though it was a file
+called "builtin". This might not be helpful as it could be, because it won't
+distinguish between multiple calls to `eval` when reporting an error.
+
+By default TAO\_IDL uses IDL3 and this will cause an error when parsing the
+annotations. Version is controlled using `--idl-version` command line argument
+and ultimately `idl_global->idl_version_`. In the example above we would have
+to pass `--idl-version 4`.
+
+We can set it to use IDL4 by default in `BE_init ()`:
+
+```C++
+int BE_init (int &, ACE_TCHAR *[])
+{
+  // ...
+  idl_global->default_idl_version_ = IDL_VERSION_4;
+  // ...
+}
+
+void BE_post_init (char *[], long)
+{
+  // Same as above ...
+}
+```
+
+In TAO\_IDL, `idl_global->default_idl_version_` sets `idl_global->idl_version_`
+after `BE_init` is called but before arguments are parsed. This gives the user
+a chance to override it if they really want to and allows them to query the
+version we're setting using `--default-idl-version`.
+
+Alternatively if it is desired to retain compatibility with older versions of
+TAO, use the `TAO_IDL_HAS_ANNOTATIONS` macro.
+
+```C++
+int BE_init (int &, ACE_TCHAR *[])
+{
+  // ...
+#ifdef TAO_IDL_HAS_ANNOTATIONS
+  idl_global->default_idl_version_ = IDL_VERSION_4;
+#endif
+  // ...
+}
+
+void BE_post_init (char *[], long)
+{
+  // ...
+#ifdef TAO_IDL_HAS_ANNOTATIONS
+  if (idl_global->idl_version_ > IDL_VERSION_3)
+    {
+      idl_global->eval (
+        // ...
+      );
+    }
+#endif
+  // ...
+}
+```
+
+This would also be used when reading the annotations later.
+
+<a name="document-usage"></a>
+Once the annotation is declared, it can be used in IDL:
+
+```
+@document("Struct with 1 member")
+struct struct1 {
+  short x;
+};
+
+@document(
+  comment="Struct with 2 members",
+  api_type=USER_API
+)
+struct struct2 {
+  short x, y;
+};
+
+@document(
+  comment="Struct with 3 members",
+  api_type=LEGACY_API,
+  deprecated=TRUE
+)
+struct struct3 {
+  short x, y, z;
+};
+```
+
+However it won't do anything because nothing using the AST is looking for it,
+so it will be ignored. To make the program aware of the annotations, see
+["Reading Annotations in the AST"](#reading-annotations-in-the-ast) below.
+
+### What Can Go in Annotations
+
+- Annotations members can be of any type that constants can be. This includes
+booleans, integers, floats, enumerations, characters, and strings.
+- Enumerations, constants, and typedefs can be declared inside the annotation
+declaration, however they can not used outside the annotation expect for when
+passing them as parameters to the same annotation. Otherwise normal scope rules
+apply: Valid constant types and values from outside the annotation can be used
+inside it.
+
+## Reading Annotations in the AST
+
+To get the annotations for most nodes types, get a reference or pointer to the
+`ACE_Vector` of Annotations using `node->annotations()` and pass it to
+`UTL_find_annotation` to get the annotation. From there use index operators
+`[]` on the annotation to get the individual members and `value()` to get the
+value. The last part is not straightforward, as we have to deal with the
+`AST_Expression` class which is the internal class of TAO\_IDL that holds
+constant values.
+
+Internally, annotation local names are prefixed with `@` to prevent clashes
+with other elements in IDL with the same name. For example when trying to use
+`UTL_find_annotation` with annotation named `bar` in a module named `foo`, the
+proper internal scoped name to pass as the second argument is either
+`foo::@bar` or `::foo::@bar` if we want to be explicit that `foo` is in the
+root module.
+
+### Reading `@document` Annotations
+
+In this example we will use the [`@document` annotation defined
+above](#document-example) to generate Doxygen comments in the C++ code
+generated. For simplicity's sake, we will limit this example to structs defined
+in TAO client headers. This can be accomplished by modifying the struct
+visitor in `be/be_visitor_structure/structure_ch.cpp`.
+
+At the top of the file, these includes should be added:
+
+```C++
+#include "ast_annotation_member.h"
+#include "utl_annotations.h"
+#include "utl_string.h"
+#include "ast_enum_val.h"
+```
+
+About midway though the file, in
+`int be_visitor_structure_ch::visit_structure (be_structure *node)`
+right before
+```C++
+*os << be_nl_2
+    << "struct " // ...
+```
+these lines would also need to be added:
+
+```C++
+  AST_Annotation_Appl *document =
+    UTL_find_annotation (node->annotations (), "::@document");
+  if (document)
+    {
+      const char *comment =
+        AST_Annotation_Member::narrow_from_decl ((*document)["comment"])->
+          value ()->ev ()->u.strval->get_string ();
+
+      bool deprecated =
+        AST_Annotation_Member::narrow_from_decl ((*document)["deprecated"])->
+          value ()->ev ()->u.bval;
+
+      /*
+       * This is more complicated because we are trying to get the name of
+       * the enumerator. If we just wanted the number value, we could treat the
+       * AST_Expresssion from the annotation member as a unsigned long by using
+       * ev()->u.ulval.
+       */
+      const char *api_type = 0;
+      AST_Expression *api_type_val =
+        AST_Annotation_Member::narrow_from_decl ((*document)["api_type"])->
+          value ();
+      AST_Enum *api_type_enum = api_type_val->enum_parent();
+      if (api_type_enum)
+        {
+          AST_EnumVal *enum_val =
+            api_type_enum->lookup_by_value (api_type_val);
+          if (enum_val)
+            {
+              api_type = enum_val->local_name ()->get_string ();
+            }
+        }
+
+      *os
+        << "/**" << be_nl
+        << " * " << comment << be_nl
+        ;
+
+      if (api_type)
+        {
+          *os
+            << " *" << be_nl
+            << " * API_TYPE: " << api_type << be_nl
+            ;
+        }
+
+      if (deprecated)
+        {
+          *os
+            << " *" << be_nl
+            << " * \\deprecated This is deprecated" << be_nl
+            ;
+        }
+      *os << " */";
+    }
+```
+
+Using the [`@document` use example from above](#document-usage), these are
+inserted into the client header file:
+
+```C++
+// ...
+/**
+ * Struct with 1 member
+ *
+ * API_TYPE: INTERNAL_API
+ */
+struct  struct1
+{
+  // ...
+};
+
+// ...
+
+/**
+ * Struct with 2 members
+ *
+ * API_TYPE: USER_API
+ */
+struct  struct2
+{
+  // ...
+};
+
+/**
+ * Struct with 3 members
+ *
+ * API_TYPE: LEGACY_API
+ *
+ * \deprecated This is deprecated
+ */
+struct  struct3
+{
+  // ...
+};
+// ...
+```
+
+#### Reading Annotations Manually
+
+`UTL_find_annotation` is convenient but only returns the last annotation of the
+passed annotation type. If we want the first one, handle multiple annotations
+of the same type, or read all the annotations, we will have to do what
+`UTL_find_annotation` is doing for us, which is just iterating over the
+`ACE_Vector` of `AST_Annotation_Appl`.
+
+```C++
+  AST_Annotation_Appls &annotations;
+  AST_Annotation_Decl *annotation;
+  for (AST_Annotation_Appls::iterator i = annotations.begin ();
+    i != annotations.end (); ++i)
+    {
+      AST_Annotation_Appl *appl = *i;
+      if (appl && appl->annotation_decl () == annotation)
+        {
+          // Do work with annotation application
+        }
+    }
+```
+
+### Reading Special Cases of Annotations
+
+Annotations placed before a definition in a scope are interpreted as annotating
+the node that is being defined. Annotations in other places require special
+grammar and special handling in the API.
+
+The following cases shows how to get the last annotation called `anno` from
+these special cases.
+
+If a type is `typedef`-ed, resolve it completely using `AST_Type
+*primitive_base_type ()` and `dynamic_cast` to the correct type before trying
+to read these special cases.
+
+#### Unions Discriminators
+
+**[See IDL Example](#unions-discriminators)**
+
+```C++
+  AST_Union *node;
+  AST_Annotation_Appl *document =
+    UTL_find_annotation (node->disc_annotations (), "::@anno");
+```
+
+#### Base Types in Sequences
+
+**[See IDL Example](#base-types-in-sequences)**
+
+```C++
+  AST_Sequence *node;
+  AST_Annotation_Appl *document =
+    UTL_find_annotation (node->base_type_annotations (), "::@anno");
+```
+
+#### Base Types in Arrays
+
+**[See IDL Example](#base-types-in-arrays)**
+
+```C++
+  AST_Array *node;
+  AST_Annotation_Appl *document =
+    UTL_find_annotation (node->base_type_annotations (), "::@anno");
+```
+
+## Limitations
+
+The current limitations exist in TAO\_IDL annotation implementation as of writing:
+
+<a name="limitation1"></a>
+1. Because of lack of a proper grammar specification in IDL for where
+   annotations can go, annotations in places other than before declarations in
+   scopes and other places listed above will result in syntax errors, even if
+   they work with other IDL tools.
+
+2. Even though this is implicitly allowed by the IDL specification, Annotations
+   whose local names clash with IDL keywords are not supported. This includes
+   the OMG standard annotations `default` and `oneway`.