Merge branch 'release/1.9'1.9

16 files changed, 269 insertions, 65 deletions

diff --git a/.gitignore b/.gitignore
index 07d32c1..ada7038 100644
--- a/.gitignore
+++ b/.gitignore
@@ -12,3 +12,4 @@ build/*
 dist/*
 .coverage
 
+.vscode
diff --git a/README.md b/README.md
index 6da4a4a..201cc93 100644
--- a/README.md
+++ b/README.md
@@ -26,6 +26,39 @@ To install the qface library you need to have python3 and pip installed.
 pip3 install qface
 ```
 
+## Install Development Version
+
+### Prerequisites
+
+To install the development version you need to clone the repository and ensure you have checkout the develop branch.
+
+```sh
+git clone git@github.com:Pelagicore/qface.git
+cd qface
+git checkout develop
+```
+
+The installation requires the python package manager called (pip) using the python 3 version. You can try:
+
+```sh
+python3 --version
+pip3 --version
+```
+
+### Installation
+
+Use the editable option of pip to install an editable version.
+
+```sh
+cd qface
+pip3 install --editable .
+```
+
+This reads the `setup.py` document and installs the package as reference to this repository. So all changes will be immediatly reflected in the installation.
+
+To update the installation just simple pull from the git repository.
+
+
 ## Download
 
 If you are looking for the examples and the builtin generators you need to download the code.
diff --git a/docs/extending.rst b/docs/extending.rst
index 042f094..5252ed0 100644
--- a/docs/extending.rst
+++ b/docs/extending.rst
@@ -140,9 +140,9 @@ The features are passed to the generator in your custom generator code. The exis
 
 Here the plugin rule will only be run when the feature set contains a 'plugin_enabled' string.
 
-.. rubric:: Preserve
+.. rubric:: Preserving Documents
 
-Documents can be marked as preserved to prevent them to be overwritten when the user has edited them. the rules documents has an own marker for this called ``preserve``. This is a list of target documents which shall be be marked preserved by the generator.
+Documents can be moved to the ``preserve`` tag to prevent them to be overwritten. The rules documents has an own marker for this called ``preserve``. This is the same dictionary of target/source documents which shall be be marked preserved by the generator.
 
 
 .. code-block:: yaml
@@ -151,12 +151,10 @@ Documents can be marked as preserved to prevent them to be overwritten when the
         interface:
             documents:
                 '{{interface|lower}}.h': 'plugin/interface.h'
-                '{{interface|lower}}.cpp': 'plugin/interface.cpp'
             preserve:
-                - '{{interface|lower}}.h'
-                - '{{interface|lower}}.cpp'
+                '{{interface|lower}}.cpp': 'plugin/interface.cpp'
 
-In the example above the two interface documents will not be overwritten during a second generator call and can be edited by the user.
+In the example above the preserve listed documents will not be overwritten during a second generator run and can be edited by the user.
 
 .. rubric:: Destination and Source
 
@@ -181,3 +179,19 @@ This is the implicit logical hierarchy taken into account:
 
 Typical you place the destination prefix on the module level if your destination depends on the module symbol. For generic templates you would place the destination on the system level. On the system level you can not use child symbols (such as the module) as at this time these symbols are not known yet.
 
+Parsing Documentation Comments
+==============================
+
+The comments are provided as raw text to the template engine. You need to parse using the `parse_doc` tag and the you can inspect the documentation object.
+
+See below for a simple example
+
+.. code-block:: html
+
+    {% with doc = property.comment|parse_doc %}
+    \brief {{doc.brief}}
+
+    {{doc.description}}
+    {% endwith %}
+
+Each tag in the JavaDoc styled comment, will be converted into a property of the object returned by `parse_doc`. All lines without a tag will be merged into the description tag.
diff --git a/docs/index.rst b/docs/index.rst
index 6e7ae75..ed5f38a 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -2,7 +2,9 @@
 QFace
 =====
 
-QFace is a flexible Qt API generator. It uses a common IDL format (called QFace interface document) to define an API. QFace comes with a set of predefined generators to generate QML Plugins. QFace can be easily extended with your own generator.
+QFace is a flexible API generator inspired by the Qt API idioms. It uses a common IDL format (called QFace interface document) to define an API. QFace is optimized to write a custom generator based on the common IDL format.
+
+There exists already several code generators for common use cases. These can be used as is or can be used as a base for a custom generator.
 
 .. toctree::
    :maxdepth: 1
@@ -18,10 +20,34 @@ QFace is a flexible Qt API generator. It uses a common IDL format (called QFace
    extending
    api
 
+
+Features
+========
+
+The list fo features is plit between features which are based on the choosen IDL and features wich are provided by the generator itself.
+
+.. rubric:: IDL Features
+
+- Common modern IDL
+- Scalable through modules
+- Structure through structs, enums, flags
+- Interface API with properties, operations and signals
+- Annotations using YAML syntax
+- Documentable IDL
+
+.. rubric:: Generator Features
+
+- Easy to install using python package manager
+- Designed to be extended
+- Well defined domain objects
+- Template based code generator
+- Simple rule based code builder
+- Well documented
+
 Quick Start
 ===========
 
-QFace is a generator framework but also comes with several reference code generator.
+QFace is a generator framework and is bundled with several reference code generator.
 
 To install qface you need to have python3 installed and typically also pip3
 
@@ -29,7 +55,7 @@ To install qface you need to have python3 installed and typically also pip3
 
     pip3 install qface
 
-This installs the python qface library and the two reference generator qface-qtcpp and qface-qtqml.
+This installs the python qface library onto your system.
 
 You can verify that you have qface installed with
 
@@ -98,22 +124,10 @@ And a "org.example.txt" file named after the module should be generated.
 * :doc:`domain`
 * :doc:`api`
 
-Builtin Generators
+Bundled Generators
 ------------------
 
-The built-in generators qface-qtcpp and qface-qtqml will generator cpp / qml code from the interface files. The generated code is source code compatible and can be used with the same QML based user interface
-
-.. code-block:: bash
-
-    mkdir cpp-out
-    qface-qtcpp sample.qface cpp-out
-
-    mkdir qml-out
-    qface-qtqml sample.qface qml-out
-
-The generators can run with one or more input files or folders and generate code for one or more modules. In case of the qtcpp generator the code needs to be open with QtCreator and compiled and installed.
-
-For the QML code the code must just made available to the QML import path.
+QFace has some gnerators which are bundled with the QFace library. They live in their own reposiutories. These generators are documented in the repositories.
 
 .. rubric:: See Also
 
diff --git a/interfaces/org.qface.meta.qface b/interfaces/org.qface.meta.qface
new file mode 100644
index 0000000..ae496fb
--- /dev/null
+++ b/interfaces/org.qface.meta.qface
@@ -0,0 +1,81 @@
+ module org.qface.meta 1.0
+
+interface MetaBuilder {
+    ESystem system;
+    void load(string path);
+    void store(string path);
+}
+
+struct EType {
+    bool isComplex
+    bool isPrimitive;
+    bool isString;
+    bool isBool;
+    bool isInt;
+    bool isReal;
+    bool isList;
+    bool isModel;
+    string name;
+}
+
+struct ESystem {
+    list<EModule> modules;
+}
+
+struct EModule {
+    list<EInterface> interfaces;
+    list<EStruct> structs;
+    list<EEnum> enums;
+    list<EFlag> flags;
+}
+
+struct EInterface {
+    string name;
+    list<EProperty> properties;
+    list<EOperation> operations;
+    list<ESignal> signals;
+}
+
+struct EProperty {
+    string name
+    EType type
+}
+
+struct EOperation {
+    string name;
+    EType type;
+    list<EParameter> parameters;
+}
+
+struct ESignal {
+    string name;
+    list<EParameter> parameters;
+}
+
+struct EStruct {
+    string name;
+    list<EField> fields;
+}
+
+struct EField {
+    string name;
+    EType type;
+}
+
+struct Enum {
+    string name
+    model<EEnumMember> members;
+}
+
+struct EEnumMember {
+    int value;
+    string name;
+}
+
+struct EFlag {
+    string name
+    model<EEnumMember> members;
+}
+
+
+
diff --git a/qface/__about__.py b/qface/__about__.py
index 7073a0f..e3fb0a8 100644
--- a/qface/__about__.py
+++ b/qface/__about__.py
@@ -9,7 +9,7 @@ except NameError:
 __title__ = "qface"
 __summary__ = "A generator framework based on a common modern IDL"
 __url__ = "https://pelagicore.github.io/qface/"
-__version__ = "1.8.1"
+__version__ = "1.9"
 __author__ = "JRyannel"
 __author_email__ = "qface-generator@googlegroups.com"
-__copyright__ = "2017 Pelagicore"
+__copyright__ = "2019 Pelagicore"
diff --git a/qface/contrib/__init__.py b/qface/contrib/__init__.py
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/qface/contrib/__init__.py
diff --git a/qface/contrib/logging.py b/qface/contrib/logging.py
new file mode 100644
index 0000000..6b7625c
--- /dev/null
+++ b/qface/contrib/logging.py
@@ -0,0 +1,28 @@
+import yaml
+import logging
+import logging.config
+import coloredlogs
+from path import Path
+import os
+
+
+def basic_log(level):
+    logging.basicConfig(level=level)
+    coloredlogs.install(level=level)
+    print('Fall back to basic logging')
+
+
+def setup_log(path='logging.yaml', level=logging.INFO, env_key='QFACE_LOG_CFG'):
+    path = Path(os.getenv(env_key, path))
+    if path.exists():
+        try:
+            config = yaml.safe_load(path.text())
+            logging.config.dictConfig(config)
+            coloredlogs.install()
+        except Exception as e:
+            print(e)
+            print('Error in logging configuration. Fall back to defaults.')
+            basic_log(level)
+    else:
+        basic_log(level)
+        print('Failed to load logging config file.')
diff --git a/qface/generator.py b/qface/generator.py
index eca36f0..eb7b24a 100644
--- a/qface/generator.py
+++ b/qface/generator.py
@@ -1,7 +1,7 @@
 
 # Copyright (c) Pelagicore AB 2016
 
-from jinja2 import Environment, Template
+from jinja2 import Environment, Template, Undefined, StrictUndefined
 from jinja2 import FileSystemLoader, PackageLoader, ChoiceLoader
 from jinja2 import TemplateSyntaxError, TemplateNotFound, TemplateError
 from path import Path
@@ -12,7 +12,7 @@ import logging
 import hashlib
 import yaml
 import click
-import sys
+import sys, os
 
 from .idl.parser.TLexer import TLexer
 from .idl.parser.TParser import TParser
@@ -22,6 +22,7 @@ from .idl.listener import DomainListener
 from .utils import merge
 from .filters import filters
 
+from jinja2.debug import make_traceback as _make_traceback
 
 try:
     from yaml import CLoader as Loader, CDumper as Dumper
@@ -31,12 +32,23 @@ except ImportError:
 logger = logging.getLogger(__name__)
 
 
-"""
-Provides an API for accessing the file system and controlling the generator
-"""
+def template_error_handler(traceback):
+    exc_type, exc_obj, exc_tb = traceback.exc_info
+    error = exc_obj
+    if isinstance(exc_type, TemplateError):
+        error = exc_obj.message
+    message = '{0}:{1}: error: {2}'.format(exc_tb.tb_frame.f_code.co_filename, exc_tb.tb_lineno, error)
+    click.secho(message, fg='red', err=True)
+
+
+class TestableUndefined(StrictUndefined):
+    """Return an error for all undefined values, but allow testing them in if statements"""
+    def __bool__(self):
+        return False
 
 
 class ReportingErrorListener(ErrorListener.ErrorListener):
+    """ Provides an API for accessing the file system and controlling the generator """
     def __init__(self, document):
         self.document = document
 
@@ -60,7 +72,7 @@ class Generator(object):
     strict = False
     """ enables strict code generation """
 
-    def __init__(self, search_path: str, context: dict={}):
+    def __init__(self, search_path, context={}):
         loader = ChoiceLoader([
             FileSystemLoader(search_path),
             PackageLoader('qface')
@@ -68,8 +80,9 @@ class Generator(object):
         self.env = Environment(
             loader=loader,
             trim_blocks=True,
-            lstrip_blocks=True
+            lstrip_blocks=True,
         )
+        self.env.exception_handler = template_error_handler
         self.env.filters.update(filters)
         self._destination = Path()
         self._source = ''
@@ -81,7 +94,7 @@ class Generator(object):
         return self._destination
 
     @destination.setter
-    def destination(self, dst: str):
+    def destination(self, dst):
         if dst:
             self._destination = Path(self.apply(dst, self.context))
 
@@ -91,7 +104,7 @@ class Generator(object):
         return self._source
 
     @source.setter
-    def source(self, source: str):
+    def source(self, source):
         if source:
             self._source = source
 
@@ -103,28 +116,30 @@ class Generator(object):
     def filters(self, filters):
         self.env.filters.update(filters)
 
-    def get_template(self, name: str):
+    def get_template(self, name):
         """Retrieves a single template file from the template loader"""
         source = name
         if name and name[0] is '/':
             source = name[1:]
         elif self.source is not None:
             source = '/'.join((self.source, name))
-            print('get_template: ', name, source)
-
         return self.env.get_template(source)
 
-    def render(self, name: str, context: dict):
+    def render(self, name, context):
         """Returns the rendered text from a single template file from the
         template loader using the given context data"""
+        if Generator.strict:
+            self.env.undefined = TestableUndefined
+        else:
+            self.env.undefined = Undefined
         template = self.get_template(name)
         return template.render(context)
 
-    def apply(self, template: str, context: dict):
+    def apply(self, template, context):
         """Return the rendered text of a template instance"""
         return self.env.from_string(template).render(context)
 
-    def write(self, file_path: Path, template: str, context: dict={}, preserve: bool = False):
+    def write(self, file_path, template, context={}, preserve=False, force=False):
         """Using a template file name it renders a template
            into a file given a context
         """
@@ -132,28 +147,28 @@ class Generator(object):
             context = self.context
         error = False
         try:
-            self._write(file_path, template, context, preserve)
+            self._write(file_path, template, context, preserve, force)
         except TemplateSyntaxError as exc:
-            message = '{0}:{1} error: {2}'.format(exc.filename, exc.lineno, exc.message)
-            click.secho(message, fg='red')
+            message = '{0}:{1}: error: {2}'.format(exc.filename, exc.lineno, exc.message)
+            click.secho(message, fg='red', err=True)
             error = True
         except TemplateNotFound as exc:
-            message = '{0} error: Template not found'.format(exc.name)
-            click.secho(message, fg='red')
+            message = '{0}: error: Template not found'.format(exc.name)
+            click.secho(message, fg='red', err=True)
             error = True
         except TemplateError as exc:
-            message = 'error: {0}'.format(exc.message)
-            click.secho(message, fg='red')
+            # Just return with an error, the generic template_error_handler takes care of printing it
             error = True
+
         if error and Generator.strict:
-            sys.exit(-1)
+            sys.exit(1)
 
-    def _write(self, file_path: Path, template: str, context: dict, preserve: bool = False):
+    def _write(self, file_path: Path, template: str, context: dict, preserve: bool = False, force: bool = False):
         path = self.destination / Path(self.apply(file_path, context))
         path.parent.makedirs_p()
         logger.info('write {0}'.format(path))
         data = self.render(template, context)
-        if self._has_different_content(data, path):
+        if self._has_different_content(data, path) or force:
             if path.exists() and preserve:
                 click.secho('preserve: {0}'.format(path), fg='blue')
             else:
@@ -220,12 +235,10 @@ class RuleGenerator(Generator):
         self.context.update(rule.get('context', {}))
         self.destination = rule.get('destination', None)
         self.source = rule.get('source', None)
-        preserved = rule.get('preserve', [])
-        if not preserved:
-            preserved = []
         for target, source in rule.get('documents', {}).items():
-            preserve = target in preserved
-            self.write(target, source, preserve=preserve)
+            self.write(target, source)
+        for target, source in rule.get('preserve', {}).items():
+            self.write(target, source, preserve=True)
 
     def _shall_proceed(self, obj):
         conditions = obj.get('when', [])
@@ -248,10 +261,10 @@ class FileSystem(object):
         try:
             return FileSystem._parse_document(document, system)
         except FileNotFoundError as e:
-            click.secho('{0}: file not found'.format(document), fg='red')
+            click.secho('{0}: error: file not found'.format(document), fg='red', err=True)
             error = True
         except ValueError as e:
-            click.secho('Error parsing document {0}'.format(document))
+            click.secho('Error parsing document {0}'.format(document), fg='red', err=True)
             error = True
         if error and FileSystem.strict:
             sys.exit(-1)
@@ -337,10 +350,13 @@ class FileSystem(object):
         document = Path(document)
         if not document.exists():
             if required:
-                click.secho('yaml document does not exists: {0}'.format(document), fg='red')
+                click.secho('yaml document does not exists: {0}'.format(document), fg='red', err=True)
             return {}
         try:
             return yaml.load(document.text(), Loader=Loader)
         except yaml.YAMLError as exc:
-            click.secho(str(exc), fg='red')
+            error = document
+            if hasattr(exc, 'problem_mark'):
+                error = '{0}:{1}'.format(error, exc.problem_mark.line+1)
+            click.secho('{0}: error: {1}'.format(error, str(exc)), fg='red', err=True)
         return {}
diff --git a/qface/helper/doc.py b/qface/helper/doc.py
index 9695136..2b1df51 100644
--- a/qface/helper/doc.py
+++ b/qface/helper/doc.py
@@ -2,7 +2,7 @@ import re
 
 translate = None
 """
-The translare function used for transalting inline tags. The
+The translate function used for translating inline tags. The
 function will be called with tag, value arguments.
 
 Example:
@@ -56,6 +56,8 @@ class DocObject:
 
 
 def parse_doc(s):
+    """ parse a comment in the format of JavaDoc and returns an object, where each JavaDoc tag
+        is a property of the object. """
     if not s:
         return
     doc = DocObject()
diff --git a/qface/helper/qtcpp.py b/qface/helper/qtcpp.py
index 83aa8fc..dc37f2a 100644
--- a/qface/helper/qtcpp.py
+++ b/qface/helper/qtcpp.py
@@ -126,7 +126,9 @@ class Filters(object):
     @staticmethod
     def close_ns(symbol):
         '''generates a closing names statement from a symbol'''
-        return ' '.join(['}' for x in symbol.module.name_parts])
+        closing = ' '.join(['}' for x in symbol.module.name_parts])
+        name = '::'.join(symbol.module.name_parts)
+        return '{0} // namespace {1}'.format(closing, name)
 
     @staticmethod
     def using_ns(symbol):
diff --git a/qface/helper/qtqml.py b/qface/helper/qtqml.py
index db070f5..e29311f 100644
--- a/qface/helper/qtqml.py
+++ b/qface/helper/qtqml.py
@@ -59,3 +59,11 @@ class Filters(object):
             return 'ListModel'
         return t
 
+    @staticmethod
+    def path(s):
+        return str(s).replace('.', '/')
+
+    @staticmethod
+    def identifier(s):
+        return str(s).lower().replace('.', '_')
+
diff --git a/qface/idl/domain.py b/qface/idl/domain.py
index d4f87ca..75a2950 100644
--- a/qface/idl/domain.py
+++ b/qface/idl/domain.py
@@ -500,6 +500,10 @@ class Property(Symbol):
         '''return the fully qualified name (`<module>.<interface>#<property>`)'''
         return '{0}.{1}#{2}'.format(self.module.name, self.interface.name, self.name)
 
+    @property
+    def writeable(self):
+        return not self.readonly and not self.const
+
     def toJson(self):
         o = super().toJson()
         if self.readonly:
diff --git a/qface/idl/listener.py b/qface/idl/listener.py
index 3a36837..34a0c64 100644
--- a/qface/idl/listener.py
+++ b/qface/idl/listener.py
@@ -200,6 +200,7 @@ class DomainListener(TListener):
         assert self.struct
         name = ctx.name.text
         self.field = Field(name, self.struct)
+        self.parse_annotations(ctx, self.field)
         contextMap[ctx] = self.field
 
     def exitStructFieldSymbol(self, ctx: TParser.StructFieldSymbolContext):
diff --git a/qface/watch.py b/qface/watch.py
index 4667fe3..9501f25 100644
--- a/qface/watch.py
+++ b/qface/watch.py
@@ -28,15 +28,15 @@ class RunScriptChangeHandler(FileSystemEventHandler):
         self.is_running = False
 
 
-def monitor(script, src, dst):
+def monitor(script, src, dst, args):
     """
     reloads the script given by argv when src files changes
     """
     src = src if isinstance(src, (list, tuple)) else [src]
     dst = Path(dst).expand().abspath()
     src = [Path(entry).expand().abspath() for entry in src]
-    script = Path(script).expand().abspath()
-    command = '{0} {1} {2}'.format(script, ' '.join(src), dst)
+    command = ' '.join(args)
+    print('command: ', command)
     event_handler = RunScriptChangeHandler(command)
     observer = Observer()
     click.secho('watch recursive: {0}'.format(script.dirname()), fg='blue')
diff --git a/tests/test_qtcpp_helper.py b/tests/test_qtcpp_helper.py
index acc034f..de83448 100644
--- a/tests/test_qtcpp_helper.py
+++ b/tests/test_qtcpp_helper.py
@@ -222,7 +222,7 @@ def test_namespace():
     assert ns == 'namespace org { namespace example {'
 
     ns = qtcpp.Filters.close_ns(module)
-    assert ns == '} }'
+    assert ns == '} } // namespace org::example'
 
     ns = qtcpp.Filters.using_ns(module)
     assert ns == 'using namespace org::example;'