added `transform` parameter to `dump`, to post-process output before writing0.15.4

7 files changed, 220 insertions, 51 deletions

diff --git a/_doc/api.rst b/_doc/api.rst
index 512d726..cf963ca 100644
--- a/_doc/api.rst
+++ b/_doc/api.rst
@@ -12,6 +12,7 @@ At the latest with 1.0, but possible earlier transition error and
 warning messages will be issued, so any packages depending on
 ruamel.yaml should pin the version with which they are testing.
 
+
 Up to 0.15.0, the loaders (``load()``, ``safe_load()``,
 ``round_trip_load()``, ``load_all``, etc.) took, apart from the input
 stream, a ``version`` argument to allow downgrading to YAML 1.1,
@@ -185,7 +186,7 @@ created that were discarded on termination of the function.
 
 This way of doing things leads to several problems:
 
-- it is impossible to return information to the caller apart from the
+- it is virtually impossible to return information to the caller apart from the
   constructed data structure. E.g. if you would get a YAML document
   version number from a directive, there is no way to let the caller
   know apart from handing back special data structures. The same
@@ -195,7 +196,11 @@ This way of doing things leads to several problems:
 - these instances were composites of the various load/dump steps and
   if you wanted to enhance one of the steps, you needed e.g. subclass
   the emitter and make a new composite (dumper) as well, providing all
-  of the parameters (i.e. copy paste
+  of the parameters (i.e. copy paste)
+
+  Alternatives, like making a class that returned a ``Dumper`` when
+  called and sets attributes before doing so, is cumbersome for
+  day-to-day use.
 
 - many routines (like ``add_representer()``) have a direct global
   impact on all of the following calls to ``dump()`` and those are
diff --git a/_doc/basicuse.rst b/_doc/basicuse.rst
index 935e83a..05b59fc 100644
--- a/_doc/basicuse.rst
+++ b/_doc/basicuse.rst
@@ -18,7 +18,9 @@ You load a YAML document using::
 in this ``doc`` can be a file pointer (i.e. an object that has the
 `.read()` method, a string or a ``pathlib.Path()``. `typ='safe'`
 accomplishes the same as what ``safe_load()`` did before: loading of a
-document without resolving unknow tags.
+document without resolving unknow tags. Provide  `pure=True` to
+enforce using the pure Python implementation (faster C libraries will be used
+when possible/available)
 
 Dumping works in the same way::
 
@@ -29,13 +31,17 @@ Dumping works in the same way::
    yaml.dump({a: [1, 2], s)
 
 in this ``s`` can be a file pointer (i.e. an object that has the
-`.write()` method, a ``pathlib.Path()`` or ``None`` (the default, which causes the
-YAML documented to be returned as a string.
+`.write()` method, or a ``pathlib.Path()``. If you want to display
+your output, just stream to `sys.stdout`.
+
+If you need to transform a string representation of the output provide
+a function that takes a string as input and returns one:
+
+   def tr(s):
+       return s.replace('\n', '<\n')  # such output is not valid YAML!
+
+   yaml.dump(data, sys.stdout, transform=tr)
 
-*If you have `yaml.dump()`
-return the YAML doc as string, do not just ``print`` that returned
-value*. In that case use `yaml.dump(data, sys.stdout)`, which is more
-efficient (and shows that you know what you are doing).
 
 More examples
 -------------
diff --git a/_doc/detail.rst b/_doc/detail.rst
index 272b70c..1a2d68e 100644
--- a/_doc/detail.rst
+++ b/_doc/detail.rst
@@ -26,6 +26,12 @@ Details
   (``lc.key('a')``, ``lc.value('a')`` resp. ``lc.item(3)``)
 - preservation of whitelines after block scalars. Contributed by Sam Thursfield.
 
+*In the following examples it is assumed you have done something like:*::
+
+  from ruamel.yaml import YAML
+  yaml = YAML()
+
+*if not explicitly specified.*
 
 Indentation of block sequences
 ------------------------------
@@ -43,7 +49,7 @@ back to::
   -   b: 1
   -   2
 
-if you specify ``indent=4``.
+if you specify ``yaml.indent = 4``.
 
 PyYAML (and older versions of ruamel.yaml) gives you non-indented
 scalars (when specifying default_flow_style=False)::
@@ -52,10 +58,10 @@ scalars (when specifying default_flow_style=False)::
   -   b: 1
   - 2
 
-The dump routine also has an additional ``block_seq_indent`` parameter that
+The dump also observes an additional ``block_seq_indent`` settingr that
 can be used to push the dash inwards, *within the space defined by* ``indent``.
 
-The above example with the often seen ``indent=4, block_seq_indent=2``
+The above example with the often seen ``yaml.indent = 4; yaml.block_seq_indent = 2``
 indentation::
 
   x:
@@ -69,14 +75,14 @@ element itself would normally be pushed to the next line (and older versions
 of ruamel.yaml did so). But this is
 prevented from happening. However the ``indent`` level is what is used
 for calculating the cumulative indent for deeper levels and specifying
-``indent=3`` resp. ``block_seq_indent=2``, migth give correct, but counter
+``yaml.indent = 3`` resp. ``yaml.block_seq_indent = 2``, migth give correct, but counter
 intuitive results.
 
 **It is best to always have** ``indent >= block_seq_indent + 2``
 **but this is not enforced**. Depending on your structure, not following
 this advice **might lead to invalid output**.
 
-Positioning ':' in top level mappings, prefix in ':'
+Positioning ':' in top level mappings, prefixing ':'
 ----------------------------------------------------
 
 If you want your toplevel mappings to look like::
@@ -85,12 +91,12 @@ If you want your toplevel mappings to look like::
   comment        : |
       this is just a first try
 
-then call ``round_trip_dump()`` with ``top_level_colon_align=True``
-(and ``indent=4``). ``True`` causes calculation based on the longest key,
+then set ``yaml.top_level_colon_align = True``
+(and ``yaml.indent = 4``). ``True`` causes calculation based on the longest key,
 but you can also explicitly set a number.
 
 If you want an extra space between a mapping key and the colon specify
-``prefix_colon=' '``::
+``yaml.prefix_colon = ' '``::
 
   - https://myurl/abc.tar.xz : 23445
   #                         ^ extra space here
@@ -98,7 +104,7 @@ If you want an extra space between a mapping key and the colon specify
 
 If you combine ``prefix_colon`` with ``top_level_colon_align``, the
 top level mapping doesn't get the extra prefix. If you want that
-anyway, specify ``top_level_colon_align=12`` where ``12`` has to be an
+anyway, specify ``yaml.top_level_colon_align = 12`` where ``12`` has to be an
 integer that is one more than length of the widest key.
 
 
@@ -122,15 +128,8 @@ The 1.2 version does **not** support:
 - Unquoted Yes and On as alternatives for True and No and Off for False.
 
 If you cannot change your YAML files and you need them to load as 1.1
-you can load with:
-
-  ruamel.yaml.load(some_str, Loader=ruamel.yaml.RoundTripLoader, version=(1, 1))
-
-or the equivalent (version can be a tuple, list or string):
-
-  ruamel.yaml.round_trip_load(some_str, version="1.1")
-
-this also works for ``load_all``/``round_trip_load_all``.
+you can load with ``yaml.version = (1, 1)``,
+or the equivalent (version can be a tuple, list or string)  ``yaml.version = "1.1"
 
 *If you cannot change your code, stick with ruamel.yaml==0.10.23 and let
 me know if it would help to be able to set an environment variable.*
@@ -154,8 +153,11 @@ for for this is::
 
   from __future__ import print_function
 
+  import sys
   import ruamel.yaml
 
+  yaml = ruamel.yaml.YAML()  # defaults to round-trip
+
   inp = """\
   abc:
     - a     # comment 1
@@ -168,14 +170,14 @@ for for this is::
     f: 6 # comment 3
   """
 
-  data = ruamel.yaml.load(inp, ruamel.yaml.RoundTripLoader)
+  data = yaml.load(inp)
   data['abc'].append('b')
   data['abc'].yaml_add_eol_comment('comment 4', 1)  # takes column of comment 1
   data['xyz'].yaml_add_eol_comment('comment 5', 'c')  # takes column of comment 2
   data['xyz'].yaml_add_eol_comment('comment 6', 'e')  # takes column of comment 3
   data['xyz'].yaml_add_eol_comment('comment 7', 'd', column=20)
 
-  print(ruamel.yaml.dump(data, Dumper=ruamel.yaml.RoundTripDumper), end='')
+  yaml.dump(data, sys.stdout)
 
 .. example code add_comment.py
 
@@ -263,6 +265,6 @@ included and you can do::
     """
 
 
-    data = yaml.load(yaml_str, Loader=yaml.RoundTripLoader)
+    data = yaml.load(yaml_str)
 
     assert data.mlget(['a', 1, 'd', 'f'], list_ok=True) == 196
diff --git a/_doc/example.rst b/_doc/example.rst
index 4e8ac4a..000b06e 100644
--- a/_doc/example.rst
+++ b/_doc/example.rst
@@ -4,8 +4,41 @@ Examples
 Basic round trip of parsing YAML to Python objects, modifying
 and generating YAML::
 
+  import sys
+  from ruamel.yaml import YAML
+
+  inp = """\
+  # example
+  name:
+    # details
+    family: Smith   # very common
+    given: Alice    # one of the siblings
+  """
+
+  yaml = YAML()
+  code = yaml.load(inp)
+  code['name']['given'] = 'Bob'
+
+  yaml.dump(code, sys.stdout)
+
+.. example code small.py
+
+Resulting in ::
+
+  # example
+  name:
+    # details
+    family: Smith   # very common
+    given: Bob      # one of the siblings
+
+
+.. example output small.py
+
+with the old API::
+
   from __future__ import print_function
 
+  import sys
   import ruamel.yaml
 
   inp = """\
@@ -19,9 +52,13 @@ and generating YAML::
   code = ruamel.yaml.load(inp, ruamel.yaml.RoundTripLoader)
   code['name']['given'] = 'Bob'
 
-  print(ruamel.yaml.dump(code, Dumper=ruamel.yaml.RoundTripDumper), end='')
+  ruamel.yaml.dump(code, sys.stdout, Dumper=ruamel.yaml.RoundTripDumper)
 
-.. example code small.py
+  # the last statement can be done less efficient in time and memory with
+  # leaving out the end='' would cause a double newline at the end
+  # print(ruamel.yaml.dump(code, Dumper=ruamel.yaml.RoundTripDumper), end='')
+
+.. example code small_o.py
 
 Resulting in ::
 
@@ -32,7 +69,8 @@ Resulting in ::
     given: Bob      # one of the siblings
 
 
-.. example output small.py
+.. example output small_o.py
+
 
 ----
 
@@ -40,7 +78,7 @@ YAML handcrafted anchors and references as well as key merging
 are preserved. The merged keys can transparently be accessed
 using ``[]`` and ``.get()``::
 
-  import ruamel.yaml
+  from ruamel.yaml import YAML
 
   inp = """\
   - &CENTER {x: 1, y: 2}
@@ -66,26 +104,33 @@ using ``[]`` and ``.get()``::
     label: center/big
   """
 
-  data = ruamel.yaml.load(inp, ruamel.yaml.RoundTripLoader)
+  yaml = YAML()
+  data = yaml.load(inp)
   assert data[7]['y'] == 2
 
-
 .. example code anchor_merge.py
 
+
 ----
 
 The ``CommentedMap``, which is the ``dict`` like construct one gets when round-trip loading,
 supports insertion of a key into a particular position, while optionally adding a comment::
 
+  import sys
+  from ruamel.yaml import YAML
+
   yaml_str = """\
   first_name: Art
   occupation: Architect  # This is an occupation comment
   about: Art Vandelay is a fictional character that George invents...
   """
 
-  data = ruamel.yaml.round_trip_load(yaml_str)
+  yaml = YAML()
+  data = yaml.load(yaml_str)
   data.insert(1, 'last name', 'Vandelay', comment="new key")
-  print(ruamel.yaml.round_trip_dump(data))
+  yaml.dump(data, sys.stdout)
+
+.. example code map_insert.py
 
 gives::
 
@@ -94,9 +139,99 @@ gives::
   occupation: Architect  # This is an occupation comment
   about: Art Vandelay is a fictional character that George invents...
 
+
+.. example output map_insert.py
+
 Please note that the comment is aligned with that of its neighbour (if available).
 
 The above was inspired by a `question <http://stackoverflow.com/a/36970608/1307905>`_
 posted by *demux* on StackOverflow.
 
 
+----
+
+By default `ruamel.yaml` indents with two positions in block style, for
+both mappings and sequences. For sequences the indent is counted to the beginning of the
+scalar, with the dash taking the first position of the indented "space".
+
+The following program with three dumps::
+
+
+  import sys
+  from ruamel.yaml import YAML
+
+  data = {1: {1: [{1: 1, 2: 2}, {1: 1, 2: 2}], 2: 2}, 2: 42}
+
+  yaml = YAML()
+  yaml.explicit_start = True
+  yaml.dump(data, sys.stdout)
+  yaml.indent = 4
+  yaml.block_seq_indent = 2
+  yaml.dump(data, sys.stdout)
+
+
+  def sequence_indent_four(s):
+      # this will fail on direclty nested lists: {1; [[2, 3], 4]}
+      levels = []
+      ret_val = ''
+      for line in s.splitlines(True):
+          ls = line.lstrip()
+          indent = len(line) - len(ls)
+          if ls.startswith('- '):
+              if not levels or indent > levels[-1]:
+                  levels.append(indent)
+              elif levels:
+                  if indent < levels[-1]:
+                      levels = levels[:-1]
+              # same -> do nothing
+          else:
+              if levels:
+                  if indent <= levels[-1]:
+                      while levels and indent <= levels[-1]:
+                          levels = levels[:-1]
+          ret_val += '  ' * len(levels) + line
+      return ret_val
+
+  yaml = YAML()
+  yaml.explicit_start = True
+  yaml.dump(data, sys.stdout, transform=sequence_indent_four)
+
+.. example code transform.py
+
+gives as output::
+
+  ---
+  1:
+    1:
+    - 1: 1
+      2: 2
+    - 1: 1
+      2: 2
+    2: 2
+  2: 42
+  ---
+  1:
+      1:
+        - 1: 1
+          2: 2
+        - 1: 1
+          2: 2
+      2: 2
+  2: 42
+  ---
+  1:
+    1:
+      - 1: 1
+        2: 2
+      - 1: 1
+        2: 2
+    2: 2
+  2: 42
+
+
+.. example output transform.py
+
+
+The transform example was inspired by a `question
+<https://stackoverflow.com/q/44388701/1307905>`_ posted by *nowox* on
+StackOverflow.
diff --git a/_doc/install.rst b/_doc/install.rst
index be3c4f4..e90f9e7 100644
--- a/_doc/install.rst
+++ b/_doc/install.rst
@@ -1,11 +1,11 @@
 Installing
 ==========
 
-``ruamel.yaml`` can be installed from PyPI_ using::
+``ruamel.yaml`` should be installed from PyPI_ using::
 
     pip install ruamel.yaml
 
-There is a a commandline utility ``yaml`` available after installing::
+There also is a commandline utility ``yaml`` available after installing::
 
    pip install ruamel.yaml.cmd
 
@@ -17,7 +17,7 @@ Optional requirements
 
 If you have the C yaml library and headers installed, as well as
 the header files for your Python executables then you can use the
-non-roundtrip but faster C loader and emitter.
+non-roundtrip, but faster, C loader and emitter.
 
 On Debian systems you should use::
 
@@ -28,4 +28,3 @@ you can leave out ``python3-dev`` if you don't use python3
 For CentOS (7) based systems you should do::
 
    sudo yum install libyaml-devel python-devel
-
diff --git a/_doc/overview.rst b/_doc/overview.rst
index 1c664e0..18af771 100644
--- a/_doc/overview.rst
+++ b/_doc/overview.rst
@@ -3,17 +3,17 @@ Overview
 
 ``ruamel.yaml`` is a YAML 1.2  loader/dumper package for Python. It is a
 derivative of Kirill Simonov's `PyYAML 3.11
-<https://bitbucket.org/xi/pyyaml>`_ 
+<https://bitbucket.org/xi/pyyaml>`_.
 
 ``ruamel.yaml`` supports `YAML 1.2`_ and has  round-trip loaders and dumpers
 that preserves, among others:
 
 - comments
-- block style and key ordering are kept, so you can diff the round-tripped 
+- block style and key ordering are kept, so you can diff the round-tripped
   source
 - flow style sequences ( 'a: b, c, d') (based on request and test by
   Anthony Sottile)
-- anchors names that are hand-crafted (i.e. not of the form``idNNN``)
+- anchor names that are hand-crafted (i.e. not of the form``idNNN``)
 - `merges <http://yaml.org/type/merge.html>`_ in dictionaries are preserved
 
 This preservation is normally not broken unless you severely alter
@@ -22,10 +22,25 @@ Reassigning values or replacing list items, etc., is fine.
 
 For the specific 1.2 differences see :ref:`yaml-1-2-support`
 
-Although individual indentation is not preserved, you can specify both
-indentation and specific offset of block sequence dashes within that
-indentation. There is a utility function that tries to determine the
-correct values for ``indent`` and ``block_seq_indent`` that can
-then be passed to the dumper.
+Although individual indentation of lines is not preserved, you can
+specify both indentation (counting for this does **not** include the
+dash for a sequence element) and specific offset of block sequence
+dashes within that indentation. There is a utility function that tries
+to determine the correct values for ``indent`` and
+``block_seq_indent`` that can then be passed to the dumper.
+
+
+Although `ruamel.yaml` still allows most of the PyYAML way of doing
+things, adding features required a different API then the transient
+nature of PyYAML's ``Loader`` and ``Dumper``.  Starting with
+``ruamel.yaml`` version 0.15.0 this new API gets introduced. Old ways
+that get in the way will be removed, after first generating warnings
+on use, then generating an error. In general a warning in version 0.N.x will become an
+error in 0.N+1.0
+
+
+Many of the bugs filed against PyYAML, but that were never
+acted upon, have been fixed in ``ruamel.yaml``
+
 
 .. include:: links.rst
diff --git a/_doc/pyyaml.rst b/_doc/pyyaml.rst
index 93dcd65..822efd8 100644
--- a/_doc/pyyaml.rst
+++ b/_doc/pyyaml.rst
@@ -47,7 +47,7 @@ PY2/PY3 reintegration
 ``ruamel.yaml`` re-integrates the Python 2 and 3 sources, running on
 Python 2.7 (CPython, PyPy), 3.3, 3.4, 3.5 (support for 2.6 has been
 dropped mid 2016). It is more easy to extend and maintain as only a
-miniscule part of the code is version specific.
+miniscule part of the code is Python version specific.
 
 Fixes
 -----
@@ -68,4 +68,11 @@ test framework is called from within ``tox`` runs.
 Before versions are pushed to PyPI, ``tox`` is invoked, and has to pass, on all
 supported Python versions, on PyPI as well as flake8/pep8
 
+API
+---
+
+Starting with 0.15 the API for using ``ruamel.yaml`` has diverged allowing easier
+addition of new features.
+
+
 .. include:: links.rst