diff options
Diffstat (limited to 'Doc/library/ast.rst')
| -rw-r--r-- | Doc/library/ast.rst | 257 |
1 files changed, 257 insertions, 0 deletions
diff --git a/Doc/library/ast.rst b/Doc/library/ast.rst new file mode 100644 index 0000000000..70840da779 --- /dev/null +++ b/Doc/library/ast.rst @@ -0,0 +1,257 @@ +.. _ast: + +Abstract Syntax Trees +===================== + +.. module:: ast + :synopsis: Abstract Syntax Tree classes and manipulation. + +.. sectionauthor:: Martin v. Löwis <martin@v.loewis.de> +.. sectionauthor:: Georg Brandl <georg@python.org> + +.. versionadded:: 2.5 + The low-level ``_ast`` module containing only the node classes. + +.. versionadded:: 2.6 + The high-level ``ast`` module containing all helpers. + + +The :mod:`ast` module helps Python applications to process trees of the Python +abstract syntax grammar. The abstract syntax itself might change with each +Python release; this module helps to find out programmatically what the current +grammar looks like. + +An abstract syntax tree can be generated by passing :data:`_ast.PyCF_ONLY_AST` +as a flag to the :func:`compile` builtin function, or using the :func:`parse` +helper provided in this module. The result will be a tree of objects whose +classes all inherit from :class:`ast.AST`. + +A modified abstract syntax tree can be compiled into a Python code object using +the built-in :func:`compile` function. + +Node classes +------------ + +.. class:: AST + + This is the base of all AST node classes. The actual node classes are + derived from the :file:`Parser/Python.asdl` file, which is reproduced + :ref:`below <abstract-grammar>`. They are defined in the :mod:`_ast` C + module and re-exported in :mod:`ast`. + + There is one class defined for each left-hand side symbol in the abstract + grammar (for example, :class:`ast.stmt` or :class:`ast.expr`). In addition, + there is one class defined for each constructor on the right-hand side; these + classes inherit from the classes for the left-hand side trees. For example, + :class:`ast.BinOp` inherits from :class:`ast.expr`. For production rules + with alternatives (aka "sums"), the left-hand side class is abstract: only + instances of specific constructor nodes are ever created. + + .. attribute:: _fields + + Each concrete class has an attribute :attr:`_fields` which gives the names + of all child nodes. + + Each instance of a concrete class has one attribute for each child node, + of the type as defined in the grammar. For example, :class:`ast.BinOp` + instances have an attribute :attr:`left` of type :class:`ast.expr`. + + If these attributes are marked as optional in the grammar (using a + question mark), the value might be ``None``. If the attributes can have + zero-or-more values (marked with an asterisk), the values are represented + as Python lists. All possible attributes must be present and have valid + values when compiling an AST with :func:`compile`. + + .. attribute:: lineno + col_offset + + Instances of :class:`ast.expr` and :class:`ast.stmt` subclasses have + :attr:`lineno` and :attr:`col_offset` attributes. The :attr:`lineno` is + the line number of source text (1-indexed so the first line is line 1) and + the :attr:`col_offset` is the UTF-8 byte offset of the first token that + generated the node. The UTF-8 offset is recorded because the parser uses + UTF-8 internally. + + The constructor of a class :class:`ast.T` parses its arguments as follows: + + * If there are positional arguments, there must be as many as there are items + in :attr:`T._fields`; they will be assigned as attributes of these names. + * If there are keyword arguments, they will set the attributes of the same + names to the given values. + + For example, to create and populate an :class:`ast.UnaryOp` node, you could + use :: + + node = ast.UnaryOp() + node.op = ast.USub() + node.operand = ast.Num() + node.operand.n = 5 + node.operand.lineno = 0 + node.operand.col_offset = 0 + node.lineno = 0 + node.col_offset = 0 + + or the more compact :: + + node = ast.UnaryOp(ast.USub(), ast.Num(5, lineno=0, col_offset=0), + lineno=0, col_offset=0) + + +.. _abstract-grammar: + +Abstract Grammar +---------------- + +The module defines a string constant ``__version__`` which is the decimal +Subversion revision number of the file shown below. + +The abstract grammar is currently defined as follows: + +.. literalinclude:: ../../Parser/Python.asdl + + +:mod:`ast` Helpers +------------------ + +.. versionadded:: 2.6 + +Apart from the node classes, :mod:`ast` module defines these utility functions +and classes for traversing abstract syntax trees: + +.. function:: parse(expr, filename='<unknown>', mode='exec') + + Parse an expression into an AST node. Equivalent to ``compile(expr, + filename, mode, PyCF_ONLY_AST)``. + + +.. function:: literal_eval(node_or_string) + + Safely evaluate an expression node or a string containing a Python + expression. The string or node provided may only consist of the following + Python literal structures: strings, numbers, tuples, lists, dicts, booleans, + and ``None``. + + This can be used for safely evaluating strings containing Python expressions + from untrusted sources without the need to parse the values oneself. + + +.. function:: get_docstring(node, clean=True): + + Return the docstring of the given *node* (which must be a + :class:`FunctionDef`, :class:`ClassDef` or :class:`Module` node), or ``None`` + if it has no docstring. If *clean* is true, clean up the docstring's + indentation with :func:`inspect.cleandoc`. + + +.. function:: fix_missing_locations(node) + + When you compile a node tree with :func:`compile`, the compiler expects + :attr:`lineno` and :attr:`col_offset` attributes for every node that supports + them. This is rather tedious to fill in for generated nodes, so this helper + adds these attributes recursively where not already set, by setting them to + the values of the parent node. It works recursively starting at *node*. + + +.. function:: increment_lineno(node, n=1) + + Increment the line number of each node in the tree starting at *node* by *n*. + This is useful to "move code" to a different location in a file. + + +.. function:: copy_location(new_node, old_node) + + Copy source location (:attr:`lineno` and :attr:`col_offset`) from *old_node* + to *new_node* if possible, and return *new_node*. + + +.. function:: iter_fields(node) + + Yield a tuple of ``(fieldname, value)`` for each field in ``node._fields`` + that is present on *node*. + + +.. function:: iter_child_nodes(node) + + Yield all direct child nodes of *node*, that is, all fields that are nodes + and all items of fields that are lists of nodes. + + +.. function:: walk(node) + + Recursively yield all child nodes of *node*, in no specified order. This is + useful if you only want to modify nodes in place and don't care about the + context. + + +.. class:: NodeVisitor() + + A node visitor base class that walks the abstract syntax tree and calls a + visitor function for every node found. This function may return a value + which is forwarded by the `visit` method. + + This class is meant to be subclassed, with the subclass adding visitor + methods. + + .. method:: visit(node) + + Visit a node. The default implementation calls the method called + :samp:`self.visit_{classname}` where *classname* is the name of the node + class, or :meth:`generic_visit` if that method doesn't exist. + + .. method:: generic_visit(node) + + This visitor calls :meth:`visit` on all children of the node. + + Note that child nodes of nodes that have a custom visitor method won't be + visited unless the visitor calls :meth:`generic_visit` or visits them + itself. + + Don't use the :class:`NodeVisitor` if you want to apply changes to nodes + during traversal. For this a special visitor exists + (:class:`NodeTransformer`) that allows modifications. + + +.. class:: NodeTransformer() + + A :class:`NodeVisitor` subclass that walks the abstract syntax tree and + allows modification of nodes. + + The `NodeTransformer` will walk the AST and use the return value of the + visitor methods to replace or remove the old node. If the return value of + the visitor method is ``None``, the node will be removed from its location, + otherwise it is replaced with the return value. The return value may be the + original node in which case no replacement takes place. + + Here is an example transformer that rewrites all occurrences of name lookups + (``foo``) to ``data['foo']``:: + + class RewriteName(NodeTransformer): + + def visit_Name(self, node): + return copy_location(Subscript( + value=Name(id='data', ctx=Load()), + slice=Index(value=Str(s=node.id)), + ctx=node.ctx + ), node) + + Keep in mind that if the node you're operating on has child nodes you must + either transform the child nodes yourself or call the :meth:`generic_visit` + method for the node first. + + For nodes that were part of a collection of statements (that applies to all + statement nodes), the visitor may also return a list of nodes rather than + just a single node. + + Usually you use the transformer like this:: + + node = YourTransformer().visit(node) + + +.. function:: dump(node, annotate_fields=True, include_attributes=False) + + Return a formatted dump of the tree in *node*. This is mainly useful for + debugging purposes. The returned string will show the names and the values + for fields. This makes the code impossible to evaluate, so if evaluation is + wanted *annotate_fields* must be set to False. Attributes such as line + numbers and column offsets are dumped by default. If this is wanted, + *include_attributes* can be set to ``True``. |