diff options
| author | Toshio Kuratomi <a.badger@gmail.com> | 2020-07-17 13:07:35 -0700 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2020-07-17 13:07:35 -0700 |
| commit | 9dda393d7036c86e69dd1a4dfbe0f72bf5f9bc5b (patch) | |
| tree | 2fa40aa168c2be4d6ee92dfcf09c5391f482e3ee | |
| parent | ca5197f784fdcb104c823476e2353e0b6af48029 (diff) | |
| download | ansible-9dda393d7036c86e69dd1a4dfbe0f72bf5f9bc5b.tar.gz | |
Collections docs generation (#59761)
* Build documentation for Ansible-2.10 (formerly known as ACD).
Builds plugin docs from collections whose source is on galaxy
The new command downloads collections from galaxy, then finds the
plugins inside of them to get the documentation for those plugins.
* Update the python syntax checks
* docs builds can now require python 3.6+.
* Move plugin formatter code out to an external tool, antsibull-docs.
Collection owners want to be able to extract docs for their own
websites as well.
* The jinja2 filters, tests, and other support code have moved to antsibull
* Remove document_plugins as that has now been integrated into antsibull-docs
* Cleanup and bugfix to other build script code:
* The Commands class needed to have its metaclass set for abstractmethod
to work correctly
* Fix lint issues in some command plugins
* Add the docs/docsite/rst/collections to .gitignore as
everything in that directory will be generated so we don't want any of
it saved in the git repository
* gitignore the build dir and remove edit docs link on module pages
* Add docs/rst/collections as a directory to remove on make clean
* Split the collections docs from the main docs
* remove version and edit on github
* remove version banner for just collections
* clarify examples need collection keyword defined
* Remove references to plugin documentation locations that no longer exist.
* Perhaps the pages in plugins/*.rst should be deprecated
altogether and their content moved?
* If not, perhaps we want to rephrase and link into the collection
documentation?
* Or perhaps we want to link to the plugins which are present in
collections/ansible/builtin?
* Remove PYTHONPATH from the build-ansible calls
One of the design goals of the build-ansible.py script was for it to
automatically set its library path to include the checkout of ansible
and the library of code to implement itself. Because it automatically
includes the checkout of ansible, we don't need to set PYTHONPATH in
the Makefile any longer.
* Create a command to only build ansible-base plugin docs
* When building docs for devel, only build the ansible-base docs for
now. This is because antsibull needs support for building a "devel
tree" of docs. This can be changed once that is implemented
* When building docs for the sanity tests, only build the ansible-base
plugin docs for now. Those are the docs which are in this repo so
that seems appropriate for now.
38 files changed, 286 insertions, 1641 deletions
diff --git a/.gitignore b/.gitignore index cdd3941dee..5590928817 100644 --- a/.gitignore +++ b/.gitignore @@ -37,6 +37,8 @@ docs/docsite/rst/cli/ansible.rst docs/docsite/rst/dev_guide/collections_galaxy_meta.rst docs/docsite/rst/dev_guide/testing/sanity/index.rst.new docs/docsite/rst/modules/*.rst +docs/docsite/rst/collections/*.rst +docs/docsite/rst/collections/*/*.rst docs/docsite/rst/playbooks_directives.rst docs/docsite/rst/plugins_by_category.rst docs/docsite/rst/plugins/*/*.rst @@ -277,7 +277,7 @@ linkcheckdocs: .PHONY: generate_rst generate_rst: lib/ansible/cli/*.py mkdir -p ./docs/man/man1/ ; \ - PYTHONPATH=./lib $(GENERATE_CLI) --template-file=docs/templates/man.j2 --output-dir=docs/man/man1/ --output-format man lib/ansible/cli/*.py + $(GENERATE_CLI) --template-file=docs/templates/man.j2 --output-dir=docs/man/man1/ --output-format man lib/ansible/cli/*.py docs: generate_rst diff --git a/docs/docsite/Makefile b/docs/docsite/Makefile index 3b0b512b24..8993da1e3e 100644 --- a/docs/docsite/Makefile +++ b/docs/docsite/Makefile @@ -1,6 +1,6 @@ OS := $(shell uname -s) SITELIB = $(shell python -c "from distutils.sysconfig import get_python_lib; print get_python_lib()"): -PLUGIN_FORMATTER=../../hacking/build-ansible.py document-plugins +PLUGIN_FORMATTER=../../hacking/build-ansible.py docs-build TESTING_FORMATTER=../bin/testing_formatter.sh KEYWORD_DUMPER=../../hacking/build-ansible.py document-keywords CONFIG_DUMPER=../../hacking/build-ansible.py document-config @@ -12,23 +12,35 @@ else CPUS ?= $(shell nproc) endif -# Sets the build output directory if it's not already specified +# Sets the build output directory for the main docsite if it's not already specified ifndef BUILDDIR BUILDDIR = _build endif -MODULE_ARGS= +# Backwards compat for separate VARS +PLUGIN_ARGS= ifdef MODULES - MODULE_ARGS = -l $(MODULES) +ifndef PLUGINS + PLUGIN_ARGS = -l $(MODULES) +else + PLUGIN_ARGS = -l $(MODULES),$(PLUGINS) endif - -PLUGIN_ARGS= +else ifdef PLUGINS PLUGIN_ARGS = -l $(PLUGINS) endif +endif + DOC_PLUGINS ?= become cache callback cliconf connection httpapi inventory lookup netconf shell strategy vars +PYTHON=python +# fetch version from project release.py as single source-of-truth +VERSION := $(shell $(PYTHON) ../../packaging/release/versionhelper/version_helper.py --raw || echo error) +ifeq ($(findstring error,$(VERSION)), error) +$(error "version_helper failed") +endif + assertrst: ifndef rst $(error specify document or pattern with rst=somefile.rst) @@ -38,17 +50,24 @@ all: docs docs: htmldocs -generate_rst: collections_meta config cli keywords modules plugins testing +generate_rst: collections_meta config cli keywords plugins testing +base_generate_rst: collections_meta config cli keywords base_plugins testing htmldocs: generate_rst CPUS=$(CPUS) $(MAKE) -f Makefile.sphinx html +base_htmldocs: base_generate_rst + CPUS=$(CPUS) $(MAKE) -f Makefile.sphinx html + singlehtmldocs: generate_rst CPUS=$(CPUS) $(MAKE) -f Makefile.sphinx singlehtml +base_singlehtmldocs: base_generate_rst + CPUS=$(CPUS) $(MAKE) -f Makefile.sphinx singlehtml + linkcheckdocs: generate_rst CPUS=$(CPUS) $(MAKE) -f Makefile.sphinx linkcheck - + webdocs: docs #TODO: leaving htmlout removal for those having older versions, should eventually be removed also @@ -58,7 +77,7 @@ clean: -rm -rf $(BUILDDIR)/html -rm -rf htmlout -rm -rf module_docs - -rm -rf _build + -rm -rf $(BUILDDIR) -rm -f .buildinfo -rm -f objects.inv -rm -rf *.doctrees @@ -70,43 +89,44 @@ clean: find . -type f \( -name "*~" -or -name "#*" \) -delete find . -type f \( -name "*.swp" \) -delete @echo "Cleaning up generated rst" - rm -f rst/modules/*_by_category.rst - rm -f rst/modules/list_of_*.rst - rm -f rst/modules/*_maintained.rst - rm -f rst/modules/*_module.rst - rm -f rst/modules/*_plugin.rst rm -f rst/playbooks_directives.rst - rm -f rst/plugins/*/*.rst rm -f rst/reference_appendices/config.rst rm -f rst/reference_appendices/playbooks_keywords.rst rm -f rst/dev_guide/collections_galaxy_meta.rst rm -f rst/cli/*.rst + rm -rf rst/collections/* + @echo "Cleaning up legacy generated rst locations" + rm -rf rst/modules + rm -f rst/plugins/*/*.rst .PHONY: docs clean collections_meta: ../templates/collections_galaxy_meta.rst.j2 - PYTHONPATH=../../lib $(COLLECTION_DUMPER) --template-file=../templates/collections_galaxy_meta.rst.j2 --output-dir=rst/dev_guide/ ../../lib/ansible/galaxy/data/collections_galaxy_meta.yml + $(COLLECTION_DUMPER) --template-file=../templates/collections_galaxy_meta.rst.j2 --output-dir=rst/dev_guide/ ../../lib/ansible/galaxy/data/collections_galaxy_meta.yml # TODO: make generate_man output dir cli option cli: mkdir -p rst/cli - PYTHONPATH=../../lib $(GENERATE_CLI) --template-file=../templates/cli_rst.j2 --output-dir=rst/cli/ --output-format rst ../../lib/ansible/cli/*.py + $(GENERATE_CLI) --template-file=../templates/cli_rst.j2 --output-dir=rst/cli/ --output-format rst ../../lib/ansible/cli/*.py keywords: ../templates/playbooks_keywords.rst.j2 - PYTHONPATH=../../lib $(KEYWORD_DUMPER) --template-dir=../templates --output-dir=rst/reference_appendices/ ./keyword_desc.yml + $(KEYWORD_DUMPER) --template-dir=../templates --output-dir=rst/reference_appendices/ ./keyword_desc.yml config: ../templates/config.rst.j2 - PYTHONPATH=../../lib $(CONFIG_DUMPER) --template-file=../templates/config.rst.j2 --output-dir=rst/reference_appendices/ ../../lib/ansible/config/base.yml - -modules: ../templates/plugin.rst.j2 - PYTHONPATH=../../lib $(PLUGIN_FORMATTER) -t rst --template-dir=../templates --module-dir=../../lib/ansible/modules -o rst/modules/ $(MODULE_ARGS) - -plugins: ../templates/plugin.rst.j2 - @echo "looping over doc plugins" - for plugin in $(DOC_PLUGINS); \ - do \ - PYTHONPATH=../../lib $(PLUGIN_FORMATTER) -t rst --plugin-type $$plugin --template-dir=../templates --module-dir=../../lib/ansible/plugins/$$plugin -o rst $(PLUGIN_ARGS); \ - done + $(CONFIG_DUMPER) --template-file=../templates/config.rst.j2 --output-dir=rst/reference_appendices/ ../../lib/ansible/config/base.yml + +# For now, if we're building on devel, just build base docs. In the future we'll want to build docs that +# are the latest versions on galaxy (using a different antsibull-docs subcommand) +plugins: + if expr "$(VERSION)" : '.*[.]dev[0-9]\+$$' &> /dev/null; then \ + $(PLUGIN_FORMATTER) base -o rst $(PLUGIN_ARGS);\ + else \ + $(PLUGIN_FORMATTER) full -o rst $(PLUGIN_ARGS);\ + fi + +# This only builds the plugin docs included with ansible-base +base_plugins: + $(PLUGIN_FORMATTER) base -o rst $(PLUGIN_ARGS);\ testing: $(TESTING_FORMATTER) diff --git a/docs/docsite/_themes/sphinx_rtd_theme/ansible_banner.html b/docs/docsite/_themes/sphinx_rtd_theme/ansible_banner.html index 7ca2bd4a0f..ba889e7186 100644 --- a/docs/docsite/_themes/sphinx_rtd_theme/ansible_banner.html +++ b/docs/docsite/_themes/sphinx_rtd_theme/ansible_banner.html @@ -10,26 +10,27 @@ element.appendChild(para); document.write('</div>'); } - - // Create a banner if we're not the latest version - current_url = window.location.href; - if ((current_url.search("latest") > -1) || (current_url.search("/{{ latest_version }}/") > -1)) { - // no banner for latest release - } else if (current_url.search("devel") > -1) { - document.write('<div id="banner_id" class="admonition caution">'); - para = document.createElement('p'); - banner_text=document.createTextNode("You are reading the *devel* version of the Ansible documentation - most module documentation is currently missing as the modules have moved to collections. Until docs catches up to this change, use the version selection to the left if you want module documentation or the latest stable release version. The *devel* version is not guaranteed stable."); - para.appendChild(banner_text); - element = document.getElementById('banner_id'); - element.appendChild(para); - document.write('</div>'); - } else { - document.write('<div id="banner_id" class="admonition caution">'); - para = document.createElement('p'); - banner_text=document.createTextNode("You are reading an older version of the Ansible documentation. Use the version selection to the left if you want the latest stable released version."); - para.appendChild(banner_text); - element = document.getElementById('banner_id'); - element.appendChild(para); - document.write('</div>'); - } + {% if (not READTHEDOCS) and (available_versions is defined) %} + // Create a banner if we're not the latest version + current_url = window.location.href; + if ((current_url.search("latest") > -1) || (current_url.search("/{{ latest_version }}/") > -1)) { + // no banner for latest release + } else if (current_url.search("devel") > -1) { + document.write('<div id="banner_id" class="admonition caution">'); + para = document.createElement('p'); + banner_text=document.createTextNode("You are reading the *devel* version of the Ansible documentation - this version is not guaranteed stable. Use the version selection to the left if you want the latest stable released version."); + para.appendChild(banner_text); + element = document.getElementById('banner_id'); + element.appendChild(para); + document.write('</div>'); + } else { + document.write('<div id="banner_id" class="admonition caution">'); + para = document.createElement('p'); + banner_text=document.createTextNode("You are reading an older version of the Ansible documentation. Use the version selection to the left if you want the latest stable released version."); + para.appendChild(banner_text); + element = document.getElementById('banner_id'); + element.appendChild(para); + document.write('</div>'); + } + {% endif %} </script> diff --git a/docs/docsite/_themes/sphinx_rtd_theme/ansible_versions.html b/docs/docsite/_themes/sphinx_rtd_theme/ansible_versions.html index 06a0271eda..3d3d7fc682 100644 --- a/docs/docsite/_themes/sphinx_rtd_theme/ansible_versions.html +++ b/docs/docsite/_themes/sphinx_rtd_theme/ansible_versions.html @@ -1,7 +1,7 @@ <!--- Based on https://github.com/rtfd/sphinx_rtd_theme/pull/438/files --> {# Creates dropdown version selection in the top-left navigation. #} <div class="version"> - {% if not READTHEDOCS %} + {% if (not READTHEDOCS) and (available_versions is defined) %} <div class="version-dropdown"> <select class="version-list" id="version-list" onchange="javascript:location.href = this.value;"> <script> x = document.getElementById("version-list"); </script> diff --git a/docs/docsite/collection-plugins.yml b/docs/docsite/collection-plugins.yml new file mode 100644 index 0000000000..499274b40b --- /dev/null +++ b/docs/docsite/collection-plugins.yml @@ -0,0 +1,17 @@ +# We also need an example of modules hosted in Automation Hub +# We'll likely move to data hosted in botmeta instead of a standalone file but +# we'll need all of these same details. +module: + purefa_user: + source: 'https://galaxy.ansible.com/' + fqcn: 'purestorage.flasharray' + purefa_vg: + source: 'https://galaxy.ansible.com/' + fqcn: 'purestorage.flasharray' + gcp_compute_firewall_info: + source: 'https://galaxy.ansible.com/' + fqcn: 'google.cloud' +module_utils: + purefa: + source: 'https://galaxy.ansible.com/' + fqcn: 'purestorage.flasharray' diff --git a/docs/docsite/requirements.txt b/docs/docsite/requirements.txt index 2672d01095..1290ac1df0 100644 --- a/docs/docsite/requirements.txt +++ b/docs/docsite/requirements.txt @@ -6,3 +6,4 @@ sphinx==2.1.2 sphinx-notfound-page Pygments >= 2.4.0 straight.plugin # Needed for hacking/build-ansible.py which is the backend build script +antsibull >= 0.15.0 diff --git a/docs/docsite/rst/conf.py b/docs/docsite/rst/conf.py index f26fdf37db..11ebe09690 100644 --- a/docs/docsite/rst/conf.py +++ b/docs/docsite/rst/conf.py @@ -280,6 +280,10 @@ autoclass_content = 'both' intersphinx_mapping = {'python': ('https://docs.python.org/2/', (None, '../python2.inv')), 'python3': ('https://docs.python.org/3/', (None, '../python3.inv')), 'jinja2': ('http://jinja.palletsprojects.com/', (None, '../jinja2.inv')), + 'collections': ('https://docs.ansible.com/collections/', + (None, '../collections.inv', + 'http://docs.testing.ansible.com/collections/objects.inv', + '../_collections_build/html/objects.inv')), 'ansible_2_9': ('https://docs.ansible.com/ansible/2.9/', (None, '../ansible_2_9.inv')), 'ansible_2_8': ('https://docs.ansible.com/ansible/2.8/', (None, '../ansible_2_8.inv')), 'ansible_2_7': ('https://docs.ansible.com/ansible/2.7/', (None, '../ansible_2_7.inv')), diff --git a/docs/docsite/rst/index.rst b/docs/docsite/rst/index.rst index 0d2c5a6612..eea338480c 100644 --- a/docs/docsite/rst/index.rst +++ b/docs/docsite/rst/index.rst @@ -74,7 +74,7 @@ Ansible releases a new major release of Ansible approximately three to four time :maxdepth: 1 :caption: Reference & Appendices - ../modules/modules_by_category + collections/index reference_appendices/playbooks_keywords reference_appendices/common_return_values reference_appendices/config diff --git a/docs/docsite/rst/network/index.rst b/docs/docsite/rst/network/index.rst index 9ae7a863f3..257563910c 100644 --- a/docs/docsite/rst/network/index.rst +++ b/docs/docsite/rst/network/index.rst @@ -10,7 +10,7 @@ Ansible Network modules extend the benefits of simple, powerful, agentless autom If you're new to Ansible, or new to using Ansible for network management, start with :ref:`network_getting_started`. If you are already familiar with network automation with Ansible, see :ref:`network_advanced`. -For documentation on using a particular network module, consult the :ref:`list of all network modules<network_modules>`. Some network modules are maintained by the Ansible community - here's a list of :ref:`network modules maintained by the Ansible Network Team<network_supported>`. +For documentation on using a particular network module, consult the :ref:`list of all network modules<network_modules>`. Network modules for various hardware are supported by different teams including the hardware vendors themselves, volunteers from the Ansible community, and the Ansible Network Team. .. toctree:: :maxdepth: 3 diff --git a/docs/docsite/rst/plugins/become.rst b/docs/docsite/rst/plugins/become.rst index 7bc181d06e..c4bbb7e8c6 100644 --- a/docs/docsite/rst/plugins/become.rst +++ b/docs/docsite/rst/plugins/become.rst @@ -47,11 +47,6 @@ Plugin List You can use ``ansible-doc -t become -l`` to see the list of available plugins. Use ``ansible-doc -t become <plugin name>`` to see specific documentation and examples. -.. toctree:: :maxdepth: 1 - :glob: - - become/* - .. seealso:: :ref:`about_playbooks` diff --git a/docs/docsite/rst/plugins/cache.rst b/docs/docsite/rst/plugins/cache.rst index ce7e848743..a13c78db04 100644 --- a/docs/docsite/rst/plugins/cache.rst +++ b/docs/docsite/rst/plugins/cache.rst @@ -118,11 +118,6 @@ Plugin List You can use ``ansible-doc -t cache -l`` to see the list of available plugins. Use ``ansible-doc -t cache <plugin name>`` to see specific documentation and examples. -.. toctree:: :maxdepth: 1 - :glob: - - cache/* - .. seealso:: :ref:`action_plugins` diff --git a/docs/docsite/rst/plugins/callback.rst b/docs/docsite/rst/plugins/callback.rst index 3160eb120f..a8de3de986 100644 --- a/docs/docsite/rst/plugins/callback.rst +++ b/docs/docsite/rst/plugins/callback.rst @@ -79,12 +79,6 @@ Plugin list You can use ``ansible-doc -t callback -l`` to see the list of available plugins. Use ``ansible-doc -t callback <plugin name>`` to see specific documents and examples. -.. toctree:: :maxdepth: 1 - :glob: - - callback/* - - .. seealso:: :ref:`action_plugins` diff --git a/docs/docsite/rst/plugins/connection.rst b/docs/docsite/rst/plugins/connection.rst index 02d2ffb4c1..0417526ff2 100644 --- a/docs/docsite/rst/plugins/connection.rst +++ b/docs/docsite/rst/plugins/connection.rst @@ -58,12 +58,6 @@ You can use ``ansible-doc -t connection -l`` to see the list of available plugin Use ``ansible-doc -t connection <plugin name>`` to see detailed documentation and examples. -.. toctree:: :maxdepth: 1 - :glob: - - connection/* - - .. seealso:: :ref:`Working with Playbooks<working_with_playbooks>` diff --git a/docs/docsite/rst/plugins/inventory.rst b/docs/docsite/rst/plugins/inventory.rst index a652f8dd3d..eda99bf3e7 100644 --- a/docs/docsite/rst/plugins/inventory.rst +++ b/docs/docsite/rst/plugins/inventory.rst @@ -162,11 +162,6 @@ Plugin List You can use ``ansible-doc -t inventory -l`` to see the list of available plugins. Use ``ansible-doc -t inventory <plugin name>`` to see plugin-specific documentation and examples. -.. toctree:: :maxdepth: 1 - :glob: - - inventory/* - .. seealso:: :ref:`about_playbooks` diff --git a/docs/docsite/rst/plugins/lookup.rst b/docs/docsite/rst/plugins/lookup.rst index cb76fa26da..31183b15c4 100644 --- a/docs/docsite/rst/plugins/lookup.rst +++ b/docs/docsite/rst/plugins/lookup.rst @@ -138,11 +138,6 @@ Plugin list You can use ``ansible-doc -t lookup -l`` to see the list of available plugins. Use ``ansible-doc -t lookup <plugin name>`` to see specific documents and examples. -.. toctree:: :maxdepth: 1 - :glob: - - lookup/* - .. seealso:: :ref:`about_playbooks` diff --git a/docs/docsite/rst/plugins/shell.rst b/docs/docsite/rst/plugins/shell.rst index 7caff1c637..b0846323e0 100644 --- a/docs/docsite/rst/plugins/shell.rst +++ b/docs/docsite/rst/plugins/shell.rst @@ -33,11 +33,6 @@ In this case, you will also want to update the :ref:`ansible_shell_executable <a You can further control the settings for each plugin via other configuration options detailed in the plugin themselves (linked below). -.. toctree:: :maxdepth: 1 - :glob: - - shell/* - .. seealso:: :ref:`about_playbooks` diff --git a/docs/docsite/rst/plugins/strategy.rst b/docs/docsite/rst/plugins/strategy.rst index 5e7d6c99ac..e3623329a6 100644 --- a/docs/docsite/rst/plugins/strategy.rst +++ b/docs/docsite/rst/plugins/strategy.rst @@ -59,11 +59,6 @@ You can use ``ansible-doc -t strategy -l`` to see the list of available plugins. Use ``ansible-doc -t strategy <plugin name>`` to see plugin-specific specific documentation and examples. -.. toctree:: :maxdepth: 1 - :glob: - - strategy/* - .. seealso:: :ref:`about_playbooks` diff --git a/docs/docsite/rst/plugins/vars.rst b/docs/docsite/rst/plugins/vars.rst index d70706a3bd..c24bdb8116 100644 --- a/docs/docsite/rst/plugins/vars.rst +++ b/docs/docsite/rst/plugins/vars.rst @@ -57,11 +57,6 @@ You can use ``ansible-doc -t vars -l`` to see the list of available plugins. Use ``ansible-doc -t vars <plugin name>`` to see specific plugin-specific documentation and examples. -.. toctree:: :maxdepth: 1 - :glob: - - vars/* - .. seealso:: :ref:`action_plugins` diff --git a/docs/docsite/rst/user_guide/become.rst b/docs/docsite/rst/user_guide/become.rst index 11552dc4b4..2ab7cba9de 100644 --- a/docs/docsite/rst/user_guide/become.rst +++ b/docs/docsite/rst/user_guide/become.rst @@ -286,7 +286,7 @@ For more information, see `this systemd issue Become and network automation ============================= -As of version 2.6, Ansible supports ``become`` for privilege escalation (entering ``enable`` mode or privileged EXEC mode) on all :ref:`Ansible-maintained platforms<network_supported>` that support ``enable`` mode. Using ``become`` replaces the ``authorize`` and ``auth_pass`` options in a ``provider`` dictionary. +As of version 2.6, Ansible supports ``become`` for privilege escalation (entering ``enable`` mode or privileged EXEC mode) on all Ansible-maintained network platforms that support ``enable`` mode. Using ``become`` replaces the ``authorize`` and ``auth_pass`` options in a ``provider`` dictionary. You must set the connection type to either ``connection: network_cli`` or ``connection: httpapi`` to use ``become`` for privilege escalation on network devices. Check the :ref:`platform_options` and :ref:`network_modules` documentation for details. diff --git a/docs/docsite/rst/user_guide/collections_using.rst b/docs/docsite/rst/user_guide/collections_using.rst index cbd1962dfa..c3f8e6bb7b 100644 --- a/docs/docsite/rst/user_guide/collections_using.rst +++ b/docs/docsite/rst/user_guide/collections_using.rst @@ -5,7 +5,8 @@ Using collections ***************** -Collections are a distribution format for Ansible content that can include playbooks, roles, modules, and plugins. +Collections are a distribution format for Ansible content that can include playbooks, roles, modules, and plugins. As modules move from the core Ansible repository into collections, the module documentation will move to the `collections documentation page <https://docs.ansible.com/collections/>`_ + You can install and use collections through `Ansible Galaxy <https://galaxy.ansible.com>`_. * For details on how to *develop* collections see :ref:`developing_collections`. diff --git a/docs/docsite/rst/user_guide/modules.rst b/docs/docsite/rst/user_guide/modules.rst index 36bdc32303..70dac88459 100644 --- a/docs/docsite/rst/user_guide/modules.rst +++ b/docs/docsite/rst/user_guide/modules.rst @@ -7,9 +7,8 @@ Working With Modules :maxdepth: 1 modules_intro - ../reference_appendices/common_return_values modules_support - ../modules/modules_by_category + ../reference_appendices/common_return_values Ansible ships with a number of modules (called the 'module library') diff --git a/docs/templates/list_of_CATEGORY_modules.rst.j2 b/docs/templates/list_of_CATEGORY_modules.rst.j2 deleted file mode 100644 index 92422406de..0000000000 --- a/docs/templates/list_of_CATEGORY_modules.rst.j2 +++ /dev/null @@ -1,48 +0,0 @@ -{# avoids rST "isn't included in any toctree" errors for module docs #} -:orphan: - -{% if title %} -.. _@{ title.lower() + '_' + plugin_type + 's' }@: -{% else %} -.. _@{ plugin_type + 's' }@: -{% endif %} - -{% if title %} -@{ title }@ @{ plugin_type + 's' }@ -@{ '`' * title | length }@```````` -{% else %} -@{ plugin_type + 's' }@ -``````` -{% endif %} - -{% if blurb %} -@{ blurb }@ - -{% endif %} - -{% if category['_modules'] %} - -{% for module in category['_modules'] | sort %} - * :ref:`@{ module }@_@{ plugin_type }@`{% if module_info[module]['deprecated'] %} **(D)**{% endif%} -{% endfor %} -{% endif %} - -{% for name, info in subcategories.items() | sort %} - -.. _@{ name.lower() + '_' + title.lower() + '_' + plugin_type + 's' }@: - -@{ name.title() }@ -@{ '-' * name | length }@ - - - -{% for module in info['_modules'] | sort %} - * :ref:`@{ module }@_@{ plugin_type }@`{% if module_info[module]['deprecated'] %} **(D)**{% endif%} -{% endfor %} - -{% endfor %} - -.. note:: - - **(D)**: This marks a module as deprecated, which means a module is kept for backwards compatibility but usage is discouraged. - The module documentation details page may explain more about this rationale. - diff --git a/docs/templates/list_of_CATEGORY_plugins.rst.j2 b/docs/templates/list_of_CATEGORY_plugins.rst.j2 deleted file mode 100644 index 0f9b611fb1..0000000000 --- a/docs/templates/list_of_CATEGORY_plugins.rst.j2 +++ /dev/null @@ -1,36 +0,0 @@ -.. _@{ title.lower() + '_' + plugin_type + 's' }@: - -@{ title }@ @{ plugin_type }@ -@{ '`' * title | length }@```````` - -{% if blurb %} -@{ blurb }@ - -{% endif %} -.. toctree:: :maxdepth: 1 -{% if category['_modules'] %} - -{% for module in category['_modules'] | sort %} - @{ module }@{% if module_info[module]['deprecated'] %} **(D)**{% endif%}{% if module_info[module]['doc']['short_description'] %} -- @{ module_info[module]['doc']['short_description'] }@{% endif %} <plugins/@{ module_info[module]['primary_category'] }@/@{ module }@> -{% endfor %} -{% endif %} - -{% for name, info in subcategories.items() | sort %} - -.. _@{ name.lower() + '_' + title.lower() + '_' + plugin_type + 's' }@: - -@{ name.title() }@ -@{ '-' * name | length }@ - -.. toctree:: :maxdepth: 1 - -{% for module in info['_modules'] | sort %} - :ref:`@{ module }@_@{ plugin_type }@`{% if module_info[module]['deprecated'] %} **(D)**{% endif%} -- @{ module_info[module]['doc']['short_description'] }@ -{% endfor %} - -{% endfor %} - -.. note:: - - **(D)**: This marks a module as deprecated, which means a module is kept for backwards compatibility but usage is discouraged. - The module documentation details page may explain more about this rationale. - diff --git a/docs/templates/modules_by_support.rst.j2 b/docs/templates/modules_by_support.rst.j2 deleted file mode 100644 index eced773f3d..0000000000 --- a/docs/templates/modules_by_support.rst.j2 +++ /dev/null @@ -1,45 +0,0 @@ -.. _@{ slug }@: - -{# avoids rST "isn't included in any toctree" errors for module index docs #} -:orphan: - -**************************@{ '*' * maintainers | length }@ -Modules Maintained by the @{ maintainers }@ -**************************@{ '*' * maintainers | length }@ - -.. contents:: - :local: - -{% for category, data in subcategories.items() | sort %} - -{% if category.lower() %} -.. _@{ category.lower() + '_' + slug.lower() + '_categories' }@: -{% else %} -.. _@{ slug.lower() + '_categories' }@: -{% endif %} - -@{ category.title() }@ -@{ '=' * category | length }@ - -{% for name, info in data.items() | sort %} - -{% if name.lower() %} -.. _@{ name.lower() + '_' + category + '_' + slug.lower() + '_' + plugin_type + 's' }@: -{% else %} -.. _@{ slug.lower() + '_' + category }@: -{% endif %} - -@{ name.title() }@ -@{ '-' * name | length }@ - -{% for module in info['_modules'] | sort %} - * :ref:`@{ module }@_@{plugin_type}@`{% if module_info[module]['deprecated'] %} **(D)** {% endif%} -{% endfor %} - -{% endfor %} - -{% endfor %} - -.. note:: - - **(D)**: This marks a module as deprecated, which means a module is kept for backwards compatibility but usage is discouraged. - The module documentation details page may explain more about this rationale. diff --git a/docs/templates/plugin.rst.j2 b/docs/templates/plugin.rst.j2 deleted file mode 100644 index f61d841b9a..0000000000 --- a/docs/templates/plugin.rst.j2 +++ /dev/null @@ -1,442 +0,0 @@ -:source: @{ source }@ - -{# avoids rST "isn't included in any toctree" errors for module docs #} -{% if plugin_type == 'module' %} -:orphan: -{% endif %} - -.. _@{ module }@_@{ plugin_type }@: -{% for alias in aliases %} -.. _@{ alias }@_@{ plugin_type }@: -{% endfor %} - -{% if short_description %} -{% set title = module + ' -- ' + short_description | rst_ify %} -{% else %} -{% set title = module %} -{% endif %} - -@{ title }@ -@{ '+' * title|length }@ - -{% if version_added is defined and version_added != '' -%} -.. versionadded:: @{ version_added | default('') }@ -{% endif %} - -.. contents:: - :local: - :depth: 1 - -{# ------------------------------------------ - # - # Please note: this looks like a core dump - # but it isn't one. - # - --------------------------------------------#} -{% if deprecated is defined -%} - - -DEPRECATED ----------- -{# use unknown here? skip the fields? #} -:Removed in Ansible: version: @{ deprecated['removed_in'] | default('') | string | rst_ify }@ -:Why: @{ deprecated['why'] | default('') | rst_ify }@ -:Alternative: @{ deprecated['alternative'] | default('') | rst_ify }@ - - -{% endif %} - -Synopsis --------- -{% if description -%} - -{% for desc in description %} -- @{ desc | rst_ify }@ -{% endfor %} - -{% endif %} - -{% if aliases is defined -%} -Aliases: @{ ','.join(aliases) }@ -{% endif %} - -{% if requirements -%} - -Requirements ------------- -{% if plugin_type == 'module' %} -The below requirements are needed on the host that executes this @{ plugin_type }@. -{% else %} -The below requirements are needed on the local master node that executes this @{ plugin_type }@. -{% endif %} - -{% for req in requirements %} -- @{ req | rst_ify }@ -{% endfor %} - -{% endif %} - -{% if options -%} - -Parameters ----------- - -.. raw:: html - - <table border=0 cellpadding=0 class="documentation-table"> - {# Pre-compute the nesting depth to allocate columns -#} - @{ to_kludge_ns('maxdepth', 1) -}@ - {% for key, value in options|dictsort recursive -%} - @{ to_kludge_ns('maxdepth', [loop.depth, from_kludge_ns('maxdepth')] | max) -}@ - {% if value.suboptions -%} - {% if value.suboptions.items -%} - @{ loop(value.suboptions.items()) -}@ - {% elif value.suboptions[0].items -%} - @{ loop(value.suboptions[0].items()) -}@ - {% endif -%} - {% endif -%} - {% endfor -%} - {# Header of the documentation -#} - <tr> - <th colspan="@{ from_kludge_ns('maxdepth') }@">Parameter</th> - <th>Choices/<font color="blue">Defaults</font></th> - {% if plugin_type != 'module' %} - <th>Configuration</th> - {% endif %} - <th width="100%">Comments</th> - </tr> - {% for key, value in options|dictsort recursive %} - <tr> - {# indentation based on nesting level #} - {% for i in range(1, loop.depth) %} - <td class="elbow-placeholder"></td> - {% endfor %} - {# parameter name with required and/or introduced label #} - <td colspan="@{ from_kludge_ns('maxdepth') - loop.depth0 }@"> - <div class="ansibleOptionAnchor" id="parameter-{% for part in value.full_key %}@{ part }@{% if not loop.last %}/{% endif %}{% endfor %}"></div> - <b>@{ key }@</b> - <a class="ansibleOptionLink" href="#parameter-{% for part in value.full_key %}@{ part }@{% if not loop.last %}/{% endif %}{% endfor %}" title="Permalink to this option"></a> - <div style="font-size: small"> - <span style="color: purple">@{ value.type | documented_type }@</span> - {% if value.get('elements') %} / <span style="color: purple">elements=@{ value.elements | documented_type }@</span>{% endif %} - {% if value.get('required', False) %} / <span style="color: red">required</span>{% endif %} - </div> - {% if value.version_added %}<div style="font-style: italic; font-size: small; color: darkgreen">added in @{value.version_added}@</div>{% endif %} - </td> - {# default / choices #} - <td> - {# Turn boolean values in 'yes' and 'no' values #} - {% if value.default is sameas true %} - {% set _x = value.update({'default': 'yes'}) %} - {% elif value.default is sameas false %} - {% set _x = value.update({'default': 'no'}) %} - {% endif %} - {% if value.type == 'bool' %} - {% set _x = value.update({'choices': ['no', 'yes']}) %} - {% endif %} - {# Show possible choices and highlight details #} - {% if value.choices %} - <ul style="margin: 0; padding: 0"><b>Choices:</b> - {% for choice in value.choices %} - {# Turn boolean values in 'yes' and 'no' values #} - {% if choice is sameas true %} - {% set choice = 'yes' %} - {% elif choice is sameas false %} - {% set choice = 'no' %} - {% endif %} - {% if (value.default is not list and value.default == choice) or (value.default is list and choice in value.default) %} - <li><div style="color: blue"><b>@{ choice | escape }@</b> ←</div></li> - {% else %} - <li>@{ choice | escape }@</li> - {% endif %} - {% endfor %} - </ul> - {% endif %} - {# Show default value, when multiple choice or no choices #} - {% if value.default is defined and value.default not in value.choices %} - <b>Default:</b><br/><div style="color: blue">@{ value.default | tojson | escape }@</div> - {% endif %} - </td> - {# configuration #} - {% if plugin_type != 'module' %} - <td> - {% if 'ini' in value %} - <div> ini entries: - {% for ini in value.ini %} - <p>[@{ ini.section }@]<br>@{ ini.key }@ = @{ value.default | default('VALUE') }@</p> - {% endfor %} - </div> - {% endif %} - {% if 'env' in value %} - {% for env in value.env %} - <div>env:@{ env.name }@</div> - {% endfor %} - {% endif %} - {% if 'vars' in value %} - {% for myvar in value.vars %} - <div>var: @{ myvar.name }@</div> - {% endfor %} - {% endif %} - </td> - {% endif %} - {# description #} - <td> - {% for desc in value.description %} - <div>@{ desc | replace('\n', '\n ') | html_ify }@</div> - {% endfor %} - {% if 'aliases' in value and value.aliases %} - <div style="font-size: small; color: darkgreen"><br/>aliases: @{ value.aliases|join(', ') }@</div> - {% endif %} - </td> - </tr> - {% if value.suboptions %} - {% if value.suboptions.items %} - @{ loop(value.suboptions|dictsort) }@ - {% elif value.suboptions[0].items %} - @{ loop(value.suboptions[0]|dictsort) }@ - {% endif %} - {% endif %} - {% endfor %} - </table> - <br/> - -{% endif %} - -{% if notes -%} -Notes ------ - -.. note:: -{% for note in notes %} - - @{ note | rst_ify }@ -{% endfor %} - -{% endif %} - -{% if seealso -%} -See Also --------- - -.. seealso:: - -{% for item in seealso %} -{% if item.module is defined and item.description is defined %} - :ref:`@{ item.module }@_module` - @{ item.description | rst_ify }@ -{% elif item.module is defined %} - :ref:`@{ item.module }@_module` - The official documentation on the **@{ item.module }@** module. -{% elif item.name is defined and item.link is defined and item.description is defined %} - `@{ item.name }@ <@{ item.link }@>`_ - @{ item.description | rst_ify }@ -{% elif item.ref is defined and item.description is defined %} - :ref:`@{ item.ref }@` - @{ item.description | rst_ify }@ -{% endif %} -{% endfor %} - -{% endif %} - -{% if examples or plainexamples -%} - -Examples --------- - -.. code-block:: yaml+jinja - -{% for example in examples %} -{% if example['description'] %}@{ example['description'] | indent(4, True) }@{% endif %} -@{ example['code'] | escape | indent(4, True) }@ -{% endfor %} -{% if plainexamples %}@{ plainexamples | indent(4, True) }@{% endif %} - -{% endif %} - -{% if not returnfacts and returndocs and returndocs.ansible_facts is defined %} -{% set returnfacts = returndocs.ansible_facts.contains %} -{% set _x = returndocs.pop('ansible_facts', None) %} -{% endif %} - -{% if returnfacts -%} - -Returned Facts --------------- -Facts returned by this module are added/updated in the ``hostvars`` host facts and can be referenced by name just like any other host fact. They do not need to be registered in order to use them. - -.. raw:: html - - <table border=0 cellpadding=0 class="documentation-table"> - {# Pre-compute the nesting depth to allocate columns #} - @{ to_kludge_ns('maxdepth', 1) -}@ - {% for key, value in returnfacts|dictsort recursive %} - @{ to_kludge_ns('maxdepth', [loop.depth, from_kludge_ns('maxdepth')] | max) -}@ - {% if value.contains -%} - {% if value.contains.items -%} - @{ loop(value.contains.items()) -}@ - {% elif value.contains[0].items -%} - @{ loop(value.contains[0].items()) -}@ - {% endif -%} - {% endif -%} - {% endfor -%} - <tr> - <th colspan="@{ from_kludge_ns('maxdepth') }@">Fact</th> - <th>Returned</th> - <th width="100%">Description</th> - </tr> - {% for key, value in returnfacts|dictsort recursive %} - <tr> - {% for i in range(1, loop.depth) %} - <td class="elbow-placeholder"></td> - {% endfor %} - <td colspan="@{ from_kludge_ns('maxdepth') - loop.depth0 }@" colspan="@{ from_kludge_ns('maxdepth') - loop.depth0 }@"> - <div class="ansibleOptionAnchor" id="return-{% for part in value.full_key %}@{ part }@{% if not loop.last %}/{% endif %}{% endfor %}"></div> - <b>@{ key }@</b> - <a class="ansibleOptionLink" href="#return-{% for part in value.full_key %}@{ part }@{% if not loop.last %}/{% endif %}{% endfor %}" title="Permalink to this fact"></a> - <div style="font-size: small"> - <span style="color: purple">@{ value.type | documented_type }@</span> - {% if value.elements %} / <span style="color: purple">elements=@{ value.elements | documented_type }@</span>{% endif %} - </div> - {% if value.version_added %}<div style="font-style: italic; font-size: small; color: darkgreen">added in @{value.version_added}@</div>{% endif %} - </td> - <td>@{ value.returned | html_ify }@</td> - <td> - {% if value.description is string %} - <div>@{ value.description | html_ify }@ - </div> - {% else %} - {% for desc in value.description %} - <div>@{ desc | html_ify }@ - </div> - {% endfor %} - {% endif %} - <br/> - {% if value.sample is defined and value.sample %} - <div style="font-size: smaller"><b>Sample:</b></div> - {# TODO: The sample should be escaped, using | escape or | htmlify, but both mess things up beyond repair with dicts #} - <div style="font-size: smaller; color: blue; word-wrap: break-word; word-break: break-all;">@{ value.sample | replace('\n', '\n ') | html_ify }@</div> - {% endif %} - </td> - </tr> - {# --------------------------------------------------------- - # sadly we cannot blindly iterate through the child dicts, - # since in some documentations, - # lists are used instead of dicts. This handles both types - # ---------------------------------------------------------#} - {% if value.contains %} - {% if value.contains.items %} - @{ loop(value.contains|dictsort) }@ - {% elif value.contains[0].items %} - @{ loop(value.contains[0]|dictsort) }@ - {% endif %} - {% endif %} - {% endfor %} - </table> - <br/><br/> - -{% endif %} - -{% if returndocs -%} - -Return Values -------------- -Common return values are documented :ref:`here <common_return_values>`, the following are the fields unique to this @{ plugin_type }@: - -.. raw:: html - - <table border=0 cellpadding=0 class="documentation-table"> - @{ to_kludge_ns('maxdepth', 1) -}@ - {% for key, value in returndocs|dictsort recursive -%} - @{ to_kludge_ns('maxdepth', [loop.depth, from_kludge_ns('maxdepth')] | max) -}@ - {% if value.contains -%} - {% if value.contains.items -%} - @{ loop(value.contains.items()) -}@ - {% elif value.contains[0].items -%} - @{ loop(value.contains[0].items()) -}@ - {% endif -%} - {% endif -%} - {% endfor -%} - <tr> - <th colspan="@{ from_kludge_ns('maxdepth') }@">Key</th> - <th>Returned</th> - <th width="100%">Description</th> - </tr> - {% for key, value in returndocs|dictsort recursive %} - <tr> - {% for i in range(1, loop.depth) %} - <td class="elbow-placeholder"> </td> - {% endfor %} - <td colspan="@{ from_kludge_ns('maxdepth') - loop.depth0 }@"> - <div class="ansibleOptionAnchor" id="return-{% for part in value.full_key %}@{ part }@{% if not loop.last %}/{% endif %}{% endfor %}"></div> - <b>@{ key }@</b> - <a class="ansibleOptionLink" href="#return-{% for part in value.full_key %}@{ part }@{% if not loop.last %}/{% endif %}{% endfor %}" title="Permalink to this return value"></a> - <div style="font-size: small"> - <span style="color: purple">@{ value.type | documented_type }@</span> - {% if value.elements %} / <span style="color: purple">elements=@{ value.elements | documented_type }@</span>{% endif %} - </div> - {% if value.version_added %}<div style="font-style: italic; font-size: small; color: darkgreen">added in @{value.version_added}@</div>{% endif %} - </td> - <td>@{ value.returned | html_ify }@</td> - <td> - {% if value.description is string %} - <div>@{ value.description | html_ify |indent(4) | trim}@</div> - {% else %} - {% for desc in value.description %} - <div>@{ desc | html_ify |indent(4) | trim}@</div> - {% endfor %} - {% endif %} - <br/> - {% if value.sample is defined and value.sample %} - <div style="font-size: smaller"><b>Sample:</b></div> - {# TODO: The sample should be escaped, using |escape or |htmlify, but both mess things up beyond repair with dicts #} - <div style="font-size: smaller; color: blue; word-wrap: break-word; word-break: break-all;">@{ value.sample | replace('\n', '\n ') | html_ify }@</div> - {% endif %} - </td> - </tr> - {# --------------------------------------------------------- - # sadly we cannot blindly iterate through the child dicts, - # since in some documentations, - # lists are used instead of dicts. This handles both types - # ---------------------------------------------------------#} - {% if value.contains %} - {% if value.contains.items %} - @{ loop(value.contains|dictsort) }@ - {% elif value.contains[0].items %} - @{ loop(value.contains[0]|dictsort) }@ - {% endif %} - {% endif %} - {% endfor %} - </table> - <br/><br/> - -{% endif %} - -Status ------- - -{% if deprecated %} - -- This @{ plugin_type }@ will be removed in version @{ deprecated['removed_in'] | default('') | string | rst_ify }@. *[deprecated]* -- For more information see `DEPRECATED`_. - -{% endif %} - -{% if author is defined -%} -Authors -~~~~~~~ - -{% for author_name in author %} -- @{ author_name }@ -{% endfor %} - -{% endif %} - -.. hint:: -{% if plugin_type == 'module' %} - If you notice any issues in this documentation, you can `edit this document <https://github.com/ansible/ansible/edit/devel/lib/ansible/modules/@{ source }@?description=%23%23%23%23%23%20SUMMARY%0A%3C!---%20Your%20description%20here%20--%3E%0A%0A%0A%23%23%23%23%23%20ISSUE%20TYPE%0A-%20Docs%20Pull%20Request%0A%0A%2Blabel:%20docsite_pr>`_ to improve it. -{% else %} - If you notice any issues in this documentation, you can `edit this document <https://github.com/ansible/ansible/edit/devel/lib/ansible/plugins/@{ plugin_type }@/@{ source }@?description=%23%23%23%23%23%20SUMMARY%0A%3C!---%20Your%20description%20here%20--%3E%0A%0A%0A%23%23%23%23%23%20ISSUE%20TYPE%0A-%20Docs%20Pull%20Request%0A%0A%2Blabel:%20docsite_pr>`_ to improve it. - - -.. hint:: - Configuration entries for each entry type have a low to high priority order. For example, a variable that is lower in the list will override a variable that is higher up. -{% endif %} diff --git a/docs/templates/plugin_deprecation_stub.rst.j2 b/docs/templates/plugin_deprecation_stub.rst.j2 deleted file mode 100644 index 5156f0bd96..0000000000 --- a/docs/templates/plugin_deprecation_stub.rst.j2 +++ /dev/null @@ -1,18 +0,0 @@ -:source: @{ source }@ - -{# avoids rST "isn't included in any toctree" errors for module docs #} -:orphan: - -.. _@{ module }@_@{ plugin_type }@_alias_@{ alias }@: - -{% if short_description %} -{% set title = alias + ' -- ' + short_description | rst_ify %} -{% else %} -{% set title = alias %} -{% endif %} - -@{ title }@ -@{ '+' * title|length }@ - -This is an alias for :ref:`@{ module }@ <@{ module }@_@{ plugin_type }@>`. -This name has been **deprecated**. Please update your tasks to use the new name ``@{ module }@`` instead. diff --git a/docs/templates/plugins_by_category.rst.j2 b/docs/templates/plugins_by_category.rst.j2 deleted file mode 100644 index 9febc09fad..0000000000 --- a/docs/templates/plugins_by_category.rst.j2 +++ /dev/null @@ -1,9 +0,0 @@ -Plugin Index -============ - - -.. toctree:: :maxdepth: 1 - -{% for name in categories %} - list_of_@{ name }@_plugins -{% endfor %} diff --git a/docs/templates/plugins_by_support.rst.j2 b/docs/templates/plugins_by_support.rst.j2 deleted file mode 100644 index fefe84a3b8..0000000000 --- a/docs/templates/plugins_by_support.rst.j2 +++ /dev/null @@ -1,15 +0,0 @@ -.. _@{ slug }@: - -Plugins Maintained by the @{ maintainers }@ -``````````````````````````@{ '`' * maintainers | length }@ - -.. toctree:: :maxdepth: 1 - -{% for module in modules | sort %} - @{ module }@{% if module_info[module]['deprecated'] %} **(D)**{% endif %} - @{ module_info[module]['doc']['short_description'] }@ <plugins/@{ module_info[module]['primary_category'] }@/@{ module }@> -{% endfor %} - -.. note:: - - **(D)**: This marks a plugin as deprecated, which means a plugin is kept for backwards compatibility but usage is discouraged. - The plugin documentation details page may explain more about this rationale. - diff --git a/hacking/build-ansible.py b/hacking/build-ansible.py index 8b151d9d23..8ebb88d33b 100755 --- a/hacking/build-ansible.py +++ b/hacking/build-ansible.py @@ -22,15 +22,25 @@ except ImportError: def build_lib_path(this_script=__file__): - """Return path to the common build library directory""" + """Return path to the common build library directory.""" hacking_dir = os.path.dirname(this_script) libdir = os.path.abspath(os.path.join(hacking_dir, 'build_library')) return libdir +def ansible_lib_path(this_script=__file__): + """Return path to the common build library directory.""" + hacking_dir = os.path.dirname(this_script) + libdir = os.path.abspath(os.path.join(hacking_dir, '..', 'lib')) + + return libdir + + +sys.path.insert(0, ansible_lib_path()) sys.path.insert(0, build_lib_path()) + from build_ansible import commands, errors @@ -47,14 +57,15 @@ def create_arg_parser(program_name): def main(): """ - Main entrypoint of the script + Start our run. "It all starts here" """ subcommands = load('build_ansible.command_plugins', subclasses=commands.Command) arg_parser = create_arg_parser(os.path.basename(sys.argv[0])) - arg_parser.add_argument('--debug', dest='debug', required=False, default=False, action='store_true', + arg_parser.add_argument('--debug', dest='debug', required=False, default=False, + action='store_true', help='Show tracebacks and other debugging information') subparsers = arg_parser.add_subparsers(title='Subcommands', dest='command', help='for help use build-ansible.py SUBCOMMANDS -h') diff --git a/hacking/build_library/build_ansible/command_plugins/collection_meta.py b/hacking/build_library/build_ansible/command_plugins/collection_meta.py index be58bd4742..08c20c940d 100644 --- a/hacking/build_library/build_ansible/command_plugins/collection_meta.py +++ b/hacking/build_library/build_ansible/command_plugins/collection_meta.py @@ -11,14 +11,13 @@ import os.path import pathlib import yaml -from jinja2 import Environment, FileSystemLoader from ansible.module_utils.six import string_types from ansible.module_utils._text import to_bytes +from antsibull.jinja2.environment import doc_environment # Pylint doesn't understand Python3 namespace modules. from ..change_detection import update_file_if_different # pylint: disable=relative-beyond-top-level from ..commands import Command # pylint: disable=relative-beyond-top-level -from ..jinja2.filters import documented_type, rst_ify # pylint: disable=relative-beyond-top-level DEFAULT_TEMPLATE_FILE = 'collections_galaxy_meta.rst.j2' @@ -61,12 +60,7 @@ class DocumentCollectionMeta(Command): normalize_options(options) - env = Environment(loader=FileSystemLoader(template_dir), - variable_start_string="@{", - variable_end_string="}@", - trim_blocks=True) - env.filters['documented_type'] = documented_type - env.filters['rst_ify'] = rst_ify + env = doc_environment(template_dir) template = env.get_template(template_file) output_name = os.path.join(output_dir, template_file.replace('.j2', '')) diff --git a/hacking/build_library/build_ansible/command_plugins/docs_build.py b/hacking/build_library/build_ansible/command_plugins/docs_build.py new file mode 100644 index 0000000000..342ad4cdab --- /dev/null +++ b/hacking/build_library/build_ansible/command_plugins/docs_build.py @@ -0,0 +1,164 @@ +# coding: utf-8 +# Copyright: (c) 2020, Ansible Project +# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt) + +# Make coding more python3-ish +from __future__ import absolute_import, division, print_function + +import glob +import os +import os.path +import pathlib +import shutil +from tempfile import TemporaryDirectory + +import yaml + +from ansible.release import __version__ as ansible_base__version__ + +# Pylint doesn't understand Python3 namespace modules. +# pylint: disable=relative-beyond-top-level +from ..commands import Command +# pylint: enable=relative-beyond-top-level + + +__metaclass__ = type + + +DEFAULT_TOP_DIR = pathlib.Path(__file__).parents[4] +DEFAULT_OUTPUT_DIR = pathlib.Path(__file__).parents[4] / 'docs/docsite' + + +# +# Subcommand base +# + +def generate_base_docs(args): + """Regenerate the documentation for all plugins listed in the plugin_to_collection_file.""" + # imports here so that they don't cause unnecessary deps for all of the plugins + from antsibull.cli import antsibull_docs + + with TemporaryDirectory() as tmp_dir: + # + # Construct a deps file with our version of ansible_base in it + # + modified_deps_file = os.path.join(tmp_dir, 'ansible.deps') + + # The _acd_version doesn't matter + deps_file_contents = {'_acd_version': ansible_base__version__, + '_ansible_base_version': ansible_base__version__} + + with open(modified_deps_file, 'w') as f: + f.write(yaml.dump(deps_file_contents)) + + # Generate the plugin rst + antsibull_docs.run(['antsibull-docs', 'stable', '--deps-file', modified_deps_file, + '--ansible-base-cache', str(args.top_dir), + '--dest-dir', args.output_dir]) + + # If we make this more than just a driver for antsibull: + # Run other rst generation + # Run sphinx build + + +# +# Subcommand full +# + +def generate_full_docs(args): + """Regenerate the documentation for all plugins listed in the plugin_to_collection_file.""" + # imports here so that they don't cause unnecessary deps for all of the plugins + import sh + from antsibull.cli import antsibull_docs + from packaging.version import Version + + ansible_base_ver = Version(ansible_base__version__) + ansible_base_major_ver = '{0}.{1}'.format(ansible_base_ver.major, ansible_base_ver.minor) + + with TemporaryDirectory() as tmp_dir: + sh.git(['clone', 'https://github.com/ansible-community/ansible-build-data'], _cwd=tmp_dir) + deps_files = glob.glob(os.path.join(tmp_dir, 'ansible-build-data', + ansible_base_major_ver, '*.deps')) + if not deps_files: + raise Exception('No deps files exist for version {0}'.format(ansible_base_major_ver)) + + # Find the latest version of the deps file for this version + latest = None + latest_ver = Version('0') + for filename in deps_files: + with open(filename, 'r') as f: + deps_data = yaml.safe_load(f.read()) + new_version = Version(deps_data['_ansible_base_version']) + if new_version > latest_ver: + latest_ver = new_version + latest = filename + + # Make a copy of the deps file so that we can set the ansible-base version to use + modified_deps_file = os.path.join(tmp_dir, 'ansible.deps') + shutil.copyfile(latest, modified_deps_file) + + # Put our version of ansible-base into the deps file + with open(modified_deps_file, 'r') as f: + deps_data = yaml.safe_load(f.read()) + + deps_data['_ansible_base_version'] = ansible_base__version__ + + with open(modified_deps_file, 'w') as f: + f.write(yaml.dump(deps_data)) + + # Generate the plugin rst + antsibull_docs.run(['antsibull-docs', 'stable', '--deps-file', modified_deps_file, + '--ansible-base-cache', str(args.top_dir), + '--dest-dir', args.output_dir]) + + # If we make this more than just a driver for antsibull: + # Run other rst generation + # Run sphinx build + + +class CollectionPluginDocs(Command): + name = 'docs-build' + _ACTION_HELP = """Action to perform. + full: Regenerate the rst for the full ansible website. + base: Regenerate the rst for plugins in ansible-base and then build the website. + named: Regenerate the rst for the named plugins and then build the website. + """ + + @classmethod + def init_parser(cls, add_parser): + parser = add_parser(cls.name, + description='Generate documentation for plugins in collections.' + ' Plugins in collections will have a stub file in the normal plugin' + ' documentation location that says the module is in a collection and' + ' point to generated plugin documentation under the collections/' + ' hierarchy.') + parser.add_argument('action', action='store', choices=('full', 'base', 'named'), + default='full', help=cls._ACTION_HELP) + parser.add_argument("-o", "--output-dir", action="store", dest="output_dir", + default=DEFAULT_OUTPUT_DIR, + help="Output directory for generated doc files") + parser.add_argument("-t", "--top-dir", action="store", dest="top_dir", + default=DEFAULT_TOP_DIR, + help="Toplevel directory of this ansible-base checkout or expanded" + " tarball.") + parser.add_argument("-l", "--limit-to-modules", '--limit-to', action="store", + dest="limit_to", default=None, + help="Limit building module documentation to comma-separated list of" + " plugins. Specify non-existing plugin name for no plugins.") + + @staticmethod + def main(args): + # normalize CLI args + + if not args.output_dir: + args.output_dir = os.path.abspath(str(DEFAULT_OUTPUT_DIR)) + + if args.action == 'full': + return generate_full_docs(args) + + if args.action == 'base': + return generate_base_docs(args) + # args.action == 'named' (Invalid actions are caught by argparse) + raise NotImplementedError('Building docs for specific files is not yet implemented') + + # return 0 diff --git a/hacking/build_library/build_ansible/command_plugins/plugin_formatter.py b/hacking/build_library/build_ansible/command_plugins/plugin_formatter.py deleted file mode 100644 index 475d86d97c..0000000000 --- a/hacking/build_library/build_ansible/command_plugins/plugin_formatter.py +++ /dev/null @@ -1,807 +0,0 @@ -# Copyright: (c) 2012, Jan-Piet Mens <jpmens () gmail.com> -# Copyright: (c) 2012-2014, Michael DeHaan <michael@ansible.com> and others -# Copyright: (c) 2017, Ansible Project - -# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt) - -from __future__ import absolute_import, division, print_function -__metaclass__ = type - - -import datetime -import glob -import json -import os -import re -import sys -import warnings -from collections import defaultdict -from copy import deepcopy -from distutils.version import LooseVersion -from functools import partial -from pprint import PrettyPrinter - -try: - from html import escape as html_escape -except ImportError: - # Python-3.2 or later - import cgi - - def html_escape(text, quote=True): - return cgi.escape(text, quote) - -import jinja2 -import yaml -from jinja2 import Environment, FileSystemLoader - -from ansible.errors import AnsibleError -from ansible.module_utils._text import to_bytes -from ansible.module_utils.common.collections import is_sequence -from ansible.module_utils.parsing.convert_bool import boolean -from ansible.module_utils.six import iteritems, string_types -from ansible.plugins.loader import fragment_loader -from ansible.utils import plugin_docs -from ansible.utils.display import Display - -# Pylint doesn't understand Python3 namespace modules. -from ..change_detection import update_file_if_different # pylint: disable=relative-beyond-top-level -from ..commands import Command # pylint: disable=relative-beyond-top-level -from ..jinja2.filters import do_max, documented_type, html_ify, rst_fmt, rst_ify, rst_xline # pylint: disable=relative-beyond-top-level - - -##################################################################################### -# constants and paths - -# if a module is added in a version of Ansible older than this, don't print the version added information -# in the module documentation because everyone is assumed to be running something newer than this already. -TOO_OLD_TO_BE_NOTABLE = 2.4 - -# Get parent directory of the directory this script lives in -MODULEDIR = os.path.abspath(os.path.join( - os.path.dirname(os.path.realpath(__file__)), os.pardir, 'lib', 'ansible', 'modules' -)) - -# The name of the DOCUMENTATION template -EXAMPLE_YAML = os.path.abspath(os.path.join( - os.path.dirname(os.path.realpath(__file__)), os.pardir, 'examples', 'DOCUMENTATION.yml' -)) - -DEPRECATED = b" (D)" - -pp = PrettyPrinter() -display = Display() - - -# kludge_ns gives us a kludgey way to set variables inside of loops that need to be visible outside -# the loop. We can get rid of this when we no longer need to build docs with less than Jinja-2.10 -# http://jinja.pocoo.org/docs/2.10/templates/#assignments -# With Jinja-2.10 we can use jinja2's namespace feature, restoring the namespace template portion -# of: fa5c0282a4816c4dd48e80b983ffc1e14506a1f5 -NS_MAP = {} - - -def to_kludge_ns(key, value): - NS_MAP[key] = value - return "" - - -def from_kludge_ns(key): - return NS_MAP[key] - - -test_list = partial(is_sequence, include_strings=False) - - -def normalize_options(value): - """Normalize boolean option value.""" - - if value.get('type') == 'bool' and 'default' in value: - try: - value['default'] = boolean(value['default'], strict=True) - except TypeError: - pass - return value - - -def write_data(text, output_dir, outputname, module=None): - ''' dumps module output to a file or the screen, as requested ''' - - if output_dir is not None: - if module: - outputname = outputname % module - - if not os.path.exists(output_dir): - os.makedirs(output_dir) - fname = os.path.join(output_dir, outputname) - fname = fname.replace(".py", "") - - try: - updated = update_file_if_different(fname, to_bytes(text)) - except Exception as e: - display.display("while rendering %s, an error occured: %s" % (module, e)) - raise - if updated: - display.display("rendering: %s" % module) - else: - print(text) - - -IS_STDOUT_TTY = sys.stdout.isatty() - - -def show_progress(progress): - '''Show a little process indicator.''' - if IS_STDOUT_TTY: - sys.stdout.write('\r%s\r' % ("-/|\\"[progress % 4])) - sys.stdout.flush() - - -def get_plugin_info(module_dir, limit_to=None, verbose=False): - ''' - Returns information about plugins and the categories that they belong to - - :arg module_dir: file system path to the top of the plugin directory - :kwarg limit_to: If given, this is a list of plugin names to - generate information for. All other plugins will be ignored. - :returns: Tuple of two dicts containing module_info, categories, and - aliases and a set listing deprecated modules: - - :module_info: mapping of module names to information about them. The fields of the dict are: - - :path: filesystem path to the module - :deprecated: boolean. True means the module is deprecated otherwise not. - :aliases: set of aliases to this module name - :metadata: The modules metadata (as recorded in the module) - :doc: The documentation structure for the module - :seealso: The list of dictionaries with references to related subjects - :examples: The module's examples - :returndocs: The module's returndocs - - :categories: maps category names to a dict. The dict contains at - least one key, '_modules' which contains a list of module names in - that category. Any other keys in the dict are subcategories with - the same structure. - - ''' - - categories = dict() - module_info = defaultdict(dict) - - # * windows powershell modules have documentation stubs in python docstring - # format (they are not executed) so skip the ps1 format files - # * One glob level for every module level that we're going to traverse - files = ( - glob.glob("%s/*.py" % module_dir) + - glob.glob("%s/*/*.py" % module_dir) + - glob.glob("%s/*/*/*.py" % module_dir) + - glob.glob("%s/*/*/*/*.py" % module_dir) - ) - - module_index = 0 - for module_path in files: - # Do not list __init__.py files - if module_path.endswith('__init__.py'): - continue - - # Do not list blacklisted modules - module = os.path.splitext(os.path.basename(module_path))[0] - if module in plugin_docs.BLACKLIST['MODULE'] or module == 'base': - continue - - # If requested, limit module documentation building only to passed-in - # modules. - if limit_to is not None and module.lower() not in limit_to: - continue - - deprecated = False - if module.startswith("_"): - if os.path.islink(module_path): - # Handle aliases - source = os.path.splitext(os.path.basename(os.path.realpath(module_path)))[0] - module = module.replace("_", "", 1) - if source.startswith("_"): - source = source.replace("_", "", 1) - aliases = module_info[source].get('aliases', set()) - aliases.add(module) - aliases_deprecated = module_info[source].get('aliases_deprecated', set()) - aliases_deprecated.add(module) - # In case we just created this via get()'s fallback - module_info[source]['aliases'] = aliases - module_info[source]['aliases_deprecated'] = aliases_deprecated - continue - else: - # Handle deprecations - module = module.replace("_", "", 1) - deprecated = True - - # - # Regular module to process - # - - module_index += 1 - show_progress(module_index) - - # use ansible core library to parse out doc metadata YAML and plaintext examples - doc, examples, returndocs, metadata = plugin_docs.get_docstring( - module_path, fragment_loader, verbose=verbose, collection_name='ansible.builtin') - - if metadata and 'removed' in metadata.get('status', []): - continue - - category = categories - - # Start at the second directory because we don't want the "vendor" - mod_path_only = os.path.dirname(module_path[len(module_dir):]) - - # Find the subcategory for each module - relative_dir = mod_path_only.split('/')[1] - sub_category = mod_path_only[len(relative_dir) + 2:] - - primary_category = '' - module_categories = [] - # build up the categories that this module belongs to - for new_cat in mod_path_only.split('/')[1:]: - if new_cat not in category: - category[new_cat] = dict() - category[new_cat]['_modules'] = [] - module_categories.append(new_cat) - category = category[new_cat] - - category['_modules'].append(module) - - # the category we will use in links (so list_of_all_plugins can point to plugins/action_plugins/*' - if module_categories: - primary_category = module_categories[0] - - if not doc: - display.error("*** ERROR: DOCUMENTATION section missing for %s. ***" % module_path) - continue - - if 'options' in doc and doc['options'] is None: - display.error("*** ERROR: DOCUMENTATION.options must be a dictionary/hash when used. ***") - pos = getattr(doc, "ansible_pos", None) - if pos is not None: - display.error("Module position: %s, %d, %d" % doc.ansible_pos) - doc['options'] = dict() - - for key, opt in doc.get('options', {}).items(): - doc['options'][key] = normalize_options(opt) - - # save all the information - module_info[module] = {'path': module_path, - 'source': os.path.relpath(module_path, module_dir), - 'deprecated': deprecated, - 'aliases': module_info[module].get('aliases', set()), - 'aliases_deprecated': module_info[module].get('aliases_deprecated', set()), - 'metadata': metadata, - 'doc': doc, - 'examples': examples, - 'returndocs': returndocs, - 'categories': module_categories, - 'primary_category': primary_category, - 'sub_category': sub_category, - } - - # keep module tests out of becoming module docs - if 'test' in categories: - del categories['test'] - - return module_info, categories - - -def jinja2_environment(template_dir, typ, plugin_type): - - env = Environment(loader=FileSystemLoader(template_dir), - variable_start_string="@{", - variable_end_string="}@", - trim_blocks=True) - env.globals['xline'] = rst_xline - - # Can be removed (and template switched to use namespace) when we no longer need to build - # with <Jinja-2.10 - env.globals['to_kludge_ns'] = to_kludge_ns - env.globals['from_kludge_ns'] = from_kludge_ns - if 'max' not in env.filters: - # Jinja < 2.10 - env.filters['max'] = do_max - - if 'tojson' not in env.filters: - # Jinja < 2.9 - env.filters['tojson'] = json.dumps - - templates = {} - if typ == 'rst': - env.filters['rst_ify'] = rst_ify - env.filters['html_ify'] = html_ify - env.filters['fmt'] = rst_fmt - env.filters['xline'] = rst_xline - env.filters['documented_type'] = documented_type - env.tests['list'] = test_list - templates['plugin'] = env.get_template('plugin.rst.j2') - templates['plugin_deprecation_stub'] = env.get_template('plugin_deprecation_stub.rst.j2') - - if plugin_type == 'module': - name = 'modules' - else: - name = 'plugins' - - templates['category_list'] = env.get_template('%s_by_category.rst.j2' % name) - templates['support_list'] = env.get_template('%s_by_support.rst.j2' % name) - templates['list_of_CATEGORY_modules'] = env.get_template('list_of_CATEGORY_%s.rst.j2' % name) - else: - raise Exception("Unsupported format type: %s" % typ) - - return templates - - -def process_version_added(version_added): - if not isinstance(version_added, string_types): - return version_added - if ':' not in version_added: - return version_added - # Strip tag from version_added. It suffices to do this here since - # this is only used for ansible-base, and there the only valid tag - # is `ansible.builtin:`. - return version_added[version_added.index(':') + 1:] - - -def too_old(added): - if not added: - return False - try: - added_tokens = str(added).split(".") - readded = added_tokens[0] + "." + added_tokens[1] - added_float = float(readded) - except ValueError as e: - warnings.warn("Could not parse %s: %s" % (added, str(e))) - return False - return added_float < TOO_OLD_TO_BE_NOTABLE - - -def process_options(module, options, full_key=None): - option_names = [] - if full_key is None: - full_key = [] - - if options: - for (k, v) in iteritems(options): - # Make sure that "full key" is contained - full_key_k = full_key + [k] - v['full_key'] = full_key_k - - # Error out if there's no description - if 'description' not in v: - raise AnsibleError("Missing required description for parameter '%s' in '%s' " % (k, module)) - - # Make sure description is a list of lines for later formatting - if isinstance(v['description'], string_types): - v['description'] = [v['description']] - elif not isinstance(v['description'], (list, tuple)): - raise AnsibleError("Invalid type for options['%s']['description']." - " Must be string or list of strings. Got %s" % - (k, type(v['description']))) - - # Error out if required isn't a boolean (people have been putting - # information on when something is required in here. Those need - # to go in the description instead). - required_value = v.get('required', False) - if not isinstance(required_value, bool): - raise AnsibleError("Invalid required value '%s' for parameter '%s' in '%s' (must be truthy)" % (required_value, k, module)) - - # Strip old version_added information for options - if 'version_added' in v: - v['version_added'] = process_version_added(v['version_added']) - if too_old(v['version_added']): - del v['version_added'] - - if 'suboptions' in v and v['suboptions']: - if isinstance(v['suboptions'], dict): - process_options(module, v['suboptions'], full_key=full_key_k) - elif isinstance(v['suboptions'][0], dict): - process_options(module, v['suboptions'][0], full_key=full_key_k) - - option_names.append(k) - - option_names.sort() - - return option_names - - -def process_returndocs(returndocs, full_key=None): - if full_key is None: - full_key = [] - - if returndocs: - for (k, v) in iteritems(returndocs): - # Make sure that "full key" is contained - full_key_k = full_key + [k] - v['full_key'] = full_key_k - - # Strip old version_added information for options - if 'version_added' in v: - v['version_added'] = process_version_added(v['version_added']) - if too_old(v['version_added']): - del v['version_added'] - - # Process suboptions - suboptions = v.get('contains') - if suboptions: - if isinstance(suboptions, dict): - process_returndocs(suboptions, full_key=full_key_k) - elif is_sequence(suboptions): - process_returndocs(suboptions[0], full_key=full_key_k) - - -def process_plugins(module_map, templates, outputname, output_dir, ansible_version, plugin_type): - for module_index, module in enumerate(module_map): - - show_progress(module_index) - - fname = module_map[module]['path'] - display.vvvvv(pp.pformat(('process_plugins info: ', module_map[module]))) - - # crash if module is missing documentation and not explicitly hidden from docs index - if module_map[module]['doc'] is None: - display.error("%s MISSING DOCUMENTATION" % (fname,)) - _doc = {plugin_type: module, - 'version_added': '2.4', - 'filename': fname} - module_map[module]['doc'] = _doc - # continue - - # Going to reference this heavily so make a short name to reference it by - doc = module_map[module]['doc'] - display.vvvvv(pp.pformat(('process_plugins doc: ', doc))) - - # add some defaults for plugins that dont have most of the info - doc['module'] = doc.get('module', module) - doc['version_added'] = process_version_added(doc.get('version_added', 'historical')) - - doc['plugin_type'] = plugin_type - - if module_map[module]['deprecated'] and 'deprecated' not in doc: - display.warning("%s PLUGIN MISSING DEPRECATION DOCUMENTATION: %s" % (fname, 'deprecated')) - - required_fields = ('short_description',) - for field in required_fields: - if field not in doc: - display.warning("%s PLUGIN MISSING field '%s'" % (fname, field)) - - not_nullable_fields = ('short_description',) - for field in not_nullable_fields: - if field in doc and doc[field] in (None, ''): - print("%s: WARNING: MODULE field '%s' DOCUMENTATION is null/empty value=%s" % (fname, field, doc[field])) - - if 'description' in doc: - if isinstance(doc['description'], string_types): - doc['description'] = [doc['description']] - elif not isinstance(doc['description'], (list, tuple)): - raise AnsibleError("Description must be a string or list of strings. Got %s" - % type(doc['description'])) - else: - doc['description'] = [] - - if 'version_added' not in doc: - # Will never happen, since it has been explicitly inserted above. - raise AnsibleError("*** ERROR: missing version_added in: %s ***\n" % module) - - # - # The present template gets everything from doc so we spend most of this - # function moving data into doc for the template to reference - # - - if module_map[module]['aliases']: - doc['aliases'] = module_map[module]['aliases'] - - # don't show version added information if it's too old to be called out - added = 0 - if doc['version_added'] == 'historical': - del doc['version_added'] - else: - added = doc['version_added'] - - # Strip old version_added for the module - if too_old(added): - del doc['version_added'] - - doc['option_keys'] = process_options(module, doc.get('options')) - doc['filename'] = fname - doc['source'] = module_map[module]['source'] - doc['docuri'] = doc['module'].replace('_', '-') - doc['now_date'] = datetime.date.today().strftime('%Y-%m-%d') - doc['ansible_version'] = ansible_version - - # check the 'deprecated' field in doc. We expect a dict potentially with 'why', 'version', and 'alternative' fields - # examples = module_map[module]['examples'] - # print('\n\n%s: type of examples: %s\n' % (module, type(examples))) - # if examples and not isinstance(examples, (str, unicode, list)): - # raise TypeError('module %s examples is wrong type (%s): %s' % (module, type(examples), examples)) - - # use 'examples' for 'plainexamples' if 'examples' is a string - if isinstance(module_map[module]['examples'], string_types): - doc['plainexamples'] = module_map[module]['examples'] # plain text - else: - doc['plainexamples'] = '' - - doc['metadata'] = module_map[module]['metadata'] - - display.vvvvv(pp.pformat(module_map[module])) - if module_map[module]['returndocs']: - doc['returndocs'] = module_map[module]['returndocs'] - process_returndocs(doc['returndocs']) - else: - doc['returndocs'] = None - - doc['author'] = doc.get('author', ['UNKNOWN']) - if isinstance(doc['author'], string_types): - doc['author'] = [doc['author']] - - display.v('about to template %s' % module) - display.vvvvv(pp.pformat(doc)) - try: - text = templates['plugin'].render(doc) - except Exception as e: - display.warning(msg="Could not parse %s due to %s" % (module, e)) - continue - - if LooseVersion(jinja2.__version__) < LooseVersion('2.10'): - # jinja2 < 2.10's indent filter indents blank lines. Cleanup - text = re.sub(' +\n', '\n', text) - - write_data(text, output_dir, outputname, module) - - # Create deprecation stub pages for deprecated aliases - if module_map[module]['aliases']: - for alias in module_map[module]['aliases']: - if alias in module_map[module]['aliases_deprecated']: - doc['alias'] = alias - - display.v('about to template %s (deprecation alias %s)' % (module, alias)) - display.vvvvv(pp.pformat(doc)) - try: - text = templates['plugin_deprecation_stub'].render(doc) - except Exception as e: - display.warning(msg="Could not parse %s (deprecation alias %s) due to %s" % (module, alias, e)) - continue - - if LooseVersion(jinja2.__version__) < LooseVersion('2.10'): - # jinja2 < 2.10's indent filter indents blank lines. Cleanup - text = re.sub(' +\n', '\n', text) - - write_data(text, output_dir, outputname, alias) - - -def process_categories(plugin_info, categories, templates, output_dir, output_name, plugin_type): - # For some reason, this line is changing plugin_info: - # text = templates['list_of_CATEGORY_modules'].render(template_data) - # To avoid that, make a deepcopy of the data. - # We should track that down and fix it at some point in the future. - plugin_info = deepcopy(plugin_info) - for category in sorted(categories.keys()): - module_map = categories[category] - category_filename = output_name % category - - display.display("*** recording category %s in %s ***" % (category, category_filename)) - - # start a new category file - - category_name = category.replace("_", " ") - category_title = category_name.title() - - subcategories = dict((k, v) for k, v in module_map.items() if k != '_modules') - template_data = {'title': category_title, - 'category_name': category_name, - 'category': module_map, - 'subcategories': subcategories, - 'module_info': plugin_info, - 'plugin_type': plugin_type - } - - text = templates['list_of_CATEGORY_modules'].render(template_data) - write_data(text, output_dir, category_filename) - - -def process_support_levels(plugin_info, categories, templates, output_dir, plugin_type): - supported_by = {'Ansible Core Team': {'slug': 'core_supported', - 'modules': [], - 'output': 'core_maintained.rst', - 'blurb': "These are :doc:`modules maintained by the" - " Ansible Core Team<core_maintained>` and will always ship" - " with Ansible itself."}, - 'Ansible Network Team': {'slug': 'network_supported', - 'modules': [], - 'output': 'network_maintained.rst', - 'blurb': "These are :doc:`modules maintained by the" - " Ansible Network Team<network_maintained>` in" - " a relationship similar to how the Ansible Core Team" - " maintains the Core modules."}, - 'Ansible Partners': {'slug': 'certified_supported', - 'modules': [], - 'output': 'partner_maintained.rst', - 'blurb': """ -Some examples of :doc:`Certified Modules<partner_maintained>` are those submitted by other -companies. Maintainers of these types of modules must watch for any issues reported or pull requests -raised against the module. - -The Ansible Core Team will review all modules becoming certified. Core committers will review -proposed changes to existing Certified Modules once the community maintainers of the module have -approved the changes. Core committers will also ensure that any issues that arise due to Ansible -engine changes will be remediated. Also, it is strongly recommended (but not presently required) -for these types of modules to have unit tests. - -These modules are currently shipped with Ansible, but might be shipped separately in the future. -"""}, - 'Ansible Community': {'slug': 'community_supported', - 'modules': [], - 'output': 'community_maintained.rst', - 'blurb': """ -These are :doc:`modules maintained by the Ansible Community<community_maintained>`. They **are -not** supported by the Ansible Core Team or by companies/partners associated to the module. - -They are still fully usable, but the response rate to issues is purely up to the community. Best -effort support will be provided but is not covered under any support contracts. - -These modules are currently shipped with Ansible, but will most likely be shipped separately in the future. - """}, - } - - # only gen support pages for modules for now, need to split and namespace templates and generated docs - if plugin_type == 'plugins': - return - # Separate the modules by support_level - for module, info in plugin_info.items(): - if not info.get('metadata', None): - display.warning('no metadata for %s' % module) - continue - if info['metadata']['supported_by'] == 'core': - supported_by['Ansible Core Team']['modules'].append(module) - elif info['metadata']['supported_by'] == 'network': - supported_by['Ansible Network Team']['modules'].append(module) - elif info['metadata']['supported_by'] == 'certified': - supported_by['Ansible Partners']['modules'].append(module) - elif info['metadata']['supported_by'] == 'community': - supported_by['Ansible Community']['modules'].append(module) - else: - raise AnsibleError('Unknown supported_by value: %s' % info['metadata']['supported_by']) - - # Render the module lists based on category and subcategory - for maintainers, data in supported_by.items(): - subcategories = {} - subcategories[''] = {} - for module in data['modules']: - new_cat = plugin_info[module]['sub_category'] - category = plugin_info[module]['primary_category'] - if category not in subcategories: - subcategories[category] = {} - subcategories[category][''] = {} - subcategories[category]['']['_modules'] = [] - if new_cat not in subcategories[category]: - subcategories[category][new_cat] = {} - subcategories[category][new_cat]['_modules'] = [] - subcategories[category][new_cat]['_modules'].append(module) - - template_data = {'maintainers': maintainers, - 'subcategories': subcategories, - 'modules': data['modules'], - 'slug': data['slug'], - 'module_info': plugin_info, - 'plugin_type': plugin_type - } - text = templates['support_list'].render(template_data) - write_data(text, output_dir, data['output']) - - -def validate_options(options): - ''' validate option parser options ''' - - if not options.module_dir: - sys.exit("--module-dir is required") - if not os.path.exists(options.module_dir): - sys.exit("--module-dir does not exist: %s" % options.module_dir) - if not options.template_dir: - sys.exit("--template-dir must be specified") - - -class DocumentPlugins(Command): - name = 'document-plugins' - - @classmethod - def init_parser(cls, add_parser): - parser = add_parser(cls.name, description='Generate module documentation from metadata') - - parser.add_argument("-A", "--ansible-version", action="store", dest="ansible_version", - default="unknown", help="Ansible version number") - parser.add_argument("-M", "--module-dir", action="store", dest="module_dir", - default=MODULEDIR, help="Ansible library path") - parser.add_argument("-P", "--plugin-type", action="store", dest="plugin_type", - default='module', help="The type of plugin (module, lookup, etc)") - parser.add_argument("-T", "--template-dir", action="append", dest="template_dir", - help="directory containing Jinja2 templates") - parser.add_argument("-t", "--type", action='store', dest='type', choices=['rst'], - default='rst', help="Document type") - parser.add_argument("-o", "--output-dir", action="store", dest="output_dir", default=None, - help="Output directory for module files") - parser.add_argument("-I", "--includes-file", action="store", dest="includes_file", - default=None, help="Create a file containing list of processed modules") - parser.add_argument("-l", "--limit-to-modules", '--limit-to', action="store", - dest="limit_to", default=None, help="Limit building module documentation" - " to comma-separated list of plugins. Specify non-existing plugin name" - " for no plugins.") - parser.add_argument('-V', action='version', help='Show version number and exit') - parser.add_argument('-v', '--verbose', dest='verbosity', default=0, action="count", - help="verbose mode (increase number of 'v's for more)") - - @staticmethod - def main(args): - if not args.template_dir: - args.template_dir = ["hacking/templates"] - validate_options(args) - display.verbosity = args.verbosity - plugin_type = args.plugin_type - - display.display("Evaluating %s files..." % plugin_type) - - # prep templating - templates = jinja2_environment(args.template_dir, args.type, plugin_type) - - # set file/directory structure - if plugin_type == 'module': - # trim trailing s off of plugin_type for plugin_type=='modules'. ie 'copy_module.rst' - outputname = '%s_' + '%s.rst' % plugin_type - output_dir = args.output_dir - else: - # for plugins, just use 'ssh.rst' vs 'ssh_module.rst' - outputname = '%s.rst' - output_dir = '%s/plugins/%s' % (args.output_dir, plugin_type) - - display.vv('output name: %s' % outputname) - display.vv('output dir: %s' % output_dir) - - # Convert passed-in limit_to to None or list of modules. - if args.limit_to is not None: - args.limit_to = [s.lower() for s in args.limit_to.split(",")] - - plugin_info, categories = get_plugin_info(args.module_dir, limit_to=args.limit_to, verbose=(args.verbosity > 0)) - - categories['all'] = {'_modules': plugin_info.keys()} - - if display.verbosity >= 3: - display.vvv(pp.pformat(categories)) - if display.verbosity >= 5: - display.vvvvv(pp.pformat(plugin_info)) - - # Transform the data - if args.type == 'rst': - display.v('Generating rst') - for key, record in plugin_info.items(): - display.vv(key) - if display.verbosity >= 5: - display.vvvvv(pp.pformat(('record', record))) - if record.get('doc', None): - short_desc = record['doc']['short_description'].rstrip('.') - if short_desc is None: - display.warning('short_description for %s is None' % key) - short_desc = '' - record['doc']['short_description'] = rst_ify(short_desc) - - if plugin_type == 'module': - display.v('Generating Categories') - # Write module master category list - category_list_text = templates['category_list'].render(categories=sorted(categories.keys())) - category_index_name = '%ss_by_category.rst' % plugin_type - write_data(category_list_text, output_dir, category_index_name) - - # Render all the individual plugin pages - display.v('Generating plugin pages') - process_plugins(plugin_info, templates, outputname, output_dir, args.ansible_version, plugin_type) - - # Render all the categories for modules - if plugin_type == 'module': - display.v('Generating Category lists') - category_list_name_template = 'list_of_%s_' + '%ss.rst' % plugin_type - process_categories(plugin_info, categories, templates, output_dir, category_list_name_template, plugin_type) - - # Render all the categories for modules - process_support_levels(plugin_info, categories, templates, output_dir, plugin_type) - - return 0 diff --git a/hacking/build_library/build_ansible/jinja2/__init__.py b/hacking/build_library/build_ansible/jinja2/__init__.py deleted file mode 100644 index e69de29bb2..0000000000 --- a/hacking/build_library/build_ansible/jinja2/__init__.py +++ /dev/null diff --git a/hacking/build_library/build_ansible/jinja2/filters.py b/hacking/build_library/build_ansible/jinja2/filters.py deleted file mode 100644 index 21436eda90..0000000000 --- a/hacking/build_library/build_ansible/jinja2/filters.py +++ /dev/null @@ -1,100 +0,0 @@ -# Copyright: (c) 2019, Ansible Project -# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt) - -# Make coding more python3-ish -from __future__ import (absolute_import, division, print_function) -__metaclass__ = type - -import re - -try: - from html import escape as html_escape -except ImportError: - # Python-3.2 or later - import cgi - - def html_escape(text, quote=True): - return cgi.escape(text, quote) - -from jinja2.runtime import Undefined - -from ansible.errors import AnsibleError -from ansible.module_utils._text import to_text -from ansible.module_utils.six import string_types - - -_ITALIC = re.compile(r"I\(([^)]+)\)") -_BOLD = re.compile(r"B\(([^)]+)\)") -_MODULE = re.compile(r"M\(([^)]+)\)") -_URL = re.compile(r"U\(([^)]+)\)") -_LINK = re.compile(r"L\(([^)]+), *([^)]+)\)") -_CONST = re.compile(r"C\(([^)]+)\)") -_RULER = re.compile(r"HORIZONTALLINE") - - -def html_ify(text): - ''' convert symbols like I(this is in italics) to valid HTML ''' - - if not isinstance(text, string_types): - text = to_text(text) - - t = html_escape(text) - t = _ITALIC.sub(r"<em>\1</em>", t) - t = _BOLD.sub(r"<b>\1</b>", t) - t = _MODULE.sub(r"<span class='module'>\1</span>", t) - t = _URL.sub(r"<a href='\1'>\1</a>", t) - t = _LINK.sub(r"<a href='\2'>\1</a>", t) - t = _CONST.sub(r"<code>\1</code>", t) - t = _RULER.sub(r"<hr/>", t) - - return t.strip() - - -def documented_type(text): - ''' Convert any python type to a type for documentation ''' - - if isinstance(text, Undefined): - return '-' - if text == 'str': - return 'string' - if text == 'bool': - return 'boolean' - if text == 'int': - return 'integer' - if text == 'dict': - return 'dictionary' - return text - - -# The max filter was added in Jinja2-2.10. Until we can require that version, use this -def do_max(seq): - return max(seq) - - -def rst_ify(text): - ''' convert symbols like I(this is in italics) to valid restructured text ''' - - try: - t = _ITALIC.sub(r"*\1*", text) - t = _BOLD.sub(r"**\1**", t) - t = _MODULE.sub(r":ref:`\1 <\1_module>`", t) - t = _LINK.sub(r"`\1 <\2>`_", t) - t = _URL.sub(r"\1", t) - t = _CONST.sub(r"``\1``", t) - t = _RULER.sub(r"------------", t) - except Exception as e: - raise AnsibleError("Could not process (%s) : %s" % (text, e)) - - return t - - -def rst_fmt(text, fmt): - ''' helper for Jinja2 to do format strings ''' - - return fmt % (text) - - -def rst_xline(width, char="="): - ''' return a restructured text line of a given length ''' - - return char * width diff --git a/test/sanity/code-smell/docs-build.py b/test/sanity/code-smell/docs-build.py index 2e9a0fb4f9..9b6cbd3f86 100755 --- a/test/sanity/code-smell/docs-build.py +++ b/test/sanity/code-smell/docs-build.py @@ -11,7 +11,7 @@ import sys def main(): base_dir = os.getcwd() + os.path.sep docs_dir = os.path.abspath('docs/docsite') - cmd = ['make', 'singlehtmldocs'] + cmd = ['make', 'base_singlehtmldocs'] sphinx = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=docs_dir) stdout, stderr = sphinx.communicate() diff --git a/test/sanity/code-smell/docs-build.requirements.txt b/test/sanity/code-smell/docs-build.requirements.txt index 3a57c92f51..5e458795dd 100644 --- a/test/sanity/code-smell/docs-build.requirements.txt +++ b/test/sanity/code-smell/docs-build.requirements.txt @@ -3,3 +3,4 @@ pyyaml sphinx sphinx-notfound-page straight.plugin +antsibull diff --git a/test/sanity/ignore.txt b/test/sanity/ignore.txt index dabb59b66c..74abc7047c 100644 --- a/test/sanity/ignore.txt +++ b/test/sanity/ignore.txt @@ -19,9 +19,6 @@ hacking/build_library/build_ansible/command_plugins/dump_keywords.py compile-3.5 hacking/build_library/build_ansible/command_plugins/generate_man.py compile-2.6!skip # docs build only, 3.6+ required hacking/build_library/build_ansible/command_plugins/generate_man.py compile-2.7!skip # docs build only, 3.6+ required hacking/build_library/build_ansible/command_plugins/generate_man.py compile-3.5!skip # docs build only, 3.6+ required -hacking/build_library/build_ansible/command_plugins/plugin_formatter.py compile-2.6!skip # docs build only, 3.6+ required -hacking/build_library/build_ansible/command_plugins/plugin_formatter.py compile-2.7!skip # docs build only, 3.6+ required -hacking/build_library/build_ansible/command_plugins/plugin_formatter.py compile-3.5!skip # docs build only, 3.6+ required hacking/build_library/build_ansible/command_plugins/porting_guide.py compile-2.6!skip # release process only, 3.6+ required hacking/build_library/build_ansible/command_plugins/porting_guide.py compile-2.7!skip # release process only, 3.6+ required hacking/build_library/build_ansible/command_plugins/porting_guide.py compile-3.5!skip # release process only, 3.6+ required |