Overhaul demo and support running it locally (#2141)

* Add Make target for building Pyodide with currently checked out Pygments

Add a target to the documentation's Makefile to make Pyodide with
Pygments from the current checkout.

Additionally, use this target when building the documentation with
the demo page.

The new target works by building Pyodide on a Docker container, and
then exporting the build artifacts back to the host.

The Dockerfile used by the new target is based on the Dockerfile which
was used for building Pyodide on the CI:
https://github.com/pygments/pyodide-artifacts/blob/master/container/Dockerfile

The main difference between the two is that the new Dockerfile uses the
newest Pyodide base image, which is built by a repository which is actively
maintained (the Iodide project is no longer maintained).

The purpose of this change is to allow running the demo locally,
which was not possible previously, as Pyodide was only built by the
CI when deploying to Github Pages.

* Add instructions on how to run the demo locally to the README

* Update demo to work with Pyodide v0.20

Remove usage of Pyodide functionality which was deprecated in Pyodide v0.17:

- Loading Pyodide using `languagePluginURL` and `languagePluginLoader`.
- Access to globals via `pyodide.globals.x` has changed to `pyodide.globals.get("x")`.

Source:
https://pyodide.org/en/stable/project/release-notes/v0.17.0.html?highlight=languageplugin#api-changes

* Fix #2137

* Fix use of `styles` variable before it's defined

The `styles` variable was being used before it's value was set (which
is done when Pyodide finishes loading).

* Remove GitHub action for building Pyodide

This action is obsolete, as building Pyodide is now done using `make pyodide`.

* Upgrade Pages deployment action version

* Add to .dockerignore all files in .gitignore

* Change `pyodide` target to be a phony target

10 files changed, 96 insertions, 28 deletions

diff --git a/.dockerignore b/.dockerignore
new file mode 100644
index 00000000..e08d9e1d
--- /dev/null
+++ b/.dockerignore
@@ -0,0 +1,16 @@
+doc
+tests
+Pygments.egg-info
+TAGS
+build
+dist
+htmlcov
+venv
+**/__pycache__
+.*
+*.rst
+*.egg
+*.pyo
+.*.sw[op]
+
+!/doc/pyodide/meta.yaml
diff --git a/.github/actions/pyodide-package/action.yml b/.github/actions/pyodide-package/action.yml
deleted file mode 100644
index 2187cd9f..00000000
--- a/.github/actions/pyodide-package/action.yml
+++ /dev/null
@@ -1,5 +0,0 @@
-name: 'Update Pyodide package'
-description: 'Update the WASM compiled Pygments with Pyodide'
-runs:
-  using: 'docker'
-  image: 'birkenfeld/pyodide-pygments-builder'
diff --git a/.github/workflows/docs.yaml b/.github/workflows/docs.yaml
index 2fd42f8a..66a57b8d 100644
--- a/.github/workflows/docs.yaml
+++ b/.github/workflows/docs.yaml
@@ -18,20 +18,19 @@ jobs:
     - name: Install Sphinx & WCAG contrast ratio
       run: pip install Sphinx wcag-contrast-ratio
     - name: Create Pyodide WASM package
-      uses: ./.github/actions/pyodide-package
+      run: cd doc && make pyodide
     - name: Sphinx build
       run: |
         cd doc
         WEBSITE_BUILD=1 make dirhtml
-        cp -a ../pyodide _build/dirhtml/_static
         touch _build/dirhtml/.nojekyll
         echo -e 'pygments.org\nwww.pygments.org' > _build/dirhtml/CNAME
         echo 'Automated deployment of docs for GitHub pages.' > _build/dirhtml/README
     - name: Deploy to repo
       if: github.repository_owner == 'pygments'
-      uses: peaceiris/actions-gh-pages@v2.5.0
-      env:
-        ACTIONS_DEPLOY_KEY: ${{ secrets.ACTIONS_DEPLOY_KEY }}
-        EXTERNAL_REPOSITORY: pygments/pygments.github.io
-        PUBLISH_BRANCH: master
-        PUBLISH_DIR: ./doc/_build/dirhtml
+      uses: peaceiris/actions-gh-pages@v3
+      with:
+        deploy_key: ${{ secrets.ACTIONS_DEPLOY_KEY }}
+        external_repository: pygments/pygments.github.io
+        publish_branch: master
+        publish_dir: ./doc/_build/dirhtml
diff --git a/README.rst b/README.rst
index 9974cbbe..e9633e05 100644
--- a/README.rst
+++ b/README.rst
@@ -16,8 +16,22 @@ Documentation
 
 ... can be found online at https://pygments.org/ or created with Sphinx by ::
 
-   cd doc
-   make html
+   make docs
+
+By default, the documentation does not include the demo page, as it requires
+having Docker installed for building Pyodide. To build the documentation with
+the demo page, use ::
+
+   WEBSITE_BUILD=1 make docs
+
+The initial build might take some time, but subsequent ones should be instant
+because of Docker caching.
+
+To view the generated documentation, serve it using Python's ``http.server``
+module (this step is required for the demo to work) ::
+
+   python3 -m http.server --directory doc/_build/html
+
 
 Development
 -----------
diff --git a/doc/Makefile b/doc/Makefile
index 7fb75411..a0dcaaa4 100644
--- a/doc/Makefile
+++ b/doc/Makefile
@@ -14,10 +14,11 @@ ALLSPHINXOPTS   = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
 # the i18n builder cannot share the environment and doctrees with the others
 I18NSPHINXOPTS  = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
 
-.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext
+.PHONY: help clean pyodide html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext
 
 help:
 	@echo "Please use \`make <target>' where <target> is one of"
+	@echo "  pyodide    to make Pyodide with currently checked out Pygments"
 	@echo "  html       to make standalone HTML files"
 	@echo "  dirhtml    to make HTML files named index.html in directories"
 	@echo "  singlehtml to make a single large HTML file"
@@ -41,12 +42,21 @@ help:
 clean:
 	-rm -rf $(BUILDDIR)/*
 
+pyodide:
+	$(if $(test ! -f docker), $(error "Could not find Docker. Please install that before continuing."))
+	# Enable the BuildKit backend to use the --output option.
+	DOCKER_BUILDKIT=1 docker build --file pyodide/Dockerfile --output $(BUILDDIR)/pyodide/pyodide ..
+	@echo
+	@echo "Pyodide build finished. The Pyodide artifacts are in $(BUILDDIR)/pyodide."
+
 html:
+	$(if $(WEBSITE_BUILD), $(MAKE) pyodide)
 	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
 	@echo
 	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
 
 dirhtml:
+	$(if $(WEBSITE_BUILD), $(MAKE) pyodide)
 	$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml
 	@echo
 	@echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."
diff --git a/doc/_static/demo-worker.js b/doc/_static/demo-worker.js
index af8b0092..22b8b3d2 100644
--- a/doc/_static/demo-worker.js
+++ b/doc/_static/demo-worker.js
@@ -1,21 +1,23 @@
-self.languagePluginUrl = '/_static/pyodide/';
 importScripts('/_static/pyodide/pyodide.js');
 
-(async function() {
-    await languagePluginLoader;
+async function loadPyodideAndPygments() {
+    self.pyodide = await loadPyodide();
     await self.pyodide.loadPackage(["Pygments"]);
     const styles = self.pyodide.runPython(`
         from pygments.formatters.html import HtmlFormatter
         from pygments.styles import STYLE_MAP
         {s: HtmlFormatter(style=s).get_style_defs('.demo-highlight') for s in STYLE_MAP}
-    `);
+    `).toJs();
     self.postMessage({loaded: {styles}})
-})();
+}
+let pyodideReadyPromise = loadPyodideAndPygments();
 
 self.onmessage = async (event) => {
+    // Make sure loading is done.
+    await pyodideReadyPromise;
     if (event.data.highlight) {
-        self.pyodide.globals['code'] = event.data.highlight.code;
-        self.pyodide.globals['lexer_name'] = event.data.highlight.lexer;
+        self.pyodide.globals.set('code', event.data.highlight.code);
+        self.pyodide.globals.set('lexer_name', event.data.highlight.lexer);
 
         self.pyodide.runPython(`
             import pygments.lexers
@@ -40,14 +42,14 @@ self.onmessage = async (event) => {
             `);
             self.postMessage({html});
         } else if (formatter == 'tokens') {
-            const tokens = self.pyodide.runPython('list(tokens)');
+            const tokens = self.pyodide.runPython('list(tokens)').toJs();
             self.postMessage({tokens});
         } else {
             console.warn('unknown formatter:', formatter);
         }
     } else if (event.data.guess_lexer) {
-        self.pyodide.globals['code'] = event.data.guess_lexer.code;
-        self.pyodide.globals['filename'] = event.data.guess_lexer.filename;
+        self.pyodide.globals.set('code', event.data.guess_lexer.code);
+        self.pyodide.globals.set('filename', event.data.guess_lexer.filename);
         const lexer = self.pyodide.runPython(`
             import sys
             sys.setrecursionlimit(1000)
diff --git a/doc/_static/demo.js b/doc/_static/demo.js
index 488ded03..b193d10c 100644
--- a/doc/_static/demo.js
+++ b/doc/_static/demo.js
@@ -29,7 +29,10 @@ if (qvars.formatter !== undefined) {
 }
 
 styleSelect.addEventListener('change', () => {
-    style.textContent = styles[styleSelect.value];
+    if (!styles)
+        // Worker has not loaded yet.
+        return;
+    style.textContent = styles.get(styleSelect.value);
     updateCopyLink();
     updateContrastWarning();
 });
@@ -89,7 +92,7 @@ highlightWorker.onmessage = (msg) => {
         outputDiv.innerHTML = msg.data.html;
         codeHeader.hidden = false;
         loadingDiv.hidden = true;
-        style.textContent = styles[styleSelect.value];
+        style.textContent = styles.get(styleSelect.value);
     } else if (msg.data.tokens) {
         const table = document.createElement('table');
         table.className = 'tokens';
diff --git a/doc/conf.py b/doc/conf.py
index 7af43e37..dbf4c142 100644
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -139,6 +139,7 @@ html_additional_pages = {
 
 if os.environ.get('WEBSITE_BUILD'):
     html_additional_pages['demo'] = 'demo.html'
+    html_static_path.append('_build/pyodide')
 
 # If false, no module index is generated.
 #html_domain_indices = True
diff --git a/doc/pyodide/Dockerfile b/doc/pyodide/Dockerfile
new file mode 100644
index 00000000..969651cc
--- /dev/null
+++ b/doc/pyodide/Dockerfile
@@ -0,0 +1,20 @@
+# Dockerfile for building Pyodide with a Pygmenets version from the current checkout.
+# For an example of how to use this image, see the `pyodide` target in the documentation's Makefile.
+FROM ghcr.io/pyodide/pyodide:0.20.0 AS build-stage
+
+WORKDIR pyodide
+
+# Copy new meta with path to local Pygments instead of pypi url.
+COPY doc/pyodide/meta.yaml packages/Pygments/
+
+COPY . /pygments
+
+# Add Pygments to the Pyodide build.
+ENV PYODIDE_PACKAGES=Pygments
+
+# Build Pyodide.
+RUN make
+
+FROM scratch AS export-stage
+
+COPY --from=build-stage /src/pyodide/build /
diff --git a/doc/pyodide/meta.yaml b/doc/pyodide/meta.yaml
new file mode 100644
index 00000000..d58e1d58
--- /dev/null
+++ b/doc/pyodide/meta.yaml
@@ -0,0 +1,8 @@
+package:
+  name: Pygments
+  version: '2.99'
+source:
+  path: /pygments
+test:
+  imports:
+  - pygments