SERVER-52576 Added libdeps graph visualizer web service with basic features.

1 files changed, 74 insertions, 0 deletions

diff --git a/buildscripts/libdeps/README.md b/buildscripts/libdeps/README.md
new file mode 100644
index 00000000000..a0fd283496e
--- /dev/null
+++ b/buildscripts/libdeps/README.md
@@ -0,0 +1,74 @@
+# Libdeps Graph Analysis Tools
+
+The Libdeps Graph analysis tools perform analysis and queries on graph representing the libdeps dependencies in the mongodb server builds.
+
+## Generating the graph file
+
+The scons build can create the graph files for analysis. To build the graphml file run the build with this minimal set of args required:
+
+    python3 buildscripts/scons.py --link-model=dynamic --build-tools=next generate-libdeps-graph
+
+The target `generate-libdeps-graph` has special meaning and will turn on extra build items to generate the graph. This target will build everything so that the graph is fully representative of the build. The graph file by default will be found at `build/opt/libdeps/libdeps.graphml` (where `build/opt` is the `$BUILD_DIR`).
+
+## Command Line Tool
+
+The Command Line tool will process a single graph file based off a list of input args. To see the full list of args run the command:
+
+    python3 buildscripts/libdeps/gacli.py --help
+
+By default it will performs some basic operations and print the output in human readable format:
+
+    python3.8 buildscripts/libdeps/gacli.py --graph-file build/opt/libdeps/libdeps.graphml
+
+Which will give an output similar to this:
+
+    Loading graph data...Loaded!
+
+    Graph built from git hash:
+    19da729e2696bbf15d3a35c340281e4385069b88
+
+    Graph Schema version:
+    1
+
+    Build invocation:
+    "/usr/bin/python3.8" "buildscripts/scons.py" "--variables-files=etc/scons/mongodbtoolchain_v3_gcc.vars" "--dbg=on" "--opt=on" "--enable-free-mon=on" "--enable-http-client=on" "--cache=all" "--cache-dir=/home/ubuntu/scons-cache" "--install-action=hardlink" "--link-model=dynamic" "--build-tools=next" "--ssl" "--modules=enterprise" "CCACHE=ccache" "ICECC=icecc" "-j50" "generate-libdeps-graph"
+
+    Nodes in Graph: 859
+    Edges in Graph: 90843
+    Direct Edges in Graph: 5808
+    Transitive Edges in Graph: 85035
+    Direct Public Edges in Graph: 3511
+    Public Edges in Graph: 88546
+    Private Edges in Graph: 2272
+    Interface Edges in Graph: 25
+    Shim Nodes in Graph: 20
+    Program Nodes in Graph: 134
+    Library Nodes in Graph: 725
+
+    LibdepsLinter: PUBLIC libdeps that could be PRIVATE: 0
+
+## Graph Visualizer Tool
+
+The graph visualizer tools starts up a web service to provide a frontend GUI to navigating and examining the graph files. The Visualizer used a Python Flask backend and React Javascript frontend. You will need to install the libdeps requirements file to python to run the backend:
+
+    python3 -m pip install -r etc/pip/libdeps-requirements.txt
+
+For installing the dependencies for the frontend, you will need node >= 12.0.0 and npm installed and in the PATH. To install the dependencies navigate to directory where package.json lives, and run:
+
+    cd buildscripts/libdeps/graph_visualizer_web_stack && npm install
+
+Alternatively if you are on linux, you can use the setup_node_env.sh script to automatically download node 12 and npm, setup the local environment and install the dependencies. Run the command:
+
+    buildscripts/libdeps/graph_visualizer_web_stack/setup_node_end.sh install
+
+Assuming you are on a remote workstation and using defaults, you will need to make ssh tunnels to the web service to access the service in your local browser. The frontend and backend both use a port (this case 3000 is the frontend and 5000 is the backend), and the default host is localhost, so you will need to open two tunnels so the frontend running in your local web browser can communicate with the backend. If you are using the default host and port the tunnel command will look like this:
+
+    ssh -L 3000:localhost:3000 -L 5000:localhost:5000 ubuntu@workstation.hostname
+
+Next we need to start the web service. It will require you to pass a directory where it will search for `.graphml` files which contain the graph data for various commits:
+
+    python3 buildscripts/libdeps/graph_visualizer.py --graphml-dir build/opt/libdeps
+
+The script will download nodejs, use npm to install all required packages, launch the backend and then build the optimized production frontend. You can supply the `--debug` argument to work in development load which allows real time updates as files are modified.
+
+After the server has started up, it should notify you via the terminal that you can access it at http://localhost:3000 locally in your browser.