docs: Migrate EC-3PO docs from Google Sites

The source for the design doc can be found at
https://sites.google.com/a/chromium.org/dev/chromium-os/chromiumos-design-docs/ec-3po,
while the summary is from
https://www.chromium.org/chromium-os/ec-development/ec-3po. A few terms
mentioned in the design document have also been added to the glossary.

BUG=none
TEST=check rendering in Gitiles
BRANCH=none

Change-Id: I59b4359c6e959e549ec31f6a75e9505758910c46
Signed-off-by: Harry Cutts <hcutts@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/ec/+/1949725
Reviewed-by: Matthew Blecker <matthewb@chromium.org>

6 files changed, 439 insertions, 1 deletions

diff --git a/docs/ec-3po-design.md b/docs/ec-3po-design.md
new file mode 100644
index 0000000000..56a217849b
--- /dev/null
+++ b/docs/ec-3po-design.md
@@ -0,0 +1,326 @@
+# EC-3PO: The EC console interpreter
+
+[TOC]
+
+## Introduction
+
+Today, the Chromium OS Embedded Controller (EC) has a very rich debug console
+that is very helpful and has features including command history, editing,
+timestamps, channels and much more. However, all of these features currently use
+up valuable flash space that a number of our boards desperately need. We
+constantly run into this problem where boards are running out of space and
+people have to hack out a lot of code just so the image will fit. It's been
+occurring with more frequency lately and I imagine it will continue to occur as
+we add more features to our EC code base. What we could do instead is move all
+of that console functionality out to a separate utility and turn the EC console
+into a binary mode which would only speak in host command packets and debug
+output packets. EC-3PO would serve as the interpreter translating from the
+traditional EC console that we all know and love to host command packets sent
+down to the EC and vice versa.
+
+## Benefits
+
+The benefits to be gained are numerous and can all occur without changing
+people's existing workflow. The only slight impact might be that we instruct
+people to `emerge hdctools` every so often. All people would notice would be
+that the EC images would be getting smaller and/or more console features.
+
+### Testing & FAFT
+
+Currently, [FAFT](https://www.chromium.org/for-testers/faft) runs its tests
+while trying to parse strings from the EC console. This method can be fairly
+fragile as debug output can be interleaved with the console input. A lot of
+items could be improved by switching to this host command packet interface.
+
+* Communicating in packets makes testing easier and more robust.
+  * When FAFT is running, the EC could be in a binary mode where it only
+    communicates in host command packets. These packets are easier to parse,
+    create, and filter on.
+  * With filtering, you get the added bonus of not having unwanted debug output.
+  * It allows us to really test the host command interface which is how the EC
+    talks to the AP anyways.
+* Better testing of existing host command handlers.
+  * By speaking in host command packets, we can reuse the existing host command
+    handlers which is nice since we'll be using the same handlers that are used
+    to respond to the AP.
+* FAFT would no longer have to worry about the console dropping characters.
+  * We can add error checking to the interpreter which would automatically retry
+    errors. This alleviates FAFT from trying to check if the EC had properly
+    received a line of input. (Ctrl+L)
+
+With better and more reliable tests, we can improve the quality of our EC
+codebase.
+
+### Space Savings
+
+By moving the console functionality off of the EC, we would be able to shave off
+a considerable amount of bytes from the EC images. People wouldn't have to worry
+as much about creating a debug console print with the fear of bloating up the
+image size. Smaller stack requirements by changing `printf` formatting to only
+count bytes while moving common strings off the EC. Additionally, most of these
+savings will come for free as it will apply to every EC with a console. We won't
+be restricted by the type of chip.
+
+### A richer EC console
+
+* We could do things like on-the-fly console channel filtering.
+* Coloring specific channels such as "mark all USB PD messages in green".
+* Adding colors in general.
+* Adding temporary console commands.
+* Longer command history which survives EC reboot
+* Searching command history
+* Redirecting debug output to log files (which causes no interleaving of command
+  and debug output)
+* Bang commands (`!foo`)
+
+### Better debuggability
+
+Sometimes, there will be an issue with the EC (or believed to be an EC issue)
+such as the keyboard locking up on certain keys or rows. At times like that, it
+would be nice to have an EC console to see what's going on. Other times maybe
+having a servo connected might make the issue not present itself.
+
+* We could do cool things like having an EC console without having to hook up
+  servo.
+* Run `ectool` from the chroot using a PTY interface.
+
+## Deployment Strategy
+
+There are many facets to this feature, but here's a deployment strategy which
+will gradually take us from the current EC console today, to a future where the
+console is completely removed from the EC. The goal will be to make this change
+as transparent as possible to developers.
+
+### Phase 1: Insertion
+
+_[[Merged to ToT](https://crrev.com/c/320629) on 2016-02-07]_
+
+Phase 1 will most likely consist of getting EC-3PO in place in between servo and
+the EC while not modifying the behavior of the console too much. In this phase,
+we can replicate the console interface and achieve the following things.
+
+* Replicate command editing.
+* Save command history.
+* Add error checking to the console commands.
+
+### Phase 2: Assimilation
+
+Phase 2 will start to introduce the host command packet communication.
+
+* Printing will be done via packets and assembled in EC-3PO.
+* Console commands now are sent using the host command packets.
+  * This will be incremental as console commands are converted.
+* Add debug output filtering and redirection/logging.
+
+### Phase 3: Expansion
+
+Phase 3 will expand the feature set of EC-3PO.
+
+* Add PTY interface to `ectool`.
+* Add on-device console without `servod`.
+* Colored output.
+* Command history search.
+* Bang commands (`!foo`)
+
+## High Level Design & Highlights
+
+![Diagram with three boxes. EC-3PO with an incoming PTY communicates with the
+Serial Driver over another PTY. The Serial Driver communicates with the EC
+UART.](./images/ec-3po-high-level-design.png)
+
+### EC Interface
+
+Each host command is a 16-bit command value. Commands which take parameters or
+return response data specify `struct`s for that data. See
+[`include/ec_commands.h`](https://chromium.googlesource.com/chromiumos/platform/ec/+/refs/heads/master/include/ec_commands.h)
+for the current format of request and replies. Currently, there are no changes
+made to the format of the host request and response structures.
+
+On the EC, we essentially need to create a UART host command handler. This
+handler will be watching the console input for a byte sequence to switch into
+this host command mode. The starting sequence for an incoming host command
+packet will be `0xDA`, a byte signifying `EC_COMMAND_PROTOCOL_3`. Once this byte
+is seen, the EC will transition to its "host command processing mode" and
+receive the host command. It will then process the host command, send the binary
+host response, and then transition back into normal mode. Ideally, there should
+be some locking of the UART to prevent other writes while this is taking place.
+
+By the end of the assimilation (Phase 2), there would be no "normal mode" and
+everything would be in the binary host command mode.
+
+### Host Interface
+
+The host interface is where the bulk of the work will be. This will be
+converting from the received host commands to console prints as well as
+converting the console inputs to host commands. It will also be responsible for
+replicating the console. This includes things like moving back and forth for
+command editing.
+
+The interpreter should also open a PTY and `dut-control` should return this PTY
+as the `ec_uart_pty`. This is to ensure that the change is as transparent as
+possible to developers.
+
+### Features
+
+The following are an explanation of a few of the planned features.
+
+#### Command Error Checking
+
+EC-3PO and the EC can perform error checking when sending the commands to the
+EC. This feature would be implemented prior to switching to the binary format.
+The interpreter can package the command in the following manner.
+
+* 2 Ampersands
+* 2 hex digits representing the length of the command
+* 2 hex digits representing the CRC-8 of the command
+* 1 Ampersand
+* The command itself
+* 2 newline characters.
+
+This is robust because no commands currently start with `&`. If the EC does not
+see ‘`&&`', then one of the ampersands has been dropped. If the EC doesn't see
+an ampersand after 4 hex digits, it either dropped a hex digit or the ampersand.
+Two newlines, so dropping one of those is alright. Once the EC gets the command
+and newline, it can verify the command string hasn't been corrupted.
+
+In the event that the command string was corrupted, the EC can return an error
+string back of `&&EE`. If the translator reads a line with at least one
+ampersand followed by one "E", then an error had occurred and the translator can
+simply retry the console command. This creates a reliable input to the console,
+a great win for FAFT.
+
+#### PTY interface to `ectool`
+
+Once the UART host command handler is functional, we could add the UART as one
+of the interfaces to `ectool`. This would allow `ectool` to be run from the
+chroot simply by having `ectool` communicate over the PTY. Since `ectool`
+communicates using host commands anyways, everything should just work™. The
+benefits of this include faster development of `ectool` and host commands as
+well as a more robust interface for FAFT.
+
+#### On-device EC console without Servo
+
+Once the transition is complete and the console speaks entirely in host
+commands, it's no longer necessary to have the console talk solely over the
+UART. EC-3PO, or a variant, could exist as a standalone application that could
+be bundled in the system image, just like `ectool`. It would then send and
+receive host commands using the same interface that `ectool` uses whether it be
+LPC or I2C. This would essentially give us a console without having to hook up
+servo.
+
+Note for security reasons, this must be locked down to only allow a subset of
+commands and debug output when the system is ready to ship.
+
+#### Replacing `cprintf()`
+
+All prints will need to become packets. In these packets will contain the format
+string, but all `cprintf()` has to do is parse to format string to determine how
+many bytes the parameters take up. Then, `cprintf()` will send the format string
+and the parameters to EC-3PO which will create the proper string using that
+information. That makes `cprintf()` on the EC smaller and use less stack space
+than it would have used for formatting.
+
+We could also have a table of common format strings which the EC could just
+provide an index and the parameters.
+
+## Internal Design
+
+EC-3PO is a Python package which aims to migrate the rich debug console from the
+EC itself to the host. It is composed of two modules: console and interpreter.
+
+![Diagram of EC-3PO internal design. Within an outer box labelled EC-3PO are the
+Console and Interpreter modules. A bidirectional command pipe links the two, and
+a debug pipe goes from the Interpreter to the
+Console.](./images/ec-3po-internal-design.png)
+
+### Console module
+
+The console module provides the interactive console interface between the user
+and the interpreter. It handles the presentation of the EC console including
+editing methods as well as session-persistent command history.
+
+The console runs in an infinite loop listening for activity on three things
+using the `select()` system call: the PTY served to the user, the command pipe,
+and the debug pipe. The debug pipe is a unidirectional pipe from the interpreter
+to the console. From this pipe are debug prints that originate from the attached
+EC and the console currently writes these strings as is to the user PTY. The
+command pipe is bidirectional and is used between the console and the
+interpreter for command traffic. An example transaction would be a host command
+request and response.
+
+#### Enhanced EC image negotiation
+
+When the user transmits a character on the PTY, the console begins to scan every
+byte and perform the appropriate actions. Since not every EC image will support
+these enhanced features, the console must perform an interrogation to determine
+what behaviour to take. If the interrogation mode is set to auto, this
+negotiation takes place every time the enter key is pressed. The interrogation
+is very simple 2 way handshake. The console sends down a byte, `EC_SYN` and
+waits a timeout period to receive a byte, `EC_ACK`. This timeout period is 300ms
+for non-enhanced EC images and 1 second for enhanced EC images. Enhanced EC
+images will try to immediately respond to an `EC_SYN` with an `EC_ACK` to
+indicate to EC-3PO that the current EC images is enhanced. The non-enhanced EC
+image timeout period is intended to be short because non-enhanced EC images will
+never reply to the `EC_SYN`. By keeping this timeout short, we are essentially
+inserting a slight pause after each command. However, this timeout is long
+enough for an enhanced EC image to send its reply of `EC_ACK`. Once the `EC_ACK`
+is received by the console, the console increases the timeout to 1 second for
+stability purposes just in case the enhanced EC image takes a bit longer to send
+its reply for some reason. This negotiation allows EC-3PO to behave correctly
+for both enhanced and non-enhanced EC images.
+
+If a user knows that they are not using an enhanced EC image, they can disable
+the interrogation by issuing a OOBM command. This will cause the console to
+never initiate a negotiation, eliminating the delay. See the "interrogate"
+command in the Out of Band Management section for usage.
+
+#### Enhanced vs. Non-Enhanced EC images
+
+All EC images which don't explicitly enable the new features (or were built
+before the features were implemented in the code base) are non-enhanced images.
+Non-enhanced EC images will be handling the presentation of the console
+including editing methods (and history if enabled). Therefore, the EC-3PO
+console and interpreter behave as a simple pipe for this case forwarding
+everything straight to the EC to handle. For the enhanced EC images, all console
+presentation (including editing methods) is handled locally by the EC-3PO
+console. Enhanced EC images will also support all of the other features
+discussed in this document.
+
+#### Out of Band Management
+
+The interactive console also has an Out of Band Management (OOBM) interface.
+This allows commands to be entered that can alter the behavior of the console
+and interpreter during runtime. From the console, one can bring up the OOBM
+prompt by pressing ‘%' . These were originally added for debug purposes.
+
+##### Supported Commands
+
+* `loglevel <integer>`
+  * Allows setting the effective loglevel of the console and interpreter.
+* `interrogate <never | always | auto> [enhanced]`
+  * Allows control of when and how often interrogation occurs.
+
+### Interpreter module
+
+The interpreter provides the interpretation layer between the EC UART and the
+user. Similar to the console module, the interpreter starts an infinite loop of
+servicing the user and the EC by means of a `select()` system call. It receives
+commands through its command pipe, formats the commands for the EC, and sends
+the command to the EC. It also presents data from the EC to either be displayed
+via the interactive console module or some other consumer.
+
+The interpreter also keeps track of whether the EC image it's communicating with
+is enhanced or not. This is required so that the interpreter can communicate
+correctly with the EC. For enhanced EC images, the interpreter will pack
+incoming commands in a particular format. This could be the "packed plaintext"
+form or the binary host command format. With the packed plaintext form, the
+interpreter also supports command retrying by monitoring the response of the EC
+and automatically retrying the command with no input from the user.
+
+### Other users
+
+Since the interpreter communicates using pipes, it's not necessary that the user
+use the console module. For example, FAFT could directly connect to the
+interpreter and send down commands and receive command responses instead of
+having to deal with the PTY and instead just deal with python objects.
+
diff --git a/docs/ec-3po.md b/docs/ec-3po.md
new file mode 100644
index 0000000000..83877374aa
--- /dev/null
+++ b/docs/ec-3po.md
@@ -0,0 +1,82 @@
+# EC-3PO
+
+[TOC]
+
+## What is EC-3PO?
+
+EC-3PO is the console interpreter that will one day replace the EC console that
+we have today.  EC-3PO aims to migrate our rich debug console from the EC itself
+to the host.  This allows us to maintain our rich debug console without
+impacting our EC image sizes while also allowing us to add new features.
+
+For more information, see [the design doc](./ec-3po-design.md).
+
+## How do I use EC-3PO?
+
+If you're using `servod` to connect to your EC, chances are you're already using
+it. EC-3PO was grafted into `servod` on Feb 7th 2016. If you're not running
+`servod`, you can run EC-3PO manually by running `console.py` in the
+`util/ec3po` directory from the EC checkout. You will need to provide the PTY
+that you get from elsewhere though.
+
+To obtain the EC console PTY, inside the chroot run:
+
+```shell
+$ dut-control ec_uart_pty
+```
+
+**NOTE: It's important to use `dut-control` to query the PTY instead of just
+eyeballing the `servod` output.** The former PTY (now known as
+`raw_ec_uart_pty`) will be sending raw binary data. Trying to use that console
+with an enhanced EC image will definitely fail and you won't be able to
+send/receive any commands.
+
+Then use your favorite serial terminal program to connect to the PTY. Since
+`servod` is run as root, you'll need to run your serial terminal program as root
+as well using `sudo`. This is because the permissions have changed from 666 to
+660.
+
+EC-3PO has been tested with `minicom`, `screen`, `socat`, and `cu`. However, if
+you're using `cu` you'll have to do the following to get it to work because
+apparently, `cu` wants group write permissions. On Ubuntu at least, the PTY is
+created with the `tty` group. If on your machine it's not, then just replace
+`tty` with whatever group it's created with.
+
+1. Create a `tty` group if you don't have one already.
+1. Add root to the `tty` group.
+1. Rerun `cu` with `sudo` and it should work now.
+
+## Why does the console seem "laggier" than before?
+
+This is because there's a ~300ms delay after entering each console command. This
+is due to the interrogation that the console interpreter performs to determine
+if the EC image it's currently talking to is enhanced or not. Debug prints
+coming from the EC should be the same speed. Since most people aren't currently
+using the enhanced EC images, you can go ahead and run this command if the 300ms
+delay is unbearable.
+
+To disable the delay:
+
+1. Open the EC console.
+1. Press `%`
+1. Enter `interrogate never`
+1. Then press enter.
+
+\**For `socat` users, due to the line buffered nature, you'll have to just enter
+`%interrogate never`. Notice the lack of the space character between `%` and the
+command.*
+
+The interrogation delay should now be gone and you can have your 300ms/cmd back.
+
+## How do I try out this "enhanced" EC image you speak of?
+
+You simply add this to your board.h file.
+
+```c
+#define CONFIG_EXPERIMENTAL_CONSOLE
+```
+
+## I can't open the EC console
+
+Make sure you try with `sudo`. If you're using `cu`, make sure root is a member
+of the group of the created PTY.
diff --git a/docs/ec_terms.md b/docs/ec_terms.md
index 8e4f705b0f..536aec4669 100644
--- a/docs/ec_terms.md
+++ b/docs/ec_terms.md
@@ -64,6 +64,11 @@
     switching, sensor management, and other functions, offloading these tasks
     from the [AP](#ap).
 
+*   **EC-3PO**{#ec-3po}
+
+    A replacement of the current UART-based console which moves much of the code
+    off the EC into a host tool, reducing the amount of flash space required.
+
 *   **E-Mark - Electronically Marked Cable** {#emark}
 
     See the [USB-C documentation](./usb-c.md#emark) for more details.
@@ -75,6 +80,13 @@
     bandwidth up to 264 Mbps. The full [eSPI Specification] is available from
     Intel.
 
+*   **FAFT - Fully Automated Firmware Tests**{#faft}
+
+    A collection of tests and related infrastructure that exercise and verify
+    capabilities of Chrome OS. See the [FAFT design doc] and [chromium.org
+    documentation](https://www.chromium.org/for-testers/faft) for more details.
+    Replaced [SAFT](#saft).
+
 *   **GMR - Giant Magnetoresistance Sensor** {#gmr}
 
     A sensor device that detects a magnetic field. These sensors differ from
@@ -142,6 +154,12 @@
     video streams from a single display port. The EC code is typically
     responsible for enabling and disabling the MST hub chipset.
 
+*   **OOBM - Out of Band Management**{#oobm}
+
+    A command in the [EC-3PO protocol](#ec-3po) that allows commands to be
+    entered to alter the behaviour of the console and interpreter during
+    runtime.
+
 *   **PD - USB Power Delivery**{#pd}
 
     See the [USB-C documentation](./usb-c.md#pd) for more details.
@@ -160,6 +178,12 @@
     typical application is to control fan speeds or the brightness of a
     backlight.
 
+*   **SAFT - Semi-Automated Firmware Tests**{#saft}
+
+    A suite of tests for firmware, succeeded by [FAFT](#faft). See the
+    [chromium.org documentation](https://www.chromium.org/for-testers/saft) for
+    more details.
+
 *   **SPI - Serial Peripheral Interconnect**{#spi}
 
     A 4-wire synchronous communication bus consisting of the signals CLK
@@ -197,7 +221,8 @@
 [CEC Wikipedia page]: <https://en.wikipedia.org/wiki/Consumer_Electronics_Control>
 [DPTF Readme]: <https://github.com/intel/dptf/blob/master/README.txt>
 [eSPI Specification]: <https://www.intel.com/content/dam/support/us/en/documents/software/chipset-software/327432-004_espi_base_specification_rev1.0.pdf>
+[FAFT design doc]: <https://chromium.googlesource.com/chromiumos/third_party/autotest/+/refs/heads/master/docs/faft-design-doc.md>
 [I2C Specification]: <https://www.nxp.com/docs/en/user-guide/UM10204.pdf>
 [RS-232]: <https://en.wikipedia.org/wiki/RS-232>
 [EC MKBP driver]: <https://chromium.googlesource.com/chromiumos/platform/ec/+/refs/heads/master/common/keyboard_mkbp.c>
-[Low Pin Count bus]: https://en.wikipedia.org/wiki/Low_Pin_Count
\ No newline at end of file
+[Low Pin Count bus]: https://en.wikipedia.org/wiki/Low_Pin_Count
diff --git a/docs/images/ec-3po-high-level-design.png b/docs/images/ec-3po-high-level-design.png
new file mode 100644
index 0000000000..2956fa777f
--- /dev/null
+++ b/docs/images/ec-3po-high-level-design.png
diff --git a/docs/images/ec-3po-internal-design.png b/docs/images/ec-3po-internal-design.png
new file mode 100644
index 0000000000..25f60793ae
--- /dev/null
+++ b/docs/images/ec-3po-internal-design.png
diff --git a/docs/sitemap.md b/docs/sitemap.md
index 8f87d84728..2ab3217257 100644
--- a/docs/sitemap.md
+++ b/docs/sitemap.md
@@ -29,6 +29,11 @@
 *   [USB-C Power Delivery and Alternate Modes](./usb-c.md)
 *   [USB-A and USB-C Policies for Sourcing Power](./usb_power.md)
 
+## EC-3PO
+
+*   [EC-3PO overview](./ec-3po.md)
+*   [EC-3PO design doc](./ec-3po-design.md)
+
 ## Miscellaneous
 
 *   [Low Battery Startup](./low_battery_startup.md)