# Packetized console mode [TOC] ## Overview Some EC board images are getting very close to their flash space size limits. A significant part of the image is often taken by the text strings printed out as debug messages or console command output. Removing text strings from the image would allow to free up a lot of space (approximately 10% in case of Cr50), on top of that moving console command support code from the image would allow to free quite a bit more. One requirement for such a facility would be that removing text strings from the image is possible without touching the source code, so that all existing boards could be retrofitted with the new approach without actual code changes. ## Post processing the source code The vast majority of printouts generated by the EC images come from invocations to console output generating functions (aka `cprintf()`, `cprints()` and `cputs()`). In many cases these functions are invoked through macros. The proposed approach is to introduce additional processing steps where the source code could be modified pre-compilation during build time. In the standard processing case `.o` files are generated directly from `.c` files by the compiler. To support the packet mode console the source files' processing has to be more involved. First the source code is taken through C preprocessor. This extends all macro invocations, and each input `.c` file gets a corresponding generated `.E` file. Then invocations of the console output generating functions are substituted such that the format strings can be replaced by their indices (see below). This produces an additional set of intermediate files, each `.E` gets a corresponding `.Ep` file. Then the set of `.Ep` files are taken through the compilation step, resulting an a set of `.o` files generated. In more details, the following steps are taken when generating the set of `.o` files necessary to generate the executable image: - run `.c` sources though C preprocessor, generating `.E` files instead of `.o` files (.i.e. use -E compiler command line option instead of -c) - once all sources are preoprocessed, invoke the `util_precompile.py` script to scan all `.E` files together. For each instance of `cprintf()`, `cprints()`, and `cputs()` found in `.E` files do the following: - save the format string in a dictionary, unless the string is already there. The keys in the dictionary are the format strings, the values - are the string indices, for each new string the value is the length of the dictionary before the string was added. - replace the console output generating function names with `cmsgX`, where `X` is the number of arguments, values from 0 to 8 are supported. - scan the format string for format characters (%), and based on number of format characters pick the `X` value for `cmsgX` function name. - based on the format attributes analyze the parameters and make sure that they can be typecasted to `uintptr_t`. This is the case for pretty much any parameter other than 64 bit values, timestamps in particular. In case there are 64 bit parameters, augment the code by adding a block around the invocation, declaring a 64 bit variable, assigning it the parameter value and using the variable address as the appropriate `cmsgX` parameter. - Wile scanning the parameters, also prepare a 32 bit descriptor, with each parameter's description packed into 4 bits, thus supporting up to 8 parameters. - complete `cmsgX` call declaration giving it as the parameters the console channel number (as is, retrieved from the source code), the index the text string got when it was added to the dictionary, the 32 bit parameter descriptor and the original parameters typecasted to `uintptr_t`, unless processing of 64 bit value(s) took place, in which case pointer(s) to the value(s) are used. - for each `.E` file generate the `.Ep` file with all console output generating functions replaced with `cmsgX()` invocations. - once all .E files have been processed, the `util_precompile.py` script converts the format string dictionary into a list of strings, each string placed in the list at its index, so knowing the index the terminal program can retrieve the format string. The list is serialized, compressed and saved in a file. ## Sending packets to the terminal Some code has to be added to process `cmsgX()` invocations. Each of these functions prepares an array of parameters and calls the common function, which processes the format descriptor and the parameters, and generates a packet sent over the console channel to the terminal. The packet has the following structure, very similar to the one used in the Acropora project: ```c struct console_packet { /* Magic number = CONSOLE_PACKET_MAGIC */ uint8_t magic; /* * Packet sequence number, incremented each time sender sends a packet. * So receiver can detect small numbers of corrupt/dropped packets. */ uint8_t sequence : 4; /* Set if the sender had to discard packets due to buffer overflow. */ uint8_t overflow : 1; uint8_t placeholder : 3; /* Channel; values from enum console_channel */ uint8_t channel; /* Bottom 48 bits of system time; enough for 8 years @ 1 us */ uint8_t timestamp[6]; /* * Length of variable-length section in bytes, not including the * packet end trailer. */ uint8_t data_len; /* Index of the format string in the string database. */ uint16_t str_index; /* Header checksum */ uint8_t crc; /* Fixed length header to here. * * Followed by variable-length data, if data_len > 0. * * params: 1-8 of objects matching the format of the string indexed by * 'str_index' above. * * CONSOLE_PACKET_END, as a validity check that we haven't dropped * anything. A checksum or CRC would be kinda expensive for debug * data. Note that it is not present if data_len == 0. */ } ``` The data part of the packet is the concatenated values of the parameters processed in accordance with the format: all integer values less than 64 bits in size and pointers are transferred as 4 byte entities. In case the format calls for a 64 bit value, the parameter is interpreted as an address or an 8 byte value, which is retrieved from memory and added to the packet. In case the format calls for a string, the string is included in the packet, along with the trailing zero. Inclusion of `__func__` in the parameter list is a special case. The name of the function is added to the string dictionary, and 3 bytes are sent in the packet, 0xff and then the number `__func__` was assigned in the string database. This allows the terminal to tell that not the actual string but the index is sent as the parameter. ## Terminal program A Python script (`acroterm.py`) was copied from the Acropora project and modified to support the changed packet format. The script receives the name of the file containing the format strings blob as one of command lie parameters. When starting, the scrip attaches to the TTY device (be it UART or USB channel) and searches for the packet header characters in the stream. Symbols received before packet header is encountered or between packets are sent to the console directly. When packets are received, header integrity is verified and the `str_index` field from the header is used to retrieve the format string from the blob. The string is scanned for the format characters, and the data section of the packet is interpreted according to the format specification, recreated strings are sent to the console. Data received out of packets is sent to the console directly and is displayed using a different color. This, among other things, allows to display text generated by early boot stages and in general support builds which do not yet deploy packet mode.