= Moving to GPSD-NG: a Guide for Client Developers = Eric S. Raymond v1.0, August 2009 == Why a new protocol? == GPSD is moving to a new request/response protocol. This move has been forced upon us because the old one ran out of namespace. It was designed with case-insensitive single-character command/response codes. 25 of 26 ASCII alphabetics were already in use by 2006, and there have been functional challenges accumulating over the last three years that will require several more request/response codes - things like reporting AIS data, reporting raw pseudoranges, and reporting RTCM3. Yes, we could as a desperate expedient have pressed non-alphabetic printables into service - but the result would have looked like line noise and only delayed the day of reckoning. Instead, we're writing a new protocol that is upward-compatible with the old one in that you can mix old and new syntax. The old protocol will continue to be available for some time, but not indefinitely. Eventually we're going to want to shed the code complexity and overhead of maintaining both protocols at once. This will happen sooner than it otherwise might have because gpsd is in part targeted for SBCs and other constrained environments where shaving a few K off the runtime image can actually matter. When it comes to keeping the codebase lean and mean, we try harder. We'll try to make the transition easy, but we cannot guarantee no problems. The sooner you start adapting your code, the less pain you are likely to experience. The rest of this document will explain both theory and practice, and give you specific questions on how to fix client code. == Virtue is rewarded == Since 2004, the way you were *supposed* to be using gpsd was through one of the client libraries (in C or Python). If you have been doing it right, you have been telling gpsd to stream data at you with the 'w' command, via application code probably looked something like this: ------------------------------------------------------------------- gpsdata = gps_open(source.server, source.port); (void)gps_query(gpsdata, "w+x\n"); // This is in your application's main event loop somewhere. // It polls gpsd for new data. ... gps_poll(gpsdata) ... gps_close() ------------------------------------------------------------------- If you have been virtuous, you need only to make one small change to your code. Replace the gps_query() call with: ------------------------------------------------------------------- gps_stream(gpsdata, WATCH_ENABLE) ------------------------------------------------------------------- This will tell whatever version of the client library your application dynamically links to emit shat it should under the hood, either old or new protocol. Unless a target system carries a version of the libgps shared library different from the gpsd version, everything should work and continue to work through future updates. There. You're probably done, unless you relied on some parts of struct gpsdata_t that application developers usually don't or issued unusual configuration commands. Here are the exceptions: * You issued other gps_query() commands, such as "J=1". If so, you'll need to learn how to say them in the new API. That is not difficult, and this document will cover what you need to know. * Your application code referenced struct gpsdata_t members that no longer exist. This will be obvious because your compilation will fail. Later in this document you will learn how to fix these problems. You can probably ignore the rest of this document, unless either (a) you want to learn about gpsd's new capabilities so you can use them in creative ways, or (b) you want to caper with unholy glee as you contemplate the trials awaiting the non-virtuous. If you are non-virtuous - that is, you rolled your own client-side interface - you've had years of warning that this choice would fail to insulate you from protocol details and cost you pain in the future -- and that future is now. In the remainder of this document we will try to help you minimize the pain. The main strategy for doing so is to *use libgps* (or its functional equivalents in languages other than C). Scrap your hand-rolled interface code! When you use libgps, compatibility issues become *our* problem to solve rather than *yours*. == When the bough breaks == Even virtuous clients have to worry about version skew. Supposing you have used libgps and not done anything exotic, you will still have problems if the client library you linked and the instance of gpsd it speaks to are using different protocols. We're working to minimize problems by scheduling the changes like this: 1. In 2.40 (shipping sometime in the fourth quarter of 2009), we will roll out a daemon that speaks both old and new protocols and a client library that speaks old protocol by default - that is, under the hood gps_stream() still says "w+" rather than "?WATCH". 2. When we have reason to believe all major Linux distros have deployed 2.40, we will change the client library to ship "?WATCH" by default. This is likely to happen in late 2010. 3. At some point in the far future, we will disable old protocol in the default builds of both client and daemon. This is probably at least three years or more out. The possible failure modes are pretty obvious. Transitions are difficult. We're essentially relying on the distribution integrators to ship libgps and gpsd updates at the same time, with sane package dependencies. If that goes smoothly, applications may not even notice the changes. We can hope... == On not doing things by halves == At the same time that pressure has been building to redesign the protocol, we've been gaining experience in gpsd's application domain that has made us rethink some of the assumptions behind the old one. Since we knew we were going to have a forced compatibility break at the wire-protocol level anyway, we decided not to do things by halves. One big break - in the application model, struct gpsdata_t, and the wire protocol behind it - is better than three or four spread out over a period of time. As a result, the new protocol is not an exact superset of the old one. It reflects a different way of carving up the behavior space in gpsd's application domain. And the shape of struct gpsdata_t, the client-side interface structure, has changed in corresponding ways. Accordingly, there are three things a client developer will need to understand about the new protocol. The first is theory: how its model of the gpsd application domain is different. The second is practice: how to issue new-style commands and interpret responses. The third, if you have relied on the structure in a way that now breaks your compile, is how that structure has changed. == How the theory has changed == === Channels are gone === In old protocol, when you requested data from the daemon, it would search for a device supplying the kind of data you had told it you wanted (GPS, by default) and connect your listening channel to *that single device*. The association between channel and device was set when channel was first bound to device and implicit; reports weren't labeled with the device name. You could request a specific device if you wanted to. In the new protocol, channels are gone. You tell gpsd to stream reports at you; thereafter, every time an attached GPS or other device generates a report, you'll get it. There may be multiple devices reporting; each report will come labeled with the name of the originating device, and that name will be left in your client structure along with the rest of the new data. In both protocols, when you poll gpsd and get data the client library takes care of interpreting what comes up the wire from the daemon, and merges the resulting data into your client structure (struct gpsdata_t). The difference is that before, the API locked you to one device during the life of the session. Now it potentially has to deal with a *set* of devices, treated symmetrically. There are multiple reasons this change is a good idea. One is that it makes coping with devices being hotplugged in or out completely trivial from the client's point of view - it can just choose to ignore the fact that the device IDs in the reports have changed. Also, the channel-management hair in the interface goes away. Also, it means that clients can treat identically the cases where (a) you have one device reporting different kinds of data (e.g. a marine navigation system reporting both GPS and AIS) and (b) you have several devices reporting different kinds of data. === From lockstep to streaming === A subtler change has to do with the difference between a lockstep or conversational interface and a streaming, stateless one. In the earliest versions of GPSD, clients requested various pieces of data by command. After each request, they would need to wait until a response came back. Then, watcher mode was added. By saying "w+", you could ask gpsd to stream GPS reports at you whenever it got them. In the new protocol, streaming is almost all there is. Every report coming up from the daemon is tagged with its device and type. Instead of issuing commands and then ewaiting for specific responses, clients should expect any kind of report at any time and merge it into client-local storage (libgps does this for you). This change is necessary to cope with devices that may send (for example) mixed GPS and AIS data. In the future, the stream from gpsd could include other kinds of data, such as the take from a digital compass, water-temperature sensors, or even aircraft transponders. == How the command set has changed == If your code issues old-protocol commands A, D, E, M, P, T, U, or V, it is a wretched hive of scum and villainy that probably hasn't changed since before the introduction of "W". You are using the oldest single-shot commands and will have to rewrite your interface significantly, as the new protocol does not support equivalents. Use libgps. If your code issues B, C, or N commands, they need to change to ?DEVICE commands. See the protocol reference for details. The old 'F' and 'G' commands do not have equivalents. It would be possible to implement these, but we probably won't do it unless there is actual demand. Consider teaching your client to ignore fix updates when they don't have a specified "device" or "class" tag, respectively. The old 'I' command has no equivalent. You probably issued it as part of an initialization string, hoping that a subtype string would later show up in gps_id so you could post it somewhere. In the new protocol, when a device sends back subtype information the daemon ships the client an object of class DEVICE with a device tag and subtype fields. Watch for that and process appropriately. The old 'J' command has been replaced by an optional attribute in ?WATCH. Include the WATCH_NOJITTER mask in the argument of gps_stream(). The old 'K' command is replaced by ?DEVICES. The old 'L' command is replaced by ?VERSION. Note that the daemon now ships a version response to each client on connect, so it will probably never be necessary for you to issue a ?VERSION request. The old 'O' and 'Y' commands are gone. Use ?WATCH and sample the stream instead. The old 'Q' and 'S' commands doe not have exact equivalents. We will implement an option to have the TPV response include DOPs and status if there is demand for them. The old 'R' command has been replaced by an optional attribute in ?WATCH. Include the WATCH_RAW mask in the argument of gps_stream(), or set a raw hook before alling gps_stream(). The old 'W' command has been replaced by ?WATCH. Call gps_stream() with whatever option you want to set. The old 'X' command is gone. Instead, you will see an object of class DEVICE from the daemon whenever a device is opened or closed. The old 'Z' and '$' commands, used by the developers for profiling, have no equivalents. We'll implement them if we need them. == How the gps_data_t structure has changed == The client libary's reporting structure, struct gpsdata_t, has a new substructure (struct devconfig_t) named "dev" that groups together information about the device that shipped the last update to the client. The members of this structure replace several top-level struct gpsdata members in older versions. Most notably, the gps_device member has been replaced by dev.path. It is valid after every response with a device tag (DEVICE, TPV, SKY, AIS, RTCM2, RTCM3). The top-level gps_id member is replaced by dev.subtype. This data should be considered valid only when DEVICEID_SET is on in the top-level set member. The dev members baudrate, parity, stopbits, cycle, mincycle, and driver_mode replace older top-level members. They should be considered valid only when DEVICE_SET is on in the top-level set member. The top-level members ndevices and devicelist (used only on the client side) have been replaced by an array of struct devconfig_t structures. Data in this structure should be considered valid only when DEVICELIST_SET is on in the top-level set member. Storage for pathnames is no longer dynamically allocated, but static; to save space, it lives in a union with several other substructures.