-- HaldunKomsuoglu? - 26 Aug 2003

As a suppliment to the SenseCompComm page this page serves as an open discussion media to determine the general requirements for communication infrastructure of RiSE independent of the implementational details. Reader can refer to USBorI2C for a comparison of two possible implementation path for communication.

The text is partitioned into blocks where each block starts with a subject header followed by a threaded bulletted list of comments concerning the subject. Please, maintain the threaded structure to make it easier to parse thru the comments. Refer to TextFormattingRules for syntax guidelines.

Types of Communications and Their Trade-offs

• Devices require different types of communication: (A1) moving periodic samples with a low jitter, either CPU->device or device->CPU; and (A2) message passing CPU<->device with "guaranteed" arrival. Messages should be error resistent (i.e. have checksums). In general, the "higher-level" a device is, the more of (A2) it will use instead of (A1). (A2) trades more processing overhead for lower average bitrate. Clarification: in the computer communication terminology used here guaranteed arrival means that messages either arrive with no errors within a finite time, or an error condition occurs and the messages are dropped; i.e. messages are never corrupted and losses are always indicated by error conditions. -- ShaiRevzen? - 26 Aug 2003
  • My presumption was that we would tend to prefer a strictly master/slave bus structure, with all communication initiated by the master (i.e. central CPU) and the slaves (sensors/actuators) simply subordonate to the needs of the main CPU. -- AlRizzi? - 26 Aug 2003
    • The actual bus arbitration (on the electrical level) is outside the scope of what I was trying to describe. There is a general tradeoff in communication between delivery time variation (jitter) and reliable delivery; what I meant to say is that our library should support both ends of the spectrum -- low jitter best effort and high reliability asynchronous delivery. Either can be realized over a variety of bus structures, e.g. I2C? with a single master. -- ShaiRevzen? - 26 Aug 2003
    • I believe the usage of the arrow (-->) in "CPU->device or device->CPU" is misleading. The looks as if in A-->B the transaction is initiated by A and accepted by B. However, I think instead Shai is only refering to the direction of information flow. This being said I just point out that I share the same view with Al on low level channel arbitration, i.e., single master multiple slave setting where both read and write transactions are initiated and controlled by the master. -- HaldunKomsuoglu? - 26 Aug 2003
  • I have also presumed that error checking and delivery assurance would all be done at the "application" level, rather than in the communication protocol. -- AlRizzi? - 26 Aug 2003
    • Here I disagree quite strongly. There should be a consistent abstraction of a "communication link" provided by our library code and the communication protocol it implements. This will evolve over time, but should be decoupled from the "application level". -- ShaiRevzen? - 26 Aug 2003
    • I presume we are refering to the integrity of a data package in this item. Then, I suppose it would be simpler to implement the error check in the communication layer. Depending on implementation this task can even be given to the bus interface circuitry which would decrease the computational load on the main CPU. Furthermore, assuming perfect communication at the application layer would simplify the design of the code at this level.
  • I am not certain what a "device" refer to in this comment but it will prove to be crucial that we have multi-casting and broad-casting capabilities as well as addressing individual modules.-- HaldunKomsuoglu? - 26 Aug 2003

Timing of Communications and Associated Trade-offs

• Control software generally requires a pre-fitered input. This input arrives in one of the following modes: (D1) periodically, throughout robot operation; (D2) periodically, for limited spurts (i.e. some corrective behavior get triggered that requires a periodic update, then the goal is reached and updates are no longer required); and (D3) irregularly. If control software inputs of the sort (D3), and of the sort (D2) with low duty cycle are predominant, we can save bandwidth and CPU load by offloading more processing onto the peripheral processors -- otherwise, there is little point in it. -- ShaiRevzen? - 26 Aug 2003
  • I think the issue that needs to be addressed is synchronous vs. async operation of the entire bus system. If we stick to a purely synch system bandwidth and communication scheduling are relatively easy to manage, at the cost of wasted communication to "poll" for what are essentially asynchronous events. Using a structure that supports async communcation will tend to be less predictable, and requires bus arbitration strategies at some level in the comm protocol. -- AlRizzi? - 26 Aug 2003
    • See remark above re. electrical arbitration of the bus; both I2C? and USB are synchronous, but USB has a standard (hardware controlled) scheduling system implemented on top of that. For I2C? and serial we need to do this ourselves -- ShaiRevzen? - 26 Aug 2003
  • Presuming that our network arbitration is done in a master/slave manner all transaction (both read and write) will be initiated by the master. In this setting to be able to recieve an asynchronous message from a slave module the master has to poll it in any case. Therefore, a synch and an asynch design would not differ in terms of their bandwidth usage in getting messages from slaves in this network. -- HaldunKomsuoglu? - 26 Aug 2003
  • In my opinion we should favor a synchronous design. As Al pointed out the scheduler design and the network efficiency would be much greater in such a case. Furthermore, the nature of the communication on a robotic platform is periodic which would be better served by a synchronous network arbitration. To handle asynchronous messages I propose to simple piggy them back to the basic synchronous messages. Note that this approach would take care of messages that needs to be passed in both directions, i.e., master to slave and slave to master.-- HaldunKomsuoglu? - 26 Aug 2003
  • That said, I see little difference between D1 and D2. In the D1 case the epoch of interest just happens to be the entire run of the robot. -- AlRizzi? - 26 Aug 2003
• Add your comment here

Computation Requirements and Tradeoffs

• If pre-filtering is computationally heavy, we can probably gain more by a stronger CPU or DSP processors for peripherals than we can by using stronger general purpose controllers for peripherals.-- ShaiRevzen? - 26 Aug 2003
  • I am not sure I follow this? Given our rapid development schedule I think we will need to chose one or two candidate processors for our "remote nodes" and quickly produce candidate designs for their integration with hardware. Mostly I am concerned that we do not get in a situation where we will need to "custom" design a new "node" for each new piece of hardware. -- AlRizzi? - 26 Aug 2003
    • What I meant was that if the sensor input require heavy processing a "general purpose" microcontroller makes less sense in the remote nodes; remote nodes should be algorithmically simple, because we only gain if they do not require many software updates. Think "DSP" rather than "CPU" for a remote node... -- ShaiRevzen? - 26 Aug 2003

• • • I agree with Al. To accelerate and ease the hardware development, which will be a key enabler in RiSE project, we should supply developers with a couple template designs for the remote nodes, hence, free their minds from the low level details of the communications. This design should include: 1) the basic circuit layout including the microcontroller that implements the hardware interface with the communication network; 2) communication protocol library for the embedded system; 3) a basic task scheduler for the jobs that each developer will add-on; and 4) the interface code for the main CPU. I can imagine making available several versions of this design utilizing chips from an MCU family with different computational and peripheral features.-- HaldunKomsuoglu? - 26 Aug 2003
  • "Remote nodes" in my mind should be pretty powerful, i.e. support multiple I/O channels; the software should represent these channels are separate entities, thereby allowing us to migrate physical devices across remote nodes. -- ShaiRevzen? - 26 Aug 2003
    • This may not be the case at all times. I can easily imagine cases where we need to place a seperate remote node due to the physical location of the sensing job which may be rather simple, for instance detection of some events. Hence, a node can be a very simple guy doing a very simple thing. -- HaldunKomsuoglu? - 26 Aug 2003
    • Ok, I think we need to separate "logical" and "physical" nodes in this case. We can probably stick two logical nodes (with their corresponding I2C? interfaces) on the same board. Also, the decompisition of these nodes in the higher level software does not need to correspond tightly to the physical implementation. Our hardware abstraction layer can provide other means of grouping these logical entities, even decompose their constituents into different abstractions.-- UlucSaranli? - 28 Aug 2003
• Given that development for deeply embedded components for sensor nodes will be significantly more painful than a general purpose CPU running QNX, I would favor relatively simple processing for these nodes. The load on the central CPU for RHex partly comes from simple tasks that add up because we have six copies of each. I suspect this will be similar in RiSE. Below is a list of sensors and local processing tasks that I would think of doing first. -- UlucSaranli? - 28 Aug 2003
  • Encoders: It makes sense to do the periodic reading of 16bit encoder counts and handle the conversion to joint angles locally.
  • Motor drives: It probably makes sense to implement basic PID control locally, as well as providing a feedforward term that can be changed from the central CPU. Depending on the electronics, back EMF compensation and implementation of a motor model can also be done locally. This would also yield estimates of motor armature current, voltage and other unmeasured quantities (see current RHexLib implementation)
  • Battery current and voltage: We could implement battery charge state monitors well as local functions such as warning lights for low battery locally. Of course, conversion to metric units would be best done locally.
  • Gyros and accelerometers: I think the only local computation we should do here is unit conversion following calibration. I believe state estimation and kalman filtering at local nodes will be very hard to implement in our time frame. One could also imagine some primitive filtering for each axis such as combining 2G and 10G measurementsto eliminate saturation and redundancy, as well as basic signal processing.
  • IR sensors? : Unit conversion and failure detection.
  • ... other sensors ...
• Add your comments here

I/O System Layers

• I suggest the following sort of I/O system layers:

  1	Hardware interface comprising the following API-s:
  1.1	class HWInputSignal implementing a getSample()-->value method
  1.2	class HWOutputSignal implementing a putSample(value) method
  2	Signal abstraction, isolating the layers above it from hardware timing, and providing a "naming" scheme (e.g. id numbers) for signals
  2.1	class InputSignal implementing getSamples()-->[times,values]
  2.2	class OutputSignal implementing putSamples([times,values])
  3	Filtering layer, providing a many-to-many remapping of InputSignal -s to their filtered values, and allowing multiple filtered signals to share the same InputSignal data source.
  3.1	class FilteredSignal with setFilter([filter coefficients]) and getValue()-->[time, value]

• • Someone need to explain this to me more. I see a sensor or actuator node as a small computational blob that can report (on command) its state (values of signals it supports and filtered versions of them). My model of how this works goes something like this
    1. At start up the main CPU resets all nodes and issues configuration commands (telling nodes what filters to run, how to paramaterize them, and what data should be reported in response to a query).
    2. At the top/bottom of every cycle the main CPU polls each remote node for data and updates its internal "state" structures, reflecting the current state of the system.
  • Of course nodes could be reconfigured at any time during operation
  • If everything I am talking about is "below" the level of abstraction described here, then ignore me. -- AlRizzi? - 26 Aug 2003
  • I saw this a bit differently:
    1. At startup the main CPU boots all remote nodes. They register their InputSignal and OutputSignal objects with the main CPU software.
    2. As part of "application startup" the application binds various FilteredSignal objects to these. This operation builds the required "remote node" processing instruction transparently.
    3. While running "remote nodes" do their magic and process the inputs and outputs, updating a "remote state". In later stages this may also include closing some control loops in the "remote node" without main CPU participation, e.g. stopping a leg when the foot's tactile sensor is engaged.
    4. When the main CPU has time to so, it may poll the FilteredSignal objects, generating a communication exchange with the remote node.
    5. Alternatively, a FilteredSignal sub-class may be set up to update periodically -- but this only makes sense if nearly all the values thus obtained are actually used (it is pointess to read a remote sensor over and over again into the same "mailbox" if you don't actually use the value....) -- ShaiRevzen? - 26 Aug 2003

• I favor a more RHexLib style interface where I think of each remote node as a "hardware equivalent" of a RHexLib software module. This approach would allow the software modules (running on board PC) and the functionalities of these remote nodes to blend very smoothly. Below I attached a base class, BusModule, definition to implement interfaces for remote nodes which is driven from the Module class from RHexLib. Obviously, for the BusModule the update( ) will have no function since these periodic actions are implemented and performed by the sensor/actuator module MCU remotely. The remaining basic module functions (init, activate, deactivate, uninit) are merely requests to the hardware module for the associated action. Here "command( )" call represents the bus access functionality where the request is represented by a standard OpCode known by all hardware modules. In addition to the standard RHexLib module functionality there are two other functions (get and set) to facilitate data exchange. -- HaldunKomsuoglu? - 27 Aug 2003

     class BusModule : public Module {

     public:

             BusModule  ( char *name, int index, bool singleUser ) : Module( char *name, int index, bool singleUser ) { };
             ~BusModule ( void ) { };

             init       ( void ) { command( INIT       ); };
             activate   ( void ) { command( ACTIVATE   ); };
             update     ( void ) { };
             deactivate ( void ) { command( DEACTIVATE ); };
             uninit     ( void ) { command( UNINIT     ); };

             get        ( char pIndex, char * val );
             set        ( char pIndex, char   val );

     private:

             enum opCode_t {
               INIT       = 0,
               ACTIVATE   = 1,
               DEACTIVATE = 2,
               UNINIT     = 3
             }

             command ( opCode_t cmd );
     }

• Add your comments here

Communication Model

* And the following sort of communication system model: (1) Each communication endpoint (regardless of communication method) will be identified by a "port number"; I suggest an API similar to the socket() bind() connect()/accept() recv() send() for these. This will make it easier to test and simulate using TCP/UDP sockets. This approach also allows us to put different devices on different busses if hardware issues require it (e.g. different electrical properties, or bandwidth limitations) without changing the software. (2) Connections may be either "message based" or "constant bitrate" (CBR) stream. Either of these is easy to implement over any infrastructure we choose. (3) Message based is similar to UDP (with a checksum) - messages are send as blocks of data; any corruption causes the related message to dropped. (4) "aligned stream" is similar to TCP, but sends data at a fixed bit-rate and guarantees data framing. For example, if you align to 4 byte boundaries and a byte is corrupted, the whole 4 byte block is lost. If you queue a 16 byte message, it will be sent as four 4 byte messages. If you omit sending a message, this is equivalent to re-sending the last message - i.e. the sampling rate is held fixed. -- ShaiRevzen? - 26 Aug 2003

• • I had been expecting a strictly messge based scheme, with a well defined standard (and extensible) protocol for all quiries/commands, but that goes back to the sync/async communication question. -- AlRizzi? - 26 Aug 2003
  • Generally speaking, messages make sense only if there is enough "intelligence" to interpret them on the other side. CBR makes sense if one of the parties is "dumb". This is because a good messaging protocol has overheads that only make sense if the data being carried is "valueble" enough. -- ShaiRevzen? - 26 Aug 2003
  • In my opinion the periodic operating nature of the remote nodes imposes the need for devoted CBR channels for each source and sink. Messages for those modules that are periodically accessed can be piggied back to their periodic transactions. A problem arises if a remote module have no periodic communication but only needs to send/rcv async messages then we will need some message passing mechanism. First of all in my opinion number of such modules will be very small. Second, choosing a sych operating regime for our network and just polling these modules at a rate suitable for their priority would be the simplest way to go. Furthermore, simplicity and efficiency of a synchronous communication scheduler at the master arbirar might even counter balance the bandwidth loss due to polling. -- HaldunKomsuoglu? - 27 Aug 2003
• Add your comments here

Communication Model Implementations

* Assuming the above architecture we could implement two different mappings for it, one based on an underlying I2C? , and another based one a "simple" serial connection. This will allow all other parts of the system to be ignorant of the underlying communication medium, and will allow us to "run" modules off a regular PC, without special I2C? interface hardware. By doing this we get the best of both the USB and the I2C? worlds. -- ShaiRevzen? - 26 Aug 2003

• • I think we can achieve this (and that it is important that we do) independent of most of the questions I have raised here. -- AlRizzi? - 26 Aug 2003
  • If I did not misunderstand this requires each remote module to implement physical interfaces for both RS232 and whatever architecture we will choose for RiSE. In my opinion, this puts unnecessary requirements on the design of the remote node hardware as well as firmware. I don't see what is fundamentally wrong with using a piece of adapter circuitry that will connect the module under development to a PC. (In the case of USB such a device exists but hidden in the chassis of the PC.) This device will mostly like the bus arbitrar in the physical robot anyways and its connection to a PC can be over a simple RS232 as well. In the development system we can very easily hide this hardware detail behind the API. -- HaldunKomsuoglu? - 27 Aug 2003
  • In my opinion, using two different physical interface, say a serial and a fancy high speed one, would be very dangerous. Suppose you use the former for development and the later for the run-time. Any unforeseen difference in their physical implementation and their firmware can cause very hard to track bugs in run-time. -- HaldunKomsuoglu? - 27 Aug 2003
• Add your comments here