TIP #409: UDP IN TCL ====================== Version: $Revision: 1.19 $ Author: Alexandre Ferrieux Colin McCormack State: Draft Type: Project Tcl-Version: 8.7 Vote: Pending Created: Friday, 17 August 2012 URL: https://tip.tcl-lang.org409.html Post-History: Obsoletes: TIP #391 ------------------------------------------------------------------------- ABSTRACT ========== This TIP adds support for UDP in Tcl, with a pure event approach to packet reception (as opposed to hijacking the existing channel infrastructure). RATIONALE =========== UDP support is a long-awaited feature in Tcl. Some extensions have traditionally supported it through the channel abstraction. This is of course doable, but there is a non-negligible cost in both complexity and performance due to the impedance mismatch between the "fluid" model behind channels and the "granular" world of datagrams (where boundaries are significant). Another discrepancy is the existence of (per-packet) metadata, like the source address and port of an incoming UDP packet, which do not fit nicely in the (per-connection) options of the channel API (via *fconfigure*). Once this mismatch is acknowledged, it is easy to identify a better home for UDP in Tcl: let it be a direct event source (for the receive side), just like Tk or Snack. Indeed, hooking a callback for packet reception is a natural fit with Tcl's event-driven tradition, while preserving packet boundaries and easily exposing per-packet metadata. Sending is trivially handled by a direct per-packet call (but not disguised as a *puts*). Again, this naturally allows for boundary preservation and metadata specification. This TIP asks for UDP-in-the core for two strong reasons: - it enables another important core feature: asynchronous (event-driven) DNS lookups - the C runtime supports it without extra libraries (unlike USB or Bluetooth) We then believe that the ubiquity of the need and the lack of external dependencies justify for UDP the first-class status that TCP enjoys (regardless of the channel vs event models). Note that an acceptable compromise would be a bundled extension, ie distributed-with-the-core (in "pkgs/"). SPECIFICATION =============== The new *udp* ensemble hosts three subcommands: *new*, *create*, and *names* *udp new* ?/options/? *udp create* /name/ ?/options/? both create an UDP endpoint ; *udp names* returns the list of existing such endpoints. LIFECYCLE ----------- Following the traditions of Tk and Snack, *udp new* and *udp create* return a Tcl command, which takes subcommands implementing the needed verbs. By analogy with the corresponding TclOO constructors, * *new* generates the command name internally, in the *::tcl::udp* namespace * *create* takes an extra /name/ argument providing the target command name, either fully qualified, or relative to the current namespace. An UDP endpoint is thus created by: set u [udp new ...] or udp create foo ... Once created, its configuration can be tweaked by: $u *configure* /-option/ /value/ and retrieved with set value [$u *configure* -/option/] or set fullconf [$u *configure*] To destroy the endpoint, use: $u *destroy* or, as in Tk, destroy the command: *rename* $u {} or the whole namespace. STATUS OF OPTIONS ------------------- Among options, a few may only be specified on creation: - local interface+port - (optional) connect() target's address+port - reuse flag (described in Advanced Features) while all the other options may be specified /both/ on creation or through [$u configure]. ADDRESSES AND PORTS --------------------- The first two creation-only options have the following syntax: set u [*udp new* ?*-local* /address_port/? ] set u [*udp new* ?*-remote* /address_port/? ] where /address_port/ pairs are represented as 2-element Tcl lists. set address_port [list $address $port] Addresses can be given either as numeric IP (v4 or v6) addresses, or as hostnames; in the latter case, a synchronous DNS resolution is performed before actual use, just like for *socket*. In the case of *-local*, $address can be specified as "*", meaning INADDR_ANY, and $port can be 0, asking for the OS to select a free port. Thus a dynamic port on all interfaces can be requested with set u [udp new -local [list * 0]] In case of port 0, after creation of the endpoint, the actual port chosen by the OS can be retrieved with [$u configure]: puts "Local port is: [lindex [$u configure -local] 1]" In the case of *-remote*, both address and port must be fully specified. The semantics, as is well known, is to tell the kernel (a) to forbid sending to any other destination, and (b) to discard all incoming packets sent by another source. SENDING A MESSAGE ------------------- Sending is done, unsurprisingly, with the *send* verb: $u *send* /payload/ ?/dest_addr_port/? where /dest_addr_port/ is a pair as above. Its blocking semantics is that of the underlying send()/sendto() system calls: it typically returns immediately, though the hardware may buffer the data for some time, and delivery is not guaranteed. The /payload/ is interpreted as a byte array that holds the entire content of the UDP message to send. The destination parameter can be omitted if the endpoint has been created with the *-remote* option (connected mode). RECEIVING A MESSAGE ASYNCHRONOUSLY ------------------------------------ Asynchronous (i.e. event-driven) message reception is done by specifying a listener callback to *new/create* or *configure*: $u *configure -listener* /command_prefix/ Subsequently, when an incoming packet hits Tcl in the event loop, the /command_prefix/ is called with the endpoint identifier, payload and metadata: {*}/command_prefix/ $u /payload/ /metadata_dict/ where /payload/ is the byte array of the UDP payload and /metadata_dict/ is a dict containing at least two options: *-remote* /address_port/ *-local* /address_port/ When read from options or metadata, all address components apart from "*" are returned in numeric form; no reverse DNS is ever performed. Note that /-local/ may carry more information than a configured /-local/ where the address part is "*", by identifying which of the system's several interfaces was targeted. Also note that /command_prefix/ is a single command, possibly with arguments, that will be expanded on invocation (hence it must be a proper list); it is *not* an arbitrary script as in Tk's *bind/ tradition. When /command_prefix/ is the empty list (which is the default), the notifier gives up interest in the underlying UDP socket; this allows to keep the port bound while letting the OS buffer any incoming packets (up to a configurable limit) without any script-level handling, while leaving the event loop active. This is similar to setting a *fileevent* on a channel to the empty string. RECEIVING A MESSAGE SYNCHRONOUSLY ----------------------------------- Synchronous message reception is done with the *recv* verb: set payload [$u *recv* /metadata_dict_var/] The payload is returned as a byte array, and the variable passed as argument receives the metadata dict just described. ADVANCED FEATURES ------------------- The following few verbs and options control extra IP features that are typically useful in popular UDP applications like media streaming: $u *configure -sendbuffer* /buffersize/ $u *configure -receivebuffer* /buffersize/ Set the OS's send (resp. receive) buffer sizes to the given values (in bytes). $u *join* /multiaddr/ ?*-source* /sourceaddr/? Joins the multicast group /multiaddr/, optionally using IGMPv3's source membership target /sourceaddr/. $u *leave* /multiaddr/ ?*-source* /sourceaddr/? Leaves the same multicast group with or without source membership. set u [*udp new -reuse 1*] Sets the SO_REUSEADDR socket option, so that multiple processes may bind to the same local port and interface, and receive the same packets. Creation-time only. $u *configure -ttl* /ttl/ Sets the TTL of subsequent outgoing packets. Some operating systems typically reduce the default TTL to one for multicast packets; this override may thus come in handy. REJECTED ALTERNATIVES ======================= * Token-based API (like for channels) instead of commands: would need extra machinery (a new Tcl_ObjType). Moreover, the command style gives reflection for free: a simple proc can emulate an UDP endpoint, and is compatible with all OO schemes. * Address+Port pairs as dicts or urls: would hamper performance for no added value * Listener as specific subcommand instead of option: R/W semantics is actually that of an option; a subcommand would preclude the direct specification of the listener on creation ; analogy with the -command of *fcopy*, *http*, or Tk widgets. SOURCES OF INSPIRATION ======================== This TIP builds on experience with previous attempts at UDP extensions: TclUDP, ceptcl, Duft, along with intensive practice with unpublished work, and borrows ideas from all of them. COPYRIGHT =========== This document has been placed in the public domain. ------------------------------------------------------------------------- TIP AutoGenerator - written by Donal K. Fellows