TIP #55: Package Format for Tcl Extensions

TIP:55
Title:Package Format for Tcl Extensions
Version:$Revision: 1.18 $
Authors: Steve Cassidy <Steve dot Cassidy at mq dot edu dot au>
Larry W. Virden <lvirden at yahoo dot com>
State:Draft
Type:Informative
Vote:No voting
Created:Thursday, 16 August 2001

Abstract

This document specifies the contents of a binary distribution of a Tcl package, especially directory structure and required files, suitable for automated installation into an existing Tcl installation.

Rationale

There is currently no standard way of distributing or installing a Tcl extension package. The TEA document defines a standard interface to building packages and includes an install target but presumes that the packages is being installed on the same machine as it was built. This TIP defines a directory structure and assorted files for the binary distribution of a package which can be placed into an archive (for example zip or tar file) and transferred for installation on another machine. A basic mechanism for installation of packages is also described.

Definitions

The following definitions are excerpted from TIP #78:

package

A collection of files providing additional functionality to a user of a Tcl interpreter when loaded into said interpreter.

Some files in a package implement the provided functionality whereas other files contain metadata required by the package management of Tcl to be able to use the package.

distribution

An encapsulation of one or more packages for transport between places, machines, organizations, and people.

shared library

A piece of binary code that provides a set of operations and data structures like a normal library, but which does not need to be physically incorporated into the executables that use it until they are actually executed. This is the normal way to distribute binary code for a Tcl package such that it can be incorporated into a Tcl interpreter with the load command. On Windows, shared libraries are known as DLLs, on the Macintosh ...

References

Much of the required structure for an installable distribution is defined by the requirements of Tcl's existing package loading methods. The structure of an installable distribution should largely mirror the structure of an installed package where possible.

The R system (a statistical package [1]) has a well defined package format which enables automatic installation of new packages and integration of documentation and demonstration programs for these with that of the main R system.

A number of packaging and installation systems (for example, Debian [2] and RPM [3]) have been developed by the Linux community which provide an interesting range of facilities. These systems commonly provide facilities for pre and post installation scripts and pre and post removal scripts to help set up and shut down packages. Also included are detailed dependency relations between packages which can be used by an installer to ensure that a package will work once it is installed or warn of potential conflicts after installation.

A significant part of this proposal is the proposed format of the package metadata which derives from other metadata standardisation efforts, mainly the Dublin Core [4] and the Resource Description Framework [5].

Requirements

The simplest case of a Tcl package is one that contains only Tcl code; these will be considered first, and the additional issues raised by packages containing compiled code will be dealt with later.

The minimum contents of a Tcl only package are defined by the requirements of [package require xyzzy]. The package needs to be placed in a directory on the auto_path and must contain one or more .tcl files which implement the functionality provided by the package.

In addition to these files, it is useful to include documentation for the commands implemented by the package and some additional metadata about the author etc. Distributions might also optionally include demonstration scripts and applications illustrating their use, these could either be incorporated into the documentation or included as stand-alone Tcl files.

Distributions which include shared libraries add an additional layer of complexity since these will only run on the platforms for which they have been compiled. There are two clear options here: either distributions are platform specific, intended for installation on one platform alone, or the structure of the distribution is extended to allow the option of including multiple shared libraries. The latter option would allow a single installation to serve multiple platforms and so should be preferred although this TIP will not require a distribution to support multiple platforms.

Proposed Directory Structure

The following directory structure is proposed for an installable distribution:

  packagename$version
      + DESCRIPTION.txt  -- Metadata, description of the package
      + doc/             -- documentation
      + examples/        -- example scripts and applications
      + $architecture/   -- shared library directories
      + pkgIndex.tcl     -- package index file (optional)

In addition, a distribution may include any additional files or directories required for its operation.

DESCRIPTION is a file containing metadata about the package(s) contained in the distribution. Its format will be described in a later section of this document.

The file pkgIndex.tcl currently required by the package-loading mechanism of the Tcl core is optionally distributed. In most cases, it will be generated by the installer; all the information which is necessary to do this is part of the distribution. Distribution authors should only include pkgIndex.tcl if special features of their distribution mean that the generated file would not work.

If the pkgIndex.tcl file is included in the distribution it should load files from their locations within the distribution directory structure. For example, Tcl files should be loaded from the tcl directory.

doc/ directory contains documentation in an accepted format. Currently Tcl documentation is delivered either in source form (nroff or TMML) or as HTML files. Given the lack of a standard cross platform solution, this TIP does not require a specific format; however, the inclusion of either a text or HTML formatted help file is strongly encouraged. If HTML formatted help is included the main file should be named index.html or index.htm so that it can be linked to a central web page. If only plain text documentation is included there should be a file called readme.txt (in either upper or lower case) which will serve as the top level documentation file.

examples/ directory contains one or more Tcl files giving examples of the use of this package. These should be complete scripts suitable for either sourcing in tclsh/wish or running from the command line. The examples should be self contained and any external data should be included in files in this directory or a sub-directory. This directory should contain a file readme.txt which explains how to run the examples and provides a commentary on what they do.

$architecture directories contain shared libraries for various platforms. The special architecture tcl is used for Tcl script files. They either implement the package or contain companion procedure definitions to the shared libraries of the package.

The distribution need not provide all possible combinations of architectures and may only provide one shared library. This structure is proposed to allow shared libraries to co-exist in a multi-platform environment and to allow binary packages to be distributed in multi-platform distributions. The architectures included in the distribution should be named in the DESCRIPTION.txt file.

The possible values of $architecture and methods for generating them are discussed in a later section.

Metadata

This section defines the metadata describing the package contained in the distribution in a format-neutral way. The model for this data is that provided by the Resource Description Framework (RDF [6]) which defines a triple based data model. The RDF model defines objects, their properties and relationships between them. In addition, where possible, element names are taken from the Dublin Core Metadata Element Set [7] which defines a standard set of element names for metadata. Dublin Core names are marked with DC in parentheses in the following list.

In a package description, the object being described is the package itself, hence the element names are all intended to describe packages. Other objects might be described including people and organisations. The package description should not include these objects but a package repository might store them separately keyed on the values stored in this description (e.g. email addresses of creators).

The following Dublin Core elements were not included in the standard set above but may be used in a package description if appropriate.

Encoding of the Metadata

The primary means of storing RDF data is using XML but it can be stored in many other formats. This TIP prescribes a simple text based encoding according to the RFC 2822 format which is described in this section. Data stored in this format can be converted to XML format for use by other tools, similarly XML formatted descriptions can be converted into this text format without loss of information.

The text format description is stored in the file DESCRIPTION.txt. The XML formatted version of the data may be stored in the file DESCRIPTION.rdf within the archive and may be automatically generated if not present.

The general format of this file is that of a RFC 2822 mail message, without body and using custom headers. The available headers are the case-independent logical names from the preceding section but may be augmented by other fields defined by repository maintainers or other applications. The headers are allowed appear in any order.

Example:

  Identifier: stemmer
  Version: 1.0.0
  Title: A stemmer for English.
  Creator: Steve Cassidy <[email protected]>
  Description:   Provides a procedure to remove any prefixes or suffixes on
         a word to give the word stem. Uses Porter's algorithm to do this
         in an intelligent manner with an accuracy of around 80%.
  Rights: BSD
  URL: http://www.shlrc.mq.edu.au/emu/tcl/
  Available: 2001-08-16
  Architecture: tcl
  Subject: linguistics
  Subject: text

Combination Distributions

It is often useful to combine a number of related packages so that they can be installed together to provide a certain kind of functionality, for example, web page production tools or database access. Perl uses the term Bundle to refer to such a group of related packages. There are two alternative mechanisms for distribution of such a package within the mechanisms suggested here. Firstly, since a distribution may contain more than one package, the set of files making up the various packages could be combined together and described by a single DESCRIPTION.txt file. This is similar to the way that tcllib is currently distributed. The disadvantage would be that all of the Tcl files implementing these packages would have to reside in the same directory which could cause name clashes.

The second alternative is to create a distribution consisting of only a DESCRIPTION.txt file to describe which Requires the component packages causing them to be installed from the repository. For example, tcllib might be described as follows:

  Identifier: tcllib
  Version: 1.0.0
  Title: The Standard Tcl Library
  Description:  This package is intended to be a collection of Tcl
             packages that provide utility functions useful to a large
             collection of Tcl programmers.
  Rights: BSD
  URL: http://sourceforge.net/projects/tcllib
  Contributor: Andreas Kupries  <andreas_kupries at users dot sourceforge dot net>
  Contributor: Don Porter <dgp at users dot sourceforge dot net>
  Require: base64
  Require: cmdline
  Require: csv
  ...

Installing tcllib would cause the installer to fetch base64, cmdline, csv etc from the repository and install them in order to satisfy the tcllib requirement. A new pkgIndex.tcl file could be constructed to load all of these packages if [package require tcllib] was called.

Architecture

Possible values for $architecture in the directory structure include:

Installing Packages

A package structured according to this TIP can be installed using the following steps:

  1. Download the package archive (eg. zip file)

  2. Locate a writable directory included on $auto_path (or ask for a installation directory)

  3. Unpack the archive in the desired location.

  4. Run pkg_mkIndex with appropriate arguments to generate a pkgIndex.tcl file if none is present. Arguments will include the appropriate Architecture directories for the platform.

  5. (optional) link help files and demos to the central index.

Alternatives

Alternatives might be considered for the package DESCRIPTION.txt file, for the documentation directory and for the location of shared libraries.

An alternative for package description file is to include an alternative package description, for example the XML based ``ppd'' format used to describe Perl packages on the ActiveState Perl package repository. The main motivation for the simple format proposed is that it is trivial for authors to write and trivial for programs to read and can be transformed into standards based RDF XML. The use of the DC element names means that search engines etc. will be able to usefully index the packages in a repository.

Note that the ppd format could still be used to describe packages stored in a repository for installation and that some of the information required to build the ppd format could be derived from the description file.

In the R package format referenced earlier, documentation is included in a standard source form and is converted to HTML or text based help pages; these might be included in the package or derived from the source forms on installation. The closest option for Tcl would be to require nroff format help files which can be converted to HTML or text files on installation. Unfortunately there is no guaranteed tool to do nroff->X conversion on Windows or Macintosh platforms. Until there is an accepted way of authoring Tcl documentation this TIP defers any standard layout of these files in an installable package.

The alternative to having shared libraries in specific directories is to have separate packages for each new platform. This has the advantage of making the packages smaller and more closely correspond to the existing directory structure of an installed package. The main motivation for the suggested directory structure is to allow multi-platform packages or to facilitate multi-platform installations.

Supporting Tools

The standards outlined in this TIP should be supported by Tcl scripts to:

In addition, the TEA standard should be extended with a package makefile target which will act like the current install target but which will copy files to a local directory and optionally build an archive of the package for distribution.

Copyright

This document has been placed in the public domain.

Powered by Tcl[Index] [History] [HTML Format] [Source Format] [LaTeX Format] [Text Format] [XML Format] [*roff Format (experimental)] [RTF Format (experimental)]

TIP AutoGenerator - written by Donal K. Fellows