3 Overview

Upipe provides an extensive set of header files, to be used by pipes and applications. Applications will also generally want to link with optional libraries, which allow to create and manage some core data structures.

Upipe's data structures can be classified into four groups, as shown by the following graph which depicts dependancy relationships between structures (dashed lines are optional):

Core structures provide services for all other groups:

• atomic operations with or without locks (uatomic_uint32_t)

• reference counters (struct urefcount)

• chained lists (struct uchain) and rings allowing pointer tagging (struct uring)

• lock-less LIFOs (struct ulifo) without thread limitations

• lock-less FIFOs (struct ufifo) without thread limitations

• lock-less buffer queues (struct uqueue) without thread limitations

• lock-less buffer pools (struct upool) without thread limitations

• lock-less exclusive access to non-reentrant resources (struct udeal)

• objects to pass event information between threads (struct ueventfd)

• dictionaries storing key/value pairs (struct udict)

• memory allocation with or without pools (struct umem)

• access to monotonic or real-time system clock (struct uclock)

Buffer management structures define how data is carried across inside a pipeline and are handled by struct ubuf and associated managers. These structures typically point to a shared, reference counted area maintained by the manager. Upipe natively supports three types of buffers, each with a dedicated API to access and modify data :

• blocks (arbitrary octet content with arbitrary and variable size), typically used for encoded data: ubuf_block_alloc ; the API supports windowing and resizing blocks, as well as merging and cutting out with zero-copy and copy-on-write semantics.

• pictures, defined by their chromatic planes, pixel sizes and subsampling, with arbitrary width and height: ubuf_pic_alloc.

• sounds, in planar or packed formats: ubuf_sound_alloc.

Pipes and applications however do not manipulate struct ubuf directly, but struct uref, which is composed of a pointer to struct ubuf and a pointer to struct udict. The dictionary allows to associate arbitrary attributes to the struct ubuf.

An attribute is defined by a category, a name, a type and a value. Available types are:

• opaque: stores a buffer of an arbitrary length

• string: stores a string terminated by \0

• void: doesn't store anything but the presence or absence of the attribute (flag)

• bool: stores a true or false value

• small_unsigned: stores a uint8_t

• small_int: stores an int8_t

• unsigned: stores a uint64_t

• int: stores an int64_t

• rational: stores a struct urational

• float: stores a double

The standard Upipe distribution provides managers for block and picture formats, relying on application memory allocation (malloc). They in turn use the struct umem_mgr facility to allocate buffer spaces.

Pipes are provided by specialized modules, possibly delivered by third-parties. A struct upipe may expose input and control methods (with standard and custom commands), and would typically take data from its input, process it and output it, possibly using a different buffer, to its output. Functions from a pipe may only be called from a single thread, so locking and reentrancy isn't required.

When created, pipes are passed one structure for logging and sending exceptions (struct uprobe) to the parent code upon certain events (end of file, fatal error, new flow, etc.), and optional arguments, in particular, for pipes that aren't sources, a flow definition packet of type struct uref describing the input. Pipe allocation is performed with upipe_void_alloc or upipe_flow_alloc, depending whether the pipe requests a flow definition packet. A few pipes may also provide their own specific allocator.

The standard Upipe distribution currently contains the following pipe types (though third-party modules may also be used):

Data is fed into a pipe using upipe_input. The struct uref argument then belongs to the callee and shouldn't be used any longer. There is an additional struct upump argument that points to the pump that generated the buffer (or NULL if unavailable).

The generic upipe_control call provides the application with an interface to modify the pipe's property. The counterpart of this function is provided by struct uprobe, which allows the pipe to send messages to the application. It is possible to build upon this messaging system to dynamically take actions on the pipe or the pipeline. All parameters to upipe_control belong to the caller. All parameters sent by subpipes via probes also belong to the subpipe. The standard Upipe distribution currently contains the following catchers:

In Upipe's design, decision taking happens inside probes, while data processing is done in pipes with no intelligence whatsoever.

Control commands and pipes are classified (that is, enum values are prepended with a prefix) into 7 categories:

• generic: commands and probes which can apply to any type of pipe (no prefix)

• source: for pipes that have no input, but rely instead on external events to retrieve incoming data

• join: for pipes that have several inputs, such as a mux

• split: for pipes that have several outputs, such as a demux

• sink: for pipes that have no output and may rely on external events

• void: for pipes that have neither input nor output (such a pipe may be used internally to create other pipes)

• pipe type-specific commands and probes which must be prefixed with the short name of the pipe

As a convenience, the Upipe distribution provides a number of "helper" macros which usually manage internal structures and control commands:

Strictly speaking, a struct upipe object has at most one input and one output (and possibly none). Split and join pipes are implemented using subpipes: the main split (resp. join) pipe is a sink (resp. a source), and each output (resp. input) requires allocating a source (resp. sink) subpipe using upipe_void_alloc_sub or upipe_flow_alloc_sub, depending whether the subpipe requests a flow definition. In a split pipe, outputs are configured with upipe_set_output and upipe_set_flow_def called on the output subpipe; however data is fed into the main pipe. In a join pipe, data is input using upipe_input on each input subpipe; however outputs are configured on the main pipe. The caller must therefore keep a constellation of objects, not only the main pipe but also all the subpipes. Outputs (resp. inputs) are closed by calling upipe_release on the related subpipe.

Source pipes and sink pipes (but not exclusively) rely on external events to retrieve or dispatch data. For instance, one may want to wait on a UDP socket for packets. Or to wait until a system pipe (mkfifo) can be written again. Or more simply, wait for a timeout.

Pipes which need those interactions can create pumps with the built-in primitives. The struct upump abstraction layer then maps the events to the API of the event loop which is used by the application. It supports the following types of events:

• idlers are executed whenever there is nothing else to do: upump_alloc_idler

• timers are executed after a given timeout: upump_alloc_timer

• read (resp. write) file descriptor watchers are executed whenever data is ready to be read from (resp. written to) a file descriptor: upump_alloc_fd_read (resp. upump_alloc_fd_write)

It is expected that more event types get added in the future, especially for Microsoft Windows(tm)-specific objects. Some core objects, which require being able to wait on a condition, propose their own API to allocate an adequate upump (uqueue_upump_alloc_pop or udeal_upump_alloc). In turn, they rely on the struct ueventfd object provided by Upipe.

A sink pipe which can no longer write to its output may block the source pump (passed to upipe_input) using upump_blocker_alloc. A call-back must be provided, which will be called upon the destruction of the source pump.

Upipe currently provides abstraction layers for these event loops:

To deal with structures efficiently, Upipe has a notion of "managers", which are similar to factories in object-oriented programming. Consequently, struct ubuf are created by a struct ubuf_mgr, struct udict by struct udict_mgr, struct uref by struct uref_mgr, struct upump by struct upump_mgr and struct upipe by struct upipe_mgr. Managers typically deal with memory pools or hardware resources such as access to video memory or hardware decoding.

Upipe provides standard implementations of struct ubuf_mgr (for blocks and pictures), struct udict_mgr, struct umem_mgr and struct uref_mgr in the libupipe library (link with -lupipe).

The struct upump_mgr implementations are supplied by dedicated, API-specific libraries such as libupump-ev. Finally, struct upipe_mgr managers allow to create the pipes themselves and are either provided by the libupipe-modules library for standard pipes, or by third-party libraries.

To instanciate managers it is necessary to link with some libraries; however there is no dependancy associated with the use of the children structures struct ubuf, struct uref and struct upump, as all code is either provided by inline functions or function pointers. Libraries providing implementations of struct upipe_mgr and struct upipe usually do not depend on libupipe.

Upon initialization, a new pipe may be passed the managers for its struct uref, struct ubuf and struct upump structures, if it requires to create them. If it has no need for a manager (for instance it just changes an attribute to all uref structures passing by), then it need not be passed.

The movement of buffers between the input and output of a pipe is called a flow. Pipes generally expect some parameters describing the flow; these are called a flow definition. A flow definition packet is a struct uref pointing to a struct udict with key/value pairs. Standard flow definitions are provided for block (uref_block_flow_alloc_def), picture (uref_pic_flow_alloc_def) and sound (uref_sound_flow_alloc_def) formats.

The input flow definition is set on a pipe using upipe_set_flow_def. It may be changed at any time, but the pipe may deny it by returning false if it considers it is too big a change; by convention, pipes only allow it if it doesn't require a full reset of the pipe's state.

Some pipes (filters) may also require a flow definition describing the requested output, for instance to change picture format. In that case the output flow definition packet is passed on allocation using upipe_flow_alloc instead of upipe_void_alloc. When a pipe changes the flow definition on its output, it must send the UPROBE_NEW_FLOW_DEF event. Upipe provides helper functions (see upipe_void_alloc_output and upipe_flow_alloc_output) to automatically forward the new flow definition to the output pipe.

It is useful to note that a flow definition packet is actually a patchwork of attributes with different purposes:

• attributes which define the format of the buffers, such as the number of planes or chroma subsampling

• attributes which describe several properties useful for the display or processing of the flow, such as aspect ratio, fps, bitrate, sample rate, etc.

• attributes which may have no relation with the data themselves but are useful to choose between elementary streams, such as the language, program name, event information, etc.

It should therefore be considered quite standard to have frequent flow definition updates, and most pipes won't probably feel the difference.

All dates in Upipe are represented, by convention, as ticks of a 27 MHz clock. The origin and pace of the clock depends on the variation of the date. There are three date variations stored in struct uref:

Please note that the three clocks may drift slightly. For instance, a 25 progressive frames per second stream should have a frame exactly every 40 ms according to the orig and prog clocks, but in the sys clock it may drift to 39 or 41 ms if the sender's clock is too fast or too slow. Also, when reading files, the trickplay module may create slow motion or fast forward effects by changing the pace of the sys clock.

In a struct uref, every date variation has a semantic attached to it (enum uref_date_type). There are three date types, which are used in different parts of the pipeline to represent different events:

A struct uref contains only one date type for each date variation. The date type should be cr for block data (input of demux and output of mux), dts for encoded frames (output of demux and input of mux), and pts for raw frames (output of decoder and input of encoder). This is called the "base". Dates are set using uref_clock_set_cr_prog, uref_clock_set_dts_prog, uref_clock_set_pts_prog, etc. Note that each call to uref_clock_set_XXX_prog will overwrite previous values for prog and rebase the prog date. It is possible to set the delay between the dts and pts using uref_clock_set_dts_pts_delay, and between the cr and dts using uref_clock_set_cr_dts_delay (typically the vbv delay). This makes it possible for a demux to set both dts and pts when it is available. uref_clock_rebase_dts_prog may be used by the decoder to rebase the prog date using dts. uref_clock_rebase_cr_sys would typically be used in a mux.

The struct uref also contains the sys date of the last random access point. It can be retrieved (resp. set) with uref_clock_get_rap_sys (resp. uref_clock_set_rap_sys).

Upipe objects do not natively deal with threads. Multithreading is supposed to be the prerogative of the application, or at least of very high level bin pipes (an exception being the libraries which themselves take advantage of multithreading, such as FFmpeg/libav and x264). However there is some minimum support to allow running a part of a pipeline in a different "worker" thread.

The first kind of support is the upipe_qsrc_mgr_alloc and upipe_qsink_mgr_alloc managers, which allow to bridge struct uref from one thread to another. In that case, each thread runs its own upump manager, which is passed to the pipes running in it.

Upipe also provides facilities to transfer pipes to another thread (ie. upump manager), and control them remotely from a "main" thread: upipe_xfer_mgr_alloc. It works the following way:

• An xfer_mgr is created, in general from the main thread, using upipe_xfer_mgr_alloc.

• A new thread is created (for instance with pthread_create). The pointer to the xfer_mgr is passed as the opaque argument to pthread_create, and retrieved in the worker thread.

• The worker thread initializes a new event loop and a new upump manager, and calls upipe_xfer_mgr_attach on the xfer_mgr passed by the main thread. It can then enter the event loop.

• The main thread creates a pipe that is supposed to run on the worker thread via upipe_void_alloc, and initializes it. The pipe is supposed to work with a upump_mgr, either directly, or through the use of upipe_qsrc_mgr_alloc and/or upipe_qsink_mgr_alloc. If a group of pipes has to be moved (for instance with queue source and queue sink), the group can be built normally with upipe_set_output, and then all members requiring a upump manager have to be transferred.

• Pipes that need a upump manager are transferred to the worked thread by calling upipe_xfer_alloc on them. In case there are several pipes, they must be transferred in the sink to source order. After upipe_xfer_alloc, the original upipe pointer shall no longer be used in the main thread; however this function returns a pointer to a pipe handle, that allows to do basic operations on the remote pipe, such as releasing it.

• At the end of the processing, all pipe handles are released, which in turn releases the remote pipes in the worker thread. The main thread calls upipe_mgr_release to stop the event loop in the worker thread, and release the xfer_mgr.