3 Overview

Upipe provides an extensive set of header files, to be used by pipes and applications. Applications will also generally want to link with optional libraries, which allow to create and manage some core data structures.

Upipe's data structures can be classified into five groups:

Core structures provide services for all other groups:

• atomic operations with or without locks (uatomic_uint32_t)

• reference counters (struct urefcount)

• chained lists (struct uchain) and rings allowing pointer tagging (struct uring)

• lock-less LIFOs (struct ulifo) without thread limitations

• lock-less FIFOs (struct ufifo) without thread limitations

• lock-less buffer queues (struct uqueue) without thread limitations

• lock-less buffer pools (struct upool) without thread limitations

• lock-less exclusive access to non-reentrant resources (struct udeal)

• blocking exclusive access to non-reentrant resources (struct umutex)

• objects to pass event information between threads (struct ueventfd)

• dictionaries storing key/value pairs (udict)

• memory allocation with or without pools (struct umem)

• access to monotonic or real-time system clock (struct uclock)

Buffer management structures define how data is carried across inside a pipeline and are handled by struct ubuf and associated managers. These structures typically point to a shared, reference counted area maintained by the manager. Upipe natively supports three types of buffers, each with a dedicated API to access and modify data :

• blocks (arbitrary octet content with arbitrary and variable size), typically used for encoded data: ubuf_block_alloc ; the API supports windowing and resizing blocks, as well as merging and cutting out with zero-copy and copy-on-write semantics.

• pictures, defined by their chromatic planes, pixel sizes and subsampling, with arbitrary width and height: ubuf_pic_alloc.

• sounds, in planar or packed formats: ubuf_sound_alloc.

Pipes and applications however do not manipulate struct ubuf directly, but struct uref, which is composed of a pointer to struct ubuf and a pointer to udict. The dictionary allows to associate arbitrary attributes to the struct ubuf.

An attribute is defined by a category, a name, a type and a value. Available types are:

• opaque: stores a buffer of an arbitrary length

• string: stores a string terminated by \0

• void: doesn't store anything but the presence or absence of the attribute (flag)

• bool: stores a true or false value

• small_unsigned: stores a uint8_t

• small_int: stores an int8_t

• unsigned: stores a uint64_t

• int: stores an int64_t

• rational: stores a struct urational

• float: stores a double

The standard Upipe distribution provides managers for block and picture formats, relying on application memory allocation (malloc). They in turn use the struct umem_mgr facility to allocate buffer spaces.

Pipes are provided by specialized modules, possibly delivered by third-parties. A struct upipe may expose input and control methods (with standard and custom commands), and would typically take data from its input, process it and output it, possibly using a different buffer, to its output. Functions from a pipe may only be called from a single thread, so locking and reentrancy isn't required.

When created, pipes are passed one structure for logging and sending exceptions (struct uprobe) to the parent code upon certain events (end of file, fatal error, new flow, etc.), and optional arguments, in particular, for pipes that aren't sources, a flow definition packet of type struct uref describing the input. Pipe allocation is performed with upipe_void_alloc or upipe_flow_alloc, depending whether the pipe requests a flow definition packet. A few pipes may also provide their own specific allocator.

The standard Upipe distribution currently contains the following pipe types (though third-party modules may also be used):

Data is fed into a pipe using upipe_input. The struct uref argument then belongs to the callee and shouldn't be used any longer. There is an additional struct upump argument that points to the pump that generated the buffer (or NULL if unavailable).

The generic upipe_control call provides the application with an interface to modify the pipe's property. The counterpart of this function is provided by struct uprobe, which allows the pipe to send messages to the application. It is possible to build upon this messaging system to dynamically take actions on the pipe or the pipeline. All parameters to upipe_control belong to the caller. All parameters sent by subpipes via probes also belong to the subpipe. The standard Upipe distribution currently contains the following catchers:

In Upipe's design, decision taking happens inside probes, while execution is done in pipes.

Control commands and pipes are classified (that is, enum values are prepended with a prefix) into 7 categories:

• generic: commands and probes which can apply to any type of pipe (no prefix)

• source: for pipes that have no input, but rely instead on external events to retrieve incoming data

• join: for pipes that have several inputs, such as a mux

• split: for pipes that have several outputs, such as a demux

• sink: for pipes that have no output and may rely on external events

• void: for pipes that have neither input nor output (such a pipe may be used internally to create other pipes)

• pipe type-specific commands and probes which must be prefixed with the short name of the pipe

As a convenience, the Upipe distribution provides a number of "helper" macros which usually manage internal structures and control commands:

Strictly speaking, a struct upipe object has at most one input and one output (and possibly none). Split and join pipes are implemented using subpipes: the main split (resp. join) pipe is a sink (resp. a source), and each output (resp. input) requires allocating a source (resp. sink) subpipe using upipe_void_alloc_sub or upipe_flow_alloc_sub, depending whether the subpipe requests a flow definition. In a split pipe, outputs are configured by allocating output subpipes; however data is fed into the main pipe. In a join pipe, data is input using upipe_input on each input subpipe; however outputs are configured on the main pipe. The caller must therefore keep a constellation of objects, not only the main pipe but also all the subpipes. Outputs (resp. inputs) are closed by calling upipe_release on the related subpipe.

Source pipes and sink pipes (but not exclusively) rely on external events to retrieve or dispatch data. For instance, one may want to wait on a UDP socket for packets. Or to wait until a system pipe (mkfifo) can be written again. Or more simply, wait for a timeout.

Pipes which need those interactions can create pumps with the built-in primitives. The struct upump abstraction layer then maps the events to the API of the event loop which is used by the application. It supports the following types of events:

• idlers are executed whenever there is nothing else to do: upump_alloc_idler

• timers are executed after a given timeout: upump_alloc_timer

• read (resp. write) file descriptor watchers are executed whenever data is ready to be read from (resp. written to) a file descriptor: upump_alloc_fd_read (resp. upump_alloc_fd_write)

It is expected that more event types get added in the future, especially for Microsoft Windows(tm)-specific objects. Some core objects, which require being able to wait on a condition, propose their own API to allocate an adequate upump (uqueue_upump_alloc_pop or udeal_upump_alloc). In turn, they rely on the struct ueventfd object provided by Upipe.

A sink pipe which can no longer write to its output may block the source pump (passed to upipe_input) using upump_blocker_alloc. A call-back must be provided, which will be called upon the destruction of the source pump.

Upipe currently provides abstraction layers for these event loops:

Buffer management structures are allocated on the fly, depending on the needs of the pipes. However, some negotiation may take place, because downstream pipes may have specific requirements, such as the alignment of data, or even a custom ubuf manager required by an external library. The process to ask for buffer management structures is therefore as follows:

• The pipe creates a struct urequest. For some request types, an additional uref is necessary.

• The pipe registers the request on its output. The standard helper UPIPE_HELPER_OUTPUT provides functions to deal with this.

• When an output is connected to the pipe, or when the output is changed, the request is automatically registered to the new output.

• The request is forwarded from pipe to pipe down to the sink. If a pipe has specific requirements such as alignment, it may modify the request.

• A sink pipe (or logical sink, for instance when a pipe changes the format so much that a new ubuf manager will be used) either provides the data structures itself, or throws the request as a probe event using upipe_throw_provide_request to ask for the required data structures.

• The call-back passed to the request is called, and the internal event bubbles upstream until it reaches the source of the request.

Thus, any change in the pipeline triggers a renegotiation of data structures. In a multi-threaded application, the pipes providing standard queues or workers have the ability to forward the requests downstream, and the replies upstream. However the reply is asynchronous, and the source pipe must not expect to receive the data structures immediately. It must buffer the packets (for instance using UPIPE_HELPER_INPUT) until the reply is received. If the pipe can really not cope with the additional delay, it may throw the upipe_throw_provide_request to get an immediate reply, which may be adjusted later. This behaviour is however discouraged.

There are four types of requests, and Upipe provides helpers to deal with them:

The following graph depicts dependency relationships between structures (dashed lines are optional):

To deal with structures efficiently, Upipe has a notion of "managers", which are similar to factories in object-oriented programming. Consequently, struct ubuf are created by a struct ubuf_mgr, udict by struct udict_mgr, struct uref by struct uref_mgr, struct upump by struct upump_mgr and struct upipe by struct upipe_mgr. Managers typically deal with memory pools or hardware resources such as access to video memory or hardware decoding.

Upipe provides standard implementations of struct ubuf_mgr (for blocks and pictures), struct udict_mgr, struct umem_mgr and struct uref_mgr in the libupipe library (link with -lupipe).

The struct upump_mgr implementations are supplied by dedicated, API-specific libraries such as libupump-ev. Finally, struct upipe_mgr managers allow to create the pipes themselves and are either provided by the libupipe-modules library for standard pipes, or by third-party libraries.

To instantiate managers it is necessary to link with some libraries; however there is no dependency associated with the use of the children structures struct ubuf, struct uref and struct upump, as all code is either provided by inline functions or function pointers. Libraries providing implementations of struct upipe_mgr and struct upipe usually do not depend on libupipe.

Upon initialization, a new pipe may be passed the managers for its struct uref, struct ubuf and struct upump structures, if it requires to create them. If it has no need for a manager (for instance it just changes an attribute to all uref structures passing by), then it need not be passed.

The movement of buffers between the input and output of a pipe is called a flow. Pipes generally expect some parameters describing the flow; these are called a flow definition. A flow definition packet is a struct uref pointing to a udict with key/value pairs. Standard flow definitions are provided for block (uref_block_flow_alloc_def), picture (uref_pic_flow_alloc_def) and sound (uref_sound_flow_alloc_def) formats.

The input flow definition is set on a pipe using upipe_set_flow_def. It may be changed at any time, but the pipe may deny it by returning an error if it considers it is too big a change; by convention, pipes only allow it if it doesn't require a full reset of the pipe's state.

Some pipes (filters) may also require a flow definition describing the requested output, for instance to change picture format. In that case the output flow definition packet is passed on allocation using upipe_flow_alloc instead of upipe_void_alloc. When a pipe changes the flow definition on its output, it must call upipe_set_flow_def on the output pipe.

It is useful to note that a flow definition packet is actually a patchwork of attributes with different purposes:

• attributes which define the format of the buffers, such as the number of planes or chroma subsampling

• attributes which describe several properties useful for the display or processing of the flow, such as aspect ratio, fps, bitrate, sample rate, etc.

• attributes which may have no relation with the data themselves but are useful to choose between elementary streams, such as the language, program name, event information, etc.

It should therefore be considered quite standard to have frequent flow definition updates, and most pipes won't probably feel the difference.

All dates in Upipe are represented, by convention, as ticks of a 27 MHz clock. The origin and pace of the clock depends on the variation of the date. There are three date variations stored in struct uref:

Please note that the three clocks may drift slightly. For instance, a 25 progressive frames per second stream should have a frame exactly every 40 ms according to the orig and prog clocks, but in the sys clock it may drift to 39 or 41 ms if the sender's clock is too fast or too slow. Also, when reading files, the trickplay module may create slow motion or fast forward effects by changing the pace of the sys clock.

In a struct uref, every date variation has a semantic attached to it (uref_date_type). There are three date types, which are used in different parts of the pipeline to represent different events:

A struct uref contains only one date type for each date variation. The date type should be cr for block data (input of demux and output of mux), dts for encoded frames (output of demux and input of mux), and pts for raw frames (output of decoder and input of encoder). This is called the "base". Dates are set using uref_clock_set_cr_prog, uref_clock_set_dts_prog, uref_clock_set_pts_prog, etc. Note that each call to uref_clock_set_XXX_prog will overwrite previous values for prog and rebase the prog date. It is possible to set the delay between the dts and pts using uref_clock_set_dts_pts_delay, and between the cr and dts using uref_clock_set_cr_dts_delay (typically the vbv delay). This makes it possible for a demux to set both dts and pts when it is available. uref_clock_rebase_dts_prog may be used by the decoder to rebase the prog date using dts. uref_clock_rebase_cr_sys would typically be used in a mux.

The struct uref also contains the sys date of the last random access point. It can be retrieved (resp. set) with uref_clock_get_rap_sys (resp. uref_clock_set_rap_sys).

Upipe objects do not natively deal with threads. Multithreading is supposed to be the prerogative of the application, or at least of very high level bin pipes (an exception being the libraries which themselves take advantage of multithreading, such as FFmpeg/libav and x264). However multithreading in Upipe is built on several levels:

• The core structures provide thread-safe queues (struct uqueue) and pools (struct upool).

• Two low-level pipes, upipe_qsrc_mgr_alloc and upipe_qsink_mgr_alloc, allow to bridge struct uref from one thread to another. In that case, each thread runs its own upump manager, which is passed to the pipes running in it.

• One low-level pipe, upipe_xfer_mgr_alloc, allows to transfer a single pipe to a upump manager running in a different thread.

• Three high-level bin pipes allow to create a thread and to transfer there whole subpipelines, while setting up appropriate queues to transfer urefs in and out of the subpipeline.

The common method to use worker threads is the following:

• A worker thread and transfer manager is created with the manager adapted to the threading system (for POSIX systems, upipe_pthread_xfer_mgr_alloc). You have to pass functions to initialize, run and clean the upump manager attached to the worker thread.

• A worker manager is created using the transfer manager. It allows to transfer whole subpipelines to the thread. You must choose the correct type of worker, whether the subpipeline is a source (upipe_wsrc_mgr_alloc), a sink (upipe_wsink_mgr_alloc) or a linear subpipeline (upipe_wlin_mgr_alloc). This controls which queues will be set up for the input and/or output of the subpipeline.

• The pipes that are to be transferred to the subpipeline are created normally in the main thread. However, some pipes may request a upump manager and start working immediately, therefore pumps would be created with the wrong upump manager. To avoid that, it is necessary to freeze the upump manager of the main thread by calling uprobe_throw on the probe that delivers upump managers (for instance uprobe_pthread_upump_mgr_alloc) with the event UPROBE_FREEZE_UPUMP_MGR.

• a worker pipe is created with a pointer to the subpipeline, and takes the place of the subpipeline, from the point of view of the main thread. It is possible to input urefs to the worker pipe (for linear and sink workers), set its output (for linear and source workers), or release it (upipe_release). The worker pipe automatically calls upipe_attach_upump_mgr on relevant pipes of the subpipeline.

• After the subpipeline has been created, the upump manager of the main thread may be thawed with UPROBE_THAW_UPUMP_MGR.