3 Overview

Upipe provides an extensive set of header files, to be used by pipes and applications. Applications will also generally want to link with optional libraries, which allow to create and manage some core data structures.

3.1 Data structures  

Upipe's data structures can be classified into five groups:

Core structures  

Core structures provide services for all other groups:

  • atomic operations with or without locks (uatomic_uint32_t)

  • reference counters (struct urefcount)

  • chained lists (struct uchain) and rings allowing pointer tagging (struct uring)

  • lock-less LIFOs (struct ulifo) without thread limitations

  • lock-less FIFOs (struct ufifo) without thread limitations

  • lock-less buffer queues (struct uqueue) without thread limitations

  • lock-less buffer pools (struct upool) without thread limitations

  • lock-less exclusive access to non-reentrant resources (struct udeal)

  • blocking exclusive access to non-reentrant resources (struct umutex)

  • objects to pass event information between threads (struct ueventfd)

  • dictionaries storing key/value pairs (udict)

  • memory allocation with or without pools (struct umem)

  • access to monotonic or real-time system clock (struct uclock)

Buffer management  

Buffer management structures define how data is carried across inside a pipeline and are handled by struct ubuf and associated managers. These structures typically point to a shared, reference counted area maintained by the manager. Upipe natively supports three types of buffers, each with a dedicated API to access and modify data :

  • blocks (arbitrary octet content with arbitrary and variable size), typically used for encoded data: ubuf_block_alloc ; the API supports windowing and resizing blocks, as well as merging and cutting out with zero-copy and copy-on-write semantics.

  • pictures, defined by their chromatic planes, pixel sizes and subsampling, with arbitrary width and height: ubuf_pic_alloc.

  • sounds, in planar or packed formats: ubuf_sound_alloc.

Pipes and applications however do not manipulate struct ubuf directly, but struct uref, which is composed of a pointer to struct ubuf and a pointer to udict. The dictionary allows to associate arbitrary attributes to the struct ubuf.

An attribute is defined by a category, a name, a type and a value. Available types are:

  • opaque: stores a buffer of an arbitrary length

  • string: stores a string terminated by \0

  • void: doesn't store anything but the presence or absence of the attribute (flag)

  • bool: stores a true or false value

  • small_unsigned: stores a uint8_t

  • small_int: stores an int8_t

  • unsigned: stores a uint64_t

  • int: stores an int64_t

  • rational: stores a struct urational

  • float: stores a double

The standard Upipe distribution provides managers for block and picture formats, relying on application memory allocation (malloc). They in turn use the struct umem_mgr facility to allocate buffer spaces.


Pipes are provided by specialized modules, possibly delivered by third-parties. A struct upipe may expose input and control methods (with standard and custom commands), and would typically take data from its input, process it and output it, possibly using a different buffer, to its output. Functions from a pipe may only be called from a single thread, so locking and reentrancy isn't required.

When created, pipes are passed one structure for logging and sending exceptions (struct uprobe) to the parent code upon certain events (end of file, fatal error, new flow, etc.), and optional arguments, in particular, for pipes that aren't sources, a flow definition packet of type struct uref describing the input. Pipe allocation is performed with upipe_void_alloc or upipe_flow_alloc, depending whether the pipe requests a flow definition packet. A few pipes may also provide their own specific allocator.

The standard Upipe distribution currently contains the following pipe types (though third-party modules may also be used):

API name
link with
source pipe opening for reading a file or special file characterized by its path
sink pipe opening for writing a file or special file characterized by its path
source pipe opening for reading a UDP socket
sink pipe opening for writing a UDP socket
sink pipe opening for writing a directory in a manner compatible with multicat
split pipe allowing to duplicate all input packets to several outputs
linear pipe outputting packets identically
upipe_qsrc_mgr_alloc (also note upipe_qsrc_alloc)
source pipe opening a thread-safe queue with one or more qsink pipes
sink pipe sending buffers to a queue opened by a qsrc pipe
sink pipe destroying all input buffers
upipe_xfer_mgr_alloc (also note upipe_xfer_alloc)
pipe manager allowing to attach another pipe to a given upump_mgr running in a different thread
pipe manager creating a pthread and allowing to attach another pipe to the remote upump_mgr
upipe_wsrc_mgr_alloc (also note upipe_wsrc_alloc)
pipe manager allowing to deport a source pipe to a different thread
upipe_wlin_mgr_alloc (also note upipe_wlin_alloc)
pipe manager allowing to deport a linear pipe to a different thread
upipe_wsink_mgr_alloc (also note upipe_wsink_alloc)
pipe manager allowing to deport a sink pipe to a different thread
pipe facilitating trick play operations
pipe evening the start and end of a stream
pipe skipping the beginning of each uref
pipe aggregating several urefs into one
pipe setting given attributes on incoming flow definitions
pipe setting given attributes on all incoming buffers
pipe setting attribute k.systime_rap on all incoming buffers
pipe creating timestamps for single streamps
pipe creating system timestamps for off-line streamps
pipe generating multicat-style auxiliary blocks
sink pipe generating multicat-style directories
source pipe opening for reading a URL and using libavformat
linear pipe decoding a video or audio flow using libavcodec
linear pipe encoding a video or audio flow using libavcodec
linear pipe scaling a flow of pictures using libswscale
linear pipe building a mosaic of thumbnails out of a picture flow
linear pipe resampling a flow of sound with libswresample
split pipe demultiplexing a TS stream (also features lots of subpipes)
join pipe multiplexing a TS stream (also features lots of subpipes)
linear pipe gathering MPEG-1 and MPEG-2 audio (including AAC ADTS) streams into frames
linear pipe gathering MPEG-1 and MPEG-2 video streams into frames
linear pipe gathering MPEG-4 AVC video streams into frames
linear pipe gathering A52 audio streams into frames
linear pipe gathering teletext streams into frames
linear pipe gathering DVB subtitling streams into frames
pipe trimming dead frames off a video stream
linear pipe transforming the format into another specified (potentially for an encoder)
linear pipe decoding a stream
linear pipe encoding a stream
linear pipe deinterlacing pictures using a blending algorithm
linear pipe encoding to MPEG-4 AVC using libx264
linear pipe encoding to H.265/HEVC using libx265
sink pipe displaying picture on a GLX output

Data is fed into a pipe using upipe_input. The struct uref argument then belongs to the callee and shouldn't be used any longer. There is an additional struct upump argument that points to the pump that generated the buffer (or NULL if unavailable).

The generic upipe_control call provides the application with an interface to modify the pipe's property. The counterpart of this function is provided by struct uprobe, which allows the pipe to send messages to the application. It is possible to build upon this messaging system to dynamically take actions on the pipe or the pipeline. All parameters to upipe_control belong to the caller. All parameters sent by subpipes via probes also belong to the subpipe. The standard Upipe distribution currently contains the following catchers:

API name
link with
print all log messages to a file stream
prefix all log messages with the given name
give a common upump manager to pipes requiring one
give a upump manager specific to each thread to pipes requiring one
give a uref manager to pipes requiring one
give a uclock to pipes requiring one
allocate and give a ubuf manager to pipes that require one
allocate and give a ubuf manager to pipes that require one, and recycle existing ubuf managers
select flows according to criteria
dejitter packets coming from a network by averaging reference clocks
forward probes from one thread to another, in conjunction with upipe_xfer_alloc

In Upipe's design, decision taking happens inside probes, while execution is done in pipes.

Control commands and pipes are classified (that is, enum values are prepended with a prefix) into 7 categories:

  • generic: commands and probes which can apply to any type of pipe (no prefix)

  • source: for pipes that have no input, but rely instead on external events to retrieve incoming data

  • join: for pipes that have several inputs, such as a mux

  • split: for pipes that have several outputs, such as a demux

  • sink: for pipes that have no output and may rely on external events

  • void: for pipes that have neither input nor output (such a pipe may be used internally to create other pipes)

  • pipe type-specific commands and probes which must be prefixed with the short name of the pipe

As a convenience, the Upipe distribution provides a number of "helper" macros which usually manage internal structures and control commands:

helper macro
very basic helper providing the upipe_foo_from_upipe and upipe_foo_to_upipe functions on which most helpers rely
helper providing allocation functions for a custom allocator
helper for pipes which require no argument to their allocation function
helper for pipes which require a flow definition packet to be passed as argument to their allocation function
helper for pipes which require a uref manager
helper for pipes which require a upump manager
helper for pipes which rely on external events
helper for pipes which requires a uclock
helper for the management of the output
helper for the management of the ubuf manager for the output
helper for pipes outputting data in chunks of a configuration size
helper for pipes having an input, which need to block input pumps and buffer urefs
helper for pipes reading data octet by octet, for instance to constitute packets or frames
helper for split pipes and join pipes to manage their subpipes
helper for pipes that wish to tell when the signal is acquired or lost
helper for pipes using the biTStream library and needing to convert strings
helper for pipes using inner pipes
helper for pipes using inner probes
helper for bin pipes (that incorporate a sub-pipeline of several pipes) dealing with the input of the bin
helper for bin pipes (that incorporate a sub-pipeline of several pipes) dealing with the output of the bin
very basic helper providing the uprobe_foo_from_uprobe and uprobe_foo_to_uprobe functions on which most helpers rely
helper providing easy probe allocation and deallocation

Strictly speaking, a struct upipe object has at most one input and one output (and possibly none). Split and join pipes are implemented using subpipes: the main split (resp. join) pipe is a sink (resp. a source), and each output (resp. input) requires allocating a source (resp. sink) subpipe using upipe_void_alloc_sub or upipe_flow_alloc_sub, depending whether the subpipe requests a flow definition. In a split pipe, outputs are configured by allocating output subpipes; however data is fed into the main pipe. In a join pipe, data is input using upipe_input on each input subpipe; however outputs are configured on the main pipe. The caller must therefore keep a constellation of objects, not only the main pipe but also all the subpipes. Outputs (resp. inputs) are closed by calling upipe_release on the related subpipe.

External events  

Source pipes and sink pipes (but not exclusively) rely on external events to retrieve or dispatch data. For instance, one may want to wait on a UDP socket for packets. Or to wait until a system pipe (mkfifo) can be written again. Or more simply, wait for a timeout.

Pipes which need those interactions can create pumps with the built-in primitives. The struct upump abstraction layer then maps the events to the API of the event loop which is used by the application. It supports the following types of events:

It is expected that more event types get added in the future, especially for Microsoft Windows(tm)-specific objects. Some core objects, which require being able to wait on a condition, propose their own API to allocate an adequate upump (uqueue_upump_alloc_pop or udeal_upump_alloc). In turn, they rely on the struct ueventfd object provided by Upipe.

A sink pipe which can no longer write to its output may block the source pump (passed to upipe_input) using upump_blocker_alloc. A call-back must be provided, which will be called upon the destruction of the source pump.

Upipe currently provides abstraction layers for these event loops:

event loop
link with
-lupump-ev -lev
-lupump-ecore -lecore

Internal events  

Buffer management structures are allocated on the fly, depending on the needs of the pipes. However, some negotiation may take place, because downstream pipes may have specific requirements, such as the alignment of data, or even a custom ubuf manager required by an external library. The process to ask for buffer management structures is therefore as follows:

  • The pipe creates a struct urequest. For some request types, an additional uref is necessary.

  • The pipe registers the request on its output. The standard helper UPIPE_HELPER_OUTPUT provides functions to deal with this.

  • When an output is connected to the pipe, or when the output is changed, the request is automatically registered to the new output.

  • The request is forwarded from pipe to pipe down to the sink. If a pipe has specific requirements such as alignment, it may modify the request.

  • A sink pipe (or logical sink, for instance when a pipe changes the format so much that a new ubuf manager will be used) either provides the data structures itself, or throws the request as a probe event using upipe_throw_provide_request to ask for the required data structures.

  • The call-back passed to the request is called, and the internal event bubbles upstream until it reaches the source of the request.

Thus, any change in the pipeline triggers a renegotiation of data structures. In a multi-threaded application, the pipes providing standard queues or workers have the ability to forward the requests downstream, and the replies upstream. However the reply is asynchronous, and the source pipe must not expect to receive the data structures immediately. It must buffer the packets (for instance using UPIPE_HELPER_INPUT) until the reply is received. If the pipe can really not cope with the additional delay, it may throw the upipe_throw_provide_request to get an immediate reply, which may be adjusted later. This behaviour is however discouraged.

There are four types of requests, and Upipe provides helpers to deal with them:


Structure dependencies  

The following graph depicts dependency relationships between structures (dashed lines are optional):

3.2 Managers  

To deal with structures efficiently, Upipe has a notion of "managers", which are similar to factories in object-oriented programming. Consequently, struct ubuf are created by a struct ubuf_mgr, udict by struct udict_mgr, struct uref by struct uref_mgr, struct upump by struct upump_mgr and struct upipe by struct upipe_mgr. Managers typically deal with memory pools or hardware resources such as access to video memory or hardware decoding.

Upipe provides standard implementations of struct ubuf_mgr (for blocks and pictures), struct udict_mgr, struct umem_mgr and struct uref_mgr in the libupipe library (link with -lupipe).

The struct upump_mgr implementations are supplied by dedicated, API-specific libraries such as libupump-ev. Finally, struct upipe_mgr managers allow to create the pipes themselves and are either provided by the libupipe-modules library for standard pipes, or by third-party libraries.

To instantiate managers it is necessary to link with some libraries; however there is no dependency associated with the use of the children structures struct ubuf, struct uref and struct upump, as all code is either provided by inline functions or function pointers. Libraries providing implementations of struct upipe_mgr and struct upipe usually do not depend on libupipe.

Upon initialization, a new pipe may be passed the managers for its struct uref, struct ubuf and struct upump structures, if it requires to create them. If it has no need for a manager (for instance it just changes an attribute to all uref structures passing by), then it need not be passed.

3.3 Flows  

The movement of buffers between the input and output of a pipe is called a flow. Pipes generally expect some parameters describing the flow; these are called a flow definition. A flow definition packet is a struct uref pointing to a udict with key/value pairs. Standard flow definitions are provided for block (uref_block_flow_alloc_def), picture (uref_pic_flow_alloc_def) and sound (uref_sound_flow_alloc_def) formats.

The input flow definition is set on a pipe using upipe_set_flow_def. It may be changed at any time, but the pipe may deny it by returning an error if it considers it is too big a change; by convention, pipes only allow it if it doesn't require a full reset of the pipe's state.

Some pipes (filters) may also require a flow definition describing the requested output, for instance to change picture format. In that case the output flow definition packet is passed on allocation using upipe_flow_alloc instead of upipe_void_alloc. When a pipe changes the flow definition on its output, it must call upipe_set_flow_def on the output pipe.

It is useful to note that a flow definition packet is actually a patchwork of attributes with different purposes:

  • attributes which define the format of the buffers, such as the number of planes or chroma subsampling

  • attributes which describe several properties useful for the display or processing of the flow, such as aspect ratio, fps, bitrate, sample rate, etc.

  • attributes which may have no relation with the data themselves but are useful to choose between elementary streams, such as the language, program name, event information, etc.

It should therefore be considered quite standard to have frequent flow definition updates, and most pipes won't probably feel the difference.

3.4 Clocks  

All dates in Upipe are represented, by convention, as ticks of a 27 MHz clock. The origin and pace of the clock depends on the variation of the date. There are three date variations stored in struct uref:

date variation
whole stream
Uses the same scale as uclock_now. The date is represented in system time. On live streams, incoming packets should be stamped with this clock, and outgoing packets (for presentation or streaming) should be scheduled according to it. A fictitious system time may be necessary even for some file to file conversion, as it is the only clock that is stream-wide.
Carries the dates coded in the incoming stream, with the same origin, scaled to 27 MHz (if necessary). Its only use is for remultiplexing a stream with the exact same timestamps. A demux should typically retrieve this value.
Is based on the orig clock, but the origin may be changed to make sure that the dates are always monotonically increasing, without large gaps. A demux should typically infer this clock, and a mux should use it to write its timestamps. upipe_trickp_mgr_alloc is also able to derive the system clock from the prog clock.

Please note that the three clocks may drift slightly. For instance, a 25 progressive frames per second stream should have a frame exactly every 40 ms according to the orig and prog clocks, but in the sys clock it may drift to 39 or 41 ms if the sender's clock is too fast or too slow. Also, when reading files, the trickplay module may create slow motion or fast forward effects by changing the pace of the sys clock.

In a struct uref, every date variation has a semantic attached to it (uref_date_type). There are three date types, which are used in different parts of the pipeline to represent different events:

date type
uref domain
Clock reference (cr)
block data
Represents the date of reception (resp. emission) of a low-level packet. It does not necessarily have a relationship with actual video or audio data. When dealing with live processing, clock references should be set on all packets entering a demux, and on all packets leaving a mux.
Decoding timestamp (dts)
encoded frame
Represents the theorical date of decoding of an encoded audio or video frame. It must be set on all frames leaving a demux, and all frames entering a mux.
Presentation timestamp (pts)
raw frame
Represents the theorical date of presentation of an audio or video frame. It must be set on all frames leaving a decoder, and all frames entering an encoder.

A struct uref contains only one date type for each date variation. The date type should be cr for block data (input of demux and output of mux), dts for encoded frames (output of demux and input of mux), and pts for raw frames (output of decoder and input of encoder). This is called the "base". Dates are set using uref_clock_set_cr_prog, uref_clock_set_dts_prog, uref_clock_set_pts_prog, etc. Note that each call to uref_clock_set_XXX_prog will overwrite previous values for prog and rebase the prog date. It is possible to set the delay between the dts and pts using uref_clock_set_dts_pts_delay, and between the cr and dts using uref_clock_set_cr_dts_delay (typically the vbv delay). This makes it possible for a demux to set both dts and pts when it is available. uref_clock_rebase_dts_prog may be used by the decoder to rebase the prog date using dts. uref_clock_rebase_cr_sys would typically be used in a mux.

The struct uref also contains the sys date of the last random access point. It can be retrieved (resp. set) with uref_clock_get_rap_sys (resp. uref_clock_set_rap_sys).

3.5 Threads  

Upipe objects do not natively deal with threads. Multithreading is supposed to be the prerogative of the application, or at least of very high level bin pipes (an exception being the libraries which themselves take advantage of multithreading, such as FFmpeg/libav and x264). However multithreading in Upipe is built on several levels:

  • The core structures provide thread-safe queues (struct uqueue) and pools (struct upool).

  • Two low-level pipes, upipe_qsrc_mgr_alloc and upipe_qsink_mgr_alloc, allow to bridge struct uref from one thread to another. In that case, each thread runs its own upump manager, which is passed to the pipes running in it.

  • One low-level pipe, upipe_xfer_mgr_alloc, allows to transfer a single pipe to a upump manager running in a different thread.

  • Three high-level bin pipes allow to create a thread and to transfer there whole subpipelines, while setting up appropriate queues to transfer urefs in and out of the subpipeline.

The common method to use worker threads is the following:

  • A worker thread and transfer manager is created with the manager adapted to the threading system (for POSIX systems, upipe_pthread_xfer_mgr_alloc). You have to pass functions to initialize, run and clean the upump manager attached to the worker thread.

  • A worker manager is created using the transfer manager. It allows to transfer whole subpipelines to the thread. You must choose the correct type of worker, whether the subpipeline is a source (upipe_wsrc_mgr_alloc), a sink (upipe_wsink_mgr_alloc) or a linear subpipeline (upipe_wlin_mgr_alloc). This controls which queues will be set up for the input and/or output of the subpipeline.

  • The pipes that are to be transferred to the subpipeline are created normally in the main thread. However, some pipes may request a upump manager and start working immediately, therefore pumps would be created with the wrong upump manager. To avoid that, it is necessary to freeze the upump manager of the main thread by calling uprobe_throw on the probe that delivers upump managers (for instance uprobe_pthread_upump_mgr_alloc) with the event UPROBE_FREEZE_UPUMP_MGR.

  • a worker pipe is created with a pointer to the subpipeline, and takes the place of the subpipeline, from the point of view of the main thread. It is possible to input urefs to the worker pipe (for linear and sink workers), set its output (for linear and source workers), or release it (upipe_release). The worker pipe automatically calls upipe_attach_upump_mgr on relevant pipes of the subpipeline.

  • After the subpipeline has been created, the upump manager of the main thread may be thawed with UPROBE_THAW_UPUMP_MGR.

Valid XHTML 1.0 StrictGenerated by cmassiot on Fri Feb 2 23:57:18 2018 using MkDoc