Sustainability of Digital Formats: Planning for Library of Congress Collections

File format for the exchange of resolution-independent, pixel-based (bitmapped) images, intended for very high quality moving image content for theatrical distribution; DPX masters provide the input for film recording (digital images back to film for projection) or D-Cinema digital projection systems. Each DPX file represents a single image with a single component, e.g., luma, or multiple components, e.g., red, green, blue; or Cb, Y, Cr (chroma-luma data). Many variations in multiple component data are supported. DPX images may be produced by scanning film or by using a camera that produces a DPX output.

DPX is intended to carry picture data; its developers assumed that sound would be carried in separate formats, e.g., WAVE files, and this is the general practice. Nevertheless, a few organizations scan an image area large enough to include optical soundtracks and at least one software application (AEO Light) can convert this image data to sound.

DPX is a pixel-based (raster) file format with attributes defined in a binary file header. Each file represents a single image or single frame of a motion picture or video data stream. Since each DPX frame is "one of tens of thousands" in a motion picture, users will track intellectual/bibliographic metadata separately from the set of DPX files. A DPX file has four sections (including one optional section):

• Generic image data (including file information, image information, and image orientation information). Within the "Generic image data" section, a set of identified "core" fields is required. This core set comprises a minimum amount of information that a reader needs to read and interpret. A core-compliant reader must read the core fields but not necessarily be able to read non-core fields. A core-compliant writer must fill in the fore fields with valid values because blank or undefined values are not permitted. Non-core fields must be filled in with UNDEFINED values if the correct values are not known. In v.1 (268M-1994), this is further outlined in section 3; in v.2 (268M-2003), this is further detailed in section 4. The core fields are: magic number ("SDPX" or "XPDS" - see Signifiers), offset to image data in bytes, version number ("V2.0" for version 2), total image size in bytes (including file header), image orientation, number of image elements, pixels per line, lines per image element, data sign, descriptor, transfer characteristic, colorimetric specification, bit depth, packing, encoding, and offset to data.
• Industry Specific Information (motion picture film information, television information) which generally describes the film and camera source from which the image frame data was derived. The motion-picture film information header outlined in section 6.1 includes helpful tags for film-related data including information derived from film edge codes such as film manufacturing ID code, film type, offset in perforations, prefix and count. Other structured data fields include format (e.g. Academy), frame position in sequence (incremental integer sequence of DPX files), sequence length (frames), held count (1 = default), frame rate of original (frames/s), shutter angle of camera in degrees, frame identification ( e.g. keyframe), and slate information.
• User defined data (optional). Format and structure is not defined in SMPTE ST 268M:2003 to allow for customized information needed by some users but typical uses might be processing logs, etc.
• Image data. SMPTE ST 268M:2003 does not include more specific information about the expected contents of this section aside from specifying "Image 8-K blocks are recommended for efficient use of tape-storage devices." See FileFormat.Info for one source of more information.

The European PRESTO project supported efforts to preserve broadcast content via digitization. A PRESTO report on a file format conversion software issued in 2002 includes two statements that pertain to DPX: "Currently the DPX file format has become an industry standard for film production. It supports resolution independence (spatial and densitometric), but does not meet the requirements of broadcasters because, for instance, it does not contain sound data or metadata. Film is stored as a number of single uncompressed image files. The MXF format intends to overcome these limitations by, for instance, support of metadata and by support of film structure information (shot information)." (p. 2)

The PRESTO report also includes a paragraph that indicates one application's approach to filenaming with one-frame-at-a-time formats like DPX, a critical issue since 30 minutes of film contains 43,200 frames and 30 minutes of video contains about 54,000: "Both input and output image sequences are expected to be stored as a number of files in the same directory in one of the following image formats: TIFF, PNM, SGI, RGB, DPX. Interlaced image sequences [video fields] are expected to be stored as a sequence of images containing two woven fields. The filename is assumed to be in the format name.n.ext, where n is the frame number with up to 8 digits and ext is the file extension. The number of digits in the frame number has to be constant for all frames of one sequence." (p. 13) Frank Wylie of the Library of Congress Film Preservation Laboratory confirms that most DPX-capable applications provide this type of filenaming as the default.

As near as the authors have been able to determine, the content differences between DPX v.1 (SMPTE 268M-1994) and DPX v.2 (SMPTE 268M-2003) specifications are, in summary, updating references to other SMPTE standards and assigning previously reserved file space for specific purposes. This document does not detail all the changes in references to external SMPTE or other standards. However the substantive content changes between the v.1 and v.2 standards are listed below:

• DPX v.2 (SMPTE 268M-2003) specification includes more definitions including ones for: 3.2 image file size, 3.7 image element data structure, 3.31 X scanned size, 3.32 Y scanned size, 3.33 film edge code information, 3.40 SMPTE time code, 3.41 SMPTE user bits, 3.44 horizontal sampling rate (Hz), 3.45 vertical sampling rate (Hz), 3.52 integration time.
• In DPX v.1 (SMPTE 268M-1994) Section 5.3 (Image Orientation Information), Field 42, Offset 1636, Length 28 is reserved for future use. In DPX v.2 (SMPTE 268M-2003) Section 5.3 (Image Source Information Header), Field 42, Offset 1636 is reassigned for Data Structure for additional source image information including Field 42.1, Offset 1636, Length 4 Type R32 = X scanned size; Field 42.2, Offset 1640, Length 4 Type R32 = Y scanned size; Field 42.3, Offset 1644, Length 20 Type TBD = Reserved for future use. In addition, these terms were added to the definitions.
• In DPX v.2 (SMPTE 268M-2003) Table 3B (Component Data Packing Method), values for Field 21 have expanded to include method A (value 1) and method B (value 2) where as in DPX v.1 (SMPTE 268M-1994), the there was only value 1. Method A: "Filling method A is normal: padding bits precede data within 32-bit word boundaries (10-bit image components) or within 16-bit word boundaries (12-bit image components. See Annex C, figs C.3 and C.4." Method B: "Filling method B is now non-standard: padding bits follow data within word boundaries. See Annex C, figs C.7/C.8." Note that this also affects the specification in other places.
• In DPX v.2 (SMPTE 268M-2003) Table 4 (Video Signal Standard), additional code vales for signal standards are added from values previously reserved for future use. New values include: 153 = YCBCR 1125-line, 2:1 interlace, 16:9 aspect ratio (SMPTE 240M) and 203=YCBCR 1125-line, 1:1 progressive, 16:9 aspect ratio (SMPTE 274M). Reserved values are adjusted to: 154–199= Reserved for future high-definition interlace and 204 – 254= Reserved for future high-definition progressive.
• Table 6 (Time code and user bits) is a new addition to DPX v.2 (SMPTE 268M-2003).
• In DPX v.2 (SMPTE 268M-2003), the Annexes are reordered due to the introduction of a new Annex A (informational): Structure of 268M file and representation in document.
• In DPX v.2 (SMPTE 268M-2003), Annex C (informative) Data Packing Diagrams - Including “Method A” Filling and Annex C (informative) Data Packing “Method B” are revised to reflect the changes in Field 21 (see above).
• In DPX v.2 (SMPTE 268M-2003), Annex D (informative) Relationship of metadata items to SMPTE metadata dictionary RP 210 is a new addition.

Moreover, Amendment 1 to DPX v.2 (SMPTE 268M-2003),was published in 2012. A summary of key changes are listed below:

• Inclusion of a normative reference to SMPTE ST 2065-3, Academy Density Exchange Encoding (ADX) Encoding Academy Printing Density (APD) Values which is a component of the Academy Color Encoding System (ACES) organized by the Academy of Motion Picture Arts and Sciences.
• In Tables 5A (Transfer Characteristics) and 5B (Colorimetric Specification), code value 13 which was previously reserved for future use has been revised to refer to SMPTE ST 2065-3, Academy Density Exchange Encoding (ADX).
• Table 7 in SMPTE 268M:2003, previously reserved for User Defined Data, is significantly revised to include header fields and value pairs for ADX values.

Finally, the SMPTE Standards Quarterly Report (link available through Internet Archive) from March 2015 indicates that the latest revision of the DPX v. 2 standard, SMPTE ST 268:2014 was found to contain "significant errors ...[and] a corrected version is being put together." The authors of this website will wait until a stable version is released before conducting any analysis.

Federal Agencies Digital Guidelines Initiative Audio-Visual Working Group has drafting guidelines for embedding metadata in the DPX header. The draft guidelines outline FADGI implementations of the SMPTE Core fields as well as other elements Strongly Recommended, Recommended or Optional for FADGI use. The non-Core fields take advantage of existing header structures as well as define new metadata elements for the User Defined fields to document, among other things, digitization process history. The draft guidelines and supporting documents are now available for public comment through February 2017.