Sustainability of Digital Formats: Planning for Library of Congress Collections

The PDF/R format was developed by the TWAIN Working Group and the PDF Association (PDFA.org) as a very limited subset of PDF, designed primarily for storing and transmitting scanned page data. The format is a key component of TWAIN Direct, an open protocol intended to allow applications to communicate with scanners without the need for proprietary scanner drivers. A specification for PDF/R (aka PDF/raster) was submitted for international standardization to ISO/TC 171/SC 2 and approved for publication as ISO 23504-1:2020 -- Document management applications -- Raster image transport and storage -- Part 1: Use of ISO 32000. The specification document covers two variants of PDF/R, differing in whether they are unencrypted and conform to ISO 32000-1 or are encrypted and conform to ISO 32000-2 (often referred to as PDF 2.0). This format description is focused on the former. For the differences specific to the encrypted variant of PDF/R-1 based on PDF 2.0, see PDF/R-1_enc.

Among the benefits and advantages over the full PDF format presented in the introduction to the PDF/R-1 specification are the following:

• Files can be read and written without a full PDF parser or generator and rendered without a complex rendering engine.
• Files can be created efficiently from raster images and generated using a fixed-size raster data buffer.
• The exact original raster image data can be recovered.
• Images can be located in the file and read efficiently with comparatively simple code.
• The specification provides a precise, well-defined target, simplifying engineering design and testing.
• PDF/R files can be quickly and easily identified as such by software. See File signifiers below.
• PDF/R supports only effective and readily available compression algorithms.

PDF/R retains optional PDF features for protecting and authenticating content:

• For implementations that need to protect document content at rest, there is an encrypted variant,which is based on ISO 32000-2 (PDF 2.0) in order to support stronger encryption than permitted in ISO 32000-1. See PDF/R-1_enc.
• One or more digital signatures may be used for implementations that require verification of the document origin, authenticity, date or time of creation, and so on.

The TWAIN Working Group considered pros and cons of PDF in comparison to JPEG and TIFF before choosing PDF as the basis for the file format for use in TWAIN Direct. See the introduction to PDF/raster slideshow or presentation video from 2017, given by the chair of the TWAIN Working Group. PDF was preferred to TIFF as an actively maintained international standard with widespread viewer support. Disadvantages of JPEG were the lack of support for multi-page documents or bitonal images.

The restrictions with which a PDF/R-1 file must comply were described informally in a 2017 Overview of PDF/raster by the PDF Association. The list of restrictions below is based largely on that overview:

• Page contents may only include images in ITU/CCITT G4 (FAX), DCT (JPEG), and uncompressed raster formats. These images are restricted to RGB color, grayscale, and bitonal formats.
• Page contents may not include anything other than raster images: no text, no line art, no forms, or other graphical elements. [Note: This excludes the incorporation of searchable text derived by OCR.]
• Transparency and layers are not permitted.
• No compression of non-image data is permitted. Compressed object streams are disallowed.
• Each page object shall contain a Contents stream that fills the required MediaBox (which defines the page size). [Note: Page-level metadata streams are allowed as well as document-level metadata.]
• The use of annotations is limited to those used for digital signatures. Digital signatures that render a visual presentation on the page are not permitted.
• Most optional structures are disallowed. For example, interactive actions, bookmarks, search indexes, and marked content are not permitted.
• Versions of PDF prior to 1.4 may not be used for PDF/R files.
• The only circumstance in which a PDF/R-1 file may include an incremental update is for the application of a digital signature.

As indicated in Annex A to the PDF/-1 specification, it is possible to create PDF/R files that also comply with PDF/A. However, the annex states that the only PDF/A profiles to which a PDF/R can conform are PDF/A-1b and PDF/A-2b; other profiles of PDF/A-1 and PDF/A-2 require searchable text.

PDF/R-1 is based on the publicly documented PDF/raster specification developed through a collaboration between two international associations that develop and promote standards, the TWAIN Working Group and the PDF Association (PDFA). In April 2020, the PDF/R-1 specification was approved as ISO 23504-1 under the auspices of ISO/TC 171/SC 2.

The TWAIN Working group describes itself as "a not-for-profit organization with the sole purpose of fostering a universal public standard which links applications and image acquisition devices." See https://twain.org/. On its website, PDFA states, "A global association, the PDF Association is an initiative of the Association for Digital Document Standards (ADDS) e.V., founded in September 2006." See https://www.pdfa.org/about-us/.

According to the introduction to PDF/Raster 1.0 at PDFA, "PDF/Raster was created in a collaboration between the TWAIN Working Group, which originated the PDF/raster concept, and the PDF Association, which provided PDF technology expertise and a means of communicating with the PDF software industry at large to ensure that a diverse range of relevant viewpoints was represented."

The specification for the unencrypted variant of PDF/R-1, which is based on ISO 32000-1, is in the same document as the specification for an encrypted variant based on ISO 32000-2 (aka PDF 2.0). The specification as originally submitted to ISO is available from the PDF Association and also from TWAIN's pdfraster.org site. An ISO version of the specification was approved as ISO 23504-1: Raster image transport and storage -- Use of ISO 32000 (PDF/R-1) in April 2020.

PDF/R is a relatively new specification. Adoption of PDF/R will be tightly coupled to adoption of the new TWAIN Direct protocol. An August 22, 2018 post from the Document Imaging Report site states,"adoption of TWAIN Direct has faced a chicken-and-egg type of challenge. Scanner vendors seem to be waiting for software vendors to release applications that support TWAIN Direct before embedding the technology in their devices. Of course, if no scanners support TWAIN Direct, there isn’t much driving software vendors to create connections to it." In 2018, TWAIN supplied a configuration for software development and testing by TWAIN members, using preliminary releases of products known as TWAIN Local and TWAIN Bridge. TWAIN Bridge is software for a PC that allows the TWAIN Direct specification to be used with existing scanners that support the widely used TWAIN specification now referred to as "TWAIN Classic."

In September 2019, the Twain Working Group made a public release of TWAIN Direct specifications. See announcements at PDF Association and in Markets Insider (Sept. 4, 2019).

The TWAIN Direct code repository on GitHub provides sample code for reading and writing PDF/R files.

The compilers of this resource would appreciate updated information related to the adoption of PDF/R. Comments welcome.

The original specification for PDF/R was developed by the TWAIN Working Group using the name "PDF/raster." In association with the PDF Association, the TWAIN Working Group submitted this specification in mid-2017 to ISO/TC 171/SC 2 for standardization. One of the changes introduced during the ISO process was to adopt the name "PDF/R." The specification was approved as ISO 23504-1 in April 2020.