• Preservation Home
• About
• Collections Care
• Conservation
• Digital Preservation
• Emergency Management
• En Español
• FAQ
• Preservation Science
• Resources
• Training Opportunities
• Contact

Related Links

• Donate
• Audio-Visual Preservation
• National Film Preservation Board
• National Recording Preservation Board

VI. Datasets/Databases

The Library is aware that, in some cases, the provision of datasets and databases for current research uses (including support for the U.S. Congress) may depend upon native formats and associated software, while preservation and long-term access may depend upon data-migration via transport or export formats, with a concomitant risk of loss of precision and accuracy. Given the focus of this document is preservation and long-term access, the following format preferences favor those outcomes.

1. Datasets
   (For Geospatial Data, see Section VI.ii below)

   Preferred:

1. Formats
1. Platform-independent, character-based formats are preferred over native or binary formats as long as data is complete, and retains full detail and precision. Preferred formats include well-developed, widely adopted, de facto marketplace standards, e.g.
1. Formats using well known schemas with public validation tools available
2. Line-oriented, e.g. TSV, CSV, fixed-width
3. Platform-independent open formats, e.g. SQLite (.sqlite, .db, .db3)
2. Character Encoding, in descending order of preference:
1. UTF-8, UTF-16 (with BOM)
2. US-ASCII or ISO 8859-1
3. Other named encoding
2. Related Materials
1. Consult the appropriate sections of this document to identify the preferred formats for supplementary material
3. Delivery Method, in descending order of preference:
1. Public download URLs
2. Automated private download URLs with any necessary API keys or credentials
3. Hard drive; CD-ROM; DVD-ROM
4. Metadata
1. Deposits should include all applicable metadata, data dictionaries, XML schemas, and technical specifications as appropriate. Discipline-specific metadata standards should be used whenever possible
2. As supported by format:
1. Title
2. Creator
3. Creation date
4. Place of publication
5. Publisher/ producer/ distributor
6. Contact information
7. A list of software used to produce, render or compress the data (if applicable)
8. Character encoding
3. Include if available:
1. Language of work
2. Other relevant identifiers (e.g., DOI, LCCN, canonical URL, etc.)
3. Subject descriptors
4. Abstracts
5. Key or reference to each data field
6. Checksums
7. Permanent version specifiers
8. Information about how the data was collected and any sampling or post-processing which has been applied
9. Known copyright terms, especially for datasets which combine data from multiple sources
4. For datasets serving as part of a database: proprietary database package and version
5. Technological Measures
1. Files must contain no measures (such as digital rights management technologies or encryption) that control access to or prevent use of the digital work.
2. Files in formats which support linking or embedding external resources (e.g. XML, JSON, Excel) should be self-contained to remain useful in the event of external service changes.
3. Files in formats which support executable code (e.g. Excel) do not contain executable code.

Acceptable:

1. Formats
1. Non-proprietary, publicly documented formats endorsed as standards by a professional community or government agency, e.g. CDF, HDF
2. Any proprietary format that is a de facto standard for a profession or supported by multiple tools (e.g. Excel .xls or .xlsx)
3. Text-based data formats with available schema
2. Related Materials
1. See Preferred
3. Delivery Method
1. See Preferred
4. Metadata
1. See Preferred
5. Technological measures
1. Files in formats which support executable code do not depend on embedded programs for purposes other than display (e.g. search, filtering, etc.); the raw data is available without executing code.

Back to Top



2. Geospatial Data

Preferred:

1. Formats, in order of preference
1. Most complete data (all layers, appendices), even if proprietary
2. Formats compatible with widely adopted GIS (e.g. ArcGIS)
3. Formats compatible with recommendations and tools from geospatial open source and open data communities
1. Formats developed or endorsed by the Open Geospatial Consortium (OGC) (e.g., GML see: http://www.opengeospatial.org/)
2. Formats supported by well supported open source software libraries such as GDAL, OGR and GeoTools
2. Related Materials
1. Consult the appropriate sections of this document to identify the preferred formats for supplementary material
3. Delivery Method, in descending order of preference:
1. Public download URLs
2. Automated private download URLs with any necessary API keys or credentials
3. Hard drive; CD-ROM; DVD-ROM
4. Metadata
1. For metadata information see Federal Geographic Data Committee (FGDC) 
2. To the extent allowed by the underlying format, include available information about how the data was collected and any post-processing which has been applied
5. Technological measures
1. Files must contain no measures (such as digital rights management technologies or encryption) that control access to or prevent use of the digital work.

Acceptable:

1. Formats, in order of preference
1. See Preferred
2. Related Materials
1. See Preferred
3. Delivery Method
1. See Preferred
4. Metadata
1. See Preferred
5. Technological measures
1. See Preferred

Back to Top



6. Databases

Preferred:

1. Preservation
1. Complete set of the content contained within the database, conforming to preferred specifications in sec. VI.i-ii
2. Access, in order of preference
1. Publisher web interface, with
1. Comprehensive and user-friendly search and discovery
2. Counter compliant usage statistics
2. Delivered preservation content

Acceptable:

1. Preservation
1. See Preferred
2. Access, in order of preference
1. Documented API

Back to Top