Sustainability of Digital Formats: Planning for Library of Congress Collections

Introduction | Sustainability Factors | Content Categories | Format Descriptions | Contact
Format Description Categories >> Browse Alphabetical List

Lotus Notes Storage Facility

>> Back
Table of Contents
Format Description Properties Explanation of format description terms

Identification and description Explanation of format description terms

Full name Lotus Notes Storage Facility
Description

The Notes Storage Facility (NSF) non-relational database file is used by IBM Lotus Notes and Domino software to store different kinds of data stored in Notes including email messages, chat and instant messages, documents, appointments and other calendar entries. A Note is similar in concept an object store because it normally contains both an application design and its associated data.

The mailbox database comprising Note types such as email messages, chat and instant messages, calendars entries and the list of To-Do as well as other archived data is stored in a single username.nsf file for each individual user. Contact Notes in the address book database are stored in the names.nsf file although contracts also can be saved into a readable format-vCard (VCF) or Comma Separated Value (CSV) file. A third important component are Notebooks, also known as Personal Journals, comprises of personal documents and information that can define the priority documents of the custodian. This database of Notes gets saved into the notebook.nsf file.

Structurally, a NSF file contains a file header (made up of a signature and the database header size), followed by the database header comprised of, among other data points, the software version information (not NSF file format version), file size and database identifier (DBID) the value of which consists an NSF date and time. Replication information (DBREPLICAINFO) is an important part of the database header. IBM Domino servers and Lotus Notes clients identify NSF files by their Replication Identifier (the value which consists an NSF date and time in local time) and keep replicate files synchronized by bi-directionally exchanging data, metadata, and application logic and design. Replication history, including direction (never, send, receive), is also stored in the database header. See General for more information about replication.

The database header is followed by various data constructs including:

  • superblock (containing a header, data and footer)
  • bucket descriptor block (containing a header, data and footer)
  • bitmap
  • Record Relocation Vector (RRV) bucket
  • summary buckets, and
  • non-summary buckets.

Local use Explanation of format description terms

LC experience or existing holdings  
LC preference  

Sustainability factors Explanation of format description terms

Disclosure Open partial documentation produced by Joachim Metz.
    Documentation Notes Storage Facility (NSF) non-relational database file format: Analysis of the NFS database file format is a working document for the NSF specification but is not yet complete.
Adoption

Highly adopted thanks to what Wikipedia calls a "long market history and large installed base." Its encryption capabilities make it a popular choice among federal agencies and corporate entities dealing with classified and sensitive material. Reports of repeated predictions of the decline or impending demise of IBM Notes and Domino have long circulated although new software releases continue. In November 2014, IBM announced a new web-based product for email services named Verse which may encourage existing Notes users migrate into Verse.

Email NSF files can be exported to a variety of formats. Starting with Release Notes 8.5.2, export to EML was introduced. Options prior to that included "Adobe Acrobat (presumably a PDF variant) or some other such file type" and export to ASCII Text or Microsoft RTF file "which exports the content of the message including the To, cc and bcc fields. The exported data can be used as an attachment."

    Licensing and patents NSF files are intended for use only with IBM Lotus Notes and Domino software. Patents are unknown but are presumed to exist. Comments welcome.
Transparency

According the to IANA registration, NSF files are binary with base64 encoding preferred.

Encryption has a long history in Notes starting with Release 1.0 which introduced with mail and port encryption. The use of encryption was extended in Release 2.0 with document encryption and secret keys for field-level encryption. Database and database design encryption were added in Release 4.0, and Release 4.5 included document encryption with public keys. With email messages, only the plaintext Body field encrypted as cyphertext, not the Subject, Sender, and Date fields. Email messages can be encrypted before sending, after receipt or before saving. Email message encryption is accomplished through "a public and private key set to encrypt and decrypt data, as well as to validate digital signatures. The public and private key in a set are mathematically related to each other and are unique to your User ID. Your public key is stored in your Notes certificate. Your certificate is stored in your User ID and the Domino Directory. Your private key is stored only in your User ID." Encrypted messages sent by other people use the public key from the intended recipients certificate located in the Domino Directory. When an encrypted messages is received, the private key in User ID decrypts the data.

IBM DeveloperWorks explains that full database encryption (defined in the DBCREATE_ENCRYPT_ field) are labeled as None, Simple, Medium, Strong adding that "the stronger the encryption, the longer it takes to open the database. So long as you're not using disk compression, in most cases, the best option is medium encryption because of its balance between security strength and speedy access; however, if you're using disk compression, select simple encryption so that the disk compression works effectively."

Security information is stored in the Lotus Notes ID file, usually ending in a *.id extension. The security information along with the user password permits access to a server and personal encrypted information.

The length of the encryption key is limited appropriately to achieve a balance between encryption process time and security. Under current US export laws, IBM Notes supports only one version of the Notes PKI with 128-bit symmetric keys, 1024-bit public keys, and no workload reduction factor. See Historyfor more background on encryption key lengths and their impact on Notes.

Self-documentation Metadata documenting important characteristics of the resource found in the NSF file and database.
External dependencies NSF files are intended for use only with IBM Lotus Notes and Domino software.
Technical protection considerations Maybe move encryption info here?

Quality and functionality factors Explanation of format description terms


File type signifiers and format identifiers Explanation of format description terms

Tag Value Note
Filename extension nsf
 
Internet Media Type application/vnd.lotus-notes
application/x-lotus-notes
From https://www.iana.org/assignments/media-types/application/vnd.lotus-notes
Magic numbers 1A 00 00 04 00 00
From Gary Kessler's File Signatures Table.
Pronom PUID x-fmt/336 (Lotus Notes Database Version 2)
See http://www.nationalarchives.gov.uk/PRONOM/x-fmt/336.
Pronom PUID x-fmt/337 (Lotus Notes Database Version 3)
See http://www.nationalarchives.gov.uk/PRONOM/x-fmt/337.
Pronom PUID x-fmt/338 (Lotus Notes Database Version 4)
See http://www.nationalarchives.gov.uk/PRONOM/x-fmt/338.
Wikidata Title ID See note.  No entry as of March 2017

Notes Explanation of format description terms

General

Although compression is not used within NSF files themselves, attachments to NSF email messages (stored outside the NSF file) have three compression options: None, CX (Huffman encoding) or LZ1 (introduced in introduced in Notes 6).

Replication between two servers, or between a client and a server, can occur over a network or a point-to-point modem connection. Creation of a local replica of an NSF file on the hard disk of an IBM Notes client enables the user to fully use IBM Notes and Domino databases while working off-line. The client synchronizes any changes when client and server next connect. Local replicas are also sometimes maintained for use while connected to the network in order to reduce network latency.

Wikipedia: A key feature of IBM Notes is that many replicas of the same database can exist at the same time on different servers and clients, across dissimilar platforms; the same storage architecture is used for both client and server replicas. Originally, replication in Notes happened at document (i.e., record) level. With release of Notes 4 in 1996, replication was changed so that it now occurs at field level. A database is a Notes Storage Facility (.nsf) file, containing basic units of storage known as a "note". Every note has a UniqueID that is shared by all its replicas. Every replica also has a UniqueID that uniquely identifies it within any cluster of servers, a domain of servers, or even across domains belonging to many organizations that are all hosting replicas of the same database. Each note also stores its creation and modification dates, and one or more Items.

History

Notes and Domino began in the work of Ray Ozzie, Tim Halvorsen, and Len Kawell, first on PLATO Notes at the University of Illinois and later on DECNotes. Lotus founder Mitch Kapor saw the potential in Ozzie's collaboration project and the rest is history. The first release of Lotus Notes shipped in 1989. Lotus Development Corporation originally developed "Lotus Notes" in 1989. IBM bought the Lotus corporation in 1995 and it became known as the Lotus Development division of IBM. As of 2015 it forms part of the IBM Software and Systems Group under the name "IBM Collaboration Solutions".

According to Wikipedia, "IBM Notes was the first widely adopted software product to use public key cryptography for client–server and server–server authentication and for encryption of data. Until US laws regulating encryption were changed in 2000, IBM and Lotus were prohibited from exporting versions of Notes that supported symmetric encryption keys that were longer than 40 bits." The US export laws restrictions on key length for encryption contributed to the development of three major editions of Notes: North American (uses 64-bits long encryption keys), International ("64-bit key for the International edition, the top 24 bits are encrypted using the U.S. Government's public key and stored in what is called the Workfactor Reduction Field [WRF]. Splitting the key in this manner results in a key that's 40 bits for the U.S. Government and 64 bits for everyone else."), and French (a less secure plain 40-bit encryption key).


Format specifications Explanation of format description terms


Useful references

URLs


Last Updated: 07/27/2017