Top of page

Notice: We are making improvements in the structure of the Web Archives that may result in intermittent unavailability. Read more about these improvements.

Program Web Archiving

For Site Owners

The Library notifies each site owner that we would like to include their content in the archive (with the exception of government websites) prior to archiving. In some cases, the email asks permission to archive or to provide off-site access to researchers.

If a link to this page appeared in your web server logs, or in an email notification from the Library, it is because your website is being crawled by the Library of Congress. The Library has selected your site for inclusion in its historic collection of Internet materials. For general information about the web archiving program, visit our About page. To learn more about the web archiving of your site, visit our site owner FAQ.

The Library of Congress (or its agents) collects content from websites at regular intervals, primarily using the Heritrix External crawler, which is an open-source archival web crawler. The crawler begins with a "seed URL" - for instance, a homepage – and follows links on the page, and subsequent pages, downloading copies of content it finds that help make up that page, so that the content can be preserved. Our crawler is instructed to bypass robots.txt External in order to obtain the most complete and accurate representation of websites. Site elements that are sometimes excluded through robots.txt instructions that are vital to the reproduction of a site’s look, feel, and functionality include images, CSS, and JavaScript, to name a few. 

If crawling is impacting the performance of your site or you have other concerns, please contact the Library of Congress Web Archiving Team immediately at contact us. The Library would prefer to refrain from crawling any site areas that are not intended for the general public, such as administrative sections. The Web Archiving Team is happy to discuss ways to facilitate capture of desired elements and content by the Library of Congress or its agents or to mitigate crawling of these sections.

The Library hopes that site owners share its vision of preserving web materials. If you have questions, comments, or recommendations concerning the collection of your website by the Library of Congress, please do not hesitate to contact the Web Archiving Team.

Contact Us

Comments, questions, and suggestions related to Web Archiving and this website can be sent to us online.


Web Archiving Team
Library of Congress
101 Independence Avenue, SE
Washington DC 20540-1310