Overview

VoxGov Downloader is an automated search engine crawler designed to index and gather publicly available data from U.S. Federal Government websites and public domain resources. VoxGov has been crawling and indexing from U.S. web sites for over 10 years, and throughout this period we have at all times remained compliant with all web site requirements. Our users include leading Universities and Not-For-Profits who rely upon our service for comprehensive, accurate, and up-to-date search results for public government information, supporting transparency and easy access to official documents and data.

Purpose

• Crawl publicly accessible US Government websites, such as federal agencies, regulatory bodies, and public data portals.
• Collect and index documents including regulations, legislation, reports, and official records.
• Improve search accessibility for researchers, legal professionals, journalists, and the general public.

Behaviour

• Respectful:The bot locates any crawl-delay in a domain's robots.txt rules on all sites it visits and adheres to well configured values.
• No Access to Private or Sensitive Data: The bot only accesses publicly available pages and never attempts to bypass security or authentication measures.
• Identification: The bot user-agent header string clearly identifies itself as "VoxgovDownloader” with a link to this page. While the version number may vary, a typical example is:

  VoxgovDownloader/2.10.73.0 ( https://downloader.voxgov.com/bot )

• Low Impact: Unlike other search bots that crawl the entire domain, this bot only requests certain pages containing news article lists/links, and generally only downloads each linked article once, skipping ones it previously has downloaded.

Additional/Further Information

For more details on our data usage or privacy practices, please see our privacy policy.

If you have any questions or feedback this bot and/or its behaviour on your site, please contact us using our contact page.