Department of Interior Logo

Department of the Interior Open Data Catalog Upgrade and Support

https://data.doi.gov/

The Department of Interior (DOI) needed to upgrade its open data platform (data.doi.gov). Through the development of both a refresh and upgrade in 2017 in line with new OMB Policy, proper operational planning and support, and a new operational data flow concept to work with bureaus, DOI is now being harvested directly from data.gov through 2022.

The Problem and Solution: Address New OMB Policy and Technical Debt

New Office of Management and Budget (OMB) level Policy required Departments to aggregate data from its component bureaus, programs, etc. In order to do this, the DOI needed to upgrade its open CKAN data platform (data.doi.gov), create standard operating procedures, requirements definitions, and platform design solutions, so that bureaus and agencies could properly store and distribute their data to the public with the platform’s aid.  Also, outdated versions of their Open Source driven CKAN open data platform could not meet the needs. As a side note, CKAN is a web-based open-source management system for the storage and distribution of open data.

The legacy platform was outdated and in need of maintenance and better practices. Also, the DOI was facing issues in publishing data onto the system. Consequently, bureaus and agencies had bypassed DOI to address public needs. As a result, the DOI catalog went unused for a few years. To address this, Xentity first supported DOI in discovering and defining new requirements. During design, rather then march ahead, the concept led to envision and coordinate with data.gov and other Government CKAN efforts to share the solution and best practices to allow for the overall CKAN Community to learn and implement together per Open Source best practices. This included coordinating with GSA, Federal Geographic Data Committee (FGDC), DOI, and all bureaus and programs to address various metadata requirements (FGDC, ISO).

Outcomes: Data.gov And Data.Doi.Gov Aligned To Latest CKAN Design

The new solution went live in 2017 resulting in a CKAN open data platform that aligns http://data.doi.gov and http://data.gov. This allows both programs to modernize together increasing the scaling and lower future technical debt for both. With new records harvesting rules implemented, the resulting dataset reached 50,000 to 2 million records to match OMB Policy guidance. The new http://data.doi.govautomates aggregation over 40+ feeds from bureaus and programs.

By addressing these issues collaboratively, the team had worked closely with data.gov to triage and address dozens of harvesting issues impacting DOI, data.gov and multiple federal agencies in harvesting, duplication, and other data flow management issues. Furthermore, this has set the stage for future transformation.

Through 2022, Xentity is providing the on-going harvesting, upgrades, and bureau/program support and documentation for open data flow management and platform of http://data.doi.gov for 50+ administrators. With the DOI CKAN platform now better maintained and the open data platform now upgraded, bureaus within the DOI are now publishing their data to data.gov.

The solution focused on efficiency first – getting back to a solid foundation. Open data is now more easily accessible. These results have provided more functionality to the DOI as a whole. DOI is fully compliant with the White House and back as a leader in the Federal Open Data efforts.