Managing the Explosive Growth in Data: Smart Strategies for Federal IT Leaders

There may be no adjectives left to describe just how fast the data universe is expanding. A white paper

by A.T. Kearney posited the data equivalent to Moore’s Law that computing power will double every two years. But data expands even faster, Kearney estimated with the volume of data worldwide doubling almost every year.

This lies at the heart of the government’s data storage challenge.  However the challenges facing government agencies are more complex than just storing all that information.  “The really challenging part of data is how to manage the information so that it can be available to the agency, to the public, and other organizations,” said Jeff Kramer, Reed Tech.

At the Digital Government Institute 930 Gov Conference held in September in Washington, D.C. similar sentiments to Kramer’s were heard frequently from speakers in the knowledge management and records management track, including Lisa Haralampus, Director of Records Management Policy and Outreach at the National Archives and Records Administration (NARA), her colleague, Director of Records Management Oversight and Reporting, and Laurence Brewer, Chief Records Officer for the U.S. Government.

“Managing information is a challenge in this era where there is so much of it and so many ways in which data can be used in everything from delivering citizens services, to FOIA requests, and use in internal investigations,” said Kramer.  “Not only are we contending with volume, but for federal agencies that contain documents spanning hundreds of years plus electronic data that is frequently in obsolete formats, the challenges are far greater.”

One of the most important steps according to the government panelists is to ensure the value of the data present and future users. Classifying data However, Kramer suggested that the technical challenges of storing, organizing, analyzing, and ensuring accessibility of the data itself is something that can be handled in partnership with organizations that specialize in knowledge management.

“In order to get to the point where agencies like NARA can implement machine-based learning strategies to help with complex tasks, like e-discovery and regulatory compliance, for example, data needs to be processed in ways that ensure the accuracy and completeness of the records,” shared Kramer.

Data cleansing is a particularly challenging issue for government agencies because they typically have several generations-worth of data stored in different formats, or have a complicated mix of structured and unstructured data to work with.  Kramer offered some hope, though, for agencies.  “Many entities, such as the United States Patent and Trademark Organization, are dedicating significant internal resources and working closely with industry and academia to standardize their data and make it more accessible and useful to everyone.”

To be most effective in meeting the demand for citizen services or elements of a data-driven mission, the agency needs to be focused on their core mission, not worrying about compliance rules, data maintenance, or accessibility,” Kramer concluded.

Related Posts