The challenge
Understanding and using data
Public data is a critically under-utilised national asset. The Australian Government's data holdings are a strategic national resource with considerable value for growing the economy - the better that the government can understand and use its data, the better it is able to make decisions and discover new opportunities. Many government agencies hold massive quantities of data, but it often gets stuck inside silos where its importance is invisible, origins untracked, and existence unknown to those elsewhere in the organisation who could improve or derive further value from it.
Outside of Government, there is considerable potential for public data to be openly published, and further utilised by the private sector, researchers and engaged citizens. The Australian Government is one of the world leaders in making government datasets open, but for the public to make use of the data, they need well-described and up-to-date data and powerful discovery tools in order to quickly find the data that is relevant to them.
Our response
Providing open-source software
Magda is a data catalog system that provides a single place where all of an organisation's data can be catalogued, enriched, searched, tracked and prioritised - whether big or small, internally or externally sourced, available as files, databases or APIs.
Magda resolves the challenge of storing and using data by providing open-source software to assist government agencies in all aspects of data management—from collection and authoring, through internal discovery and use, all the way to sharing inter-organisation or publishing on an open data portal. In this manner it unlocks the potential of Australia's data, making it easier to find, access and reuse from both inside and outside the Australian Government.
Magda is supported under the Platforms for Open Data (PFoD) program which is a National Innovation and Science Agency (NISA) initiative. PFoD partners include the Digital Transformation Agency and the Department of Agriculture, Water and the Environment.
The results
Improving data use and sharing within the Australian Government
As part of discovery, the Magda team undertook extensive research in order to understand the current state of data sharing, pain points and future potential, conducting nearly 100 interviews with data users across the public, private and research sectors. The Magda team identified twelve opportunity areas where Magda could alleviate current state pain points and gaps around data sharing and access - this research has gone on to help inform the Office of the National Data Commissioner's efforts around improving data use and sharing within the Australian Government.
Since January 2019, Magda has been powering Australia's national data catalogue, data.gov.au. Data.gov.au now makes over 89,000 datasets drawn from sources all over the Australian government available for search, and makes these available to 10,000-20,000 individual users per week.
As an open-source project, Magda has also been used by other teams within CSIRO, including for the Knowledge Network project in Land and Water, and the NSW Spatial Digital Twin developed by the Terria team within CSIRO's Data61.
Over the past 18 months, Magda has been extending the functionality initially developed for data.gov.au into a more general-purpose data cataloguing solution for use within government agencies. Working prototypes have been developed and deployed for the use of the Department of Agriculture, Water and Environment (DAWE), and will continue to be added to in an agile manner. Of particular note is tackling the problem of making metadata around datasets easier and quicker to create for non-expert users.
The Magda team are looking forward to continuing to support data.gov.au , enhancing its reputation as an open-source project and rounding out its new metadata generation tool to DAWE, which is of particular use to the team writing the next State of the Environment report.