As much as we like to imagine we live in a world in which technology is king and open data prevails, the reality is often more Stone Age than Silicon Valley. This is a reality which Marc DaCosta, co-founder and chairman of data management company Enigma, is acutely aware of, having spent the last four years mining public data from across the U.S. and Europe.
“We’ve had experiences calling some secretaries of state in the United States and asking, ‘What is the process for getting machine-readable data or a data dump of all of the companies in your state?’ and them having really no idea what we’re talking about and it being a big process to even articulate the request,” he says. “On the other hand, there are jurisdictions where it’s really clean, really well published, and immediately available.”
Enigma is in the business of making public data usable and accessible to everyone—something that seems implicit in the name “public data,” but often requires dedicated research to uncover, access, and make available. Once the public data is found and parsed, it’s published on Public, Enigma’s own open platform that aggregates relevant data sets and facilitates the analysis of that data for journalists, researchers, data scientists, and curious members of the public.
“One thing that we’ve got very good at is finding where the data is,” says DaCosta, “which can be challenging if you don’t know immediately who is the publisher of a certain thing. Sometimes we have to file Freedom of Information Act requests or go through other means to get data that’s otherwise not available.”
Enigma’s goal is to promote transparency and increase accountability within government, which “despite the shift in administration and the way in which some things are becoming perhaps less transparent, other things, like government contracting and spending are continuously getting better in terms of the information that’s available—but it’s definitely a long game.”
Enigma’s public catalog now numbers some 100,000 datasets, meaning the navigation and use of the original Public site had become increasingly complex. Cue a platform redesign that focuses on enabling users to more readily make connections between these datasets and chance upon information they wouldn’t otherwise be able to find.
“Our users should always ask why and be empowered to find an answer,” says Melody Hammer, Enigma’s design manager. “We strive to offer clarity and complexity, empower the expertise of our users, and provide an experience that feels alive and involved with our users’ learning.”
With that in mind, the new iteration of Public has been through rigorous user testing with groups from diverse professional backgrounds. “One group was looking at data journalists and people who work with data in various investigative ways,” says DaCosta. “We also spoke with a lot of our customers who come at things from more of a quantitative and data analyst perspective.
Then, thirdly, we engaged a cohort of people without any particular expertise in data, but who are avid news consumers. What we found was this feedback that really drove us towards wanting to focus on providing context to people.”
The findings resulted in a new homepage that earmarks notable and newsworthy datasets, and additional tools for users to more effectively find comparable datasets through Enigma’s meticulous tagging and arrangement of metadata. It has also beefed up its integration with external tools, and is working on the addition of maps and charts to aid users in their discovery and interpretation of data.
“We started to understand the ways people ask questions of data and what their research process is,” says DaCosta. “Later in people’s workflow they’re trying to bring the data into tools for visualisation or statistical analysis; it became clear to us that we needed to have a certain design for our API protocol so that it would be easy to integrate into those sorts of contexts.
“We were constantly looking to find a happy medium for both the experienced pro user and the more generally curious public user.”
Although the first version of the redesign is complete, Hammer says Enigma will continuously develop Public, implementing new updates and addressing user feedback as it is received, not to mention adding new datasets pertaining to different areas of interest.
“We have a couple of different perspectives that inform our data acquisition efforts,” says DaCosta. “One of them is certainly related to sources of information that we think are important to have accessible. In that respect we’ve been doing a lot of work lately around trying to gather information related to corruption and datasets that help to contextualise the activity of politicians and governments.
“Another tranche is really understanding the kind of the data that our customers and clients need. For instance, we worked with the city of New Orleans on a project that was trying to fight deaths from fires, and we were thinking about what kinds of datasets could speak to addressing that risk. We identified data that the United States Census Bureau had published on fire incidence reporting, which we made available to everyone who uses the platform, and were able to use it to make something for U.S. fire departments to direct their outreach efforts.
Our belief is that data is an enabler, it’s a tool, it’s something that helps us contextualize the world, but ultimately it’s only useful and relevant when it’s applied to a particular problem.