It’s the middle of the workday in New York City and I’m watching a shaky phone video of a transit worker being rescued from a subway track. I scroll through the list of 76 comments, which range from “wow hope the person is ok,” to “I’m just tryna get to work.” Most just are just one word: “Damn.”

Over 25,000 people have seen the video, but my fellow rubberneckers and I aren’t on Instagram or Twitter. We’re on the crime-spotting app Citizen, which listens to public 911 calls and turns them into real-time alerts about local emergencies. With a foothold in five U.S. cities and over a million users in New York alone, Citizen is part of a new generation of security apps using quasi-public spatial data to transform the way people inhabit their communities. 

Until now, the company has been building a solution in search of a problem. Citizen has sparked controversy in its two years of operation for spreading both paranoia and stereotypes about urban safety. In New York, its largest market, a Brooklyn city councilman recently criticized the app for reigniting misplaced fears of violent crime. (Major felonies in the city have decreased by almost half since 2000 and continued to fall since the NYPD’s controversial “stop and frisk” policy was ended in 2014.) Meanwhile, activists have charged that Citizen enables new kinds of racial profiling.

“The first version of Citizen was definitely a prototype,” said Keith Peiris, Citizen’s head of product. “The challenge is that when you take the open-air radio system and you pipe those incidents to everyone around, sometimes you’re sharing too much. You can certainly create a lot of paranoia and anxiety.”

Until now, Citizen has been building a solution in search of a problem.

Citizen aims to address these criticisms head on by rolling out a major redesign this month—its first since launching as “Vigilante” in 2017 and getting itself promptly banned from the App Store due to concerns about its content. According to Peiris, the new interface is a sign that the company is finally locating its mission. His team is working to transform the app away from simply reporting crime towards a tool that helps neighbors help each other when real emergencies happen. But as it redesigns and rebrands the app, Citizen must balance calls for change with the habits of its current users and the expectations of its investors. It remains to be seen whether revising Citizen’s interface will address underlying concerns about its ethics, and whether its shortcomings are too deep-seated for designers to solve.

The redesigned version has at least minimized the app’s most ominous feature: a dark map that sprouts red dots wherever new emergencies are reported. Where users previously saw a panoptic view of crime across their city, they will now find a simple summary for their immediate surroundings. When someone’s neighborhood is quiet, Citizen de-prioritizes attention hooks like notifications and comment threads. A user could still upload that video of the subway track rescue, for example, but it would only be visible to their neighbors. Even then, it probably wouldn’t warrant an appearance on anyone’s home screen.

“The big thing is we just want you to check [Citizen] and close it,” Peiris tells me. “We’re not a company that profits from engagement, we just want to be this thing that you use to stay safe; we don’t really care how many times you open it.”

Though the main map will remain dark, the team is using color to deliver more nuanced safety information. Incidents are now coded by severity: green for minor advisories, yellow for situations to avoid, and red for emergencies where someone is likely to get hurt. “We’re looking at all colors, in all situations, at all times of day,” said graphic designer Mackey Saturday. “Is there something about bright colors that encourage clarity and optimism, and less a feeling of ‘Oh no, something’s wrong right now?’”

The new map also comes with a subtle new social component: streets with higher densities of Citizen users appear bolder and brighter. The feature has already influenced beta testers’ walking habits, Peiris says, leading them to take streets where they might broadcast to more users for help.

It’s in moments like these that Citizen exemplifies the tenuous balance between good intentions and powerful downstream effects, which can turn product design into a high-stakes social experiment.

Peiris described this as “a heuristic for safety in numbers,” but in another light, it’s not hard to see it as a behavioral nudge in the direction of “digital redlining,” in which algorithms reinforce existing geographies of racial and economic inequality. Peiris told me that for now, Citizen’s density map is representative of the city’s demographics. “But I think as things deviate,” he said,  “we’re going to have to make it really clear to people who use the app, what to use it for and what to not use it for.” It’s in moments like these that Citizen exemplifies the tenuous balance between good intentions and powerful downstream effects, which can turn product design into a high-stakes social experiment. Exciting new features get the green light while the onus falls on third parties to point out their consequences.

Citizen may face greater pressure to change its practices because the platform has shown more willingness than its competitors to implement ethical safeguards. As opposed to the neighborhood social network NextDoor, Citizen employs a team of moderators who approve every comment. Citizen also forbids reports about specific individuals in order to prevent hate and harassment. This is in stark contrast to Neighbors, the app from Amazon’s security-camera subsidiary Ring, which allows users to tag video subjects as “suspicious,” publishes images of alleged suspects in social media advertisements, and built a platform for law enforcement to request footage directly from Ring owners. 

That’s not to say that Citizen is free from abuse, though; a casual perusal of the comments turns up plenty of discriminatory language. But the product team at least acknowledges the app’s ability to instill bias, and is taking steps to reduce it by localizing the experience and moderating comments more proactively. More significantly, the company has also pledged not to share user data with third parties or run any form of advertising. 

Citizen is trying to toe a more responsible line while diving headlong into the space where law enforcement, activism, and municipal policy intersect.

The company’s long-term path to profit has more to do with its infrastructural value than its engagement numbers. Peiris says his goal is to create a “global safety network” composed of billions of Citizen users. In that future, anyone would be able to ask the network for help (or volunteer to help someone else) at a moment’s notice. “I love the idea of a universal help signal,” he says. “It doesn’t matter whether you’ve broken your ankle, you need an EpiPen, you need immediate help—there’s a way to mobilize the people around you so the right person can come to you and give you what you need.”

But since startups are both unpredictable and unaccountable, it’’s difficult to assess how this blue-sky aspiration will translate into a concrete business with societal impacts. Take Uber as an example: while the company predicted a future full of of hyper-efficient transit and autonomous driving, its track record of strong-arming and undermining public institutions tells a very different story. Uber sought to bully its way into a monopoly position while its core business hemorrhaged cash. Citizen is betting that responding to critics with a measure of good faith will prove that it’s a very different kind of company, but the pressure from VCs to multiply their investment will never let up. The company’s financial obligations could one day present a crossroads when its global ambitions come into deeper conflict with the public sphere.

It’s perhaps for this reason that Citizen frames its new features in terms of mutual aid. To help ordinary people respond to all kinds of emergencies, it will eventually allow users to configure their own groups with friends and neighbors. Saturday described these as “overlapping micro-communities” that make up the larger network. “The more and more those overlap,” he says, “the more area we cover, and the more people universally are protected.”

Citizen is trying to toe a more responsible line while diving headlong into the space where law enforcement, activism, and municipal policy intersect. And while the team has designed solutions to its preliminary missteps, its most serious challenges will come when product design runs up against thornier issues of governance and community accountability.

“We know that in the space that we’re in, this is a thing where we need to do this together,” said Peiris. “We know that the only way to get to a city and actually elevate is to work with everyone.”