Real-World Gaming Experiences with Responsible Safety

As Niantic introduced more social features, both offline and online, the team faced multiple scenarios that could lead to safety and player experience risks: toxicity in content-based interactions, undesirable player behavior during offline meetups, and risks to the physical safety of players walking outside while playing.

As Jen Weedon, Head of Adversarial Planning and Safety by Design at Niantic, explains: the technology is constantly evolving, the potential for harm is enormous, and not fully understood yet, making detection extremely difficult.

Niantic's Trust & Safety team turned to Alice for its researchers, policy analysts, and circumvention experts, requesting deep threat intelligence and red teaming to proactively detect and prevent unknown risks and policy loopholes early in the product development cycle.

Niantic needed to understand all the scenarios that could go wrong and ensure they could either prevent them by creating new policies or understand the different safety needs for different demographics. For Pokémon GO, a family-friendly game, Niantic's tolerance for profane behavior is much lower than for a game aimed at adults. Customization in risk assessments was critical.

Alice combined abuse-scenario mapping and red teaming to stress test features before launch. First, Alice mapped potential risks ahead of product development so the team could build protections into the product before development begins. Then Alice red teamed those abuse scenarios through various player personas, testing across multiple geo-locations and mimicking threat activities to identify potential harms. This uncovered risks associated with physical gameplay, in-app abuse, security and privacy concerns, and vulnerabilities in product or moderation capabilities. Scenarios were also tested from a non-threat actor perspective to identify gameplay issues and potential dangers caused by normal gameplay.

Alice provided investigations into threat actor TTPs, reports, and actionable insights on how players could exploit features to carry out abuse or increase risk to their own safety. Niantic's product and development teams reviewed these findings and made several changes to ensure family safety, including adding user controls to customize who can see a player's real-life locations and updating codes of conduct for specific games. Alice also enabled Niantic to understand the cross-platform threat actor ecosystem, helping identify and stop bad actors misusing the platform for fraudulent activity and cheating.

By leveraging Alice's cross-platform intelligence,  Niantic now considers safety by design principles for every new feature development. They partner with game teams early in the design and development process to ensure they are considering privacy and physical safety and are prepared for safer launches.

As Jen describes it, developers expect a certain level of Trust & Safety support, and Niantic has been able to provide guidance so developers don't have to do the mental heavy lifting when building features. They use these findings to design features, make changes, or be as prepared as possible. This process has led Niantic employees to think more about responsible development in general, getting in early to reduce risks and create safer player experiences.

As Jen puts it: if you let toxic or sketchy behavior flourish, it's hard to walk back from that, so getting people thinking about these things early leads to better long-term outcomes.