Koko and Stony Brook

Koko was a project initially launched for its potential in mental health support in 2022 to help teens who were suffering from mental health issues to feel less alone and have access to a 24/7 friend. 

 

The app, however, was accused of covertly subjecting vulnerable young adults to nonconsensual trials, which sparked widespread condemnation. The experiment targeted distressed individuals aged 18 to 25 on social media platforms, directing them to a chatbot intervention without informed consent. This ethical breach, coupled with the collection of sensitive data highlights Koko's disregard for ethical standards and user autonomy. Not to mention the experiment's classification as "nonhuman subjects research" further undermines participant safety and privacy. 

 

The idea of Koko exploiting these users underscores the goal of prioritizing technological advancement over ethical integrity in mental health interventions, emphasizing the need for ethical oversight and informed consent. Koko's misstep serves as a stark reminder of the potential consequences of neglecting ethical considerations in the pursuit of AI-driven solutions for mental health challenges, risking both credibility and the well-being of vulnerable individuals.

​

Sources:

 

National Academies of Sciences, Engineering, and Medicine. "The National Academies of Sciences, Engineering, and Medicine." National Academies. [https://www.nationalacademies.org/documents/embed/link/LF2255DA3DD1C41C0A42D3BEF0989ACAECE3053A6A9B/file/D22A23920299B125F1085AB4DAF3A807167481F7E0BF?noSaveAs=1]

​

Tsai, Alexander C., et al. "Pharmaceutical industry marketing of psychotropic medications and patient care." NCBI - PMC. [https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9361144/ ]

 

​Muncy, Joel. "Just awful experiment points suicidal teens at chatbot." Malwarebytes Labs Blog. [https://www.malwarebytes.com/blog/news/2023/03/just-awful-experiment-points-suicidal-teens-at-chatbot]

 

Mole, Beth. "Koko used GPT-3 AI without informed consent from users." NEPSY. [https://www.nepsy.com/articles/leading-stories/koko-used-gpt-3-ai-without-informed-consent-from-users/]

 

Cox, Joseph. "Horribly unethical startup experimented on suicidal teens on Facebook, Tumblr with chatbot." Vice. [https://www.vice.com/en/article/5d9m3a/horribly-unethical-startup-experimented-on-suicidal-teens-on-facebook-tumblr-with-chatbot]

​

Image: "Koko: The Innovative AI Platform That's Revolutionizing Mental Health Help", Good Good Good Co. [https://www.goodgoodgood.co/articles/koko-mental-health-help]