the predatory side of gen AI apps

What happens when misogyny, incel language, and GPTs collide.

Abha’s note: This week’s issue is a guest post by Liz King. Liz is an American living in Europe since 2016. She writes about feminism, trauma, and healing, and hopes to publish a memoir.

---

I’d first heard of GenAI apps like ChatGPT and Gemini when they launched in 2022. They were still rudimentary at best then. I had a colleague who posted overly formal, clearly AI-generated text and I had tried it out myself but not in a serious way. We could generate goofy images where text was muddled and fingers on hands were not rendered correctly. Its potential had not been fully realized and I had not thought of how GenAI apps could be predatory.

For context, I work for a company that does tech conferences. One day, I was looking at some sessions for an upcoming conference and happened upon a speaker who was brand new to us. Something seemed off, right away. His headshot was giving uncanny valley vibes, definitely AI-generated in an unsettling way — eyes cast downward, too-perfect skin, eerily sharp jawlines and cheekbones. I thought to myself, there's no way this is a real photo.

I turned my attention to his bio. He mentioned a project in his bio that was immediately a red flag to me. Let's call it "Generic Dating App GPT". I googled the project, opened the project's source code and my jaw dropped. Now, I'd read the book, "Men Who Hate Women," by British journalist Laura Bates earlier in 2024. It illuminated the toxic 'Manosphere' and the language that incels use.

I immediately recognized the incel and PUA (pick-up artist) speak in this project. It promised dates with "girls" (aka adult women) without any effort at all. Messages were written in edgy language and this bot would arrange dates with said girls by negging them. There were even instructions in the source code to use the “pick up rules” and derogatory advice on how to “pick up” women.

My stomach dropped, “How had no one caught this?” This project was in the second line of the speaker’s bio. So it was something this guy was clearly proud of working on — and upon further research, I found — been promoting on many occasions.

In a podcast promoting the project, this speaker laments about spending time on dating apps for results that “leave something to be desired” when you can have "all the babes without wasting your time. Apparently, Tinder GPT was the solution to draining your time on dating apps.

In the most sinister way, the project says that besides negging, the users had to share more information to create “emotional intimacy.” At the same time, they would not act "like a simp” this would share enough information to make the women trust them enough to secure a date. This emotional bond is forged by sharing personal information and conversations around intimate topics like asking women about their feelings and telling stories (which are fabricated — great foundation for a relationship).

Why bother getting to know an actual human woman when you can get a neat AI-generated summary of her interests before meeting her in person despite never actually talking to her? This whole fiasco reminded me of this quote: “Girls are not machines that you put kindness coins into until sex falls out.”

This person’s solution to modern dating problems wasn’t to be authentic and actually talk to women, it was to dehumanize them. It’s like these men have just discovered that women are human beings with feelings, emotions, dreams and desires, but just don't really care about them. They pretend to listen to us and say the right things so they can get our phone number and eventually, get laid. It's like they think emotional vulnerability is this magical key to unlocking women and dates.

This is not the only project of this kind, of course, there are many many projects out there like this on GitHub and other places online that I would safely say are predatory. These kinds of third-party apps blatantly go against the terms and conditions of nearly all modern dating apps. So they'll always have a small disclaimer that the users should disclose they're using a bot to potential dates, but this rule is rarely enforced. You can try to report the creators of such forbidden third-party apps to said dating apps, to no avail, unfortunately.

After the initial shock, I assembled a group of everyone involved in the planning of the conference. It was mostly the women who agreed to meet about this. Our boss agreed that it had to be addressed, we had a meeting and voiced our concerns and were told then that the speaker would be disinvited from the event. I felt relieved it was addressed quickly and appropriately. End of story, all set!

However, things took a turn for the worse, when our boss and two female colleagues joined the speaker for a meeting to talk about it. Our boss thought it would be common courtesy since it was short notice and he thought the speaker deserved to know why. In the meeting, the speaker made it known that he did not see any problem whatsoever with his application. He and my boss took turns talking over my female colleagues and the meeting ended with my boss asking the speaker out for a beer after the conference.

When I heard how the meeting went down, I was furious, absolutely fuming, and sick to my stomach. Was I overreacting? Was it really not that big of a deal? I assembled an even bigger team of women and gathered more information. I wasn't going to back down without a fight and I didn't want any of my female colleagues to have to deal with this guy.

Every subsequent colleague I talked to was also outraged about this and reassured me that we were fighting for the right thing. We decided to have another meeting involving HR and our boss. I made an impassioned speech about how if we kept this speaker on board, we would be co-signing his project and his predatory values. I reminded them how that doesn't match the diversity and inclusion we aim for at our conferences.

What I left out was that I had been assaulted in college by someone I met on the app and it's what lit the fire beneath me to protect women and to stand up for what I believe in. Our boss (kind of) finally saw our side of the story and agreed that he should have talked with us before he made a final decision. In the end, the speaker was disinvited from the conference. It remains to be seen if our boss actually understood exactly why we had a problem with the speaker’s project.

By the time our boss saw our side of the story, a lot of damage was already done. One of my female colleagues said it was the last straw to her quitting and many of us were left with a bad taste in our mouths. We all had just got the confirmation that tech is a boys’ club and maybe always will be, especially in the AI world.

Maybe it shouldn’t be surprising that the world’s social media giant, Facebook started as a website to rate the attractiveness of women on Harvard’s campus. Or that IMDB was once a form for to users discuss how attractive (or not) actresses were. Now that AI is the next best thing, you can create deepfake porn videos. There are AI photo generators that can create bizarre kissing videos. The 100 most popular Generative AI apps include so-called ‘spicy chats’ and such photo and video editors. With a global rise in fascism and misogyny, these apps like this will only get more popular.

I am not writing this to scare anyone, but there are so many predatory projects out there, and we need to be aware of the not-so-positive sides of GenAI — the predatory and misogynist side. As girls online, we can only spread the word and protect each other, valuing genuine connections in places that try to create fake intimacy with GPTs. In the world of AI-generated everything, with every company having some AI feature and demanding more money for it, Meta promising to use fake accounts to drive up engagement, and other dystopian scenarios; genuine connections, putting your own words on a page and not using AI to generate them, are more important than ever.

At the end of the day, you could say many things about the speaker and creator of this app. Insecure, perhaps. It is really sad to me that he had to turn to AI to forge meaningful relationships with women. This is part of a larger movement of disgruntled young men, and while I don’t have the answers on how to de-radicalize them, I can only bring further awareness to what is out there.

To end on a brighter note, organizations are working to address the inherent biases of GenAI such as the Algorithmic Justice League founded by Dr. Joy Boulwamini. I highly recommend reading her wonderful book, “Unmasking AI.” You can also report creepy projects like this to the Algorithmic Justice League. What else can you do? Speak out when you see things like this in your life and workplace and encourage men to do the same. Continue to educate yourself on AI and inform other women, femmes, and any LGBTQ friends in your life about the horrors out there.