The news I read yesterday blew my mind! It made me question: why do we humans always find ways to misuse even the best technological tools? And in light of this… What are the limits of synthetic content marketing? Let me share the most relevant news of the week with you, and then let’s analyze it from a marketing and ethics perspective.
Every day, thanks to ChatGPT’s scheduled tasks, I receive a summary of artificial intelligence (AI) news and updates; mostly tech or marketing news, but yesterday they included a case that blew my mind.
Basically, it was the story of a recent international operation to catch a group of individuals (25 in total!) who were using AI tools to create and market synthetic (i.e., artificially generated) child pornography images.
The same technologies you and I use to generate AI-powered content —images, videos, audio, and text for marketing— are being exploited in illicit markets. This, while unfortunate, might not be surprising.
What’s surprising is the defense these criminals tried to make, arguing that the images they created didn’t depict real victims—that is, no one was harmed. They were the creative product of AI.
The reality is that while there is no tangible victim, the ethical, cultural, psychological, and social impact of these activities is enormous and undeniable. The creation and marketing of this content perpetuates the demand for child abuse and normalizes criminal behavior.
Let’s analyze this paradigm a little, from a marketing perspective, and of course, let’s add ethics.
What is synthetic content?
Synthetic content is any type of digital material—images, videos, audios or texts— generated in whole or in part by artificial intelligence.
Tools like ChatGPT, Midjourney, DALL·E, and ElevenLabs allow the creation of everything from articles to hyper-realistic faces without human intervention. While this technology has opened up a world of possibilities for digital marketing, entertainment, and education, it has also been exploited for darker purposes, such as disinformation, media manipulation, and the distribution of illegal content.
Creating these visuals is incredibly simple. This image, for example, is completely realistic, but it’s synthetic content, and you can see it was developed with Grok AI (bottom right corner).
1. A smart and scalable business model
From a business perspective, using AI to generate synthetic content is a highly efficient business model. It requires no photographers, actors, or physical sets, and can be produced in seconds at near-zero cost.
How does it work?
Image generation tools like Midjourney, DALL·E, and Stable Diffusion have proven that it’s possible to create high-quality visual content with just a few clicks. This has led to a boom in synthetic content marketing, where brands and creators use AI to produce advertising materials, social media posts, and optimized web content.
However, this same business model has been adopted in illicit markets, where the ease of production and global distribution has allowed for the proliferation of highly damaging content.
Many will remember the photo of Katy Perry at the Met Gala, which she hadn’t even attended; it wasn’t an illicit act, but the photo went viral, fooling thousands, including the media, who thought the artist was indeed there.
The keys to the illicit business model
- Low production costs: AI generates content without the need for physical resources.
- Massive scalability: thousands of images can be created in minutes.
- Minimalist operations: a single person with technical knowledge can manage production.
- Unrestricted global distribution: no storage costs or logistical barriers.
Legitimate companies like Synthesia, which creates videos with digital presenters, or ElevenLabs, which develops artificial voices, use these technologies to optimize synthetic content marketing in legal settings. In fact, both apps can clone real-life people, both in video and audio. However, this type of technology, or similar ones, in the wrong hands, is the same one that is fueling shadowy, hard-to-trace markets.
A few weeks ago, a viral video created using artificial intelligence sparked a heated debate in which several celebrities, including Scarlett Johansson, Jerry Seinfeld, Steven Spielberg, and Sacha Baron Cohen, allegedly criticized Kanye West for his anti-Semitic behavior and other controversial actions.
The video, although fake, has sparked intense debate about the use of AI to create misleading content that can confuse audiences and amplify negative narratives.
As we can see, from a purely marketing perspective, the creation and distribution of synthetic content is a multi-million dollar idea, but social impact and ethical issues are the key. Let’s talk about them.
2. Social and ethical impact: who is the victim?
One of the most common arguments used by those who defend the use of AI to generate synthetic pornography is that “there are no real victims. ”
But this statement ignores a larger problem: such content perpetuates the demand for child abuse and normalizes criminal behavior.
Why is this a problem?
- Reinforces the culture of abuse: Those who consume this content could be encouraged to seek out real people.
- It makes it difficult to prosecute crime: The thin line between synthetic and real content complicates the work of the authorities.
- Create self-sustaining markets: Ease of production eliminates barriers to entry and facilitates immediate access to content for very small amounts of money but at massive levels.
So even if there is no direct victim, the social damage is profound. Children in general become victims of a digital ecosystem that blurs the lines between reality and the artificial.
3. Synthetic content marketing
The success of any digital product lies in its ability to be found and distributed. From simple wallpapers to songs, movies, and even NFTs, each of them is capable of being created with AI. Of course, these products use digital marketing to market themselves: search engine ads, social media, mailings, affiliate marketing, guest blogging, display ads, etc.
In the case of AI-generated illicit content, this is clearly not possible, so marketing strategies are more sophisticated.
How is this content promoted?
- SEO in dark markets: Using encrypted keywords to avoid detection.
- Dark web and private forums: Distribution through anonymous networks such as TOR.
- Covert advertising on social media: Using hashtags and fake accounts to attract customers without alerting moderators.
- Cryptocurrency subscription models: Monetization through Bitcoin and other currencies to avoid tracking.
Digital marketing techniques that have been used in legal synthetic content marketing (including campaign optimization with generative AI) are being used in a rigged manner and with other tools in the illegal market, adapting tactics to evade regulations and exploit vulnerabilities.
The dilemma between innovation and regulation
X’s Grok has virtually no filters for its use, which is why it’s so risky. However, other platforms such as OpenAI, Google, and Meta, illustration and/or video apps, have implemented policies to prevent the use of their models in the creation of harmful content. However, open source allows anyone with technical knowledge to train models without restrictions.
This poses a huge challenge: AI has the potential to revolutionize digital marketing, but its misuse requires urgent regulation.
The ethical challenge of digital marketing in the age of AI
AI has proven to be a powerful tool for digital marketing, and especially content marketing. But its misuse can lead to markets that perpetuate abuse and exploitation.
OmniHUman-1, ByteDance’s new AI, can create videos of even singers from a single image and a motion signal, such as audio or video. This segment of Taylor Swift singing “Ao No Sora” is a testament to how brutally powerful and risky the technology is becoming.
Synthetic content marketing is a gold mine, but it’s urgent to draw the line as to how far we can go… because it’s currently very blurred.
The inconvenient truth is: AI doesn’t distinguish between a legal and an illegal product; it all depends on who’s using it and for what purpose.
For the marketing industry, this represents a critical ethical challenge.
As AI continues to evolve, digital marketing will need to find a balance between innovation, profitability, and social responsibility.
If we don’t act now, we risk the same algorithms that optimize our campaigns being used to normalize unprecedented crimes in the digital world.
A speaker, whose name I can’t recall, once expressed an idea that is completely true and applicable to this technology: we, as human beings, hold the future in our hands, and we can decide whether it’s a civilizational utopia like Star Trek or a dystopia like Terminator. How right he was!
