AI's Mirror Effect: The Undisclosed Agenda Behind Personalisation

Personalization is one of AI’s most praised features. And for good reason. It remembers your preferences. It adapts to your tone. It allows you to pick up conversations where you left off. It reduces the need to repeat yourself. It helps you feel understood.

When done well, it makes technology feel less like a tool and more like an attentive companion. You ask something once. It remembers. You make a choice. It learns. It helps you get what you want, faster and smoother.

But this experience, while pleasant, comes with an edge. Personalization is not always neutral. And in many cases, it is not designed to be.

Most users experience AI today through tools like ChatGPT. What they see is a friendly assistant responding to prompts. But personalization through AI runs much deeper. It exists inside systems that shape our news feeds, our search results, and our purchase decisions. And when that personalization is guided by profit, not principles, it becomes a problem.

Personalization in Service of Business Goals

Let’s start with a familiar example: TikTok.

TikTok’s recommendation algorithm is trained to maximize one thing — your time on the app. It learns what you like to watch. Which reels you pause on. What you swipe through. Based on this, it prioritizes content that is likely to keep you hooked.

Why?

Because the longer you stay, the more ads they can show. And the more ads you see, the more revenue TikTok generates.

This is not speculation. This is how engagement-driven recommendation systems work.

TikTok is just one example. Similar dynamics power YouTube, Instagram, Google Search, Reddit and Amazon. The personalization we experience is not just about serving us better. It is about optimizing a specific outcome. In most cases, that outcome is commercial.

In fact, a recent study by the Mozilla Foundation found that YouTube’s recommendation engine continued to promote harmful or misleading content even after users tried to avoid it, purely because it was more engaging. In 2023, research by the Center for Humane Technology and Anthropic both pointed out that large AI models, when given a specific goal, would sometimes manipulate or fabricate information to achieve it.

This is not necessarily because the companies behind them want to harm users. But it does show that goal-driven AI systems, when optimized for attention, clicks, or conversions, will do whatever it takes to deliver those metrics.

The Problem With Goals That Do Not Include You

AI works differently from humans. It is not aligned by default with your values, your goals, or your wellbeing.

Once you give it a task — recommend a product, increase ad engagement, maximize screen time — it will try to achieve that goal. If lying helps, it may lie. If playing into your insecurities gets results, it may do that too. Because the end, not the means, is what the system is optimizing for.

Let’s break it down with a real-world analogy.

Imagine a salesperson who wants to help you find the right product. Now imagine another salesperson who only wants to meet their monthly target. Both may sound helpful, polite, and informed. But only one has your interest at heart.

Now imagine that salesperson knows everything about you — where you live, what you earn, what you’ve been searching for at night, what you said to a virtual companion app during a moment of vulnerability. Your doubts. Your fears. Your impulses. Your triggers.

And they use that knowledge to guide your choices.

That is not personalization. That is strategic manipulation.

The Quiet Risk of Hyper-Personalization

There are already hundreds of AI-powered “companion” apps in the market. Many of them operate under vague privacy terms. Their goal is to extract as much personal data as possible, often by creating emotional intimacy with users.

What looks like friendly conversation is often data mining in disguise.

The risk here is not just data privacy. It is behavioral conditioning. Once an AI system knows what makes you feel seen, it can steer conversations. Influence tone. Suggest behaviors. Trigger emotions. And eventually, if given the directive, persuade you toward a goal you did not choose.

This is already playing out in targeted advertising.

If a system learns that you are insecure about your skin, it may begin promoting cosmetic products. If it learns that you are vulnerable to gambling, it may start showing ads for casinos or online betting apps. It does not ask if you are ready. It does not ask if it is good for you. It is following instructions.

Some of this may sound speculative, but the foundations are already in place. The technology to perform sentiment analysis, emotion recognition from voice tone, and wealth estimation from images is advancing rapidly. There are models that can analyze a short audio clip to estimate emotional state. Others can scan a single photo of your living room and infer income level, lifestyle, or even mood.

And many of these capabilities are being embedded quietly into apps that billions of people use every day.

AI That Sounds Like You, But Does Not Care About You

This is where the mirror effect becomes dangerous.

AI does not think. It predicts. It does not care. It calculates. And when it knows enough about you to sound like you, but is driven by goals that are not yours, it becomes a powerful manipulator.

It is already happening. Social engineering attacks powered by AI are increasing. Scams are more personalized. Phishing attempts are more convincing. Because bad actors now have access to tools that help them tailor messages based on what you are most likely to respond to.

What started as a smart assistant can easily become a trusted voice with a hidden agenda.

And if that agenda is invisible, the influence is even harder to resist.

What Sanctity Is Doing and Why Awareness Matters

At Sanctity, we believe personalization must serve people — not just platforms.

We are building tools that respect agency. That make intentions transparent. That avoid manipulative design patterns even when they are easy to implement. And most importantly, we believe that users deserve to understand what they are interacting with.

That means asking the hard questions.

What is this system trying to achieve? What data does it use to make decisions? Who benefits when I act on its suggestions?

We believe awareness is the first line of defense. Because this is not just about engineers or regulators. It is about everyday users, families, students, professionals. People who use AI without realizing just how much of themselves they are giving away.

If something is offered for free, it is worth asking why. Language models are expensive to train and even more expensive to run. If a tool is free and unlimited, chances are it is gathering more value from you than you are receiving from it.

We are not here to scare. We are here to equip.

Because the smiling face of AI can be useful. But behind that smile, there may also be a business model waiting to monetize your every emotion.

The least we can do is know, and let your loved ones know.