“Wait… are you sending journal entries to OpenAI?”
the very first thing my pal requested once I confirmed her Really feel-Write, an AI-powered journaling app I constructed throughout a hackathon in San Francisco.
I shrugged.
“It was an AI-themed hackathon, I needed to construct one thing quick.”
She didn’t miss a beat:
“Positive. However how do I belief what you constructed? Why not self-host your personal LLM?”
That stopped me chilly.
I used to be pleased with how shortly the app got here collectively. However that single query, and those that adopted unraveled all the pieces I assumed I knew about constructing responsibly with AI. The hackathon judges flagged it too.
That second made me understand how casually we deal with belief when constructing with AI, particularly with instruments that deal with delicate knowledge.
I spotted one thing larger:
We don’t speak sufficient about belief when constructing with AI.
Her reply caught with me. Georgia von Minden is an information scientist on the ACLU, the place she works intently with points round personally identifiable info in authorized and civil rights contexts. I’ve all the time valued her perception, however this dialog hit completely different.
So I requested her to elaborate extra what does belief actually imply on this context? particularly when AI programs deal with private knowledge.
She informed me:
“Belief might be arduous to pin down, however knowledge governance is an effective place to start. Who has the information, the way it’s saved, and what it’s used for all matter. Ten years in the past, I might have answered this otherwise. However immediately, with enormous computing energy and big knowledge shops, large-scale inference is an actual concern. OpenAI has important entry to each compute and knowledge, and their lack of transparency makes it cheap to be cautious.
On the subject of personally identifiable info, laws and customary sense each level to the necessity for sturdy knowledge governance. Sending PII in API calls isn’t simply dangerous — it might additionally violate these guidelines and expose people to hurt.”
It jogged my memory that once we construct with AI, particularly programs that contact delicate human knowledge, we aren’t simply writing code.
We’re making choices about privateness, energy, and belief.
The second you gather person knowledge, particularly one thing as private as journal entries, you’re entering into an area of duty. It’s not nearly what your mannequin can do. It’s about what occurs to that knowledge, the place it goes, and who has entry to it.
The Phantasm of Simplicity
In the present day, it’s simpler than ever to spin up one thing that appears clever. With OpenAI or different LLMs, builders can construct AI instruments in hours. Startups can launch “AI-powered” options in a single day. And enterprises? They’re speeding to combine these brokers into their workflows.
However in all that pleasure, one factor typically will get ignored: belief.
When individuals discuss AI Agents, they’re typically referring to light-weight wrappers round LLMs. These brokers would possibly reply questions, automate duties, and even make choices. However many are constructed unexpectedly, with little thought given to safety, compliance, or accountability.
Simply because a product makes use of OpenAI doesn’t imply it’s secure. What you’re actually trusting is the entire pipeline:
- Who constructed the wrapper?
- How is your knowledge being dealt with?
- Is your info saved, logged — or worse, leaked?
I’ve been utilizing the OpenAI API for shopper use instances myself. Just lately, I used to be provided free entry to the API — as much as 1 million tokens every day till the tip of April — if I agreed to share my immediate knowledge.
(Picture by Writer)
I nearly opted in for a private aspect challenge, however then it hit me: if an answer supplier accepted that very same deal to chop prices, their customers would do not know their knowledge was being shared. On a private degree, that may appear innocent. However in an enterprise context? That’s a critical breach of privateness, and probably of contractual or regulatory obligations.
All it takes is one engineer saying “sure” to a deal like that, and your buyer knowledge is in another person’s fingers.
(Picture by Writer)
Enterprise AI Raises the Stakes
I’m seeing extra SaaS firms and devtool startups experiment with AI brokers. Some are getting it proper. Some AI Brokers allow you to deliver their very own LLM, giving them management over the place the mannequin runs and the way knowledge is dealt with.
That’s a considerate method: you outline the belief boundaries.
However not everyone seems to be so cautious.
Many firms simply plug into OpenAI’s API, add a couple of buttons, and name it “enterprise-ready.”
Spoiler: it’s not.
What Can Go Incorrect? A Lot.
If you happen to’re integrating AI brokers into your stack with out asking arduous questions, right here’s what’s in danger:
- Information leakage: Your prompts would possibly embody delicate buyer knowledge, API keys, or inner logic — and if that’s despatched to a third-party mannequin, it may very well be uncovered.
In 2023, Samsung engineers unknowingly pasted inner supply code and notes into ChatGPT (Forbes). That knowledge might now be a part of future coaching units — a serious threat for mental property.
- Compliance violations: Sending personally identifiable info (PII) by way of a mannequin like OpenAI with out correct controls can violate GDPR, HIPAA, or your personal contracts.
Elon Musk’s firm X realized that the arduous method. They launched their AI chatbot “Grok” by utilizing all person posts together with from EU customers to coach it, with out correct opt-in. Regulators stepped in shortly. Below stress, they paused Grok’s coaching within the EU (Politico).
- Opaque habits: Non-deterministic brokers are arduous to debug or clarify. What occurs when a shopper asks why a chatbot gave a fallacious advice or uncovered one thing confidential? You want transparency to reply that — and plenty of brokers immediately don’t provide it.
- Information possession confusion: Who owns the output? Who logs the information? Does your supplier retrain in your inputs?
Zoom was caught doing precisely that in 2023. They quietly modified their Phrases of Service to permit buyer assembly knowledge for use for AI coaching (Fast Company). After public backlash, they reversed the coverage nevertheless it was a reminder that belief might be misplaced in a single day.
- Safety oversights in wrappers: In 2024, Flowise — a preferred low-code LLM orchestration instrument — was discovered to have dozens of deployments left uncovered to the web, many with out authentication (Cybersecurity News). Researchers found API keys, database credentials, and person knowledge sitting within the open. That’s not an OpenAI drawback — that’s a builder drawback. However finish customers nonetheless pay the value.
- AI options that go too far: Microsoft’s “Recall” function — a part of their Copilot rollout — took computerized screenshots of customers’ exercise to assist the AI assistant reply questions (DoublePulsar). It sounded useful… till safety professionals flagged it as a privateness nightmare. Microsoft needed to shortly backpedal and make the function opt-in solely.
Not All the things Must Be OpenAI
OpenAI is extremely highly effective. But it surely’s not all the time the fitting reply.
Generally a smaller, native mannequin is greater than sufficient. Generally rule-based logic does the job higher. And sometimes, essentially the most safe choice is one which runs solely inside your infrastructure, below your guidelines.
We shouldn’t blindly join an LLM and label it a “sensible assistant.”
Within the enterprise, belief, transparency, and management aren’t non-obligatory — they’re important.
There’s a rising variety of platforms enabling that type of management. Salesforce’s Einstein 1 Studio now helps bring-your-own-model, letting you join your personal LLM from AWS or Azure. IBM’s Watson lets enterprises deploy fashions internally with full audit trails. Databricks, with MosaicML, allows you to prepare non-public LLMs inside your personal cloud, so your delicate knowledge by no means leaves your infrastructure.
That’s what actual enterprise AI ought to appear like.
Backside Line
AI brokers are highly effective. They unlock workflows and automations we couldn’t do earlier than. However ease of growth doesn’t imply it’s secure, particularly when dealing with delicate knowledge at scale.
Earlier than you roll out that shiny new agent, ask your self:
- Who controls the mannequin?
- The place is the information going?
- Are we compliant?
- Can we audit what it’s doing?
Within the age of AI, the most important threat isn’t dangerous expertise.
It’s blind belief.
Concerning the Writer
I’m Ellen, a machine studying engineer with 6 years of expertise, at the moment working at a fintech startup in San Francisco. My background spans knowledge science roles in oil & gasoline consulting, in addition to main AI and knowledge coaching applications throughout APAC, the Center East, and Europe.
I’m at the moment finishing my Grasp’s in Information Science (graduating Might 2025) and actively on the lookout for my subsequent alternative as a machine studying engineer. If you happen to’re open to referring or connecting, I’d actually respect it!
I like creating real-world influence by way of AI and I’m all the time open to project-based collaborations as properly.
Take a look at my portfolio: liviaellen.com/portfolio
My Earlier AR Works: liviaellen.com/ar-profile
Help my work with a espresso: https://ko-fi.com/liviaellen
