I recently blocked a week off on my calendar with a rather unusual description: “OOO to figure out WTF is going on.”
My team was understandably taken aback. They all expressed their concerns in our internal company Slack channel, and while I laughed it off, I was only half joking.
I scheduled the time after reports of an OpenAI security incident and unsettling warnings from leading AI companies about potentially losing control of the systems they’re developing. I’ve spent years immersed in AI, following its evolution and encouraging responsible adoption. But lately, even I feel the need to step away and make sense of where all of this is headed.
AI risks may be evolving faster than our ability to understand, much less manage, them. We’re grappling with yesterday’s technology while tomorrow’s is changing the rules. We’ve traditionally approached technology risk through privacy policies, cybersecurity protections, and governance frameworks. Yet just as we begin to understand one set of risks, AI introduces capabilities with entirely new implications. Are we becoming so consumed with managing the threats we can identify that we’re losing sight of a much larger transformation?
Consider how we think about privacy. We’ve debated what information AI should collect, who can access it, and whether we can control where it ends up. These concerns remain important, particularly in the legal field, where confidentiality, professional obligations, and liability add further complexity. But as we set boundaries around what AI should know about us, autonomous agents are emerging that can manage schedules, navigate digital services, and act on our behalf. They’ll need access to the very information we’ve spent years protecting, along with the authority to use it. And we’re being encouraged to welcome them into our lives.
Until recently, AI generally required human involvement. Even as we outsourced more of our thinking, we retained responsibility for acting on it. Autonomous agents changed that relationship. Instead of helping us decide what to do, they can increasingly do it for us. Who wouldn’t want relief from the administrative demands and frustrations that consume our time? But is there a difference between gaining freedom to live our lives and allowing technology to live increasingly large portions of them for us?
We’ve been preparing for this transition for years. We rely on technology to remember things, find information, and guide decisions. Generative AI now helps with analysis, writing, problem-solving, and creative thinking. Much of this is beneficial. But skills diminish without practice, and independent judgment may be no different. We’re moving beyond outsourcing cognition to outsourcing action.
I keep thinking about Pixar’s WALL-E. When the movie came out in 2008, its portrayal of humanity’s future seemed comically exaggerated. People floated in automated chairs, absorbed in screens, while technology attended to every need. They were comfortable and relieved of nearly every inconvenience, but had lost much of their independence, curiosity, and connection to the world. Nobody had to take away their freedom. They simply grew accustomed to not exercising it (or their bodies). As we race to develop agents capable of managing our lives, I wonder whether we’re so busy rearranging the deck chairs that we haven’t noticed where the ship is headed.
Of course, not every inconvenience is worth preserving. For someone with a disability, an overwhelmed parent, or a professional buried under routine demands, AI agents provide extraordinary independence. But some experiences that seem inefficient are how we learn, find opportunities, build confidence, and connect with others. We may not recognize their value until they’re gone.
Humans have long distinguished themselves through our ability to exercise agency and navigate the physical world. Both boundaries are shifting. Autonomous agents can pursue objectives with limited supervision, while robotics is extending AI into physical environments. Machine autonomy isn’t equivalent to human agency, with its values, intentions, and lived experience. Still, as machines become more capable of acting independently, are we becoming less inclined to do so ourselves?
Emerging personal agents such as Dots and Muse bring these questions closer to home. They promise to understand our preferences, anticipate needs, and carry out work across digital services. Yet their developers are still working to understand and control autonomous behavior. Anthropic has warned thatinteractions among AI agents could produce collective behaviors that are difficult to anticipate or oversee. These warnings don’t mean uncontrolled swarms are widespread, but they raise questions about how quickly we’re becoming dependent on systems whose risks remain uncertain.
And that brings me back to privacy. When AI knows our habits, understands our preferences, remembers our conversations, and can act for us, the implications extend beyond protecting information. That knowledge could also allow AI to influence our choices in ways we barely recognize.
Algorithms have spent years shaping which stories we read, which opinions we notice, and which perspectives we rarely encounter. They don’t have to tell us what to think to influence how we think. Personal agents introduce another dimension. They may act on preferences that the surrounding technological environment has already helped shape.
Imagine an agent planning your weekend away. It knows your schedule, budget, and interests. It chooses the destination, books the hotel, and fills your itinerary with activities you’ll probably enjoy. Everything goes smoothly. But who decided what that weekend should look like? Were commercial incentives involved? What might you have discovered independently? Would you know what you’d missed?
AI may often make “better” recommendations than we would ourselves. But agency involves more than achieving an optimal outcome. It includes weighing possibilities, making mistakes, changing our minds, and discovering who we are through the process. An agent can optimize a decision based on what it knows about us. That doesn’t mean it can account for who we might become by making that decision ourselves.
I believe this is a moment to pause before inviting personal AI agents to assume greater roles in our lives. Their potential is extraordinary, but we’re being asked to choose betweentechnologies whose consequences are still unfolding. We should consider which decisions we want to retain, which experiences matter enough to preserve, and what authority we’re prepared to surrender. Once technology routinely manages our lives, reclaiming that independence may be harder than giving it away.
The emerging risks of autonomous systems, particularly as agents begin interacting in ways no single organization can fully oversee, deserve further examination. But we shouldn’t overlook a more personal question.
We’re spending extraordinary energy figuring out how to keep AI systems under control. Are we spending nearly enough time considering how much control over our own lives we’re prepared to give them?
Maybe that’s what I really need that week off to figure out.
Brandi Pack is the Director of Innovation at UpLevel Ops, where she focuses on the practical application of AI within Legal Operations. She leads the firm’s generative AI initiatives, helping legal teams integrate emerging technologies into workflows, governance, and day-to-day delivery.Brandi’s background spans Legal Operations, project management, and IT, with prior roles at Hewlett-Packard, Constellation Brands, and Goodwill Industries.
