Source note: This essay was prompted by—and draws part of its conceptual framework on AI strategy, technical trade-offs, project delivery, and organizational adoption from—the Udemy course AI Leader: Generative AI & Agentic AI for Leaders & Founders, taught by Ed Donner and produced by Ligency. After completing the course, I combined those ideas with my own experience in AI projects and technical discussions to develop this independent synthesis and extension. It is not a lecture-by-lecture summary and does not represent the views of the instructors or Udemy. If you are responsible for AI strategy, architecture decisions, project governance, or organizational adoption, I recommend taking the original course for the complete framework and context.
I have seen plenty of AI projects begin with an impressive demo and an ambitious story, only to become strangely quiet after launch.
In the postmortem, the explanation usually sounds technical. The model was unreliable. The data was messy. Integration took too long. Those problems may all be real, but they often hide an earlier failure: the organization began building before it had made several consequential decisions.
I think AI transformation is best understood as a stress test of organizational judgment. Buying a tool is the easy part. The difficult work is turning a vague ambition to “do AI” into decisions that can be tested, owned, revised, and stopped.
Why are we doing this? Which part of the work should change? Who owns the consequences of a wrong answer? What evidence would justify further investment? Who will use the system repeatedly after launch?
To keep that work from collapsing into another model comparison, I now begin with five questions.
1. Are We Discussing the Same Thing?
Many AI meetings become misaligned in the first minute.
When executives say, “We need AI,” they may be responding to competitors, looking for a strategic signal, or imagining a step change in efficiency. Business teams may be thinking about repetitive work in support, sales, research, or operations. Technical teams may already be debating RAG, fine-tuning, agents, vector databases, and GPU costs.
The phrase is shared. The implied project is not.
That mismatch is expensive. An “AI customer service system” might mean a mature commercial product, a cloud model connected to a company knowledge base, or a proprietary model trained from scratch. All three can appear on the same strategy slide. They have radically different budgets, timelines, data requirements, and failure modes.
Alignment is therefore not a vocabulary lesson. It is the work of translating technical language into boundaries:
- Which tasks will the system handle, and which will remain explicitly out of scope?
- Can the relevant data leave the company boundary?
- Will the model provide information, or will it be allowed to take action?
- Who reviews its output, and who absorbs the failure when it is wrong?
- Are we testing a general productivity gain or building a capability that truly needs to be proprietary?
RAG, fine-tuning, and agents should be discussed inside those boundaries. RAG helps a model work with changing organizational knowledge. Fine-tuning changes model behavior for a particular task. An agent gives the model more control over deciding what to do next and which tools to use. Each step toward greater autonomy expands the evaluation surface, permission model, cost exposure, and number of possible failures.
Anthropic makes a useful distinction between workflows, whose paths are predefined in code, and agents, whose processes are dynamically directed by the model. Its practical recommendation is to begin with the simplest approach that meets the need. Leaders should understand that distinction because autonomy changes accountability, not because they need to choose an orchestration framework. See Anthropic, “Building Effective AI Agents.”
Technical language becomes an organizational decision only after it has been translated into data, permissions, actions, and responsibility.
2. Is This Worth Changing the Workflow For?
Technical teams naturally ask whether something can be built. The organization should first ask whether the current way of working is worth changing.
I look for value in three kinds of change:
- Removing friction: less waiting, searching, copying, formatting, or repeated verification.
- Improving an outcome: helping the same people detect problems earlier, miss less information, or make more consistent decisions.
- Making an uneconomic service viable: extending a scarce or expensive capability to people who previously could not receive it.
These may resemble a feature list, but they require the team to identify where the change will occur.
A customer-support assistant does not create value merely because it can generate an answer. Value becomes visible when it reduces search time, improves first-contact resolution, or routes a difficult case to the right person earlier. A research tool that can summarize a hundred-page report is still incomplete. The team must explain what analysts will stop doing, what they will inspect more carefully, and how a flawed summary will be detected.
I usually continue with five more concrete questions:
- Which existing cost or delay disappears?
- Which workflow does the system enter or change?
- Which observable result should improve?
- What new risk are we accepting in exchange?
- Will we still be able to tell whether it works three months later?
The second question matters most.
A demo can rely on a clean document and a carefully chosen prompt. A workflow must survive inconsistent data, complicated permissions, unusual inputs, users with different levels of expertise, and a chain of responsibility when something goes wrong.
A compelling demo is not product value. Value must become visible in work that happens repeatedly.
If a team can describe what the model can do but not which part of the work will change, it is still demonstrating capability rather than building a product.
3. Where Is the Risk Boundary?
Many project proposals quantify the benefit and reduce the risk section to a promise to “strengthen security and compliance.”
That is not enough.
Hallucination is only one source of risk. Data may cross a boundary it should not cross. Output may contain bias. The knowledge base may become stale. A provider may change the model. Usage may push cost far beyond the pilot estimate. The harder problem is often misplaced responsibility: the system recommends an action, an employee follows it, and only after a failure does the organization discover that nobody had defined who held the final decision.
Risk should be written as a design constraint before the architecture is chosen:
- Which data must never enter an external model?
- Which decisions must never be executed automatically?
- Which outputs require human confirmation?
- Which errors are tolerable, and which should stop the project?
- How will the workflow degrade when the model is unavailable, quality falls, or cost grows too quickly?
Those answers should determine the technical path.
For general, low-risk productivity, a mature commercial product may be the rational choice. A new feature that needs rapid validation may begin with an API. Private infrastructure or self-hosting may become worth the additional GPU, operations, upgrade, and hiring costs only when sensitive data or deep control makes them necessary.
The same principle applies to autonomy. If a deterministic workflow can solve the problem, there is no virtue in giving the model control over the next step merely to make the system feel more agentic. Autonomy adds latency, cost, testing combinations, and exception paths alongside capability.
Architecture is no longer an engineering preference at this point. It is a business trade-off among speed, control, cost, and accountability.
If an AI project cannot state its risk boundary, it is not ready for production.
4. How Will We Prove That It Works?
I have joined technical discussions that begin with GPT versus Claude, RAG versus fine-tuning, or LangGraph versus CrewAI.
The debate sounds advanced. Without real examples, a success standard, and a failure threshold, it is mostly an exchange of preferences.
AI projects should earn larger engineering investment with evidence.
A practical sequence looks like this:
- Select a small set of examples from real work rather than preparing only demonstration-friendly cases.
- Ask domain experts to define what counts as correct, usable, and unacceptable.
- Measure quality alongside cost, latency, stability, and the rate of human intervention.
- Set an evidence checkpoint at which the team will continue, change direction, or stop.
- Engineer and deploy only the approach that passes that checkpoint, then monitor it in production.
Different workflows need different evidence. Customer support may care about incorrect-answer rate, escalation, and first-contact resolution. Sales may care about follow-up speed, conversion, and CRM completeness. Software teams may care about delivery time, defect rate, and maintainability. Metrics do not replace judgment. They force the team to say what it is trying to improve.
Research uncertainty must also be managed honestly. On the first day of planning, nobody knows whether the model will reach the required quality, retrieval will remain stable, or the available data will be sufficient. Management does not need a confidently invented launch date. It needs an explicit evidence date: when will the team know whether this path deserves more investment?
Every project also needs an exit. If quality misses the target, can the workflow fall back to human review, partial automation, a rules-based system, or a mature commercial product? A system that can only succeed and cannot degrade safely is not a robust production design.
The more human an AI system appears, the more disciplined its evaluation must become. The more uncertain the project, the clearer its stopping conditions should be.
5. Who Will Use It in Real Work?
Launch is not adoption.
Sometimes a model is good enough, but a single unstable experience exhausts user trust. Sometimes the tool saves time in isolation but sits outside the actual workflow, forcing employees to open another page, copy the input, and paste the result back into the system of record. Sometimes leaders talk only about job replacement, and employees rationally treat low adoption as self-protection.
Morgan Stanley’s wealth-management assistant offers a useful contrast. It began by helping advisers retrieve internal knowledge and prepare for client conversations rather than making investment decisions for them. Domain experts participated in evaluation, quality controls were built into the release process, and the scope expanded with evidence. An OpenAI case study reports that more than 98 percent of adviser teams now use the tools. See the Morgan Stanley AI case study.
The important lesson is not the model. The organization connected three things: a clearly defined job, a visible quality standard, and a place in the existing workflow.
Adoption usually depends on more concrete questions:
- Does the tool remove a step, or add another system?
- Can users understand where an answer came from and when not to trust it?
- Does feedback from failure enter the next round of improvement?
- Do employees know which work will change and which judgments remain their responsibility?
- Are the new behaviors reflected in training, evaluation, and incentives?
Leaders also need to use these systems themselves. If executives ask the organization to “embrace AI” while changing nothing about how they gather information, prepare decisions, or collaborate, employees quickly recognize the initiative as a slogan.
Adoption is not a communications task at the end of the project. It begins when the use case is selected.
Return to the Five Questions
At the next AI project meeting, it may be more useful to put five questions on the whiteboard before debating models:
- Are we discussing the same thing?
- Is this worth changing the workflow for?
- Where is the risk boundary?
- How will we prove that it works?
- Who will use it in real work?
Together, they constrain the project’s language, value, accountability, evidence, and adoption. If any one of them remains vague, stronger technology may simply accelerate the wrong decision.
Models will improve. Tools will get cheaper. Integration will become easier.
Organizational judgment will not improve automatically.
That is where the serious investment belongs.