<?xml version="1.0" encoding="UTF-8"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Adam Shedivy</title><description>Blogs, essays, and other works by Adam Shedivy.</description><link>https://adamshedivy.com/</link><language>en-us</language><item><title>Building AI agents on IBM i, from SQL to MCP</title><link>https://adamshedivy.com/blog/building-ai-agents-on-ibm-i/</link><guid isPermaLink="true">https://adamshedivy.com/blog/building-ai-agents-on-ibm-i/</guid><description>The core patterns from my TechXchange 2026 sessions: why SQL works so well for data agents, and how to build one on IBM i a layer at a time, up to MCP.</description><pubDate>Fri, 06 Nov 2026 00:00:00 GMT</pubDate><content:encoded>&lt;blockquote&gt;
&lt;p&gt;This post was originally published on &lt;a href=&quot;https://www.linkedin.com/pulse/building-ai-agents-ibm-i-from-sql-mcp-adam-shedivy-xx9pc/&quot;&gt;LinkedIn&lt;/a&gt;.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Last week at IBM TechXchange 2026, I gave a number of sessions on AI and IBM i. This article walks through some of the core patterns from those sessions: why SQL is unreasonably effective for data agents, what makes IBM i unique for building them, and what it takes to build one from scratch, step by step. The code, slides and demos are all in one &lt;a href=&quot;https://github.com/ajshedivy/IBMTechXchange2026&quot;&gt;GitHub repo&lt;/a&gt; linked at the end, so you can run everything yourself.&lt;/p&gt;
&lt;h2&gt;SQL as an agent interface&lt;/h2&gt;
&lt;p&gt;An agent is a system that uses an AI model to autonomously reason, make decisions, and take actions (using tools) to achieve a certain goal. Tools are a major part of the agentic system. Think of AI tools the same way you think of tools a human uses to get work done. Just as we use computers, text editors, web browsers, and software applications to accomplish tasks, AI agents use tools to interact with the world around them. In the context of databases and enterprise systems (like IBM i and Db2 for i), what tools or interfaces can we use for our agents?&lt;/p&gt;
&lt;p&gt;The answer (which shouldn&apos;t surprise IBM i folks) is SQL. Db2 for i holds the core business data, so if we want an agent to help us gather insights, it needs to know how to query the system.&lt;/p&gt;
&lt;p&gt;IBM also provides SQL services that make the operating system accessible and manageable through SQL. This means an agent can inspect active jobs consuming resources, manage user profiles and authorities, work with PTFs, monitor performance, audit events, and interact with the file system, all through SQL.&lt;/p&gt;
&lt;p&gt;For example, answering a question like &quot;Which user profiles with *ALLOBJ authority haven&apos;t signed on in the last 90 days?&quot; takes a single SQL query against &lt;code&gt;QSYS2.USER_INFO&lt;/code&gt;, which holds useful information about user profiles:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;SELECT AUTHORIZATION_NAME, PREVIOUS_SIGNON
  FROM QSYS2.USER_INFO
 WHERE SPECIAL_AUTHORITIES LIKE &apos;%*ALLOBJ%&apos;
   AND PREVIOUS_SIGNON &amp;lt; CURRENT TIMESTAMP - 90 DAYS;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Two things make this a good fit for agents. Models already write SQL well and understand the syntax and complex rules of the SQL language. And every statement runs as an IBM i user profile, so object authority and auditing apply to the agent the same way they apply to a person. This means you don&apos;t have to build a separate permission layer for the agent.&lt;/p&gt;
&lt;h3&gt;Why IBM i suits agents&lt;/h3&gt;
&lt;p&gt;Running a query is the easy part. An agent that operates a production system also needs an identity, limits on what it can touch, a record of what it did, and points where a person has to approve the next step. On most platforms, those pieces have to be built around the agent. IBM i has had them for decades: work runs as a user profile, the operating system enforces object authority, and the audit journal records what happened.&lt;/p&gt;
&lt;figure&gt;
  &lt;img src=&quot;https://adamshedivy.com/blog/building-ai-agents-on-ibm-i/five-requirements.png&quot; width=&quot;1488&quot; height=&quot;837&quot; alt=&quot;Slide titled 5 Requirements of an Enterprise AI OS: 1. Governed data access, access business data without crossing system boundaries. 2. Object-level enforcement, enforce what an agent can do at the object level. 3. Intrinsic audit trails, every action traceable to a decision, identity and policy. 4. Hardware abstraction, absorb new hardware without rewriting core applications. 5. Bounded autonomy, agents operate within autonomy the system enforces. Footer: IBM i answers yes to all five, by design.&quot; /&gt;
  &lt;figcaption&gt;The five requirements of an &quot;Enterprise AI OS&quot;.&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;This is why I describe IBM i as the first &lt;a href=&quot;https://adamshedivy.com/blog/ibm-i-was-built-for-agents/&quot;&gt;&quot;agent-native&quot; operating system&lt;/a&gt;. Agents built on these architectural foundations can do real operational work.&lt;/p&gt;
&lt;h2&gt;Building an agent for IBM i&lt;/h2&gt;
&lt;h3&gt;The connection layer: Mapepire&lt;/h3&gt;
&lt;p&gt;SQL is an awesome tool, but the agent needs a way to connect to a system and run SQL. &lt;a href=&quot;https://mapepire-ibmi.github.io/&quot;&gt;Mapepire&lt;/a&gt; is a server job on IBM i that accepts encrypted connections on a single port, with client SDKs for Python, Node.js and Java.&lt;/p&gt;
&lt;p&gt;In Python, the &lt;code&gt;mapepire-python&lt;/code&gt; client gets you from a connection to query results in a few lines. &lt;code&gt;SQLJob&lt;/code&gt; opens the connection to the Mapepire server. On top of that sits a small helper, &lt;code&gt;run_sql&lt;/code&gt;, which binds any parameters, runs the statement and caps the number of rows that come back. The tools we build next all use this helper.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;from mapepire_python import DaemonServer
from mapepire_python.client.sql_job import SQLJob

server = DaemonServer.from_env()  # host, user, password
job = SQLJob()
job.connect(server)

def run_sql(sql, params=None, rows=50):
    opts = {&quot;parameters&quot;: params or []}
    with job.query(sql, opts=opts) as q:
        return q.run(rows_to_fetch=rows).data
&lt;/code&gt;&lt;/pre&gt;
&lt;h3&gt;The layers of an agent&lt;/h3&gt;
&lt;p&gt;The clearest way to see what each piece of an agent contributes is to build one a layer at a time. The example here is an IBM i agent in Python with &lt;a href=&quot;https://github.com/agno-agi/agno&quot;&gt;Agno&lt;/a&gt;, an open-source framework for building agents. Each layer adds a few lines to the one before it.&lt;/p&gt;
&lt;h3&gt;Layer 1: a model, instructions and one tool&lt;/h3&gt;
&lt;p&gt;A tool is just a function: a SQL statement with parameters, plus a docstring (or description) of what the tool does. The model uses that description to decide how to use the tool.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;@tool
def monthly_revenue(year: int) -&amp;gt; str:
    &quot;&quot;&quot;Revenue, profit and order count for each month of a year,
    from the TOYSTORE3 sales data.&quot;&quot;&quot;
    return run_sql(
        &quot;&quot;&quot;SELECT MONTH, TOTAL_REVENUE_W_TAX AS REVENUE,
                  TOTAL_PROFIT_W_TAX AS PROFIT, ORDER_COUNT
             FROM TOYSTORE3.MQT_MONTHLY_REVENUE_SUMMARY
            WHERE YEAR = ?
            ORDER BY MONTH&quot;&quot;&quot;,
        [year],
    )
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;An agent is that tool plus a model and instructions. Agno&apos;s &lt;code&gt;Agent&lt;/code&gt; class makes it easy to put them together:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;sales_agent = Agent(
    model=Claude(id=&quot;claude-sonnet-5-5&quot;),
    instructions=(
        &quot;You answer questions about a toy store&apos;s sales. &quot;
        &quot;Answer only from your tools, with the numbers.&quot;
    ),
    tools=[monthly_revenue],
)

sales_agent.print_response(
    &quot;What were the best and worst months of 1997, and by how much?&quot;
)
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;That&apos;s a working agent. To answer the question, the model calls &lt;code&gt;monthly_revenue(1997)&lt;/code&gt;, reads the twelve rows that come back, and compares them. The model only picks the year. The SQL is fixed, which makes this the safest kind of tool, and for many tasks it&apos;s all you need.&lt;/p&gt;
&lt;h3&gt;Layer 2: let the agent write its own SQL&lt;/h3&gt;
&lt;p&gt;Fixed tools only answer the questions you thought of in advance. For everything else, the agent needs to write its own SQL, and it needs guardrails while it does. The &lt;code&gt;IBMiSQLToolkit&lt;/code&gt; gives the agent a small set of tools that follow the order an experienced developer works in. It finds the right tables and data it has access to and describes their columns. Then it generates an SQL statement, checks the statement with &lt;code&gt;QSYS2.PARSE_STATEMENT&lt;/code&gt;, which rejects anything that isn&apos;t a read-only query, and runs it with a row cap. If the check fails, the error goes back to the model so it can fix the statement.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;toolkit = IBMiSQLToolkit(server)

agent = Agent(
    model=Claude(id=&quot;claude-sonnet-5-5&quot;),
    instructions=INSTRUCTIONS,
    tools=[toolkit],
)
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The &lt;code&gt;IBMiSQLToolkit&lt;/code&gt; is an example of progressive disclosure, a popular technique for database agents. All the code is in the repo linked at the end.&lt;/p&gt;
&lt;h3&gt;Layer 3: give it an administrator&apos;s know-how with skills&lt;/h3&gt;
&lt;p&gt;A model knows SQL, but it doesn&apos;t know how your best administrator runs a security review. Skills hold that knowledge. A skill is a folder with a &lt;code&gt;SKILL.md&lt;/code&gt; file and optional reference files. The agent sees only each skill&apos;s name and one-line description until a task matches. Then it loads the full instructions. Loading skills from a &lt;code&gt;skills&lt;/code&gt; folder takes one line:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;agent.skills = Skills(loaders=[LocalSkills(&quot;skills/&quot;)])
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The repo has two: &lt;code&gt;ibmi-performance&lt;/code&gt; and &lt;code&gt;ibmi-security&lt;/code&gt;. The security skill follows Scott Forstie&apos;s &quot;Inject more security into Db2 for i&quot;. It maps each risk, such as profiles anyone can impersonate, files any user can change, and weak passwords, to the SQL service that finds it, with tested queries in a reference file. The skill tells the agent to start from those tested queries instead of writing new ones.&lt;/p&gt;
&lt;h3&gt;Layer 4: identity, approval, audit and memory&lt;/h3&gt;
&lt;p&gt;The last layer is what you need before an agent changes anything: it should know who it is, ask before it acts, and leave a record.&lt;/p&gt;
&lt;p&gt;First, tell the agent who it is. &lt;code&gt;profile_context&lt;/code&gt; runs once at startup and reads the agent&apos;s own user profile from &lt;code&gt;QSYS2.USER_INFO&lt;/code&gt;: its name, special authorities, group profile and home directory. Those go into the model&apos;s context through &lt;code&gt;additional_context&lt;/code&gt;. When someone asks about &quot;my jobs&quot; or &quot;my files&quot;, the agent knows which profile that means, and it knows what that profile is allowed to do before it tries.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;agent = Agent(
    name=&quot;IBM i Agent&quot;,
    model=Claude(id=&quot;claude-sonnet-5-5&quot;),
    instructions=INSTRUCTIONS,
    # add profile information to context
    additional_context=profile_context(toolkit),
    tools=[toolkit],
    skills=Skills(loaders=[LocalSkills(&quot;skills/&quot;)]),
)
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Next, make certain changes wait for human approval. A tool marked &lt;code&gt;requires_confirmation&lt;/code&gt; pauses the run and shows the call to a person, who approves or rejects it. Here the tool runs a SQL statement that changes data or authority, and the person sees the exact statement the agent wrote before it runs:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;@tool(requires_confirmation=True)
def run_sql_change(statement: str) -&amp;gt; str:
    &quot;&quot;&quot;Run one SQL statement that changes data, such as UPDATE or REVOKE.&quot;&quot;&quot;
    ...

run = agent.run(
    &quot;Remove public update access from the TOYSTORE3 tables any user can change&quot;
)
for req in run.active_requirements:
    if req.needs_confirmation:
        print(req.tool_execution.tool_args)  # the REVOKE statement the model wrote
        req.confirm()                        # or req.reject()
run = agent.continue_run(run_id=run.run_id, requirements=run.requirements)
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The &lt;code&gt;for&lt;/code&gt; loop at the end handles the confirmations: the agent loop is interrupted, and the tool call waits until it is approved.&lt;/p&gt;
&lt;p&gt;A tool hook logs every call with its arguments, and a database keeps the conversation history. Put together, the whole agent is still one &lt;code&gt;Agent&lt;/code&gt; object:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;def audit_sql(function_name, func, args):
    log.info(&quot;tool=%s args=%s&quot;, function_name, args)
    return func(**args)

agent = Agent(
    name=&quot;IBM i Agent&quot;,
    model=Claude(id=&quot;claude-sonnet-5-5&quot;),
    instructions=INSTRUCTIONS,
    additional_context=profile_context(toolkit),
    tools=[toolkit],
    skills=Skills(loaders=[LocalSkills(&quot;skills/&quot;)]),
    # store agent session information in its own db
    db=SqliteDb(db_file=&quot;agent.db&quot;),
    add_history_to_context=True,
    # function that runs on each tool call
    tool_hooks=[audit_sql],
    tool_call_limit=20,
)
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The &lt;code&gt;audit_sql&lt;/code&gt; hook records every SQL statement the agent runs. On IBM i, a database monitor can also capture the SQL statements run by the agent&apos;s user profile, so you can audit activity at both the agent level and the IBM i level.&lt;/p&gt;
&lt;p&gt;The user profile stays the same through every layer, and its authority is the hard limit. If the model tries something the profile isn&apos;t allowed to do, the statement fails, even if it is approved.&lt;/p&gt;
&lt;h2&gt;Choosing a model, and running one on IBM i&lt;/h2&gt;
&lt;p&gt;The layers above don&apos;t depend on a particular model. For the most part, agents are &quot;model agnostic,&quot; meaning that the core agent can run with any LLM. This is useful because building an agent really comes down to figuring out what problems you want to solve, what data you can use, and how you want the agent to behave.&lt;/p&gt;
&lt;p&gt;This means you can build your agent and then test and evaluate it across models and deployment configurations without rebuilding it for each model. Choosing which model to use is still an important decision.&lt;/p&gt;
&lt;p&gt;There are two main camps: using a third-party provider (like Anthropic, OpenAI or OpenRouter), or hosting the model yourself.&lt;/p&gt;
&lt;p&gt;Each has pros and cons. Hosted models through an API are easy to set up, but some shops can&apos;t send data off the system. Self-hosted models require you to manage compute resources and the full model stack. Luckily, there is a vibrant local LLM developer community.&lt;/p&gt;
&lt;p&gt;You can run a model on IBM i itself with &lt;a href=&quot;https://github.com/ggml-org/llama.cpp&quot;&gt;llama.cpp&lt;/a&gt;, which is plain C/C++ and runs in PASE without a GPU. llama.cpp comes with &lt;code&gt;llama-server&lt;/code&gt;, which exposes a model through an OpenAI-compatible API, so you can call it from SQL with &lt;code&gt;QSYS2.HTTP_POST&lt;/code&gt;. The SQL is essentially a curl request:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;VALUES JSON_VALUE(
  QSYS2.HTTP_POST(
    DEMO.LLM_BASE_URL CONCAT &apos;/chat/completions&apos;,
    JSON_OBJECT(
      &apos;model&apos;    VALUE DEMO.LLM_MODEL,
      &apos;messages&apos; VALUE JSON_ARRAY(
        JSON_OBJECT(&apos;role&apos; VALUE &apos;user&apos;,
          &apos;content&apos; VALUE
            &apos;What does CPF1164 mean?&apos;) FORMAT JSON
      ) FORMAT JSON),
    JSON_OBJECT(&apos;headers&apos; VALUE JSON_OBJECT(
      &apos;Authorization&apos; VALUE &apos;Bearer &apos;
        CONCAT DEMO.LLM_API_KEY) FORMAT JSON)),
  &apos;$.choices[0].message.content&apos;);
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;And the model&apos;s answer comes back as the query result:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;CPF1164 is the completion message IBM i logs when a job ends.
It records when the job ended, how many CPU seconds it used, and an end code: 0
means it ended normally.
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This pattern is extremely useful. Again with the power of SQL, you can embed local LLM inference directly into SQL workloads.&lt;/p&gt;
&lt;h2&gt;Standardizing tools with MCP&lt;/h2&gt;
&lt;p&gt;The tools we built earlier live inside one Python agent, so only that agent can use them. When a second team builds a second agent, they rewrite the same tools. MCP standardizes this. It&apos;s an open protocol for exposing tools, so the tools live in one server that any client can connect to.&lt;/p&gt;
&lt;figure&gt;
  &lt;img src=&quot;https://adamshedivy.com/blog/building-ai-agents-on-ibm-i/one-server-every-client.png&quot; width=&quot;1488&quot; height=&quot;837&quot; alt=&quot;Diagram titled One server, every client: Bob, an internal assistant and a custom agent all connect over MCP to the IBM i MCP server, which connects through Mapepire to IBM i, where Db2 for i runs with your profiles and your authorities.&quot; /&gt;
  &lt;figcaption&gt;Build tools once: N agents times M tools becomes N plus M integrations.&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;The &lt;a href=&quot;https://github.com/IBM/ibmi-mcp-server&quot;&gt;IBM i MCP server&lt;/a&gt; defines each tool in YAML instead of Python. Here is the &lt;code&gt;monthly_revenue&lt;/code&gt; tool from the agent, written as an MCP tool:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;tools:
  monthly_revenue:
    source: ibmi-system
    description: &amp;gt;-
      Revenue, profit and order count for each month of a year,
      from the TOYSTORE3 sales data.
    statement: |
      SELECT MONTH, TOTAL_REVENUE_W_TAX AS REVENUE,
             TOTAL_PROFIT_W_TAX AS PROFIT, ORDER_COUNT
        FROM TOYSTORE3.MQT_MONTHLY_REVENUE_SUMMARY
       WHERE YEAR = :year
       ORDER BY MONTH
    parameters:
      - name: year
        type: integer
        description: The year to report on
        required: true
    security:
      readOnly: true
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The SQL is the same statement the Python tool ran, with a named &lt;code&gt;:year&lt;/code&gt; parameter. Everything around it tells the server how to use that statement: which system to connect to, how to describe the tool to the model, what parameters it takes, and that it only reads data. There&apos;s no server code to write, so building a tool comes down to writing the SQL. If you can write a query, you can build a tool.&lt;/p&gt;
&lt;p&gt;Bob, Claude, VS Code and custom agents all see the same tool, and every call runs as an IBM i user profile through Mapepire. You can serve a curated set of tools like this one, or enable the built-in text-to-SQL toolset, which uses progressive disclosure. It includes tools for exploring the database and running queries, with its own guardrails.&lt;/p&gt;
&lt;h2&gt;Try it yourself&lt;/h2&gt;
&lt;p&gt;Everything behind this article is in one repo: the slides from all six sessions, demo recordings, and five demos you can run against your own IBM i, each with setup steps.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://github.com/ajshedivy/IBMTechXchange2026&quot;&gt;github.com/ajshedivy/IBMTechXchange2026&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;Each demo lines up with a part of this article:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;mapepire-notebook:&lt;/strong&gt; connect to Db2 for i from Python and run queries through Mapepire.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;sql-services-agent:&lt;/strong&gt; run SQL services in a notebook, turn the same SQL into tools, and give them to agents.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;sql-agents:&lt;/strong&gt; the text-to-SQL agent built on &lt;code&gt;IBMiSQLToolkit&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;llama-cpp-on-ibm-i:&lt;/strong&gt; run a model on IBM i with llama.cpp and call it from SQL.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;mcp-tool-builder:&lt;/strong&gt; build YAML SQL tools with Bob and test them in &lt;a href=&quot;https://www.mcpjam.com/&quot;&gt;MCPJam Inspector&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;If you were at one of the sessions at TechXchange, thanks for coming and for the questions. If you build something with this, I&apos;d like to hear about it.&lt;/p&gt;
</content:encoded></item><item><title>The Enterprise AI Operating System: Why IBM i&apos;s Architecture Anticipated the Agentic Era</title><link>https://adamshedivy.com/blog/enterprise-ai-operating-system/</link><guid isPermaLink="true">https://adamshedivy.com/blog/enterprise-ai-operating-system/</guid><description>The governance that agent frameworks are now building from scratch is already part of IBM i&apos;s architecture, which makes it a natural foundation for enterprise AI agents.</description><pubDate>Wed, 25 Feb 2026 00:00:00 GMT</pubDate><content:encoded>&lt;h2&gt;1. The Shift&lt;/h2&gt;
&lt;p&gt;A few months ago I watched an agent answer a question about a failing batch job on an IBM i system. The question was vague, something like &quot;why did last night&apos;s order processing run slow?&quot; The agent queried more than one source. It checked job logs, pulled active job statistics, looked at disk utilization, cross-referenced recent PTF activity, and came back with a coherent explanation in about fifteen seconds. A senior admin could have done the same investigation. It would have taken them an hour, maybe two, and they would have needed to know exactly where to look.&lt;/p&gt;
&lt;p&gt;What stuck with me was less the AI itself than the system it was talking to, which has been running business-critical workloads since the late 1980s. The agent didn&apos;t care about that. It issued SQL, received structured data, reasoned over it, and acted. The age of the platform made no difference to it; the architecture did.&lt;/p&gt;
&lt;p&gt;I think we&apos;re at the start of a structural transition in enterprise computing, and I don&apos;t think most people have fully worked out what it means. For decades, enterprise systems operated under a basic assumption: humans drive and software executes. We design workflows, configure rules, review reports, and make decisions, and then we tell the system what to do. Software handles the transactions, and humans provide the judgment.&lt;/p&gt;
&lt;p&gt;That assumption is breaking now.&lt;/p&gt;
&lt;p&gt;AI models did get more capable, but the bigger change is that the &lt;em&gt;interface problem&lt;/em&gt; got solved. Model Context Protocol, structured tool use, and agent-to-agent communication give AI systems a new kind of connection to the software infrastructure that runs businesses. MCP alone has seen over 97 million monthly SDK downloads and more than 10,000 active servers since Anthropic open-sourced it in late 2024. Every major platform adopted it: OpenAI, Google, Microsoft, AWS, IBM, SAP, Salesforce. Block deployed MCP to 12,000 employees in under two months.&lt;/p&gt;
&lt;p&gt;So agents can now talk to enterprise systems. They can query databases, invoke APIs, read logs, and trigger workflows. The problem is that &lt;em&gt;talking to&lt;/em&gt; a system and &lt;em&gt;operating within&lt;/em&gt; a system are different things. An agent that can issue SQL queries is useful. An agent that can manage workloads on its own, enforce security policies, and make bounded decisions on a production system is something else. The distance between the two is an infrastructure gap more than a gap in model capability.&lt;/p&gt;
&lt;p&gt;If agents are going to operate enterprise systems, and not only assist the humans who operate them, then we need to ask a question that I think the industry has been avoiding: what does the operating system itself need to become?&lt;/p&gt;
&lt;p&gt;This post is my attempt to work through that question. I believe the answer points toward something I&apos;m calling the Enterprise AI Operating System: a persistent runtime layer that coordinates autonomous agents, enforces governance, and maintains the state and observability that enterprise operations demand. And I believe that IBM i, a platform most people associate with &quot;legacy&quot; computing, possesses architectural properties that align with this paradigm more naturally than anything else being built today.&lt;/p&gt;
&lt;p&gt;That&apos;s a bold claim, and I want to earn it rather than just assert it. I&apos;ll start with what the industry is converging on and why the current approaches have a structural problem. Then I&apos;ll define what an Enterprise AI Operating System actually requires. Then I&apos;ll make the case for IBM i, including the tradeoffs. And I&apos;ll close with what we&apos;re building, what we&apos;ve learned, and where I think this goes.&lt;/p&gt;
&lt;p&gt;Before getting into any of that, I want to give a sense of the scale. Gartner predicts that by the end of 2026, 40% of enterprise applications will embed task-specific AI agents, up from less than 5% in 2025. That is a fast shift. Yet Menlo Ventures found that only about 16% of current enterprise AI deployments qualify as true agents, meaning ones where the model plans, executes, observes feedback, and adapts. The rest are fixed-sequence workflows with an LLM call somewhere in the middle. The gap between where the market is heading and where the infrastructure actually is should concern anyone responsible for production systems.&lt;/p&gt;
&lt;p&gt;I think that gap is where the real work is. The work is less in making models smarter or building more tools, and more in building the runtime infrastructure that makes autonomous enterprise agents safe, observable, and governable. That&apos;s the problem this post is about.&lt;/p&gt;
&lt;h2&gt;2. Everyone Is Building an AgentOS&lt;/h2&gt;
&lt;p&gt;I&apos;ve been tracking how the industry is responding to this problem, and I keep seeing the same pattern. Independent groups, without coordinating, have arrived at the same metaphor: the AI operating system.&lt;/p&gt;
&lt;p&gt;At Rutgers, the AIOS project built what they call an &quot;LLM kernel.&quot; Agents make system calls into it, just like applications make syscalls into Linux. It has an agent scheduler, a context manager that can pause and resume agents, a storage manager, a tool manager, and an access manager. The companion paper was titled &quot;LLM as OS, Agents as Apps.&quot; They mapped the full analogy: the language model is the kernel, the context window is memory, external storage is the filesystem, and agents are the applications running on top.&lt;/p&gt;
&lt;p&gt;PwC shipped a commercial product called &quot;Agent OS&quot;, a cloud-agnostic orchestration layer they describe as the &quot;central nervous system and switchboard&quot; for enterprise AI. Agno AgentOS, formerly Phidata, offers an open-source three-layer stack with a production runtime, session isolation, and team execution modes. And at CES 2026, Siemens and NVIDIA announced what they explicitly called an &quot;Industrial AI Operating System,&quot; built on digital twins where agents continuously test and optimize factory operations.&lt;/p&gt;
&lt;p&gt;What I find more interesting than the shared metaphor is that they&apos;re all building the same components from scratch. Each of these frameworks ends up implementing some version of the same core capabilities: scheduling and runtime management for agents, identity and access control, state and memory persistence, observability and audit logging, and policy enforcement.&lt;/p&gt;
&lt;p&gt;I think the reason everyone is reinventing this is architectural. The platforms these frameworks run on (Linux, cloud VMs, Kubernetes) were designed for a different model of computing. They give you process isolation, but not governed object-level access. They give you filesystems, but not integrated data where the database and the operating system share the same address space. They give you containers, but not typed system resources where the platform itself knows what kind of object something is and what operations are allowed on it.&lt;/p&gt;
&lt;p&gt;So every AgentOS framework has to build its own governance layer on top of infrastructure that wasn&apos;t designed to govern autonomous actors. AIOS builds an &quot;Access Manager&quot; from scratch to enforce privacy between agents. PwC&apos;s Agent OS integrates across AWS, Azure, Google Cloud, Oracle, Salesforce, and SAP just to provide a unified view of enterprise data. Microsoft created Entra Agent ID and bolted Defender and Purview onto Copilot Studio to get identity, protection, and auditing. AWS added natural-language policy enforcement and episodic memory to Bedrock. Kyndryl launched &quot;Policy as Code&quot; to transform organizational rules into machine-readable agent constraints. Each of them is solving the same underlying problem: the platform doesn&apos;t do this natively, so the vendor has to build it.&lt;/p&gt;
&lt;p&gt;I don&apos;t want to overstate this. These are smart teams doing serious engineering, and the products they&apos;re shipping work. But when everybody independently arrives at the same set of missing capabilities, I read that as a signal about the foundation they&apos;re building on rather than about the builders.&lt;/p&gt;
&lt;p&gt;The security story makes this more urgent. MCP adoption exploded before authentication and registry mechanisms were enterprise-ready. Simon Willison identified the core issue early: MCP creates what he called &quot;toxic combinations of private data, untrusted instructions, and exfiltration vectors.&quot; The risk turned out to be real. CVE-2025-6514 compromised over 437,000 developer environments. Microsoft&apos;s own Copilot suffered a zero-click prompt injection, dubbed EchoLeak, that allowed data exfiltration through poisoned emails. In December 2025, OWASP published a Top 10 specifically for agentic applications, cataloging threats like Agent Goal Hijack, Tool Misuse, Identity and Privilege Abuse, Memory Poisoning, and Rogue Agents. The Cloud Security Alliance published an Agentic Trust Framework in February 2026 applying Zero Trust principles to agents. NIST launched an AI Agent Standards Initiative the same month. Forrester predicts a publicly disclosed agentic AI breach in 2026.&lt;/p&gt;
&lt;p&gt;I see these incidents as predictable consequences of running autonomous agents on platforms that weren&apos;t designed for them.&lt;/p&gt;
&lt;h2&gt;3. What Does an Enterprise AI Operating System Actually Require?&lt;/h2&gt;
&lt;p&gt;I think the requirements become clear once you stop thinking about features and start thinking about architectural properties. An Enterprise AI Operating System is a set of properties that the platform either has or doesn&apos;t, rather than a product, framework, orchestration layer, or set of APIs that you install on top of Linux or Kubernetes. If the platform doesn&apos;t have those properties natively, you end up spending years reconstructing them and fighting the abstraction at every step.&lt;/p&gt;
&lt;p&gt;I think this distinction gets lost in most of the AgentOS conversation. The frameworks I described in the previous section aren&apos;t wrong. They&apos;re compensating for what the underlying platform should have provided in the first place. Compensation works until you hit a security boundary that your governance layer can&apos;t enforce because it sits above the kernel, or an audit trail has a gap because the agent framework forgot to log a syscall, or a data integration fails silently because the information an agent needed lived in a different system behind a different authentication model.&lt;/p&gt;
&lt;p&gt;The better starting point is the platform level: what properties would a system need natively to serve as a real operating environment for autonomous enterprise agents?&lt;/p&gt;
&lt;p&gt;I&apos;ve written these as questions rather than a checklist, because each question exposes a gap that the industry is struggling with right now.&lt;/p&gt;
&lt;h3&gt;Can an agent access business data without crossing system boundaries?&lt;/h3&gt;
&lt;p&gt;This is the data integration problem, and it&apos;s more severe than most people acknowledge. MIT Sloan found that roughly 80% of the work in deploying AI agents goes to what they called &quot;unglamorous&quot; tasks: data engineering, stakeholder alignment, governance, workflow integration. Deloitte&apos;s research is starker: their findings suggest that agents make enterprise decisions on approximately 20% of the information they actually need. The other 80% (contracts, email threads, negotiated rates, historical patterns) sits in fragmented systems behind incompatible APIs. ServiceNow&apos;s CEO put it bluntly: &quot;The journey to agentic AI heaven goes through a data hell.&quot;&lt;/p&gt;
&lt;p&gt;This matters for the OS question because every system boundary an agent crosses adds latency, failure modes, and security surface area. If an agent needs to query a database, then call out to a separate document store, then hit an API to get configuration data, then authenticate against a different service to check permissions, that&apos;s four trust boundaries, and each one is a place where the call can fail or context can be lost or corrupted. And the agent has to hold all of that state in its context window while reasoning about a business problem.&lt;/p&gt;
&lt;p&gt;From an architectural standpoint, the ideal is an environment where the data and the execution live in the same space, and where the database is a component of the OS rather than a service the OS connects to. I realize this is a throwback to an older model of computing, and I think that older model is the right one for this problem.&lt;/p&gt;
&lt;h3&gt;Can the system enforce what an agent is allowed to do at the object level?&lt;/h3&gt;
&lt;p&gt;I mean the &lt;em&gt;object&lt;/em&gt; level, not the network level or the API gateway level. When an agent attempts to read a database table, modify a configuration, or invoke a system function, does the platform itself know whether that specific agent identity has authority over that specific resource?&lt;/p&gt;
&lt;p&gt;This is where the probabilistic nature of agents creates a new security challenge. Traditional software is deterministic. You can audit the code, trace every possible execution path, and verify that the program will never attempt an unauthorized operation. Agents don&apos;t work that way. Their behavior emerges from model weights, prompt context, and runtime observations. You can&apos;t pre-audit what an agent will decide to do. You can only constrain what the system will &lt;em&gt;allow&lt;/em&gt; it to do.&lt;/p&gt;
&lt;p&gt;The MiniScope framework from recent academic work provides the first rigorous definition of least privilege for tool-calling agents, drawing on mobile platform permission models. The AGENTSAFE framework operationalizes governance into design-time, runtime, and audit controls with dynamic authorization. Both are trying to solve the same problem: how do you enforce boundaries on actors whose behavior you can&apos;t fully predict?&lt;/p&gt;
&lt;p&gt;Most platforms today enforce boundaries at the API or service mesh level. An agent with access to an API has access to &lt;em&gt;everything that API exposes&lt;/em&gt;. That isn&apos;t least privilege; it relies on the agent behaving well.&lt;/p&gt;
&lt;p&gt;An Enterprise AI OS needs something closer to what traditional single-level storage systems provide, where every object in the system has an identity, a type, and an authority model. Beyond knowing that an agent is making a request, the platform knows what object the agent is requesting, what operations are valid on that object type, and whether this specific agent identity has been granted those specific operations. The enforcement happens below the agent, at the system level, where the agent can&apos;t circumvent it.&lt;/p&gt;
&lt;h3&gt;Is every agent action traceable to a decision, an identity, and a policy?&lt;/h3&gt;
&lt;p&gt;Auditability in enterprise systems is a regulatory requirement. The EU AI Act becomes fully applicable in August 2026 for high-risk systems, but analysis from The Future Society found that the Act&apos;s static pre-market conformity approach conflicts directly with agents&apos; dynamic runtime behavior. No agent-specific guidance is expected before 2027 at the earliest. In that regulatory gap, enterprises need platforms where audit trails are an intrinsic property of operations rather than a logging framework added after the fact.&lt;/p&gt;
&lt;p&gt;By &quot;intrinsic&quot; I mean this: if an agent makes a decision at 2 AM that affects a production workload, the system needs to have captured what the agent did, what policy authorized it, what data informed the decision, and what identity the agent operated under. It needs to capture all of it automatically, without relying on the agent framework to log itself, because the agent is the actor you&apos;re trying to audit. Letting the actor control its own audit trail is precisely the pattern that enterprise governance exists to prevent.&lt;/p&gt;
&lt;p&gt;The OS-level distinction matters here too. A logging framework that an agent writes to is different from a system journal that records every operation on every object, whoever or whatever initiated it. The first has gaps by design, because it depends on the agent&apos;s cooperation. The second has no gaps, because it operates below the layer where the agent makes decisions.&lt;/p&gt;
&lt;p&gt;The audit question also extends to something less discussed: reproducibility. When an agent makes a bad decision, can you reconstruct the full context that led to it? What was in the agent&apos;s context window? What tools did it call, in what order? What data came back? If the audit system captures only the final action and not the reasoning chain, you can&apos;t diagnose failures; you can only observe them after the damage is done.&lt;/p&gt;
&lt;h3&gt;Can new hardware capabilities be absorbed without rewriting agent code?&lt;/h3&gt;
&lt;p&gt;This might seem like a secondary concern, but I think it matters more than people realize. AI hardware is evolving fast: matrix math accelerators, dedicated inference cards, and new memory architectures, with each generation optimized for different workload profiles. If your agent code is compiled for specific hardware, every hardware generation requires migration work. In enterprise environments, that means testing, validation, change management, and risk.&lt;/p&gt;
&lt;p&gt;The alternative is a platform that abstracts hardware behind a stable execution interface. Agents written today then benefit automatically from hardware shipped five years from now. Neither the code nor the compilation target changes; the platform absorbs the new capability and exposes it through the same interfaces.&lt;/p&gt;
&lt;p&gt;For enterprises that plan in decades rather than quarters, this is a strategic question: does your investment in agent development compound over time, or does it depreciate with every hardware cycle?&lt;/p&gt;
&lt;p&gt;This property gets the least attention in the current AgentOS conversation, because everyone is focused on the immediate problems of governance, security, and orchestration. But hardware abstraction is a decision that looks insignificant today and becomes load-bearing ten years from now.&lt;/p&gt;
&lt;h3&gt;Can agents operate within bounded autonomy that the system enforces?&lt;/h3&gt;
&lt;p&gt;This is the governed autonomy question, and I think it&apos;s the most important one. It&apos;s also the one where the distinction between &quot;agent framework governance&quot; and &quot;platform governance&quot; is sharpest.&lt;/p&gt;
&lt;p&gt;The Cloud Security Alliance&apos;s Agentic Trust Framework proposes a maturity model using human role titles: Intern agents get read-only access. Junior and Senior agents operate with supervised autonomy. Principal agents get trusted autonomy with monitoring. That&apos;s a useful mental model, but it raises an immediate question: who enforces those boundaries?&lt;/p&gt;
&lt;p&gt;If the answer is &quot;the agent framework,&quot; then governance is only as reliable as the framework. A misconfigured agent, a prompt injection, or a tool that exposes more than intended can each bypass framework-level governance, because the framework and the agent operate at the same layer. The framework is asking the agent to govern itself, or at best, wrapping the agent in guardrails that run in the same execution context.&lt;/p&gt;
&lt;p&gt;If the answer is &quot;the operating system,&quot; then governance is an architectural guarantee. The agent cannot exceed its boundaries because the platform won&apos;t execute operations that violate the authority model. Whatever the agent decides to do, whatever prompt injection it receives, and whatever tool it tries to invoke, the operation doesn&apos;t happen if the system-level authority says no.&lt;/p&gt;
&lt;p&gt;An agent that &lt;em&gt;chooses&lt;/em&gt; to respect its boundaries is different from an agent that &lt;em&gt;cannot exceed&lt;/em&gt; them because the platform won&apos;t let it. I believe enterprise systems need the latter, and I believe that requirement, more than any other, determines what kind of platform can serve as a real Enterprise AI Operating System.&lt;/p&gt;
&lt;h3&gt;Properties, not products&lt;/h3&gt;
&lt;p&gt;All of this changes how I think we should evaluate platforms for the agentic era. The useful question is &quot;which platform possesses these properties natively?&quot; rather than &quot;which product has the best agent framework?&quot; The properties are integrated data that agents can access without crossing trust boundaries, object-level security enforced below the agent layer, intrinsic audit trails that don&apos;t depend on agent cooperation, hardware abstraction that protects long-term investment, and governed autonomy enforced at the system level.&lt;/p&gt;
&lt;p&gt;You can&apos;t add these as features to a platform that lacks them. They&apos;re architectural properties, and architecture is decided early. You can build an agent framework on top of any platform, but you can&apos;t easily retrofit the platform&apos;s relationship to data, security, and governance after the fact. Every AgentOS framework described in the previous section is trying to do that, and the complexity cost is real and grows over time.&lt;/p&gt;
&lt;p&gt;I realize that framing the requirements this way makes it obvious where I&apos;m heading, but I think the framing is honest. I didn&apos;t reverse-engineer these requirements from a specific platform&apos;s feature set. The industry is converging on them independently, through security failures, governance frameworks, and the practical experience of trying to deploy autonomous agents in production. The question is which platforms meet them natively and which require them to be reconstructed from parts.&lt;/p&gt;
&lt;h2&gt;4. The Accidental Alignment&lt;/h2&gt;
&lt;p&gt;I believe IBM i possesses architectural properties that the rest of the industry is trying to recreate. I mean this as a structural observation, and I want to walk through it carefully, because the claim sounds bigger than it is if you don&apos;t understand &lt;em&gt;why&lt;/em&gt; the alignment exists.&lt;/p&gt;
&lt;p&gt;It starts in 1978. The designers of System/38 made a series of engineering decisions that, at the time, were responses to specific problems in commercial computing. They weren&apos;t thinking about AI agents. They were thinking about how a system should manage shared resources safely when multiple users and programs need concurrent access to business-critical data. They chose single-level storage, where everything lives in one virtual address space. They chose object-based typing, where the system itself knows what kind of thing every resource is. They chose capability-based pointers validated in hardware, so programs cannot forge access to objects they haven&apos;t been authorized to touch. They integrated the database into the operating system rather than running it as a separate service. And they built a technology-independent machine interface, TIMI, that decouples programs from the hardware they run on.&lt;/p&gt;
&lt;p&gt;These were pragmatic decisions, and they turn out to be the decisions you&apos;d make if you were designing an operating system for autonomous agents. The problems have the same structure: multiple actors, shared state, the need for enforceable boundaries, and a strict requirement for auditability. The System/38 designers were solving for untrusted programs, and we&apos;re solving for unpredictable agents, but the architectural answer is the same.&lt;/p&gt;
&lt;p&gt;I don&apos;t think that&apos;s a coincidence. Governed concurrent access to shared enterprise resources is a specific engineering challenge, and the set of good solutions is smaller than people assume. The industry is rediscovering this now, which is why every AgentOS framework ends up building the same components. They&apos;re converging on answers that IBM i arrived at decades ago. IBM i didn&apos;t predict agentic computing; it solved the underlying problem correctly.&lt;/p&gt;
&lt;p&gt;Let me walk through the alignment concretely, requirement by requirement.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://adamshedivy.com/blog/enterprise-ai-operating-system/ibm-i-built-for-agents.png&quot; alt=&quot;Diagram titled &amp;quot;IBM i: Built for Agents&amp;quot;: IBM i architecture (integrated OS + Db2, SQL services for system management, role-based security) enables agent superpowers (one tool call touches data, files, and system; every system action is a SQL statement; agent inherits user authority), which map to MCP server tools such as run_sql, run_cl, get_job_log, and manage_ifs&quot; /&gt;&lt;/p&gt;
&lt;h3&gt;Governed access without bolted-on layers&lt;/h3&gt;
&lt;p&gt;Every object on IBM i has a type and an authority model enforced at the system level. Data areas, programs, files, queues, and user spaces are all typed objects with explicit authorities governing who can do what to it. Programs cannot masquerade as other object types. You can&apos;t trick the system into treating a data area as a program or a file as a command. The hardware validates capability-based pointers, which means even low-level memory manipulation can&apos;t forge unauthorized access.&lt;/p&gt;
&lt;p&gt;For agents, this means that an agent running under a user profile inherits exactly the authorities that profile has been granted. If the profile can read INVENTORY but not PAYROLL, the agent can read INVENTORY but not PAYROLL. The operating system enforces that constraint, not the agent framework, so the agent doesn&apos;t get a choice.&lt;/p&gt;
&lt;p&gt;This is what the Cloud Security Alliance&apos;s Agentic Trust Framework calls &quot;Zero Trust for agents&quot;: the idea that every agent action should be verified against explicit policies before execution. IBM i has been implementing this model since before &quot;Zero Trust&quot; existed as a term. The platform&apos;s five security levels provide graduated enforcement, from no security at level 10 to full C2-certified resource protection at level 50, and most production IBM i systems run at level 40 or above.&lt;/p&gt;
&lt;p&gt;Compare this to what other platforms are building. Microsoft created Entra Agent ID to give agents distinct identities within Azure Active Directory. That&apos;s necessary work, but it&apos;s an identity layer added to a platform that didn&apos;t originally distinguish between human and programmatic actors at the OS level. AWS added natural-language policy enforcement to Bedrock because the underlying infrastructure doesn&apos;t natively express fine-grained resource authorities in terms agents can inherit. These are good compensations, but they compensate for architectural properties the platform lacks.&lt;/p&gt;
&lt;p&gt;I want to be careful here. I&apos;m not saying IBM i&apos;s security model is superior to modern cloud security in every dimension. Cloud platforms have capabilities IBM i doesn&apos;t, such as federated identity across organizational boundaries, dynamic scaling of security policies, and integration with modern authentication protocols. But for the specific problem of &quot;an autonomous agent operating on shared enterprise resources needs enforceable, object-level boundaries,&quot; IBM i&apos;s model is architecturally native in a way that other platforms have to reconstruct.&lt;/p&gt;
&lt;h3&gt;Integrated data without crossing trust boundaries&lt;/h3&gt;
&lt;p&gt;Earlier I quoted ServiceNow&apos;s CEO saying that the journey to agentic AI heaven goes through data hell. On IBM i, there is much less of that data hell.&lt;/p&gt;
&lt;p&gt;With single-level storage, Db2 tables, IFS files, program objects, data areas, data queues, and spool files all exist in one virtual address space. The database is a component of the operating system, not a service the operating system connects to. When an agent issues a SQL query against IBM i, it isn&apos;t crossing a trust boundary to reach a database server. It&apos;s accessing objects that live in the same governed space as everything else on the system.&lt;/p&gt;
&lt;p&gt;In practice, think about the scenario I described at the beginning of this post. The agent investigating a slow batch job needed to check job logs, pull active job statistics, look at disk utilization, and cross-reference PTF activity. On most platforms, that&apos;s four different data sources, possibly with four different APIs and four different authentication contexts. On IBM i, it&apos;s SQL. IBM i SQL services expose the entire system through a single interface, including performance data, security configuration, developer tools, job management, system health, and PTF status. The agent uses one protocol, one authentication context, and one governed access model.&lt;/p&gt;
&lt;p&gt;This is why the MCP server architecture I&apos;ve been building works as cleanly as it does. An agent connected to an IBM i MCP server can issue SQL that reaches any of these system services. It doesn&apos;t need ten different tool integrations for ten different system functions. It needs SQL and the authority to run the relevant queries. The integration complexity that consumes 80% of agent deployment effort on other platforms, according to MIT Sloan, largely doesn&apos;t exist here because the integration happened at the platform level forty years ago.&lt;/p&gt;
&lt;p&gt;I don&apos;t want to pretend this is a complete solution to the enterprise data problem. Plenty of IBM i shops have data spread across multiple systems, multiple IBM i partitions, cloud services, and third-party applications, so the data integration challenge doesn&apos;t vanish. But the &lt;em&gt;on-platform&lt;/em&gt; data, which represents the core of what these systems do, is natively accessible in a way that other platforms can&apos;t match without significant middleware.&lt;/p&gt;
&lt;h3&gt;Audit trails as a system property&lt;/h3&gt;
&lt;p&gt;IBM i&apos;s security audit journal captures every security-relevant event at the OS level, including object creation, deletion, and modification, authority changes, program invocations, file access, and user profile operations. It isn&apos;t a logging framework that you configure and hope agents write to. It&apos;s part of how the system works: every operation on every object gets recorded, whoever or whatever initiated it.&lt;/p&gt;
&lt;p&gt;In the previous section I said that the actor should not control its own audit trail. On IBM i, it can&apos;t. The audit journal operates below the layer where agents, programs, or users make decisions. An agent can&apos;t selectively log its actions, omit an embarrassing operation, or &quot;forget&quot; to record something. The system captures everything because auditing is a property of the platform, not a behavior of the application.&lt;/p&gt;
&lt;p&gt;This matters a great deal for regulation. The EU AI Act&apos;s requirements for high-risk AI systems include maintaining logs of operations for traceability. The challenge, as The Future Society&apos;s analysis pointed out, is that the Act&apos;s static pre-market conformity approach doesn&apos;t map well to agents whose behavior emerges dynamically at runtime. You can&apos;t pre-audit what an agent will decide. What you &lt;em&gt;can&lt;/em&gt; do is guarantee that whatever it decides is recorded completely, automatically, and in a way the agent itself cannot tamper with. IBM i&apos;s journal-based auditing provides this guarantee.&lt;/p&gt;
&lt;p&gt;Audit completeness is not the same as audit usefulness, though. Capturing every operation is necessary but not sufficient. For real diagnostic value, you also need to reconstruct the reasoning chain: what was in the agent&apos;s context, what tools were called, what data came back, why the agent chose one action over another. That reasoning-level audit is an agent framework responsibility, not an OS responsibility. But the OS-level guarantee that the &lt;em&gt;effects&lt;/em&gt; of every decision are captured completely and immutably is the foundation everything else builds on, and IBM i provides it today.&lt;/p&gt;
&lt;h3&gt;Hardware abstraction through TIMI&lt;/h3&gt;
&lt;p&gt;This may be the most underappreciated property for agentic computing, and I think it will matter more over time in ways people aren&apos;t considering yet.&lt;/p&gt;
&lt;p&gt;Programs on IBM i don&apos;t compile to native machine code. They compile to an intermediate representation defined by the Technology Independent Machine Interface. When IBM ships new hardware (Power10, Power11 and whatever comes after), programs don&apos;t need to be recompiled. TIMI absorbs the new hardware capabilities and exposes them through the same interfaces.&lt;/p&gt;
&lt;p&gt;For agents, this matters because of where AI hardware is going. IBM&apos;s Power11 includes Matrix Math Accelerator cores designed for AI inference workloads. The Spyre Accelerator cards add dedicated AI processing. Future generations will bring capabilities we can&apos;t predict today. On platforms where agent code compiles to specific hardware targets, every hardware generation triggers a migration cycle: recompile, retest, revalidate, redeploy. In enterprise environments, that cycle is expensive and risky.&lt;/p&gt;
&lt;p&gt;On IBM i, that migration doesn&apos;t happen. An agent&apos;s underlying programs, written today, will automatically benefit from hardware capabilities that don&apos;t exist yet. Because of the TIMI layer, the platform can use Matrix Math Accelerators, Spyre cards, or future AI silicon without anyone touching the agent code. No other enterprise platform offers this kind of temporal decoupling between software investment and hardware evolution.&lt;/p&gt;
&lt;p&gt;I realize this sounds like a theoretical concern when the immediate pressure is getting agents working &lt;em&gt;at all&lt;/em&gt;. But I&apos;ve seen enough enterprise technology cycles to know that the decisions that feel secondary today become the ones you&apos;re grateful for, or cursing, a decade later. Enterprises plan in decades, and the value of hardware abstraction grows over that time.&lt;/p&gt;
&lt;h3&gt;SQL as the universal agentic interface&lt;/h3&gt;
&lt;p&gt;I touched on this in the data integration section, but it deserves its own treatment because I think SQL&apos;s role as the agent-to-system interface on IBM i is a real differentiator.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://adamshedivy.com/blog/enterprise-ai-operating-system/sql-universal-agentic-interface.png&quot; alt=&quot;Slide titled &amp;quot;IBM i: The first agentic-native operating system&amp;quot; listing three pillars (integrated OS and database, command-driven environment, security and role-based access) above a diagram of SQL as the universal agentic interface: a human talks to an LLM call, which runs SQL against Db2 for i on IBM i and gets feedback until it stops&quot; /&gt;&lt;/p&gt;
&lt;p&gt;IBM i SQL services expose the &lt;em&gt;operating system&lt;/em&gt;, not only database tables. Performance metrics, security configuration, system values, active jobs, message queues, PTF status, IFS contents, and authority structures are all accessible through standard SQL. IBM has been investing heavily in this design, and it makes a large difference for agents.&lt;/p&gt;
&lt;p&gt;Consider what this means for agent architecture. On most platforms, an agent that needs to manage system resources requires a collection of platform-specific tools: one for monitoring, one for security, one for job management, and one for configuration. Each tool has its own API, its own authentication model, its own error handling. The agent needs to know which tool to call for which task, and the MCP server needs to expose and manage all of them.&lt;/p&gt;
&lt;p&gt;On IBM i, the agent needs one tool: SQL. The MCP server I&apos;ve built translates natural language questions into SQL queries against IBM i SQL services. The agent gets one protocol and one interface, with full system access within the bounds of its authority. I see this simplicity as an architectural advantage. Fewer integration points mean fewer failure modes, fewer security surfaces, and fewer things that can go wrong at 2 AM when the agent is making autonomous decisions.&lt;/p&gt;
&lt;p&gt;This is also why the text-to-SQL agent pattern works so well on IBM i specifically. The agent layer converts natural language to SQL, the MCP server routes it to the IBM i system, the system enforces authorities and returns structured data, and the agent reasons over the results. Every step in that chain is governed and auditable. And because the interface between the agent and the system is SQL, which both humans and machines understand, the people who need to review what the agent did can read the audit trail.&lt;/p&gt;
&lt;h3&gt;The honest tradeoffs&lt;/h3&gt;
&lt;p&gt;I said I wanted to earn the claim rather than just assert it, and that means acknowledging where IBM i falls short.&lt;/p&gt;
&lt;p&gt;The ecosystem is small, and the developer community skews older. The talent pipeline is a real concern, because fewer people are entering the IBM i workforce than leaving it. The &quot;legacy&quot; label creates perception problems that go beyond branding and affect hiring, investment decisions, and strategic planning conversations. When you tell a CIO that the best platform for their AI agent strategy is a system that traces its lineage to 1978, you can feel the skepticism before you finish the sentence.&lt;/p&gt;
&lt;p&gt;Some of those concerns are legitimate. The tooling ecosystem is narrower than what you&apos;d find on Linux or in cloud-native environments. The learning curve for IBM i system administration is steep. Modern developer expectations around CI/CD, containerization, and infrastructure-as-code are only partially met.&lt;/p&gt;
&lt;p&gt;But I think the counterarguments are stronger than the objections, and they&apos;re getting stronger fast.&lt;/p&gt;
&lt;p&gt;PASE, the Portable Application Solutions Environment, runs standard AIX binaries on IBM i. Python and Node.js, the tools that AI development depends on, run there as supported runtimes. An agent developer doesn&apos;t need to learn RPG or CL to build agents that operate on IBM i. They need Python and SQL.&lt;/p&gt;
&lt;p&gt;MCP also makes IBM i much more accessible. Any MCP client, such as Claude Desktop, VS Code with an AI extension, or a custom orchestration platform, can connect to an IBM i MCP server and interact with the system without any IBM i expertise on the client side. The agent doesn&apos;t know or care that it&apos;s talking to an IBM i system. It sees tools, issues calls, and gets responses. The platform knowledge lives in the MCP server and the agent&apos;s instructions, not in the developer&apos;s head.&lt;/p&gt;
&lt;p&gt;IBM Bob, the AI-powered assistant for IBM i development, is showing 45% productivity gains for developers working with legacy codebases. That doesn&apos;t solve the talent pipeline problem entirely, but it changes the calculus. If AI can make one developer as productive as two on legacy code, the talent shortage becomes less existential.&lt;/p&gt;
&lt;p&gt;The hardware helps too. With the Matrix Math Accelerator cores in Power11, you can run AI inference locally, next to the data, without shipping anything to the cloud. The data stays where it is, the governance stays intact, and the latency drops. For enterprises with data sovereignty requirements or simply massive on-premises datasets, this is significant.&lt;/p&gt;
&lt;p&gt;None of this makes the tradeoffs disappear. The ecosystem needs to grow, the developer experience needs to modernize, and the perception gap needs sustained effort to close. But the core thesis, that IBM i possesses the properties an Enterprise AI Operating System requires, stands regardless of the ecosystem challenges. An ecosystem can grow, and architecture is much harder to change after the fact.&lt;/p&gt;
&lt;h2&gt;5. From Architecture to Runtime&lt;/h2&gt;
&lt;p&gt;Everything I&apos;ve described so far has been a structural argument: IBM i has these properties, and the industry needs them. But a thesis about architecture only matters if you can show it working. So I want to move from the theoretical case to what we&apos;ve actually built, what it has taught us, and where the practical edges of the idea are.&lt;/p&gt;
&lt;p&gt;The system is called iAssist. It&apos;s an agentic platform for IBM i that currently runs 28 agents organized into 5 coordinated teams, executing 3 multi-step workflows with shared learning across agents and flexible deployment options. The system is growing, so those numbers will be outdated by the time you read this, but the architecture behind them is what matters here.&lt;/p&gt;
&lt;h3&gt;What an agent actually looks like on IBM i&lt;/h3&gt;
&lt;p&gt;The AgentOS conversation tends to describe agents abstractly, with agent taxonomies, capability matrices, and autonomy levels. That&apos;s useful for framework design, but it doesn&apos;t tell you what it&apos;s like to build and operate agents on a real enterprise system. So here is the anatomy of an iAssist agent, from the perspective of someone who built them.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://adamshedivy.com/blog/enterprise-ai-operating-system/iassist-agentos-architecture.png&quot; alt=&quot;iAssist AgentOS for IBM i high-level architecture: UI clients (web UI, CLI/TUI, MCP clients, A2A) connect to an Agno AgentOS runtime with agents, teams, workflows, memory, knowledge, guardrails and observability, which calls the IBM i MCP server over MCP, which runs SQL against Db2 for i, QSYS2 SQL services and system APIs&quot; /&gt;&lt;/p&gt;
&lt;p&gt;An agent has four components: tools, instructions, memory, and skills.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Tools&lt;/strong&gt; are the agent&apos;s interface to the system. In our architecture, the primary tool interface is the IBM i MCP server, which exposes IBM i&apos;s capabilities through Model Context Protocol. Through MCP tool calls, an agent can issue SQL queries against Db2, interact with IFS files, call programs, read message queues, and check system values. The agent doesn&apos;t need to know RPG or CL or how IBM i system APIs work. It needs to know what question to ask and how to express it in SQL, and the MCP server handles the translation to IBM i&apos;s native interfaces.&lt;/p&gt;
&lt;p&gt;This is where the modular architecture becomes important. The IBM i MCP server provides raw tools, and raw tools alone aren&apos;t enough for production use. An agent with unrestricted access to SQL and no guardrails is dangerous for the same reason that giving a new hire direct database access on day one is dangerous. They might be smart and mean well, but they don&apos;t have the context to know what they shouldn&apos;t touch.&lt;/p&gt;
&lt;p&gt;So we build in layers. I can take an agent, say the &lt;code&gt;ibmi-text2sql&lt;/code&gt; agent, preconfigure it with instructions, safeguards, and tools via the IBM i MCP, and then expose that configured agent as its own MCP server. Any MCP client can connect to it: Claude Desktop, VS Code, a custom orchestration platform, even IBM Bob. The request flows from the client to the agent MCP, which routes to the IBM i MCP, and the response flows back up. The agent layer adds the structure, the guardrails and the specialized logic, and the IBM i MCP provides the raw system access. Each component stays separate, and they work together.&lt;/p&gt;
&lt;p&gt;You can plug the IBM i MCP server directly into a client. But then you get only the primitives, with no safeguards, specialized instructions, or domain-specific reasoning around them. It&apos;s the difference between giving someone a set of wrenches and giving them a mechanic who knows which wrench to use when.&lt;/p&gt;
&lt;p&gt;MCP was designed to enable this layered pattern (tools → agent → MCP endpoint → any client), where you build your agents and tools once and use them anywhere. On IBM i, the &quot;anywhere&quot; part matters because the IBM i expertise doesn&apos;t need to live in the client or the user&apos;s head. It lives in the agent layer.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Instructions&lt;/strong&gt; are the agent&apos;s domain knowledge expressed as behavioral guidance. What kind of questions can this agent answer? What SQL services should it prefer for different query types? What are the boundaries of its authority? What should it do when it encounters ambiguity? Instructions are structured prompts that shape how the agent uses its tools, not model weights or fine-tuning. Think of them as the runbook a senior admin would hand to a junior one: here&apos;s what you&apos;re responsible for, here&apos;s how to approach common scenarios, here&apos;s when to escalate.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Memory&lt;/strong&gt; is where things get interesting, and where we&apos;re still figuring out the right patterns. An agent that forgets everything between sessions is limited in ways that become obvious fast. The first time you ask an agent about a system issue and it has no context about the resolution you worked through with it yesterday, you feel the gap. We&apos;re building memory capabilities that let agents accumulate domain-specific knowledge across conversations, so an agent that helped diagnose a performance issue last week carries that context into this week&apos;s investigation. But how memory should work across sessions, teams of agents, and different user contexts is still an open question, and we haven&apos;t solved it.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Skills&lt;/strong&gt; are loadable capabilities that agents can acquire on demand based on the task at hand. Rather than building monolithic agents that try to do everything, we build focused skill modules that agents pull in when needed, such as a performance analysis skill, a security audit skill, or a PTF management skill. The agent assesses what the task requires and loads the relevant skills. This keeps individual agents focused while allowing the overall system to handle a broad range of scenarios.&lt;/p&gt;
&lt;p&gt;The packaging question is one I keep coming back to: what&apos;s the best way to consolidate tools, skills, custom knowledge, and instructions so we&apos;re not constantly rebuilding? The IBM i MCP server is only part of the equation. The full picture includes the agent&apos;s specialized intelligence, its accumulated knowledge, its skills library, and the guardrails that keep it operating safely. How you package all of that into something reusable and maintainable is, I think, one of the harder unsolved problems in the agentic space right now, on IBM i and everywhere else.&lt;/p&gt;
&lt;h3&gt;The thesis in practice&lt;/h3&gt;
&lt;p&gt;From what I&apos;ve built and observed, the architectural argument from Section 4 does hold up when agents are running on the system, with some nuances.&lt;/p&gt;
&lt;p&gt;The security model works as described. When an agent operates under a user profile, it inherits that profile&apos;s authorities. Early on, an agent was asked to pull data from a table that the user didn&apos;t have authority to. The query failed cleanly, with no data leak, no escalation, and no ambiguous failure mode. That wasn&apos;t because of clever guardrails in the agent framework. IBM i&apos;s object-level security did what it has always done. The agent never had a chance to access that data because the platform prevented it at a level below where the agent operates.&lt;/p&gt;
&lt;p&gt;That experience taught me something I now consider basic: the most reliable form of agent governance is governance the agent can&apos;t opt out of. You can build the smartest guardrails imaginable into your agent framework, but if the agent operates on a platform where it &lt;em&gt;could&lt;/em&gt; circumvent them given the right sequence of tool calls, you&apos;re trusting the agent to be good rather than ensuring it can&apos;t be bad. IBM i&apos;s architecture removes that trust requirement for the security layer.&lt;/p&gt;
&lt;p&gt;The data integration advantage is just as concrete. When I built the first version of the MCP server, I expected to need separate tool interfaces for different IBM i subsystems: one for database queries, one for system monitoring, one for job management, and one for security auditing. Instead, I found that SQL services cover almost everything, including performance metrics, active jobs, PTF status, authority structures, and IFS file operations. Having one interface to the whole system makes agent development much simpler. Fewer tools mean fewer failure modes, simpler testing, and a smaller surface area for things to go wrong.&lt;/p&gt;
&lt;p&gt;The audit story has also held up in practice. Every SQL query an agent issues through the MCP server passes through IBM i&apos;s normal security audit mechanisms. We don&apos;t maintain a separate agent audit log, because the system journal captures everything. When someone asks &quot;what did the agent do last night?&quot;, the answer is in the same place it would be if a human had done the same operations. The audit infrastructure that IBM i customers have used for decades works unchanged for agent operations, and we didn&apos;t have to build any of it.&lt;/p&gt;
&lt;h3&gt;What building this has taught me&lt;/h3&gt;
&lt;p&gt;I want to close this section with a few less structured observations from building an agentic platform on IBM i.&lt;/p&gt;
&lt;p&gt;First, the gap between &quot;agent that can talk to a system&quot; and &quot;agent that can operate within a system&quot; is much larger than it looks from the outside. Getting an agent to issue a SQL query is a weekend project. Getting an agent to reliably triage issues across a team of specialist agents, with appropriate escalation, memory of past interactions, and correct authority scoping, is months of work. MCP solved a real interface problem, but that was only the start.&lt;/p&gt;
&lt;p&gt;Second, the modular architecture matters more than I expected. Early on, I tried building comprehensive agents that handled everything. They were brittle, hard to maintain, and difficult to reason about. Moving to focused agents with loadable skills, organized into teams with clear responsibilities, made the system much more reliable. This mirrors how human teams work, which probably shouldn&apos;t be surprising, but it was a lesson I had to learn through building rather than theorizing.&lt;/p&gt;
&lt;p&gt;Third, IBM i&apos;s architectural properties reduce the engineering burden in practice. Every governance capability I didn&apos;t have to build in the agent framework is one that I can&apos;t break, introduce bugs into, or misconfigure. The security model, the audit trail, and the data integration all worked without extra effort from us. That freed up engineering time for the problems that need new solutions: agent reasoning, memory management, skill composition, and team coordination. The platform handles the infrastructure problems, so the agent framework can focus on the intelligence problems.&lt;/p&gt;
&lt;p&gt;Fourth, and this connects back to the thesis, I don&apos;t think you could build what we&apos;ve built on a platform without these properties in the same timeframe with the same team size. The compensating engineering that other AgentOS frameworks require (building governance layers, integrating data sources, constructing audit infrastructure) adds more than time. It adds ongoing maintenance, security surface area, and complexity that grows with every new agent you deploy. On IBM i, the platform absorbs that complexity, and the agents inherit governed behavior rather than implementing it.&lt;/p&gt;
&lt;p&gt;For me, that is the practical case for IBM i as an Enterprise AI Operating System: it is operationally simpler, because the hard problems were solved at the right layer forty years ago.&lt;/p&gt;
&lt;h2&gt;6. The Road Ahead&lt;/h2&gt;
&lt;p&gt;I want to end with specifics: where I think this goes, what I think comes next, and what I&apos;m still working through, including the open questions as well as the opportunity.&lt;/p&gt;
&lt;h3&gt;The MCP infrastructure play&lt;/h3&gt;
&lt;p&gt;IBM is targeting 500 MCP tools by the end of 2026. I think that&apos;s the right move, and the pace matters. MCP adoption has grown very fast, with 97+ million monthly SDK downloads and every major platform on board, and the organizations that establish rich MCP tool ecosystems earliest will have an advantage as agents become the default interface to enterprise systems.&lt;/p&gt;
&lt;p&gt;But tools alone aren&apos;t the full picture. What turns IBM i from &quot;a system agents can talk to&quot; into &quot;the system agents should run on&quot; is the layer between MCP and the platform: the governed runtime, or AgentOS. Five hundred tools exposed through MCP give agents access to IBM i. A governed agent runtime on IBM i gives agents a &lt;em&gt;home&lt;/em&gt; on IBM i, where they operate within the security model, accumulate knowledge, coordinate with other agents, and benefit from all the architectural properties I&apos;ve described in this post.&lt;/p&gt;
&lt;p&gt;I think the tool expansion and the runtime development need to happen in parallel. Tools without a runtime give you access but not governance, and a runtime without tools gives you architecture but not capability.&lt;/p&gt;
&lt;h3&gt;The governed autonomy window&lt;/h3&gt;
&lt;p&gt;Timing matters here. The industry is moving fast on agent deployment and slowly on agent governance. Gartner predicts that by 2028, 25% of enterprise breaches will be traced to AI agent misuse. Forrester expects a publicly disclosed agentic AI breach in 2026. OWASP published a Top 10 specifically for agentic applications. The Cloud Security Alliance built an entire Agentic Trust Framework. NIST launched an AI Agent Standards Initiative. The regulatory pressure from the EU AI Act is building, with full applicability for high-risk systems in August 2026 and no agent-specific guidance expected before 2027.&lt;/p&gt;
&lt;p&gt;Based on the security track record so far, I think the first major public agentic AI breach is a question of when rather than if. When it happens, the market conversation will move from &quot;how fast can we deploy agents&quot; to &quot;how safely can we deploy agents.&quot; That shift is IBM i&apos;s window.&lt;/p&gt;
&lt;p&gt;IBM i isn&apos;t immune to security problems; no system is. The window exists because IBM i&apos;s answer to the governed autonomy question is architecturally different from everyone else&apos;s. When a CISO asks &quot;how do I know this agent can&apos;t access data it shouldn&apos;t?&quot;, most platforms answer with layers of compensating controls, such as API gateways, policy engines, and monitoring frameworks, all sitting above the OS. IBM i answers at the hardware level, with capability-based pointers, object-level authorities, and a security model that has been C2-certified and running production workloads for decades.&lt;/p&gt;
&lt;p&gt;I expect companies to look for that kind of answer once the market gets nervous about agent security.&lt;/p&gt;
&lt;h3&gt;Knowledge and memory&lt;/h3&gt;
&lt;p&gt;MCP gave agents tools. The next step is giving them knowledge and memory, and I think this is where the platform can change the most for IBM i customers specifically.&lt;/p&gt;
&lt;p&gt;Knowledge through RAG (Retrieval-Augmented Generation) means agents that understand context as well as query databases. An agent investigating a performance issue doesn&apos;t start from scratch every time. It has access to a knowledge base of past incidents, resolution patterns, system-specific quirks, and tribal knowledge that currently lives in the heads of senior admins who are retiring faster than they&apos;re being replaced. The IBM i talent pipeline problem is real, and the community acknowledges it openly. RAG-powered agents don&apos;t replace that expertise, but they can preserve and distribute it in ways that weren&apos;t possible before.&lt;/p&gt;
&lt;p&gt;Memory means agents that learn. An agent that helped diagnose a subsystem bottleneck last month carries that context into this month&apos;s investigation. It remembers what worked and what didn&apos;t, and it builds a model of how &lt;em&gt;this specific system&lt;/em&gt; behaves under different conditions. Over time, the agent becomes more valuable as its understanding of the environment grows.&lt;/p&gt;
&lt;p&gt;I&apos;m excited about this direction, but it&apos;s early. The patterns for how agent memory should work across sessions, teams of agents, and different user authorization contexts aren&apos;t settled. How do you scope memory so that an agent remembers relevant context without leaking information across authorization boundaries? How do you version knowledge bases so that outdated information gets retired? How do you handle conflicting memories when two agents learned different things from similar situations? These are real problems without standard solutions yet.&lt;/p&gt;
&lt;p&gt;Still, IBM i&apos;s architecture makes several of these problems easier than they&apos;d be on other platforms. Memory scoping maps naturally to IBM i&apos;s authority model, because an agent&apos;s memory should be governed by the same authorities as its data access. Knowledge bases stored on IBM i inherit the platform&apos;s object-level security. Audit trails for memory operations flow through the same journal infrastructure as everything else. The governance questions around agent memory are hard on platforms where governance is added on top, and more tractable on a platform where it is built in.&lt;/p&gt;
&lt;h3&gt;What I&apos;m still figuring out&lt;/h3&gt;
&lt;p&gt;Some of the hardest questions are still open, and I think being clear about that is more useful than false confidence.&lt;/p&gt;
&lt;p&gt;The packaging problem is the one I think about most. How do you consolidate tools, skills, knowledge, instructions, and guardrails into composable, maintainable, reusable units? Right now, building a new agent means assembling these pieces from scratch or copying and modifying an existing agent&apos;s configuration. That works at the scale of 28 agents, but it won&apos;t work at the scale of 200. The IBM i MCP server covers only the tools layer. The full picture requires a packaging model for agent intelligence that I haven&apos;t seen anyone, on any platform, get right yet.&lt;/p&gt;
&lt;p&gt;The developer experience gap is real. IBM i&apos;s architectural advantages are invisible to a developer who&apos;s struggling with green-screen interfaces, unfamiliar toolchains, and documentation that assumes decades of platform knowledge. PASE and MCP lower the barrier significantly, because a developer can build IBM i agents with Python and SQL without learning RPG. But the barrier is lower, not gone. Making the development experience feel modern while preserving the architectural advantages is a design challenge that requires sustained investment.&lt;/p&gt;
&lt;p&gt;The organizational adoption question might be harder than the technical one. There&apos;s a real fear of AI in enterprise environments that comes partly from legitimate concerns and partly from misunderstanding. An architectural argument is necessary but not sufficient. People need to see agents operating safely on their own systems, handling real scenarios, and governed by policies they understand and control. The path from &quot;interesting idea&quot; to &quot;approved for production&quot; runs through trust, and demonstrations build that trust better than arguments do.&lt;/p&gt;
&lt;p&gt;And the competitive positioning needs to be clear-eyed. Microsoft&apos;s Agent 365, Salesforce Agentforce, Google&apos;s Agent Development Kit, and AWS Bedrock Agents are well-funded, well-marketed platforms with massive ecosystems. IBM i&apos;s AgentOS story isn&apos;t going to win on ecosystem breadth or developer mindshare. Its strength is architectural depth, specifically on the governed autonomy problem that all those platforms are trying to solve through compensating layers. The argument has to be precise: for high-stakes, governed enterprise operations where the cost of an agent acting outside its boundaries is severe, IBM i offers something architecturally different. It isn&apos;t better at everything, but it is different where it matters most.&lt;/p&gt;
&lt;h3&gt;The vision, stated plainly&lt;/h3&gt;
&lt;p&gt;I see IBM i as the trusted kernel of the agentic enterprise.&lt;/p&gt;
&lt;p&gt;It isn&apos;t a replacement for cloud-native frameworks, or a competitor to every AgentOS being built on Kubernetes, Azure, or AWS. Its role is more specific: it&apos;s the place where high-stakes, governed agent operations run. That means the platform where agents manage payroll, touch financial data, and operate production systems that can&apos;t go down and can&apos;t leak data. It&apos;s the only platform where governance is an intrinsic property of the operating system rather than an aftermarket addition.&lt;/p&gt;
&lt;p&gt;Forty years ago, the designers of System/38 made architectural decisions about how to govern shared resources safely. They couldn&apos;t have imagined autonomous AI agents. But the problem they solved, multiple actors operating concurrently on shared enterprise resources with enforceable boundaries and complete auditability, is the problem the agentic era needs solved.&lt;/p&gt;
&lt;p&gt;I think they got the architecture right, and now it&apos;s on us to build what that architecture makes possible.&lt;/p&gt;
&lt;h2&gt;References &amp;amp; Further Reading&lt;/h2&gt;
&lt;h3&gt;IBM i Agent Infrastructure&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;IBM i MCP Server. &lt;a href=&quot;https://github.com/IBM/ibmi-mcp-server&quot;&gt;github.com/IBM/ibmi-mcp-server&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;AgentOS for IBM i. &lt;a href=&quot;https://github.com/ajshedivy/ibmi-agentos&quot;&gt;github.com/ajshedivy/ibmi-agentos&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;Model Context Protocol (MCP)&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;Anthropic. &quot;Introducing the Model Context Protocol.&quot; November 2024. &lt;a href=&quot;https://www.anthropic.com/news/model-context-protocol&quot;&gt;anthropic.com/news/model-context-protocol&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Model Context Protocol Specification (2025-11-25). &lt;a href=&quot;https://modelcontextprotocol.io/specification/2025-11-25&quot;&gt;modelcontextprotocol.io/specification/2025-11-25&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Model Context Protocol GitHub Repository. &lt;a href=&quot;https://github.com/modelcontextprotocol/modelcontextprotocol&quot;&gt;github.com/modelcontextprotocol&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;Agent Operating System Research &amp;amp; Platforms&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;Mei, Kai et al. &quot;AIOS: LLM Agent Operating System.&quot; Rutgers University, 2024. &lt;a href=&quot;https://arxiv.org/abs/2403.16971&quot;&gt;arxiv.org/abs/2403.16971&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;AIOS GitHub Repository. &lt;a href=&quot;https://github.com/agiresearch/AIOS&quot;&gt;github.com/agiresearch/AIOS&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;PwC. &quot;Agent OS: AI Agent Operating System for Enterprises.&quot; &lt;a href=&quot;https://www.pwc.com/us/en/services/ai/agent-os.html&quot;&gt;pwc.com/us/en/services/ai/agent-os.html&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;PwC. &quot;PwC Launches AI Agent Operating System for Enterprises.&quot; &lt;a href=&quot;https://www.pwc.com/us/en/about-us/newsroom/press-releases/pwc-launches-ai-agent-operating-system-enterprises.html&quot;&gt;pwc.com/us/en/about-us/newsroom/press-releases/pwc-launches-ai-agent-operating-system-enterprises.html&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Siemens. &quot;Siemens Unveils Technologies to Accelerate Industrial AI Revolution at CES 2026.&quot; &lt;a href=&quot;https://press.siemens.com/global/en/pressrelease/siemens-unveils-technologies-accelerate-industrial-ai-revolution-ces-2026&quot;&gt;press.siemens.com&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;NVIDIA. &quot;Siemens and NVIDIA Expand Partnership for Industrial AI Operating System.&quot; &lt;a href=&quot;https://nvidianews.nvidia.com/news/siemens-and-nvidia-expand-partnership-industrial-ai-operating-system&quot;&gt;nvidianews.nvidia.com&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Agno (formerly Phidata). AgentOS Framework. &lt;a href=&quot;https://www.agno.com/&quot;&gt;agno.com&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Agno GitHub Repository. &lt;a href=&quot;https://github.com/agno-agi/agno&quot;&gt;github.com/agno-agi/agno&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;Industry Research &amp;amp; Market Data&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;Gartner. &quot;Gartner Predicts 40 Percent of Enterprise Apps Will Feature Task-Specific AI Agents by 2026.&quot; August 2025. &lt;a href=&quot;https://www.gartner.com/en/newsroom/press-releases/2025-08-26-gartner-predicts-40-percent-of-enterprise-apps-will-feature-task-specific-ai-agents-by-2026-up-from-less-than-5-percent-in-2025&quot;&gt;gartner.com/en/newsroom/press-releases/2025-08-26&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Gartner. &quot;Strategic Predictions for 2026.&quot; &lt;a href=&quot;https://www.gartner.com/en/articles/strategic-predictions-for-2026&quot;&gt;gartner.com/en/articles/strategic-predictions-for-2026&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Menlo Ventures. &quot;2025: The State of Generative AI in the Enterprise.&quot; December 2025. &lt;a href=&quot;https://menlovc.com/perspective/2025-the-state-of-generative-ai-in-the-enterprise/&quot;&gt;menlovc.com/perspective/2025-the-state-of-generative-ai-in-the-enterprise&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;Security Frameworks &amp;amp; Standards&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;OWASP. &quot;Top 10 for Agentic Applications (2026).&quot; December 2025. &lt;a href=&quot;https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/&quot;&gt;genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;OWASP GenAI Security Project. &lt;a href=&quot;https://genai.owasp.org/&quot;&gt;genai.owasp.org&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Cloud Security Alliance. &quot;The Agentic Trust Framework: Zero Trust Governance for AI Agents.&quot; February 2026. &lt;a href=&quot;https://cloudsecurityalliance.org/blog/2026/02/02/the-agentic-trust-framework-zero-trust-governance-for-ai-agents&quot;&gt;cloudsecurityalliance.org/blog/2026/02/02/the-agentic-trust-framework&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Cloud Security Alliance. &quot;Agentic AI Identity and Access Management: A New Approach.&quot; &lt;a href=&quot;https://cloudsecurityalliance.org/artifacts/agentic-ai-identity-and-access-management-a-new-approach&quot;&gt;cloudsecurityalliance.org/artifacts/agentic-ai-identity-and-access-management&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Willison, Simon. &quot;Anyone Using Agentic AI Needs to Understand Toxic Flows.&quot; Referenced in Dark Reading. &lt;a href=&quot;https://www.darkreading.com/cyber-risk/anyone-using-agentic-ai-needs-understand-toxic-flows&quot;&gt;darkreading.com/cyber-risk/anyone-using-agentic-ai-needs-understand-toxic-flows&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;Additional Reading&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;Wikipedia. &quot;Model Context Protocol.&quot; &lt;a href=&quot;https://en.wikipedia.org/wiki/Model_Context_Protocol&quot;&gt;en.wikipedia.org/wiki/Model_Context_Protocol&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;PwC. &quot;Expanding Agent OS to Oracle Cloud.&quot; &lt;a href=&quot;https://www.pwc.com/us/en/technology/alliances/library/expanding-agent-os-to-oracle-cloud.html&quot;&gt;pwc.com/us/en/technology/alliances/library/expanding-agent-os-to-oracle-cloud.html&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;PwC. &quot;Agent OS on AWS Bedrock.&quot; &lt;a href=&quot;https://www.pwc.com/us/en/technology/alliances/library/agent-os-aws-bedrock.html&quot;&gt;pwc.com/us/en/technology/alliances/library/agent-os-aws-bedrock.html&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Rutgers Tech Finder. &quot;Large Language Model Agent Operating Systems.&quot; &lt;a href=&quot;https://techfinder.rutgers.edu/tech/Large_Language_Model_Agent_Operating_Systems&quot;&gt;techfinder.rutgers.edu/tech/Large_Language_Model_Agent_Operating_Systems&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
</content:encoded></item><item><title>IBM i was built for agents</title><link>https://adamshedivy.com/blog/ibm-i-was-built-for-agents/</link><guid isPermaLink="true">https://adamshedivy.com/blog/ibm-i-was-built-for-agents/</guid><description>The properties every agent platform is now building from scratch were designed into IBM i in 1978.</description><pubDate>Wed, 25 Feb 2026 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;A few months ago I watched an agent investigate a slow batch job on an IBM i system. The question it got was vague: &quot;why did last night&apos;s order processing run slow?&quot; In about fifteen seconds it checked the job logs, pulled active job statistics, looked at disk utilization, cross-referenced recent PTF activity, and came back with a coherent explanation. A senior admin could have done the same investigation, but it would have taken them an hour or two, and they would have needed to know exactly where to look.&lt;/p&gt;
&lt;p&gt;What stuck with me was the system on the other end. Its lineage goes back to the late 1970s, and none of that mattered to the agent. It issued SQL, got structured data back, and reasoned over it.&lt;/p&gt;
&lt;p&gt;This post makes the case that IBM i&apos;s architecture is a good fit for running AI agents in production, and that much of what the industry is building for agents already exists in it. It&apos;s a shorter version of a longer post, and the &lt;a href=&quot;https://adamshedivy.com/blog/enterprise-ai-operating-system/&quot;&gt;full version&lt;/a&gt; has the references and more detailed examples.&lt;/p&gt;
&lt;h2&gt;Every agent platform is building the same layer&lt;/h2&gt;
&lt;p&gt;Over the last year, groups working independently of each other have reached for the same metaphor: the AI operating system. Rutgers built an LLM kernel (AIOS). PwC shipped a commercial Agent OS. Siemens and NVIDIA announced an Industrial AI Operating System at CES 2026.&lt;/p&gt;
&lt;p&gt;What I find more interesting is what&apos;s inside them. Each one builds the same components from scratch:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Scheduling and runtime management&lt;/strong&gt;, to decide what runs when.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Identity and access control&lt;/strong&gt;, to decide what an agent may touch.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;State and memory&lt;/strong&gt;, so work survives between steps.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Audit logging&lt;/strong&gt;, so every action can be traced afterward.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Policy enforcement&lt;/strong&gt;, so limits hold even when the model doesn&apos;t follow instructions.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;They build these because the platforms underneath don&apos;t provide them in a form that fits agents. Linux gives you process isolation, but not governed access to individual objects. A cloud VM gives you a filesystem, but the database is a separate system with its own permissions. Kubernetes gives you containers, but the platform doesn&apos;t know what kind of object a file or table is or which operations are allowed on it. So each framework builds its own governance layer on top. Because the gap is in the platform, every team ends up closing it again on its own.&lt;/p&gt;
&lt;h2&gt;Five questions an agent platform has to answer&lt;/h2&gt;
&lt;p&gt;I find it more useful to think of an enterprise agent platform as a set of architectural properties than as a product you install. In the full post I frame them as five questions:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Can an agent reach business data without crossing system boundaries?&lt;/li&gt;
&lt;li&gt;Can the system enforce what an agent may do at the level of individual objects?&lt;/li&gt;
&lt;li&gt;Can every agent action be traced to a decision, an identity, and a policy?&lt;/li&gt;
&lt;li&gt;Can the platform take on new hardware without agent code being rewritten?&lt;/li&gt;
&lt;li&gt;Can agents run with bounded autonomy that the system enforces, rather than the agent itself?&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;A platform either has these properties built in, or it has to approximate them in an application layer, and they&apos;re hard to add after the fact.&lt;/p&gt;
&lt;h2&gt;IBM i answers them in its architecture&lt;/h2&gt;
&lt;figure&gt;
  &lt;img src=&quot;https://adamshedivy.com/blog/ibm-i-was-built-for-agents/architecture.png&quot; width=&quot;1600&quot; height=&quot;900&quot; alt=&quot;Diagram in three layers. IBM i architecture (integrated OS and Db2, SQL services for system management, role-based security) enables agent capabilities (one tool call touches data, files and the system; every system action is a SQL statement; the agent inherits user authority), which the MCP server exposes as tools such as run_sql, run_cl, get_job_log and manage_ifs.&quot; /&gt;
  &lt;figcaption&gt;Fig. 1 — How IBM i&apos;s architecture maps to what an agent can do, and to the MCP server tools that expose it.&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;IBM i descends from System/38, announced in 1978. Its designers weren&apos;t thinking about AI. They were working out how a system could let many users and programs share business-critical data safely, and they made four decisions that are still part of the platform today:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Single-level storage.&lt;/strong&gt; Memory and disk form one address space, so programs work with objects rather than managing files and buffers.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Typed objects.&lt;/strong&gt; Everything on the system is an object with a type, and the type determines which operations are allowed on it.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;An integrated database.&lt;/strong&gt; Db2 for i is part of the operating system, not a separate server.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;TIMI&lt;/strong&gt;, the Technology Independent Machine Interface. Programs compile to an interface above the hardware, which is how IBM i has moved across processor generations without recompiling applications.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Those choices were about governing shared resources, which is the same problem agents bring back: several actors, shared state, boundaries that have to hold, and a requirement to audit everything. IBM i didn&apos;t anticipate agents, but because it solved that problem, the solution carries over. In practice, three consequences matter most.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Authority comes from the user profile.&lt;/strong&gt; An agent connected under an IBM i user profile has exactly the authorities that profile has been granted. The operating system enforces this, not the agent framework, so the agent can&apos;t go beyond those authorities even if a prompt tells it to. Those authorities are also data the agent can inspect. This tool from the &lt;a href=&quot;https://github.com/IBM/ibmi-mcp-server&quot;&gt;IBM i MCP server&lt;/a&gt; checks whether any user on the system can run the command that changes the system library list:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;SELECT object_authority
  FROM qsys2.object_privileges
 WHERE system_object_schema = &apos;QSYS&apos;
   AND system_object_name = &apos;CHGSYSLIBL&apos;
   AND object_type = &apos;*CMD&apos;
   AND authorization_name = &apos;*PUBLIC&apos;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;strong&gt;SQL services expose the whole system through one interface.&lt;/strong&gt; Performance data, security configuration, job management, and PTF status can all be queried with SQL, over one connection and one authentication context. That&apos;s what the agent in the opening example relied on. Here is the MCP server&apos;s tool for finding the top CPU consumers, defined in YAML so the agent picks a question rather than writing SQL:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;active_job_info:
  source: ibmi-system
  description: &quot;Find the top 10 consumers of CPU in the QUSRWRK and QSYSWRK subsystems&quot;
  parameters:
    - name: limit
      type: integer
      default: 10
  statement: |
    SELECT CPU_TIME, A.* FROM
    TABLE(QSYS2.ACTIVE_JOB_INFO(SUBSYSTEM_LIST_FILTER =&amp;gt; &apos;QUSRWRK,QSYSWRK&apos;)) A
    ORDER BY CPU_TIME DESC
    FETCH FIRST :limit ROWS ONLY
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;On other platforms, much of the work of deploying an agent goes into connecting it to each system it needs. On IBM i, that integration was built into the platform decades ago.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Auditing happens below the agent.&lt;/strong&gt; The audit journal records operations at the operating system level, beneath the layer where an agent decides what to do. An agent can&apos;t choose what gets logged or leave an operation out, because the platform does the recording, not the application.&lt;/p&gt;
&lt;h2&gt;Where IBM i falls short&lt;/h2&gt;
&lt;p&gt;The argument has real weaknesses, and most of them are about people, not architecture. The ecosystem is small. The developer community skews older, and the talent pipeline is a real concern. The &quot;legacy&quot; label hurts too: if you tell a CIO that the best platform for their agent strategy traces back to 1978, you can see the skepticism before you finish the sentence.&lt;/p&gt;
&lt;p&gt;The Model Context Protocol (MCP) changes part of this. Any MCP client can connect to an IBM i MCP server and work with the system, with no IBM i expertise needed on the client side. An agent developer needs Python and SQL, not RPG or CL. That doesn&apos;t fix the ecosystem, but it does mean the people building agents don&apos;t have to come from it.&lt;/p&gt;
&lt;h2&gt;Where this goes&lt;/h2&gt;
&lt;p&gt;I see IBM i as the trusted core for the agent operations that carry the most risk: payroll, financial data, production systems that can&apos;t go down and can&apos;t leak data. It doesn&apos;t replace the agent frameworks being built on Kubernetes, Azure, or AWS. It&apos;s where the high-stakes actions run, with governance enforced by the platform itself.&lt;/p&gt;
&lt;p&gt;Right now the industry is moving quickly on deploying agents and slowly on governing them. I expect that to change after the first major public breach caused by an agent. At that point the question becomes how safely agents can be deployed, not how fast, and governance built into the platform starts to matter when companies choose where to run them.&lt;/p&gt;
&lt;p&gt;The &lt;a href=&quot;https://adamshedivy.com/blog/enterprise-ai-operating-system/&quot;&gt;full post&lt;/a&gt; covers the architectural analysis in more depth, the security frameworks it draws on, what we&apos;ve built with iAssist, and the questions I&apos;m still working through. If you&apos;re running agents against IBM i, or thinking about it, I&apos;d like to hear what you&apos;re seeing.&lt;/p&gt;
</content:encoded></item><item><title>How to Run LLMs on IBM i Pt. 2</title><link>https://adamshedivy.com/blog/run-llms-on-ibm-i-part-2/</link><guid isPermaLink="true">https://adamshedivy.com/blog/run-llms-on-ibm-i-part-2/</guid><description>Llama.cpp now has an official RPM for IBM i, so you can install it with yum, download a compatible model, and run an LLM from PASE.</description><pubDate>Fri, 14 Nov 2025 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;In a post from earlier this year, &lt;a href=&quot;https://adamshedivy.com/blog/run-llms-on-ibm-i/&quot;&gt;How to run LLMs on IBM i&lt;/a&gt;, I showed how to run Large Language Models (LLMs) directly on IBM i using the open source framework &lt;a href=&quot;https://github.com/ggml-org/llama.cpp&quot;&gt;Llama.cpp&lt;/a&gt;. In this Part 2, I’ll share a big update: there’s now an official RPM for Llama.cpp on IBM i .&lt;/p&gt;
&lt;p&gt;Below, you’ll install the RPM, download a compatible model, and run an LLM from a PASE terminal.&lt;/p&gt;
&lt;hr /&gt;
&lt;h2&gt;1) Install Llama.cpp&lt;/h2&gt;
&lt;blockquote&gt;
&lt;p&gt;The llama.cpp package is available here: &lt;a href=&quot;https://public.dhe.ibm.com/software/ibmi/products/pase/rpms/repo-base-7.3&quot;&gt;https://public.dhe.ibm.com/software/ibmi/products/pase/rpms/repo-base-7.3&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;Run &lt;code&gt;yum upgrade&lt;/code&gt; to get the latest repo metadata. If you don’t see updates, try &lt;code&gt;yum clean metadata&lt;/code&gt;.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;From a PASE terminal, install the package:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;yum install llama-cpp
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;After installation, these commands will be available:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;llama-cli
llama-quantize
llama-run
llama-server
llama-simple
llama-simple-chat
&lt;/code&gt;&lt;/pre&gt;
&lt;hr /&gt;
&lt;h2&gt;2) Download a compatible model&lt;/h2&gt;
&lt;p&gt;You’ll need a model in GGUF format that works with Llama.cpp on IBM i. Use the public Hugging Face repo: &lt;a href=&quot;https://huggingface.co/models-for-i&quot;&gt;models-for-i (Models for IBM i)&lt;/a&gt;&lt;/p&gt;
&lt;h3&gt;Available models:&lt;/h3&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;strong&gt;Model&lt;/strong&gt;&lt;/th&gt;
&lt;th&gt;&lt;strong&gt;Provider&lt;/strong&gt;&lt;/th&gt;
&lt;th&gt;&lt;strong&gt;Parameters&lt;/strong&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Llama-3.2&lt;/td&gt;
&lt;td&gt;Meta&lt;/td&gt;
&lt;td&gt;1 Billion&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;blockquote&gt;
&lt;p&gt;Note: We only have one publicly compatible model: Llama-3.2-1B&lt;/p&gt;
&lt;p&gt;You can manually prepare models for IBM i using this repo: &lt;a href=&quot;https://github.com/ajshedivy/models-for-i&quot;&gt;ajshedivy/models-for-i&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h3&gt;Prerequisites&lt;/h3&gt;
&lt;p&gt;Before downloading, create a &lt;code&gt;models&lt;/code&gt; directory on IBM i:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;mkdir -p ~/models
&lt;/code&gt;&lt;/pre&gt;
&lt;h2&gt;Download Methods&lt;/h2&gt;
&lt;p&gt;There are two ways to download models from Hugging Face. The &lt;strong&gt;Hugging Face CLI method is recommended&lt;/strong&gt; for easier management and future downloads.&lt;/p&gt;
&lt;h3&gt;Method 1: Hugging Face CLI (Recommended)&lt;/h3&gt;
&lt;p&gt;&lt;strong&gt;Step 1: Set up the Hugging Face environment&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;Create a dedicated directory for Hugging Face tools:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;mkdir -p huggingface
cd huggingface
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;strong&gt;Step 2: Create a Python virtual environment&lt;/strong&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;python3.13 -m venv . --system-site-packages
source ./bin/activate
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;strong&gt;Step 3: Install the Hugging Face CLI&lt;/strong&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;pip install &quot;huggingface_hub[cli]&quot;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;strong&gt;Step 4: Authenticate with Hugging Face&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;First, create a Hugging Face account if you don&apos;t have one:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Visit &lt;a href=&quot;https://huggingface.co/&quot;&gt;https://huggingface.co/&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Sign up for a free account&lt;/li&gt;
&lt;li&gt;Navigate to Settings → Access Tokens&lt;/li&gt;
&lt;li&gt;Click &quot;New token&quot; and create a token with &quot;Read&quot; permissions&lt;/li&gt;
&lt;li&gt;Copy the token (you&apos;ll need it in the next step)&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Now authenticate on IBM i:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;hf auth login
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;When prompted, paste your access token and press Enter.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Step 5: Download model&lt;/strong&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;hf download --local-dir $HOME/models models-for-i/Llama-3.2-1B-Instruct
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;strong&gt;The download will begin and show progress. This may take several minutes depending on your connection speed.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Step 6: Verify Download&lt;/strong&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;ls -lh ~/models/Llama-3.2-1B-Instruct
&lt;/code&gt;&lt;/pre&gt;
&lt;hr /&gt;
&lt;h3&gt;Method 2: Curl&lt;/h3&gt;
&lt;p&gt;If you prefer a simpler approach or can&apos;t use the CLI, download directly with curl:&lt;/p&gt;
&lt;p&gt;Step 1: Natigate to your &lt;code&gt;models&lt;/code&gt; directory:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;cd ~/models
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;strong&gt;Step 2: Download the model file&lt;/strong&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;curl -L -o Llama-3.2-1B-Instruct-f32-be.gguf \
  https://huggingface.co/models-for-i/Llama-3.2-1B-Instruct/resolve/main/Llama-3.2-1B-Instruct-f32-be.gguf
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;strong&gt;Step 3: Verify the download&lt;/strong&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;ls -lh Llama-3.2-1B-Instruct-f32-be.gguf
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Once you&apos;ve successfully downloaded your model using either method, you&apos;re ready to proceed to the next section where we&apos;ll configure and run the model with Llama.cpp.&lt;/p&gt;
&lt;hr /&gt;
&lt;h2&gt;3) Next steps (optional)&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;Verify the binary runs:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;llama-cli --help
&lt;/code&gt;&lt;/pre&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;Try the simple run command to confirm everything is wired up:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;llama-run models/Llama-3.2-1B-Instruct-f32-be.gguf 
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;img src=&quot;https://adamshedivy.com/blog/run-llms-on-ibm-i-part-2/llama-run-hello-there.png&quot; alt=&quot;Terminal running llama-run with the Llama-3.2-1B-Instruct model on IBM i: the prompt &amp;quot;Hello there!&amp;quot; gets the reply &amp;quot;Hello! It&apos;s nice to meet you. Is there something I can help you with, or would you like to chat?&amp;quot;&quot; /&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Congrats, you are running an open-source LLM on IBM i !! In the next series, we will dive into more complex use cases, different models, etc.&lt;/p&gt;
&lt;hr /&gt;
&lt;h2&gt;Closing thoughts&lt;/h2&gt;
&lt;p&gt;Running LLMs natively on IBM i is a practical option for exploring different AI integrations and use cases. The official RPM for Llama.cpp reduces setup friction, and the GGUF ecosystem gives you flexibility to choose models that fit your hardware profile.&lt;/p&gt;
&lt;h2&gt;What’s next in this series&lt;/h2&gt;
&lt;p&gt;Coming up, we will be releasing more content around using llama.cpp:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;1) Running Different Models on IBM i&lt;/strong&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Compare small and mid-sized models that actually run well on Power hardware (Llama 3, DeepSeek-Coder, Phi-3, Mistral, etc.).&lt;/li&gt;
&lt;li&gt;Show when to pick each model depending on the workload: chat, coding help, SQL assist, or lightweight agents.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;2) Quantization, MMA, and Performance Tuning&lt;/strong&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Break down the practical tradeoffs between q4, q5, q8, and FP16—and which formats deliver the best latency on Power10.&lt;/li&gt;
&lt;li&gt;Cover simple system-level tweaks (threads, memory layout, loading behavior) that consistently improve inference speed.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;3) llama-server APIs and Example Apps&lt;/strong&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Walk through exposing a model as an HTTP endpoint on IBM i using llama-server.&lt;/li&gt;
&lt;li&gt;Build a few real examples: a SQL rewrite API, a green-screen text analyzer, and a small log-summary service.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Stay tuned!&lt;/p&gt;
&lt;h3&gt;Links&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href=&quot;https://adamshedivy.com/blog/run-llms-on-ibm-i/&quot;&gt;How to run LLMs on IBM i&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://huggingface.co/models-for-i&quot;&gt;models-for-i (Models for IBM i)&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
</content:encoded></item><item><title>How to run LLMs on IBM i</title><link>https://adamshedivy.com/blog/run-llms-on-ibm-i/</link><guid isPermaLink="true">https://adamshedivy.com/blog/run-llms-on-ibm-i/</guid><description>Run LLMs on IBM i with Llama.cpp</description><pubDate>Mon, 07 Apr 2025 00:00:00 GMT</pubDate><content:encoded>&lt;blockquote&gt;
&lt;p&gt;Check out Part 2!&lt;/p&gt;
&lt;p&gt;Part 2 has the latest instructions on installing llama.cpp using yum: &lt;a href=&quot;https://adamshedivy.com/blog/run-llms-on-ibm-i-part-2/&quot;&gt;How to Run LLMs on IBM i Pt. 2&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Note: This is experimental work.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;This guide is a work in process and is intended for experimentation and prototyping.&lt;/p&gt;
&lt;p&gt;Expect rough edges, manual steps, and a few patches along the way.&lt;/p&gt;
&lt;p&gt;Happy building!&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;In this post, I’ll walk you through how I successfully built &lt;code&gt;llama.cpp&lt;/code&gt; on &lt;strong&gt;IBM i&lt;/strong&gt;, step-by-step—from configuring the environment and compiling the source to running a local LLM right on the system. You’ll also learn how to prep a compatible model and test it using the built-in CLI, HTTP server, and web UI.&lt;/p&gt;
&lt;h2&gt;What is Llama.cpp?&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;llama.cpp&lt;/code&gt; is a lightweight, blazing-fast C++ implementation for running Large Language Models (LLMs) &lt;em&gt;locally&lt;/em&gt; on a wide range of hardware—no internet, no cloud, no external dependencies. It’s the engine powering many popular tools like &lt;strong&gt;Ollama&lt;/strong&gt;, &lt;strong&gt;LM Studio&lt;/strong&gt;, and other open-source AI apps. And now—for the first time—you can run it directly on &lt;strong&gt;IBM i&lt;/strong&gt;&lt;/p&gt;
&lt;h2&gt;Why run LLMs locally on IBM i?&lt;/h2&gt;
&lt;p&gt;If you’ve been curious about integrating LLMs into your applications, this is the perfect way to get started. Running LLMs directly on IBM i enables quick, cost-free &lt;strong&gt;prototyping&lt;/strong&gt; of AI-powered features, all within your existing environment:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Build a &lt;strong&gt;chat-style interface&lt;/strong&gt; for interacting with system data&lt;/li&gt;
&lt;li&gt;Experiment with &lt;strong&gt;text generation, summarization&lt;/strong&gt;, or &lt;strong&gt;code suggestions&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;Package and test ideas locally—no need to send data off the box&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;Build Llama.cpp on IBM i&lt;/h2&gt;
&lt;p&gt;First make sure you have the open source environment installed on your IBM i system. you can follow directions here: &lt;a href=&quot;https://ibmi-oss-docs.readthedocs.io/en/latest/yum/README.html#installation&quot;&gt;RPM pile for IBM i releases in standard support — IBM i OSS Docs  documentation&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;to configure your environment.&lt;/p&gt;
&lt;h3&gt;1. Install Open source packages&lt;/h3&gt;
&lt;p&gt;Install the required open source packages:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;gcc12&lt;/li&gt;
&lt;li&gt;gcc12-cplusplus&lt;/li&gt;
&lt;li&gt;libutil2&lt;/li&gt;
&lt;li&gt;git&lt;/li&gt;
&lt;li&gt;cmake&lt;/li&gt;
&lt;li&gt;openblas&lt;/li&gt;
&lt;li&gt;openblas-devel&lt;/li&gt;
&lt;li&gt;libpthread-stubs-devel&lt;/li&gt;
&lt;li&gt;binutils-gnu&lt;/li&gt;
&lt;li&gt;libutil-devel&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Optional (but recommended):&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;ccache&lt;/li&gt;
&lt;/ul&gt;
&lt;pre&gt;&lt;code&gt;# Required packages
yum install gcc12 gcc12-cplusplus libutil2 git cmake openblas openblas-devel \
            libpthread-stubs-devel binutils-gnu libutil-devel

# Optional (but recommended)
yum install ccache
&lt;/code&gt;&lt;/pre&gt;
&lt;h4&gt;Note&lt;/h4&gt;
&lt;p&gt;On some P10 systems,  you may need to explicitly use &lt;code&gt;binutils-gnu&lt;/code&gt; :&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;cd $HOME
mkdir bin
ln -s /QOpenSys/pkgs/bin/gnu-as bin/as
export PATH=$PWD/bin:$PATH
&lt;/code&gt;&lt;/pre&gt;
&lt;h3&gt;2. Clone Llama.cpp Repo&lt;/h3&gt;
&lt;p&gt;Clone the llama.cpp repo on IBM i&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Checkout tag release&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;git checkout -b ibmi tags/b5043 
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;this is the latest tag I used. Cloning the most recent release will likely work just fine.&lt;/p&gt;
&lt;h3&gt;3. Make a small patch&lt;/h3&gt;
&lt;p&gt;Once you have the llama.cpp repo cloned, we need to make a small change to the source code so that it will work on IBM i. go to &lt;code&gt;llama.cpp/src/llama-mmap.cpp&lt;/code&gt; and comment out the &lt;code&gt;RLIMIT_MEMLOCK&lt;/code&gt; logic found on around line 484-490:&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://github.com/ggml-org/llama.cpp/blob/e391d3ee8ddae86be70c034de1082ad51c55e211/src/llama-mmap.cpp#L484-L490&quot;&gt;llama.cpp/src/llama-mmap.cpp at e391d3e · ggml-org/llama.cpp&lt;/a&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;# Open this file in your preferred text editor:
vim src/llama-mmap.cpp
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Here is the exact code:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;struct rlimit lock_limit;
if (suggest &amp;amp;&amp;amp; getrlimit(RLIMIT_MEMLOCK, &amp;amp;lock_limit)) {
    suggest = false;
}
if (suggest &amp;amp;&amp;amp; (lock_limit.rlim_max &amp;gt; lock_limit.rlim_cur + size)) {
    suggest = false;
}
&lt;/code&gt;&lt;/pre&gt;
&lt;h3&gt;4. Build&lt;/h3&gt;
&lt;p&gt;Now we are ready to run the build. Navigate to the root of the llama.cpp repo and run the following:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;cmake -B build_cpu \
  -DCMAKE_AR=&quot;/QOpenSys/pkgs/bin/gcc-ar-12&quot; \
  -DCMAKE_RANLIB=&quot;/QOpenSys/usr/bin/ranlib&quot; \
  -DCMAKE_C_COMPILER=/QOpenSys/pkgs/bin/gcc-12 \
  -DCMAKE_CXX_COMPILER=/QOpenSys/pkgs/bin/g++-12 \
  -DCMAKE_CXX_FLAGS=&quot;-pthread&quot; \
  -DCMAKE_EXE_LINKER_FLAGS=&quot;-pthread -lutil&quot; \
  -DCMAKE_CXX_ARCHIVE_CREATE=&quot;&amp;lt;CMAKE_AR&amp;gt; -X64 qc &amp;lt;TARGET&amp;gt; &amp;lt;LINK_FLAGS&amp;gt; &amp;lt;OBJECTS&amp;gt;&quot; \
  -DCMAKE_CXX_ARCHIVE_APPEND=&quot;&amp;lt;CMAKE_AR&amp;gt; -X64 q  &amp;lt;TARGET&amp;gt; &amp;lt;LINK_FLAGS&amp;gt; &amp;lt;OBJECTS&amp;gt;&quot; \
  -DCMAKE_CXX_ARCHIVE_FINISH=&quot;&amp;lt;CMAKE_RANLIB&amp;gt; &amp;lt;TARGET&amp;gt;&quot; \
  -DCMAKE_BUILD_TYPE=Release
&lt;/code&gt;&lt;/pre&gt;
&lt;h4&gt;Current Build Summary&lt;/h4&gt;
&lt;ul&gt;
&lt;li&gt;native CPU build for either Power 9 or Power 10&lt;/li&gt;
&lt;li&gt;Threading and OpenMP enabled&lt;/li&gt;
&lt;li&gt;Release build type&lt;/li&gt;
&lt;/ul&gt;
&lt;h4&gt;Run build&lt;/h4&gt;
&lt;pre&gt;&lt;code&gt;cmake --build build_cpu --config Release
&lt;/code&gt;&lt;/pre&gt;
&lt;h4&gt;Note&lt;/h4&gt;
&lt;p&gt;If you encounter errors with &lt;code&gt;sgemm.cpp&lt;/code&gt; :&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;error: static assertion failed: RN/RM values not supported
static_assert(false, &quot;RN/RM values not supported&quot;);
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;You’re likely hitting an issue with the &lt;strong&gt;TinyBLAS&lt;/strong&gt; backend used by &lt;code&gt;llamafile&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;This error occurs on some PowerPC architectures due to unsupported instruction patterns in &lt;code&gt;sgemm.cpp&lt;/code&gt;, a minimal, highly optimized matrix multiplication routine introduced in &lt;a href=&quot;https://github.com/ggerganov/llama.cpp/pull/6414&quot;&gt;#6414&lt;/a&gt; and originally written by &lt;a href=&quot;https://justine.lol/matmul/&quot;&gt;Justine Tunney&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;How to fix it&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;Disable the &lt;code&gt;GGML_LLAMAFILE&lt;/code&gt; flag during your cmake step:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;cmake -B build_cpu \
	... \
	-DGGML_LLAMAFILE=OFF
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;If you encounter errors with the assembler:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;Assembler:
line 1: Invalid -m flag assembly mode operand: PWR10
Valid values are:COM PWR PWR2 PPC 601 603 604 PPC64 620 A35 PWR4 PWR5 PWR5X 970 PPC970 PWR6 PWR6E PWR7 PWR8 PWR9 ANY
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Ensure you&apos;re using &lt;strong&gt;GNU binutils&lt;/strong&gt; and not the default system assembler. See the note above on symlinking &lt;code&gt;as&lt;/code&gt;. See &lt;a href=&quot;#note&quot;&gt;Note&lt;/a&gt;&lt;/p&gt;
&lt;h4&gt;Verify Build&lt;/h4&gt;
&lt;pre&gt;&lt;code&gt;./build_cpu/bin/llama-cli --help
&lt;/code&gt;&lt;/pre&gt;
&lt;h3&gt;Prepare a model for Llama.cpp&lt;/h3&gt;
&lt;p&gt;Next, we need to obtain a model compatible with Llama.cpp and transfer it to the IBM i system. Since IBM i does not support some of the required Python dependencies, we’ll first download the model locally and then manually transfer it to the system.&lt;/p&gt;
&lt;p&gt;Clone the repo on your local machine to get started:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;git clone https://github.com/ajshedivy/models-for-i.git
cd models-for-i
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The instructions can be found in the README of the repo: &lt;a href=&quot;https://github.com/ajshedivy/models-for-i/tree/main?tab=readme-ov-file#models-for-ibm-i&quot;&gt;GitHub - ajshedivy/models-for-i&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;Once you are done, you should have an IBM i compatible model (&lt;code&gt;Llama-3.2-1B-Instruct&lt;/code&gt;) transferred to your system.&lt;/p&gt;
&lt;h2&gt;Run Llama.cpp on IBM i&lt;/h2&gt;
&lt;p&gt;We are almost there! So far, you have:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;built Llama.cpp from source on IBM i&lt;/li&gt;
&lt;li&gt;Downloaded a model from Hugging Face&lt;/li&gt;
&lt;li&gt;Transferred the model to IBM i&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Now we are ready to run the model!&lt;/p&gt;
&lt;h3&gt;Llama.cpp CLI&lt;/h3&gt;
&lt;pre&gt;&lt;code&gt;cd $HOME
./llama.cpp/build_cpu/bin/llama-cli \
	-m ~/models/Llama-3.2-1B-Instruct-f32-be.gguf \
	-p &quot;Hello there!&quot;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Arguments:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;-m&lt;/code&gt; is the model which we transferred to IBM i&lt;/li&gt;
&lt;li&gt;&lt;code&gt;-p&lt;/code&gt; is the prompt text&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Here is a sample output of the command:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;== Running in interactive mode. ==
 - Press Ctrl+C to interject at any time.
 - Press Return to return control to the AI.
 - To return control without starting a new line, end your input with &apos;/&apos;.
 - If you want to submit another line, end your input with &apos;\&apos;.
 - Not using system message. To change it, set a different value via -sys PROMPT

system

Cutting Knowledge Date: December 2023
Today Date: 07 Apr 2025

user

Hello there!assistant

Hello! It&apos;s nice to meet you. Is there something I can help you with or would you like to chat?

&amp;gt; 
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;There are a ton of options for the main llama-cli:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href=&quot;https://github.com/ggml-org/llama.cpp/tree/master/examples/main#llamacppexamplesmain&quot;&gt;llama.cpp/examples/main at master · ggml-org/llama.cpp&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Check them out and experiment with different combinations. I plan to write a separate blog on performance metrics and more in depth demos&lt;/p&gt;
&lt;h3&gt;Llama Run (llama-run)&lt;/h3&gt;
&lt;p&gt;This example is meant to demonstrate a minimal interactive chat:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;cd $HOME
./llama.cpp/build_cpu/bin/llama-run ~/models/Llama-3.2-1B-Instruct-f32-be.gguf
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;img src=&quot;https://adamshedivy.com/blog/run-llms-on-ibm-i/llama-run-chat.png&quot; alt=&quot;Terminal running llama-run with the Llama-3.2-1B-Instruct model on IBM i: the prompt &amp;quot;Hello there!&amp;quot; gets the reply &amp;quot;Hello! How can I assist you today?&amp;quot;&quot; /&gt;&lt;/p&gt;
&lt;p&gt;&lt;code&gt;llama-run&lt;/code&gt; docs:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href=&quot;https://github.com/ggml-org/llama.cpp/tree/master/examples/run&quot;&gt;llama.cpp/examples/run at master · ggml-org/llama.cpp&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;Llama Server&lt;/h3&gt;
&lt;p&gt;Llama.cpp comes with a built in fast, lightweight, pure C/C++ HTTP server and a set of LLM REST APIs and a simple web front end to interact with llama.cpp&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Features&lt;/strong&gt;:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;LLM inference of F16 and quantized models on GPU and CPU&lt;/li&gt;
&lt;li&gt;Parallel decoding with multi-user support&lt;/li&gt;
&lt;li&gt;Continuous batching&lt;/li&gt;
&lt;li&gt;Multimodal (wip)&lt;/li&gt;
&lt;li&gt;Monitoring endpoints&lt;/li&gt;
&lt;li&gt;Schema-constrained JSON response format&lt;/li&gt;
&lt;/ul&gt;
&lt;h4&gt;Start the server&lt;/h4&gt;
&lt;pre&gt;&lt;code&gt;cd $HOME
./llama.cpp/build_cpu/bin/llama-server \
	-m ~/models/Llama-3.2-1B-Instruct-f32-be.gguf 
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This will start the server on port &lt;code&gt;8080&lt;/code&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;main: server is listening on http://127.0.0.1:8080 - starting the main loop
&lt;/code&gt;&lt;/pre&gt;
&lt;h4&gt;Test the server&lt;/h4&gt;
&lt;p&gt;In another terminal, ssh to the same system and run the following command:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;curl --request POST \
     --url http://localhost:8080/completion \
     --header &quot;Content-Type: application/json&quot; \
     --data &apos;{
         &quot;prompt&quot;: &quot;once upon a time, &quot;,
         &quot;stream&quot;: true,
         &quot;n_predict&quot;: 50
     }&apos;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;video src=&quot;https://adamshedivy.com/blog/run-llms-on-ibm-i/llama-server-curl-test.mp4&quot; controls muted playsinline&gt;&lt;/video&gt;&lt;/p&gt;
&lt;h3&gt;Llama.cpp UI&lt;/h3&gt;
&lt;p&gt;Starting the server will also create a Web UI on port 8080. In a browser, navigate to:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;http://HOST:8080&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;video src=&quot;https://adamshedivy.com/blog/run-llms-on-ibm-i/llama-server-web-ui.mp4&quot; controls muted playsinline&gt;&lt;/video&gt;&lt;/p&gt;
&lt;p&gt;This is still under active development, more info can be found here:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href=&quot;https://github.com/ggml-org/llama.cpp/tree/master/examples/server&quot;&gt;llama.cpp/examples/server at master · ggml-org/llama.cpp&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;Closing thoughts&lt;/h2&gt;
&lt;p&gt;Running &lt;code&gt;llama.cpp&lt;/code&gt; on IBM i is not only possible—it&apos;s &lt;em&gt;fun&lt;/em&gt; and incredibly rewarding! With just a few tweaks, you can bring cutting-edge LLM tech to your favorite midrange system Here’s a quick recap of what we covered:&lt;/p&gt;
&lt;h3&gt;Setup &amp;amp; Build&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;Installed necessary open source packages&lt;/li&gt;
&lt;li&gt;Patched the &lt;code&gt;RLIMIT_MEMLOCK&lt;/code&gt; logic&lt;/li&gt;
&lt;li&gt;Configured and built &lt;code&gt;llama.cpp&lt;/code&gt; from source on Power9/Power10&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;Model Preparation&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;Downloaded IBM i–compatible model locally&lt;/li&gt;
&lt;li&gt;Transferred to IBM i manually for use with &lt;code&gt;llama.cpp&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;Running Llama.cpp&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;Used the CLI (&lt;code&gt;llama-cli&lt;/code&gt;) for quick tests and prompts&lt;/li&gt;
&lt;li&gt;Launched &lt;code&gt;llama-run&lt;/code&gt; for interactive chat&lt;/li&gt;
&lt;li&gt;Hosted the built-in HTTP server with &lt;code&gt;llama-server&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Sent test requests via &lt;code&gt;curl&lt;/code&gt; and explored the Web UI in a browser&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;Highlights&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;Built &lt;em&gt;natively&lt;/em&gt; for IBM i&lt;/li&gt;
&lt;li&gt;Works with &lt;code&gt;gguf&lt;/code&gt; models like &lt;code&gt;Llama-3.2-1B-Instruct&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Fully local, zero dependencies, minimal configuration&lt;/li&gt;
&lt;li&gt;Supports HTTP API and interactive usage modes&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Stay tuned for a follow-up post with &lt;strong&gt;performance tips&lt;/strong&gt;, &lt;strong&gt;token benchmarks&lt;/strong&gt;, and &lt;strong&gt;real-world IBM i use cases&lt;/strong&gt;!&lt;/p&gt;
</content:encoded></item><item><title>Setup a local code assistant for Db2 for i, a developer guide</title><link>https://adamshedivy.com/blog/local-code-assistant-db2-for-i/</link><guid isPermaLink="true">https://adamshedivy.com/blog/local-code-assistant-db2-for-i/</guid><description>Setup a local assistant to help you write SQL!</description><pubDate>Tue, 18 Mar 2025 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;As developers, we want tools that are help maximize productivity but also allow us to customize our experience for our specific use case. When it comes to AI tools such as code assistants, there are a number of options, but unfortunately many require premium subscriptions for models and LLM providers. Continue, the leading open source code assistant, offers a much more flexible code assistant solution. It seamlessly integrates with top LLM providers like OpenAI, Anthropic, and Gemini, while also supporting local hosting options such as Ollama and Llama.cpp, giving developers the freedom to choose the setup that best suits their workflow.&lt;/p&gt;
&lt;p&gt;In this blog, we will go over how to setup a local code assistant specifically tailored for Db2 for i and SQL. Here are some of the main advantages of local code assistants:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Data privacy:&lt;/strong&gt; your code stays on your machine, not sent to 3rd-party servers&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Customization:&lt;/strong&gt; choose which models to use for specific coding tasks&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Configuration control&lt;/strong&gt;: choose exactly how you want the model to behave by configuring model parameters&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Offline usage&lt;/strong&gt;: work offline or without an internet connection&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No subscription costs:&lt;/strong&gt; experiment and use powerful open-source models without committing to premium subscriptions&lt;/li&gt;
&lt;/ol&gt;
&lt;h2&gt;Requirements&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;A computer capable of running LLMs locally&lt;/li&gt;
&lt;li&gt;VS Code IDE with &lt;a href=&quot;https://marketplace.visualstudio.com/items?itemName=Continue.continue&amp;amp;ref=blog.continue.dev&quot;&gt;Continue extension&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;IBM i VS Code extensions:
&lt;ul&gt;
&lt;li&gt;&lt;a href=&quot;https://marketplace.visualstudio.com/items?itemName=HalcyonTechLtd.code-for-ibmi&quot;&gt;Code for i&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://marketplace.visualstudio.com/items?itemName=HalcyonTechLtd.vscode-db2i&quot;&gt;Db2 for i&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;a Continue Account (&lt;a href=&quot;https://hub.continue.dev/signup?ref=blog.continue.dev&quot;&gt;https://hub.continue.dev/signup?ref=blog.continue.dev&lt;/a&gt;)&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;Step 1: install Ollama&lt;/h2&gt;
&lt;p&gt;Ollama allows you to run LLMs locally on your laptop. Here is how to set it up:&lt;/p&gt;
&lt;h3&gt;MacOS/Linux&lt;/h3&gt;
&lt;pre&gt;&lt;code&gt;curl -fsSL https://ollama.ai/install.sh | sh
&lt;/code&gt;&lt;/pre&gt;
&lt;h3&gt;Windows&lt;/h3&gt;
&lt;p&gt;Download installer from &lt;a href=&quot;https://ollama.com/download?ref=blog.continue.dev&quot;&gt;Ollama&lt;/a&gt;&lt;/p&gt;
&lt;h2&gt;Step 2: Pull model for SQL&lt;/h2&gt;
&lt;p&gt;Next, we need to pull a model to help us write SQL. Most modern models are fairly good at SQL. I find that the Granite family of models works well for small use cases:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;ollama pull granite3.2:8b
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This will download the model to your machine. To verify it works, test it out with this simple command:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;ollama run granite3.2:8b &quot;write a python function to convert a string to base64&quot;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Other models that work well with SQL (from personal experience):&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href=&quot;https://ollama.com/library/llama3.1&quot;&gt;llama3.1&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://ollama.com/library/llama3.2&quot;&gt;llama3.2&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://ollama.com/library/qwen2.5&quot;&gt;qwen2.5&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://ollama.com/library/gemma3&quot;&gt;gemma3&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;Step 3: Set up your Continue account&lt;/h2&gt;
&lt;p&gt;Before we setup the Db2 for i Assistant, first create your Continue Account&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Visit: &lt;a href=&quot;https://hub.continue.dev/signup?ref=blog.continue.dev&quot;&gt;Sign up&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Verify email address&lt;/li&gt;
&lt;li&gt;Log in to access Continue Hub&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;What is Continue Hub?&lt;/h3&gt;
&lt;p&gt;Continue Hub is a platform for building and sharing open source developer tools like code assistants, models, prompts and MCP tools. Read more about the launch of Continue 1.0:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href=&quot;https://blog.continue.dev/continue-1-0/&quot;&gt;Continue 1.0&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;Step 4: Log into Continue in VS Code&lt;/h2&gt;
&lt;p&gt;Before we open up the assistant in VSCode, first install the Continue Extension&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href=&quot;https://marketplace.visualstudio.com/items?itemName=Continue.continue&amp;amp;ref=blog.continue.dev&quot;&gt;Continue extension&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;For detailed instructions, see the &lt;a href=&quot;https://docs.continue.dev/getting-started/install?ref=blog.continue.dev&quot;&gt;official documentation&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Once installed, log in with your Continue account created in step 3.&lt;/p&gt;
&lt;h2&gt;Step 5: Add the Db2 for i Assistant&lt;/h2&gt;
&lt;p&gt;Once your Continue account is setup, you can add the Db2 for i assistant&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;From Continue Hub Explore page, search for “Db2 for i Assistant”, or follow this link:
&lt;ul&gt;
&lt;li&gt;&lt;a href=&quot;https://hub.continue.dev/adam-shedivy/db2i-assistant&quot;&gt;hub.continue.dev&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;&lt;img src=&quot;https://adamshedivy.com/blog/local-code-assistant-db2-for-i/db2i-assistant-hub-page.png&quot; alt=&quot;The Db2 for i Assistant page on Continue Hub, listing its models (qwen2.5-coder 1.5b, Granite 3.2 and Ollama), rules, prompts and context&quot; /&gt;&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Select “Add Assistant”&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;&lt;img src=&quot;https://adamshedivy.com/blog/local-code-assistant-db2-for-i/add-assistant-button.png&quot; alt=&quot;The Star, Remix and Add Assistant buttons on the Continue Hub assistant page&quot; /&gt;&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Once added, the assistant will appear in your collection of custom assistants&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;&lt;img src=&quot;https://adamshedivy.com/blog/local-code-assistant-db2-for-i/assistant-added.png&quot; alt=&quot;Continue Hub after the assistant is added: the button reads Assistant Added and a side panel shows the Db2 for i Assistant with its models&quot; /&gt;&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Select “Open VS Code” to add the assistant to your Continue extension&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;&lt;img src=&quot;https://adamshedivy.com/blog/local-code-assistant-db2-for-i/open-vs-code.png&quot; alt=&quot;An arrow points to the Open VS Code button at the bottom of the Continue Hub side panel&quot; /&gt;&lt;/p&gt;
&lt;p&gt;Here is a quick demo:&lt;/p&gt;
&lt;p&gt;&lt;video src=&quot;https://adamshedivy.com/blog/local-code-assistant-db2-for-i/add-assistant-demo.mp4&quot; controls muted playsinline&gt;&lt;/video&gt;&lt;/p&gt;
&lt;h2&gt;Step 6: Connect to an IBM i system and invoke &lt;code&gt;@Db2i&lt;/code&gt; context provider&lt;/h2&gt;
&lt;p&gt;Once we have the assistant added, now all we need to do is connect to an IBM i system and start asking questions!&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Install the Code for IBM i extensions:
&lt;ol&gt;
&lt;li&gt;&lt;a href=&quot;https://marketplace.visualstudio.com/items?itemName=HalcyonTechLtd.code-for-ibmi&quot;&gt;Code for i&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://marketplace.visualstudio.com/items?itemName=HalcyonTechLtd.vscode-db2i&quot;&gt;Db2 for i&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;/li&gt;
&lt;li&gt;Connect to an IBM i system using the Code for i extension
&lt;ol&gt;
&lt;li&gt;&lt;a href=&quot;https://codefori.github.io/docs/quickstart/&quot;&gt;Quick Start&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;/li&gt;
&lt;li&gt;Switch to Continue, and type &lt;code&gt;@Db2i&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;You can now ask questions about objects in your library list!&lt;/li&gt;
&lt;/ol&gt;
&lt;h3&gt;Quick Demo:&lt;/h3&gt;
&lt;p&gt;&lt;video src=&quot;https://adamshedivy.com/blog/local-code-assistant-db2-for-i/db2i-query-demo.mp4&quot; controls muted playsinline&gt;&lt;/video&gt;&lt;/p&gt;
&lt;p&gt;To reproduce this demo:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;first install the SAMPLE schema&lt;/li&gt;
&lt;/ul&gt;
&lt;pre&gt;&lt;code&gt;CALL QSYS.CREATE_SQL_SAMPLE(&apos;SAMPLE&apos;)
&lt;/code&gt;&lt;/pre&gt;
&lt;ul&gt;
&lt;li&gt;Add &lt;code&gt;SAMPLE&lt;/code&gt; to your library list in the active SQL job:&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;img src=&quot;https://adamshedivy.com/blog/local-code-assistant-db2-for-i/sql-job-library-list.png&quot; alt=&quot;The Db2 for i SQL Job Manager in VS Code, editing a job&apos;s options with SAMPLE, QGPL and QTEMP in the library list&quot; /&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;save the edited Job, then ask the following query:&lt;/li&gt;
&lt;/ul&gt;
&lt;pre&gt;&lt;code&gt; @Db2i how many employees are in each department?
&lt;/code&gt;&lt;/pre&gt;
</content:encoded></item><item><title>Use MAX with Continue as a local Code Assistant</title><link>https://adamshedivy.com/blog/max-with-continue/</link><guid isPermaLink="true">https://adamshedivy.com/blog/max-with-continue/</guid><description>Serve a model locally with Modular&apos;s MAX Serve and connect it to the Continue extension for a private code assistant in VS Code.</description><pubDate>Wed, 29 Jan 2025 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;This guide explores how to set up a powerful local code assistance solution by combining Modular&apos;s MAX Serve with the Continue VS Code extension.&lt;/p&gt;
&lt;p&gt;We&apos;ll walk through the process of deploying a local language model using MAX Serve and integrating it with Continue, creating a seamless development experience that operates entirely on your local machine. This setup provides the benefits of AI-powered code assistance while maintaining full control over your development environment.&lt;/p&gt;
&lt;h2&gt;MAX Serve&lt;/h2&gt;
&lt;p&gt;MAX Serve is a Python-based inference server that provides an OpenAI-compatible REST endpoint for both local and cloud environments. At its core, MAX Serve uses MAX Engine, which features a high-performance graph compiler and runtime for model acceleration.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://adamshedivy.com/blog/max-with-continue/max-serve-architecture.png&quot; alt=&quot;MAX Serve architecture: an HTTP client sends requests to the MAX container, where MAX Serve exposes an OpenAI REST API with scheduling, batching and model loading on top of MAX Engine, running on cloud providers and CPU, GPU or xPU hardware&quot; /&gt;&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;source: &lt;a href=&quot;https://docs.modular.com/max/serve&quot;&gt;https://docs.modular.com/max/serve&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h2&gt;Continue&lt;/h2&gt;
&lt;p&gt;The open-source code assistant Continue can connect directly to MAX Serve&apos;s OpenAI-compatible REST endpoint.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://www.continue.dev/&quot;&gt;Continue&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;To get started, we&apos;ll launch the MAX Serve endpoint and then configure it in Continue.&lt;/p&gt;
&lt;h2&gt;Deploy local Endpoint&lt;/h2&gt;
&lt;p&gt;The following is from the MAX documentation: &lt;a href=&quot;https://docs.modular.com/max/tutorials/deploy-pytorch-llm#deploy-to-a-local-endpoint&quot;&gt;Deploy a PyTorch model from Hugging Face | Modular&lt;/a&gt;&lt;/p&gt;
&lt;h3&gt;Prerequisites&lt;/h3&gt;
&lt;p&gt;Install the Hugging Face CLI: &lt;a href=&quot;https://huggingface.co/docs/huggingface_hub/en/guides/cli&quot;&gt;Command Line Interface (CLI)&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;Authenticate using &lt;code&gt;huggingface-cli&lt;/code&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;huggingface-cli login
&lt;/code&gt;&lt;/pre&gt;
&lt;h3&gt;Setup MAX repository&lt;/h3&gt;
&lt;p&gt;Clone the MAX repo:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;git clone git@github.com:modularml/max.git
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;After cloning, navigate to &lt;code&gt;max/pipelines/python&lt;/code&gt; directory&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;cd max/pipelines/python/
&lt;/code&gt;&lt;/pre&gt;
&lt;h3&gt;Serve Endpoint&lt;/h3&gt;
&lt;p&gt;Use the &lt;code&gt;magic&lt;/code&gt; CLI to start the server endpoint. If you don&apos;t have the &lt;a href=&quot;https://docs.modular.com/magic/&quot;&gt;magic&lt;/a&gt; CLI yet, you can install it on macOS and Ubuntu Linux with this command:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;curl -ssL https://magic.modular.com/deb1e9f0-447e-4b92-a807-29d62898936f | bash
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Then run the &lt;code&gt;source&lt;/code&gt; command that&apos;s printed in your terminal.&lt;/p&gt;
&lt;p&gt;The command we&apos;ll run is &lt;code&gt;magic run serve&lt;/code&gt;, which takes supports a&lt;br /&gt;&lt;code&gt;huggingface-repo-id&lt;/code&gt; parameter. This parameter allows you to specify an LLM&lt;br /&gt;model hosted on Hugging Face.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;magic run serve --huggingface-repo-id=Qwen/Qwen2.5-1.5B-Instruct
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This command downloads the model and sets up MAX Serve to host a local endpoint.&lt;/p&gt;
&lt;p&gt;The endpoint is ready when you see output similar to the following in your terminal:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;uvicorn running on http://0.0.0.0:8000
&lt;/code&gt;&lt;/pre&gt;
&lt;h3&gt;Test the endpoint&lt;/h3&gt;
&lt;p&gt;once the server is up and running, we can test with this &lt;code&gt;curl&lt;/code&gt; command:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;curl -N http://localhost:8000/v1/chat/completions \
    -H &quot;Content-Type: application/json&quot; \
    -d &apos;{
        &quot;model&quot;: &quot;Qwen/Qwen2.5-1.5B-Instruct&quot;,
        &quot;stream&quot;: true,
        &quot;messages&quot;: [
            {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;You are a helpful assistant.&quot;},
            {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;What is the capital of Mongolia?&quot;}
        ]
    }&apos; | grep -o &apos;&quot;content&quot;:&quot;[^&quot;]*&quot;&apos; | sed &apos;s/&quot;content&quot;:&quot;//g&apos; | sed &apos;s/&quot;//g&apos; | tr -d &apos;\n&apos; | sed &apos;s/\\n/\n/g&apos;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Here is the expected output:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;The capital city of Mongolia is Ulaanbaatar.
&lt;/code&gt;&lt;/pre&gt;
&lt;h2&gt;Configure MAX Serve with Continue&lt;/h2&gt;
&lt;p&gt;Now that we have our MAX Serve endpoint running locally, lets connect our endpoint to Continue in VSCode!&lt;/p&gt;
&lt;h3&gt;Setup VS Code&lt;/h3&gt;
&lt;ol&gt;
&lt;li&gt;Click &lt;code&gt;Install&lt;/code&gt; on the &lt;a href=&quot;https://marketplace.visualstudio.com/items?itemName=Continue.continue&quot;&gt;Continue extension page in the Visual Studio Marketplace&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;This will open the Continue extension page in VS Code, where you will need to click &lt;code&gt;Install&lt;/code&gt; again&lt;/li&gt;
&lt;li&gt;The Continue logo will appear on the left sidebar. For a better experience, move Continue to the right sidebar&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;&lt;img src=&quot;https://adamshedivy.com/blog/max-with-continue/continue-in-vs-code-sidebar.png&quot; alt=&quot;VS Code with the Continue chat panel moved to the right sidebar&quot; /&gt;&lt;/p&gt;
&lt;h3&gt;Configure MAX Serve Endpoint&lt;/h3&gt;
&lt;p&gt;Next, we need to configure our MAX Serve endpoint.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Open the VSCode command pallet (cmd+shift+p) and run &lt;code&gt;Continue: Open Continue Config&lt;/code&gt; This will open the configuration settings for the Extension: &lt;code&gt;~./continue/config.json&lt;/code&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;&lt;img src=&quot;https://adamshedivy.com/blog/max-with-continue/open-continue-config.png&quot; alt=&quot;The VS Code command palette showing the Continue: Open Continue Config command&quot; /&gt;&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;under the &lt;code&gt;models&lt;/code&gt; section of the config file, add the following:&lt;/li&gt;
&lt;/ol&gt;
&lt;pre&gt;&lt;code&gt;{
  &quot;models&quot;: [
    {
      &quot;model&quot;: &quot;Qwen/Qwen2.5-1.5B-Instruct&quot;,
      &quot;title&quot;: &quot;MAX - Qwen/Qwen2.5-1.5B-Instruct&quot;,
      &quot;apiBase&quot;: &quot;http://localhost:8000/v1/&quot;,
      &quot;apiKey&quot;: &quot;&quot;,
      &quot;provider&quot;: &quot;openai&quot;
    }
  ]
}
&lt;/code&gt;&lt;/pre&gt;
&lt;blockquote&gt;
&lt;p&gt;the &lt;code&gt;apibase&lt;/code&gt; is the same of the server endpoint from before: &lt;code&gt;http://0.0.0.0:8000&lt;/code&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Once you save the config file, the model should be available from dropdown in  the chat window&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://adamshedivy.com/blog/max-with-continue/max-model-dropdown.png&quot; alt=&quot;The Continue chat model dropdown with MAX - Qwen/Qwen2.5-1.5B-Instruct listed alongside other models&quot; /&gt;&lt;/p&gt;
&lt;h3&gt;Continue Features&lt;/h3&gt;
&lt;p&gt;here are some Continue features using MAX Serve&lt;/p&gt;
&lt;h4&gt;Chat&lt;/h4&gt;
&lt;p&gt;The chat interface allows you to get AI assistance directly within your IDE environment&lt;/p&gt;
&lt;p&gt;&lt;video src=&quot;https://adamshedivy.com/blog/max-with-continue/chat-demo.mp4&quot; controls muted playsinline&gt;&lt;/video&gt;&lt;/p&gt;
&lt;h4&gt;Edit&lt;/h4&gt;
&lt;p&gt;The Edit feature provides a convenient way to modify code directly within your current file&lt;/p&gt;
&lt;p&gt;&lt;video src=&quot;https://adamshedivy.com/blog/max-with-continue/edit-demo.mp4&quot; controls muted playsinline&gt;&lt;/video&gt;&lt;/p&gt;
&lt;h2&gt;Key takeaways&lt;/h2&gt;
&lt;p&gt;Using MAX Serve with Continue provides a powerful local code assistant solution that combines the efficiency of local model inference with the user-friendly features of Continue. By leveraging MAX&apos;s OpenAI-compatible REST endpoint, developers can access AI assistance directly within their IDE without relying on cloud services. The setup process involves deploying a local endpoint using MAX Serve and configuring Continue to communicate with this endpoint, resulting in a seamless development experience with features like contextual chat and direct code editing capabilities.&lt;/p&gt;
&lt;h2&gt;Sources&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href=&quot;https://www.modular.com/blog/use-max-with-open-webui-for-rag-and-web-search&quot;&gt;Modular: Use MAX with Open WebUI for RAG and Web Search&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://docs.modular.com/max/serve&quot;&gt;MAX Serve | Modular&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://docs.continue.dev/&quot;&gt;Introduction | Continue&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
</content:encoded></item><item><title>Running LLMs on IBM i: An experiment</title><link>https://adamshedivy.com/blog/running-llms-on-ibm-i-experiment/</link><guid isPermaLink="true">https://adamshedivy.com/blog/running-llms-on-ibm-i-experiment/</guid><description>I built llama.cpp on IBM i and got a model running, but its output was unusable, possibly because of byte order, model conversion or sampling problems.</description><pubDate>Fri, 24 Jan 2025 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;I recently built &lt;a href=&quot;https://github.com/ggerganov/llama.cpp&quot;&gt;llama.cpp&lt;/a&gt; on an IBM i system with mixed success. In this article, I&apos;ll detail the build process and challenges I encountered. While building llama.cpp itself was straightforward, I faced several issues during the inference phase that I&apos;ll explain.&lt;/p&gt;
&lt;p&gt;I&apos;ll start by outlining the steps to build llama.cpp. Then I&apos;ll describe the process of running an LLM, followed by the challenges I encountered. Finally, I&apos;ll reflect on and analyze this experiment—was it worthwhile? What lessons can we draw from it?&lt;/p&gt;
&lt;h2&gt;What is Llama.cpp&lt;/h2&gt;
&lt;p&gt;Llama.cpp is an open-source C++ implementation designed to run Large Language Models (LLMs) efficiently on multiple hardware architectures, completely local, with no third party dependencies. Llama.cpp is the backend for many popular open source AI projects such as Ollama and LMStudio. It is a key component of local LLM hosting and inference.&lt;/p&gt;
&lt;h2&gt;Steps to Build on IBM i&lt;/h2&gt;
&lt;p&gt;First install the required open source packages:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;yum install gcc12 openblas openblas-devel libutil git cmake
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Clone llama.cpp on ibm i&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
&lt;/code&gt;&lt;/pre&gt;
&lt;h3&gt;Make a small patch:&lt;/h3&gt;
&lt;p&gt;go to &lt;code&gt;llama.cpp/src/llama-mmap.cpp&lt;/code&gt; and comment out the following lines (476-481 on latest commit):&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://github.com/ggerganov/llama.cpp/blob/c5d9effb49649db80a52caf5c0626de6f342f526/src/llama-mmap.cpp#L476&quot;&gt;llama.cpp/src/llama-mmap.cpp at c5d9eff · ggerganov/llama.cpp&lt;/a&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;        // if (suggest &amp;amp;&amp;amp; getrlimit(RLIMIT_MEMLOCK, &amp;amp;lock_limit)) {
        //     suggest = false;
        // }
        // if (suggest &amp;amp;&amp;amp; (lock_limit.rlim_max &amp;gt; lock_limit.rlim_cur + size)) {
        //     suggest = false;
        // }
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This is because &lt;code&gt;RLIMIT_MEMLOCK&lt;/code&gt; is not supported on IBM i&lt;/p&gt;
&lt;h3&gt;OpenBLAS&lt;/h3&gt;
&lt;p&gt;set environment variable for cmake:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;export CC=/QOpenSys/pkgs/bin/gcc-12
export CXX=/QOpenSys/pkgs/bin/g++-12
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Run Build:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;cmake -B build -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS -DCMAKE_CXX_FLAGS=&quot;-pthread&quot; -DCMAKE_EXE_LINKER_FLAGS=&quot;-pthread -lutil&quot;
cmake --build build --config Release -- VERBOSE=1 -j 8
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Once complete, multiple executables will be available in &lt;code&gt;llama.cpp/build/bin/*&lt;/code&gt; or wherever you specified the build location.&lt;/p&gt;
&lt;h2&gt;Prepare models for inference&lt;/h2&gt;
&lt;p&gt;We can download model weights from huggingface. For this example I tried 2 lightweight models&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;gpt2: &lt;a href=&quot;https://huggingface.co/openai-community/gpt2?language=python&quot;&gt;openai-community/gpt2 · Hugging Face&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;TinyLlama: &lt;a href=&quot;https://huggingface.co/Maykeye/TinyLLama-v0?language=python&quot;&gt;Maykeye/TinyLLama-v0 · Hugging Face&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;These models are small, which makes them good for quick testing.&lt;/p&gt;
&lt;h3&gt;Download models from huggingface&lt;/h3&gt;
&lt;blockquote&gt;
&lt;p&gt;the following steps should be done on your local device (not IBM i)&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Install &lt;code&gt;huggingface_hub&lt;/code&gt; and use following script to download&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;pip install huggingface_hub 
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Create directory&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;mkdir hf-models
cd hf-models
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Create Python script&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;import argparse
from huggingface_hub import snapshot_download

def parse_model_name(model_id):
    return model_id.split(&apos;/&apos;)[-1]

def main():
    parser = argparse.ArgumentParser(description=&quot;Download model from Hugging Face Hub&quot;)
    parser.add_argument(&apos;model_id&apos;, type=str, help=&apos;The model ID from Hugging Face Hub&apos;)
    args = parser.parse_args()

    model_id = args.model_id
    local_dir = parse_model_name(model_id)
    
    snapshot_download(repo_id=model_id, local_dir=f&apos;models/{local_dir}&apos;, local_dir_use_symlinks=False, revision=&quot;main&quot;)

if __name__ == &quot;__main__&quot;:
    main()
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Run the script to download the models:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;python download.py Maykeye/TinyLLama-v0
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This will download each model into its own directory:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;ls -lash models/TinyLlama-v0
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Directory structure after downloading:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ tree .
.
├── download.py
└── models
    ├── TinyLLama-v0
    │   ├── README.md
    │   ├── backup
    │   │   ├── do_backup.py
    │   │   └── step-1-10300.bin
    │   ├── config.json
    │   ├── demo.py
    │   ├── generation_config.json
    │   ├── model.safetensors
    │   ├── pytorch_model.bin
    │   ├── special_tokens_map.json
    │   ├── tokenizer.json
    │   ├── tokenizer.model
    │   ├── tokenizer_config.json
    │   ├── train.ipynb
    │   └── valid.
&lt;/code&gt;&lt;/pre&gt;
&lt;h3&gt;Convert Models to GGUF&lt;/h3&gt;
&lt;p&gt;Now it&apos;s time to convert the downloaded HuggingFace model to a GGUF model.&lt;br /&gt;Llama.cpp comes with a converter script to do this.&lt;/p&gt;
&lt;p&gt;Get the script by cloning the llama.cpp repo:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;git clone https://github.com/ggerganov/llama.cpp.git
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Install the required python libraries:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;pip install -r llama.cpp/requirements.txt
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Verify the script is there and understand the various options:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;python llama.cpp/convert_hf_to_gguf.py -h
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Convert the HF model to GGUF model:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;cd hf-models
python llama.cpp/convert_hf_to_gguf.py \
    models/TinyLLama-v0 \
    --outfile TinyLLama-v0.gguf \
    --bigendian \
    --outtype f16
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;In this case we&apos;re also quantizing the model using default &lt;code&gt;f16&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Verify the GGUF model was created:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;ls -lash TinyLLama-v0.gguf
&lt;/code&gt;&lt;/pre&gt;
&lt;h3&gt;Transfer GGUF model to IBM i&lt;/h3&gt;
&lt;p&gt;You can use &lt;code&gt;scp&lt;/code&gt; to transfer the model to IBM i host:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;scp TinyLLama-v0.gguf USER@pHOST:~/models  
&lt;/code&gt;&lt;/pre&gt;
&lt;h2&gt;Run Model&lt;/h2&gt;
&lt;p&gt;Back on IBM i, you can then run the `llama-cli` to test the model&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;./llama.cpp/build/bin/llama-cli -m models/TinyLlama-v0.gguf --prompt &quot;once upon a time,&quot; -n 20
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Here is the output&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;build: 4537 (f7fb43cd) with gcc-10 (IBM i) 10.5.0 20230707 (IBM 10.5.0-2) for powerpc-ibm-aix6.1.0.0
main: llama backend init
main: load the model and apply lora adapter, if any
llama_model_loader: loaded meta data with 26 key-value pairs and 75 tensors from models/TinyLlama-v0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = TinyLlama v0
llama_model_loader: - kv   3:                            general.version str              = v0
llama_model_loader: - kv   4:                           general.basename str              = TinyLlama
llama_model_loader: - kv   5:                         general.size_label str              = 4.6M
llama_model_loader: - kv   6:                            general.license str              = apache-2.0
llama_model_loader: - kv   7:                          llama.block_count u32              = 8
llama_model_loader: - kv   8:                       llama.context_length u32              = 2048
llama_model_loader: - kv   9:                     llama.embedding_length u32              = 64
llama_model_loader: - kv  10:                  llama.feed_forward_length u32              = 256
llama_model_loader: - kv  11:                 llama.attention.head_count u32              = 16
llama_model_loader: - kv  12:     llama.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  13:                          general.file_type u32              = 1
llama_model_loader: - kv  14:                           llama.vocab_size u32              = 32000
llama_model_loader: - kv  15:                 llama.rope.dimension_count u32              = 4
llama_model_loader: - kv  16:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  17:                         tokenizer.ggml.pre str              = default
llama_model_loader: - kv  18:                      tokenizer.ggml.tokens arr[str,32000]   = [&quot;&amp;lt;unk&amp;gt;&quot;, &quot;&amp;lt;s&amp;gt;&quot;, &quot;&amp;lt;/s&amp;gt;&quot;, &quot;&amp;lt;0x00&amp;gt;&quot;, &quot;&amp;lt;...
llama_model_loader: - kv  19:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  20:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv  21:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  22:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  23:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  24:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  25:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   17 tensors
llama_model_loader: - type  f16:   58 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = F16
print_info: file size   = 8.82 MiB (16.00 BPW) 
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: special tokens cache size = 3
load: token to piece cache size = 0.1914 MB
print_info: arch             = llama
print_info: vocab_only       = 0
print_info: n_ctx_train      = 2048
print_info: n_embd           = 64
print_info: n_layer          = 8
print_info: n_head           = 16
print_info: n_head_kv        = 16
print_info: n_rot            = 4
print_info: n_swa            = 0
print_info: n_embd_head_k    = 4
print_info: n_embd_head_v    = 4
print_info: n_gqa            = 1
print_info: n_embd_k_gqa     = 64
print_info: n_embd_v_gqa     = 64
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-06
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: n_ff             = 256
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 0
print_info: rope scaling     = linear
print_info: freq_base_train  = 10000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 2048
print_info: rope_finetuned   = unknown
print_info: ssm_d_conv       = 0
print_info: ssm_d_inner      = 0
print_info: ssm_d_state      = 0
print_info: ssm_dt_rank      = 0
print_info: ssm_dt_b_c_rms   = 0
print_info: model type       = ?B
print_info: model params     = 4.62 M
print_info: general.name     = TinyLlama v0
print_info: vocab type       = SPM
print_info: n_vocab          = 32000
print_info: n_merges         = 0
print_info: BOS token        = 1 &apos;&amp;lt;s&amp;gt;&apos;
print_info: EOS token        = 2 &apos;&amp;lt;/s&amp;gt;&apos;
print_info: UNK token        = 0 &apos;&amp;lt;unk&amp;gt;&apos;
print_info: PAD token        = 0 &apos;&amp;lt;unk&amp;gt;&apos;
print_info: LF token         = 13 &apos;&amp;lt;0x0A&amp;gt;&apos;
print_info: EOG token        = 2 &apos;&amp;lt;/s&amp;gt;&apos;
print_info: max token length = 18
load_tensors:   CPU_Mapped model buffer size =     8.82 MiB
llama_init_from_model: n_seq_max     = 1
llama_init_from_model: n_ctx         = 4096
llama_init_from_model: n_ctx_per_seq = 4096
llama_init_from_model: n_batch       = 2048
llama_init_from_model: n_ubatch      = 512
llama_init_from_model: flash_attn    = 0
llama_init_from_model: freq_base     = 10000.0
llama_init_from_model: freq_scale    = 1
llama_init_from_model: n_ctx_pre_seq (4096) &amp;gt; n_ctx_train (2048) -- possible training context overflow
llama_kv_cache_init: kv_size = 4096, offload = 1, type_k = &apos;f16&apos;, type_v = &apos;f16&apos;, n_layer = 8, can_shift = 1
llama_kv_cache_init:        CPU KV buffer size =     8.00 MiB
llama_init_from_model: KV self size  =    8.00 MiB, K (f16):    4.00 MiB, V (f16):    4.00 MiB
llama_init_from_model:        CPU  output buffer size =     0.12 MiB
llama_init_from_model:        CPU compute buffer size =   136.51 MiB
llama_init_from_model: graph nodes  = 262
llama_init_from_model: graph splits = 114 (with bs=512), 1 (with bs=1)
common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
main: llama threadpool init, n_threads = 4
main: model was trained on only 2048 context tokens (4096 specified)

system_info: n_threads = 4 (n_threads_batch = 4) / 8 | CPU : LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 | 

sampler seed: 1830645824
sampler params: 
        repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
        dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 4096
        top_k = 40, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, temp = 0.800
        mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampler chain: logits -&amp;gt; logit-bias -&amp;gt; penalties -&amp;gt; dry -&amp;gt; top-k -&amp;gt; typical -&amp;gt; top-p -&amp;gt; min-p -&amp;gt; xtc -&amp;gt; temp-ext -&amp;gt; dist 
generate: n_ctx = 4096, n_batch = 2048, n_predict = 20, n_keep = 1

 once upon a time,####################

llama_perf_sampler_print:    sampling time =       0.80 ms /    26 runs   (    0.03 ms per token, 32459.43 tokens per second)
llama_perf_context_print:        load time =     123.39 ms
llama_perf_context_print: prompt eval time =      12.79 ms /     6 tokens (    2.13 ms per token,   469.01 tokens per second)
llama_perf_context_print:        eval time =     160.09 ms /    19 runs   (    8.43 ms per token,   118.69 tokens per second)
llama_perf_context_print:       total time =     179.67 ms /    25 tokens
&lt;/code&gt;&lt;/pre&gt;
&lt;h3&gt;Outputs&lt;/h3&gt;
&lt;p&gt;Now you might notice that the model results are not very useful:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;once upon a time,####################
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;mainly, the&lt;code&gt;#&lt;/code&gt;token is repeatedly sampled. I tried multiple different parameter combinations—adjusting the repetition penalty, temperature, and sampling parameters—but I could not get the model to generate anything besides repeated &apos;#&apos;.&lt;/p&gt;
&lt;h4&gt;Test llama-server&lt;/h4&gt;
&lt;p&gt;To investigate I started the &lt;code&gt;llama-server&lt;/code&gt; to test the model&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;./llama.cpp/build/bin/llama-server -m models/TinyLlama-v0.gguf 
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;I sent the following request to the server:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;curl --request POST \
     --url http://localhost:8080/completion \
     --header &quot;Content-Type: application/json&quot; \
     --data &apos;{
         &quot;prompt&quot;: &quot;once upon a time,&quot;,
         &quot;n_predict&quot;: 3,
         &quot;n_probs&quot;: 3
     }&apos;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;with the &lt;code&gt;n_probs&lt;/code&gt;  set to 3, the response will contain the probabilities of the top 3 tokens for each generated token given in the sampling. Here was part of the response from the server&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;{
  &quot;index&quot;: 0,
  &quot;content&quot;: &quot;###&quot;,
  ...,
  &quot;completion_probabilities&quot;: [
    {
      &quot;id&quot;: 38,
      &quot;token&quot;: &quot;#&quot;,
      &quot;bytes&quot;: [35],
      &quot;logprob&quot;: null,
      &quot;top_logprobs&quot;: [
        {
          &quot;id&quot;: 21328,
          &quot;token&quot;: &quot;anticipate&quot;,
          &quot;bytes&quot;: [97, 110, 116, 105, 99, 105, 112, 97, 116, 101],
          &quot;logprob&quot;: null
        },
        {
          &quot;id&quot;: 21343,
          &quot;token&quot;: &quot;Erin&quot;,
          &quot;bytes&quot;: [69, 114, 105, 110],
          &quot;logprob&quot;: null
        },
        {
          &quot;id&quot;: 21342,
          &quot;token&quot;: &quot;ochemistry&quot;,
          &quot;bytes&quot;: [111, 99, 104, 101, 109, 105, 115, 116, 114, 121],
          &quot;logprob&quot;: null
        }
      ]
    },
    {
      &quot;id&quot;: 38,
      &quot;token&quot;: &quot;#&quot;,
      &quot;bytes&quot;: [35],
      &quot;logprob&quot;: null,
      &quot;top_logprobs&quot;: [
        {
          &quot;id&quot;: 21328,
          &quot;token&quot;: &quot;anticipate&quot;,
          &quot;bytes&quot;: [97, 110, 116, 105, 99, 105, 112, 97, 116, 101],
          &quot;logprob&quot;: null
        },
        {
          &quot;id&quot;: 21343,
          &quot;token&quot;: &quot;Erin&quot;,
          &quot;bytes&quot;: [69, 114, 105, 110],
          &quot;logprob&quot;: null
        },
        {
          &quot;id&quot;: 21342,
          &quot;token&quot;: &quot;ochemistry&quot;,
          &quot;bytes&quot;: [111, 99, 104, 101, 109, 105, 115, 116, 114, 121],
          &quot;logprob&quot;: null
        }
      ]
    },
    {
      &quot;id&quot;: 38,
      &quot;token&quot;: &quot;#&quot;,
      &quot;bytes&quot;: [35],
      &quot;logprob&quot;: null,
      &quot;top_logprobs&quot;: [
        {
          &quot;id&quot;: 21328,
          &quot;token&quot;: &quot;anticipate&quot;,
          &quot;bytes&quot;: [97, 110, 116, 105, 99, 105, 112, 97, 116, 101],
          &quot;logprob&quot;: null
        },
        {
          &quot;id&quot;: 21343,
          &quot;token&quot;: &quot;Erin&quot;,
          &quot;bytes&quot;: [69, 114, 105, 110],
          &quot;logprob&quot;: null
        },
        {
          &quot;id&quot;: 21342,
          &quot;token&quot;: &quot;ochemistry&quot;,
          &quot;bytes&quot;: [111, 99, 104, 101, 109, 105, 115, 116, 114, 121],
          &quot;logprob&quot;: null
        }
      ]
    }
  ]
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Observations:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;The ids are the same (38 for token #)&lt;/li&gt;
&lt;li&gt;the &lt;code&gt;top_logprobs&lt;/code&gt; are the same for each token&lt;/li&gt;
&lt;li&gt;the &lt;code&gt;logprob&lt;/code&gt; is null for all entries&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;After investigating this issue, it appears there&apos;s a problem with the logprob calculations. The issue likely stems from either a failure in the calculation process or an incompatibility between the GGUF model, IBM i, and the Llama.cpp build.&lt;/p&gt;
&lt;p&gt;Another possibility is the precision we used earlier (&lt;code&gt;f16&lt;/code&gt;) is not compatible. To test this, I ran the same conversion script as above, but using the f32 precision:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;cd hf-models
python llama.cpp/convert_hf_to_gguf.py \
    models/TinyLLama-v0 \
    --outfile TinyLLama-v0-f32.gguf \
    --bigendian \
    --outtype f32
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Then I transferred this model version to IBM i and ran the same server test. With some better luck:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;{
  &quot;index&quot;: 0,
  &quot;content&quot;: &quot;\u0019\u000f&quot;,
	...,
  &quot;completion_probabilities&quot;: [
    {
      &quot;id&quot;: 28,
      &quot;token&quot;: &quot;\u0019&quot;,
      &quot;bytes&quot;: [25],
      &quot;logprob&quot;: -10.373491287231445,
      &quot;top_logprobs&quot;: [
        {
          &quot;id&quot;: 21328,
          &quot;token&quot;: &quot;anticipate&quot;,
          &quot;bytes&quot;: [97, 110, 116, 105, 99, 105, 112, 97, 116, 101],
          &quot;logprob&quot;: -10.373491287231445
        },
        {
          &quot;id&quot;: 21343,
          &quot;token&quot;: &quot;Erin&quot;,
          &quot;bytes&quot;: [69, 114, 105, 110],
          &quot;logprob&quot;: -10.373491287231445
        },
        {
          &quot;id&quot;: 21342,
          &quot;token&quot;: &quot;ochemistry&quot;,
          &quot;bytes&quot;: [111, 99, 104, 101, 109, 105, 115, 116, 114, 121],
          &quot;logprob&quot;: -10.373491287231445
        }
      ]
    },
    {
      &quot;id&quot;: 18,
      &quot;token&quot;: &quot;\u000f&quot;,
      &quot;bytes&quot;: [15],
      &quot;logprob&quot;: -10.373491287231445,
      &quot;top_logprobs&quot;: [
        {
          &quot;id&quot;: 21328,
          &quot;token&quot;: &quot;anticipate&quot;,
          &quot;bytes&quot;: [97, 110, 116, 105, 99, 105, 112, 97, 116, 101],
          &quot;logprob&quot;: -10.373491287231445
        },
        {
          &quot;id&quot;: 21343,
          &quot;token&quot;: &quot;Erin&quot;,
          &quot;bytes&quot;: [69, 114, 105, 110],
          &quot;logprob&quot;: -10.373491287231445
        },
        {
          &quot;id&quot;: 21342,
          &quot;token&quot;: &quot;ochemistry&quot;,
          &quot;bytes&quot;: [111, 99, 104, 101, 109, 105, 115, 116, 114, 121],
          &quot;logprob&quot;: -10.373491287231445
        }
      ]
    },
    {
      &quot;id&quot;: 2,
      &quot;token&quot;: &quot;&quot;,
      &quot;bytes&quot;: [],
      &quot;logprob&quot;: -10.373491287231445,
      &quot;top_logprobs&quot;: [
        {
          &quot;id&quot;: 21328,
          &quot;token&quot;: &quot;anticipate&quot;,
          &quot;bytes&quot;: [97, 110, 116, 105, 99, 105, 112, 97, 116, 101],
          &quot;logprob&quot;: -10.373491287231445
        },
        {
          &quot;id&quot;: 21343,
          &quot;token&quot;: &quot;Erin&quot;,
          &quot;bytes&quot;: [69, 114, 105, 110],
          &quot;logprob&quot;: -10.373491287231445
        },
        {
          &quot;id&quot;: 21342,
          &quot;token&quot;: &quot;ochemistry&quot;,
          &quot;bytes&quot;: [111, 99, 104, 101, 109, 105, 115, 116, 114, 121],
          &quot;logprob&quot;: -10.373491287231445
        }
      ]
    }
  ]
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;There are our probabilities! But they are the same value… -10.373491287231445&lt;/p&gt;
&lt;p&gt;lets try with &lt;code&gt;n_probs 10&lt;/code&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;{
  &quot;index&quot;: 0,
  &quot;content&quot;: &quot;\u0018&quot;,
  &quot;completion_probabilities&quot;: [
    {
      &quot;id&quot;: 27,
      &quot;token&quot;: &quot;\u0018&quot;,
      &quot;bytes&quot;: [24],
      &quot;logprob&quot;: -10.373491287231445,
      &quot;top_logprobs&quot;: [
        {
          &quot;id&quot;: 21328,
          &quot;token&quot;: &quot;anticipate&quot;,
          &quot;bytes&quot;: [97, 110, 116, 105, 99, 105, 112, 97, 116, 101],
          &quot;logprob&quot;: -10.373491287231445
        },
        {
          &quot;id&quot;: 21343,
          &quot;token&quot;: &quot;Erin&quot;,
          &quot;bytes&quot;: [69, 114, 105, 110],
          &quot;logprob&quot;: -10.373491287231445
        },
        {
          &quot;id&quot;: 21342,
          &quot;token&quot;: &quot;ochemistry&quot;,
          &quot;bytes&quot;: [111, 99, 104, 101, 109, 105, 115, 116, 114, 121],
          &quot;logprob&quot;: -10.373491287231445
        },
        {
          &quot;id&quot;: 21341,
          &quot;token&quot;: &quot;Azerbai&quot;,
          &quot;bytes&quot;: [65, 122, 101, 114, 98, 97, 105],
          &quot;logprob&quot;: -10.373491287231445
        },
        {
          &quot;id&quot;: 21340,
          &quot;token&quot;: &quot;Effective&quot;,
          &quot;bytes&quot;: [69, 102, 102, 101, 99, 116, 105, 118, 101],
          &quot;logprob&quot;: -10.373491287231445
        },
        {
          &quot;id&quot;: 21339,
          &quot;token&quot;: &quot;Cay&quot;,
          &quot;bytes&quot;: [67, 97, 121],
          &quot;logprob&quot;: -10.373491287231445
        },
        {
          &quot;id&quot;: 21338,
          &quot;token&quot;: &quot;Electronics&quot;,
          &quot;bytes&quot;: [69, 108, 101, 99, 116, 114, 111, 110, 105, 99, 115],
          &quot;logprob&quot;: -10.373491287231445
        },
        {
          &quot;id&quot;: 21337,
          &quot;token&quot;: &quot;canal&quot;,
          &quot;bytes&quot;: [99, 97, 110, 97, 108],
          &quot;logprob&quot;: -10.373491287231445
        },
        {
          &quot;id&quot;: 21336,
          &quot;token&quot;: &quot;glucose&quot;,
          &quot;bytes&quot;: [103, 108, 117, 99, 111, 115, 101],
          &quot;logprob&quot;: -10.373491287231445
        },
        {
          &quot;id&quot;: 21335,
          &quot;token&quot;: &quot;pixels&quot;,
          &quot;bytes&quot;: [112, 105, 120, 101, 108, 115],
          &quot;logprob&quot;: -10.373491287231445
        }
      ]
    }
  ]
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;All the same &lt;code&gt;logprob&lt;/code&gt; !!&lt;/p&gt;
&lt;h3&gt;What could be going on&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Flat or Malformed Logits&lt;/strong&gt;: The raw scores (logits) before applying softmax might be identical or nearly identical, leading to uniform probabilities.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Endianness Issues&lt;/strong&gt;: On IBM i&apos;s big-endian architecture, improperly handled floating-point values could corrupt logits.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Improper Model Conversion&lt;/strong&gt;: Errors during model conversion (e.g., &lt;code&gt;f16&lt;/code&gt; to &lt;code&gt;f32&lt;/code&gt;) might corrupt weights, resulting in invalid logits.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Softmax or Sampling Bug&lt;/strong&gt;: Faulty implementation of softmax or token sampling could produce uniform probabilities regardless of logits.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Given the uniform distribution, It appears that the model is selecting next tokens randomly, as shown in this demo&lt;/p&gt;
&lt;p&gt;&lt;video src=&quot;https://adamshedivy.com/blog/running-llms-on-ibm-i-experiment/random-token-demo.mp4&quot; controls muted playsinline&gt;&lt;/video&gt;&lt;/p&gt;
&lt;h4&gt;Expected output on Mac&lt;/h4&gt;
&lt;p&gt;I ran the same model with llama.cpp on my mac and the results are much better:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;{
  &quot;index&quot;: 0,
  &quot;content&quot;: &quot; there&quot;,
	...,
  &quot;completion_probabilities&quot;: [
    {
      &quot;id&quot;: 635,
      &quot;token&quot;: &quot; there&quot;,
      &quot;bytes&quot;: [32, 116, 104, 101, 114, 101],
      &quot;logprob&quot;: -0.22087286412715912,
      &quot;top_logprobs&quot;: [
        {
          &quot;id&quot;: 635,
          &quot;token&quot;: &quot;there&quot;,
          &quot;bytes&quot;: [116, 104, 101, 114, 101],
          &quot;logprob&quot;: -0.22087286412715912
        },
        {
          &quot;id&quot;: 260,
          &quot;token&quot;: &quot;a&quot;,
          &quot;bytes&quot;: [97],
          &quot;logprob&quot;: -2.958108901977539
        },
        {
          &quot;id&quot;: 288,
          &quot;token&quot;: &quot;in&quot;,
          &quot;bytes&quot;: [105, 110],
          &quot;logprob&quot;: -2.986186981201172
        },
        {
          &quot;id&quot;: 266,
          &quot;token&quot;: &quot;the&quot;,
          &quot;bytes&quot;: [116, 104, 101],
          &quot;logprob&quot;: -5.346122741699219
        },
        {
          &quot;id&quot;: 357,
          &quot;token&quot;: &quot;it&quot;,
          &quot;bytes&quot;: [105, 116],
          &quot;logprob&quot;: -5.935047149658203
        },
        {
          &quot;id&quot;: 1419,
          &quot;token&quot;: &quot;something&quot;,
          &quot;bytes&quot;: [115, 111, 109, 101, 116, 104, 105, 110, 103],
          &quot;logprob&quot;: -6.294443130493164
        },
        {
          &quot;id&quot;: 31844,
          &quot;token&quot;: &quot;,&quot;,
          &quot;bytes&quot;: [44],
          &quot;logprob&quot;: -6.348053932189941
        },
        {
          &quot;id&quot;: 350,
          &quot;token&quot;: &quot;he&quot;,
          &quot;bytes&quot;: [104, 101],
          &quot;logprob&quot;: -6.7392497062683105
        },
        {
          &quot;id&quot;: 328,
          &quot;token&quot;: &quot;on&quot;,
          &quot;bytes&quot;: [111, 110],
          &quot;logprob&quot;: -6.799753665924072
        },
        {
          &quot;id&quot;: 484,
          &quot;token&quot;: &quot;all&quot;,
          &quot;bytes&quot;: [97, 108, 108],
          &quot;logprob&quot;: -7.111557960510254
        }
      ]
    }
  ]
}

&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Here is the full response on Mac:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt; once upon a time, there was a little girl named Lily. She was very alert and loved to play outside. One day, she saw a big box in the park. It was very tall and had many colors. She wanted to see what was inside.
Lily asked her mom for help. Her mom said, &quot;We can help the box open, but you can&apos;t go far.&quot; Lily was happy and took a nap. She ran to the box and picked it up. The box was very soft and pretty.
When Lily opened the box, she saw a big, mean bear. The bear had a long tail. Lily said, &quot;Don&apos;t worry, dog. I will help you.&quot; The bear took the ball back to Lily and said, &quot;Thank you for being so brave.&quot; 
Lily was happy. She thanked the bear and thanked the bear. From that day on, Lily and the bear became best friends. They played together and became the best of friends.
&lt;/code&gt;&lt;/pre&gt;
&lt;h2&gt;Summary&lt;/h2&gt;
&lt;p&gt;Here are the key findings from running llama.cpp on IBM i:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Successfully built llama.cpp on IBM i system with minimal modifications&lt;/li&gt;
&lt;li&gt;Faced challenges with model inference, specifically in logprob calculations&lt;/li&gt;
&lt;li&gt;Tested both f16 and f32 precision models, with f32 achieving limited success&lt;/li&gt;
&lt;li&gt;Several potential root causes: endianness issues, model conversion errors, and sampling implementation bugs&lt;/li&gt;
&lt;li&gt;Proved that while LLM deployment is feasible on IBM i, substantial technical challenges persist for usability and actual use cases&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;Sources&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href=&quot;https://www.substratus.ai/blog/converting-hf-model-gguf-model/&quot;&gt;Converting HuggingFace Models to GGUF/GGML | Substratus.AI&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
</content:encoded></item><item><title>Connect Granite 3.0 running on Power to VSCode</title><link>https://adamshedivy.com/blog/granite-on-power-in-vscode/</link><guid isPermaLink="true">https://adamshedivy.com/blog/granite-on-power-in-vscode/</guid><description>Run IBM Granite 3.0 on Power10 with Ollama and use it as a code assistant in VS Code through Continue.</description><pubDate>Fri, 17 Jan 2025 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;IBM Granite 3.0 is the latest evolution of large language models (LLMs) designed specifically for enterprise use. In this short blog, I want to demonstrate how to deploy the new Granite 3.0 models on IBM Power and how to connect them to your VSCode workspace locally.&lt;/p&gt;
&lt;h2&gt;Granite 3.0 on P10&lt;/h2&gt;
&lt;p&gt;Shout out to &lt;a href=&quot;https://www.linkedin.com/in/marvingiessing/&quot;&gt;Marvin Gießing&lt;/a&gt; and his team for helping me get granite3 deployed on Power!&lt;/p&gt;
&lt;p&gt;To get started with &lt;code&gt;granite3&lt;/code&gt; on Power, all you need is a P9/P10 system with podman. We can leverage Ollama which has already adopted the &lt;code&gt;granite3&lt;/code&gt; family of models into their library! Here is a step-by-step guide to run the models on Power:&lt;/p&gt;
&lt;h3&gt;1. start the ollama container&lt;/h3&gt;
&lt;pre&gt;&lt;code&gt;podman run -d --privileged \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  quay.io/mgiessing/ollama:v0.3.14
&lt;/code&gt;&lt;/pre&gt;
&lt;h3&gt;2. Run the granite3-dense2b model&lt;/h3&gt;
&lt;pre&gt;&lt;code&gt;podman exec -it ollama ollama run granite3-dense:2b
&lt;/code&gt;&lt;/pre&gt;
&lt;h3&gt;3. Use all available threads on your P10 (I have 8cores@SMT2 =&amp;gt; 16 threads)&lt;/h3&gt;
&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; /set parameter num_thread 16
&lt;/code&gt;&lt;/pre&gt;
&lt;h3&gt;4. Ask the model something!&lt;/h3&gt;
&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; Write a 300 word blurb about IBM Power10
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;watch it in action:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://adamshedivy.com/blog/granite-on-power-in-vscode/granite-p10-terminal-demo.gif&quot; alt=&quot;Terminal recording of granite3-dense:2b running in Ollama on a Power10 system, generating a blurb about IBM Power10&quot; /&gt;&lt;/p&gt;
&lt;p&gt;Check out that Power10 performance&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;total duration:       8.551203323s
load duration:        9.001096ms
prompt eval count:    23 token(s)
prompt eval duration: 208.571ms
prompt eval rate:     110.27 tokens/s
eval count:           333 token(s)
eval duration:        8.287601s
eval rate:            40.18 tokens/s
&lt;/code&gt;&lt;/pre&gt;
&lt;h2&gt;Continue&lt;/h2&gt;
&lt;p&gt;Now that we have the Granite 3.0 models running on Power, lets use the Ollama API to use the models from inside VSCode using the Continue extension.&lt;/p&gt;
&lt;p&gt;Continue is an open-source AI-powered code assistant designed to integrate seamlessly with your IDE, such as VSCode or JetBrains. It helps developers by offering features like code autocompletion, chat-based code referencing, and natural language code rewriting. Continue supports various AI models, making it adaptable to different programming needs, and it&apos;s designed to accelerate development by keeping developers in their flow without interruptions. You can customize it to fit specific use cases, making it a valuable tool for enhancing productivity in software development.&lt;/p&gt;
&lt;h3&gt;Install Continue&lt;/h3&gt;
&lt;p&gt;Follow these instructions to download the VSCode extension:&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://docs.continue.dev/getting-started/install#vs-code&quot;&gt;Install | Continue&lt;/a&gt;&lt;/p&gt;
&lt;h3&gt;Configure&lt;/h3&gt;
&lt;p&gt;Next, we need to add our configuration for the &lt;code&gt;granite3&lt;/code&gt; model. Continue works seamlessly with many different LLM providers and includes built-in support for Ollama!&lt;/p&gt;
&lt;p&gt;If you&apos;re using the extension for the first time, you&apos;ll see a helpful walk-through that guides you through setting up your first model.&lt;/p&gt;
&lt;p&gt;To access the configuration settings, open the command pallet in VSCode (cmd+shift+P):&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://adamshedivy.com/blog/granite-on-power-in-vscode/open-continue-config.png&quot; alt=&quot;The VS Code command palette filtered to &amp;quot;continue:&amp;quot; with Continue: Open Continue Config highlighted&quot; /&gt;&lt;/p&gt;
&lt;p&gt;This opens the &lt;code&gt;~/.continue/config.json&lt;/code&gt; file in your editor. In the &lt;code&gt;models&lt;/code&gt; section, add the configuration for Granite running on your P10 instance:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;&quot;models&quot;: [
   {
     &quot;title&quot;: &quot;Power10 - ollama:granite3-dense:2b&quot;,
     &quot;provider&quot;: &quot;ollama&quot;,
     &quot;apiBase&quot;: &quot;http://HOST:11434&quot;,
     &quot;model&quot;: &quot;granite3-dense:2b&quot;
   },
]
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;where &lt;code&gt;HOST&lt;/code&gt; is the IP of your system.&lt;/p&gt;
&lt;p&gt;Once you update your &lt;code&gt;config.json&lt;/code&gt;, save the file and then select the model &lt;code&gt;Power10 - ollama:granite3-dense:2b&lt;/code&gt; from the drop down in the chat interface. And that&apos;s it! You can now use the &lt;code&gt;granite3&lt;/code&gt; models deployed on Power right within your VSCode workspace&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://adamshedivy.com/blog/granite-on-power-in-vscode/power10-model-dropdown.png&quot; alt=&quot;The Continue chat model dropdown with Power10 - ollama:granite3-dense:2b selected&quot; /&gt;&lt;/p&gt;
&lt;h2&gt;Continue Features&lt;/h2&gt;
&lt;p&gt;Continue has a lot of great features out of the box, here are some highlights&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Code Explanation using Granite 3.0 on P10:&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://adamshedivy.com/blog/granite-on-power-in-vscode/code-explanation-demo.gif&quot; alt=&quot;Recording of Continue in VS Code using Granite 3.0 on Power10 to explain a block of code&quot; /&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Inline editing (cmd+I) using Granite 3.0 in P10:&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;&lt;video src=&quot;https://adamshedivy.com/blog/granite-on-power-in-vscode/inline-edit-demo.mp4&quot; controls muted playsinline&gt;&lt;/video&gt;&lt;/p&gt;
&lt;h2&gt;Future work&lt;/h2&gt;
&lt;p&gt;This should be enough to get you started on using Continue and Ollama running on Power10. In the future, I plan to release more tutorials and demos on using Continue for IBM Power specific use cases, such as:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;using AI for IBM i programming&lt;/li&gt;
&lt;li&gt;SQL code assistant and Db2 context provider&lt;/li&gt;
&lt;li&gt;RAG solutions for IBM Power&lt;/li&gt;
&lt;/ul&gt;
&lt;h4&gt;Links&lt;/h4&gt;
&lt;ul&gt;
&lt;li&gt;IBM release Blog: &lt;a href=&quot;https://www.ibm.com/new/ibm-granite-3-0-open-state-of-the-art-enterprise-models&quot;&gt;https://www.ibm.com/new/ibm-granite-3-0-open-state-of-the-art-enterprise-models&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;The models are open source and available on &lt;a href=&quot;https://huggingface.co/ibm-granite&quot;&gt;Hugging Face&lt;/a&gt;!&lt;/li&gt;
&lt;li&gt;ollama granite3 library: &lt;a href=&quot;https://ollama.com/blog/ibm-granite&quot;&gt;https://ollama.com/blog/ibm-granite&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Continue docs: &lt;a href=&quot;https://docs.continue.dev/&quot;&gt;https://docs.continue.dev/&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;original post: &lt;a href=&quot;https://github.com/ajshedivy/blog/issues/1&quot;&gt;https://github.com/ajshedivy/blog/issues/1&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
</content:encoded></item></channel></rss>