I recently built llama.cpp on an IBM i system with mixed success. In this article, I’ll detail the build process and challenges I encountered. While building llama.cpp itself was straightforward, I faced several issues during the inference phase that I’ll explain.
I’ll start by outlining the steps to build llama.cpp. Then I’ll describe the process of running an LLM, followed by the challenges I encountered. Finally, I’ll reflect on and analyze this experiment—was it worthwhile? What lessons can we draw from it?
What is Llama.cpp
Llama.cpp is an open-source C++ implementation designed to run Large Language Models (LLMs) efficiently on multiple hardware architectures, completely local, with no third party dependencies. Llama.cpp is the backend for many popular open source AI projects such as Ollama and LMStudio. It is a key component of local LLM hosting and inference.
Steps to Build on IBM i
First install the required open source packages:
yum install gcc12 openblas openblas-devel libutil git cmake
Clone llama.cpp on ibm i
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
Make a small patch:
go to llama.cpp/src/llama-mmap.cpp and comment out the following lines (476-481 on latest commit):
llama.cpp/src/llama-mmap.cpp at c5d9eff · ggerganov/llama.cpp
// if (suggest && getrlimit(RLIMIT_MEMLOCK, &lock_limit)) {
// suggest = false;
// }
// if (suggest && (lock_limit.rlim_max > lock_limit.rlim_cur + size)) {
// suggest = false;
// }
This is because RLIMIT_MEMLOCK is not supported on IBM i
OpenBLAS
set environment variable for cmake:
export CC=/QOpenSys/pkgs/bin/gcc-12
export CXX=/QOpenSys/pkgs/bin/g++-12
Run Build:
cmake -B build -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS -DCMAKE_CXX_FLAGS="-pthread" -DCMAKE_EXE_LINKER_FLAGS="-pthread -lutil"
cmake --build build --config Release -- VERBOSE=1 -j 8
Once complete, multiple executables will be available in llama.cpp/build/bin/* or wherever you specified the build location.
Prepare models for inference
We can download model weights from huggingface. For this example I tried 2 lightweight models
- gpt2: openai-community/gpt2 · Hugging Face
- TinyLlama: Maykeye/TinyLLama-v0 · Hugging Face
These models are small, which makes them good for quick testing.
Download models from huggingface
the following steps should be done on your local device (not IBM i)
Install huggingface_hub and use following script to download
pip install huggingface_hub
Create directory
mkdir hf-models
cd hf-models
Create Python script
import argparse
from huggingface_hub import snapshot_download
def parse_model_name(model_id):
return model_id.split('/')[-1]
def main():
parser = argparse.ArgumentParser(description="Download model from Hugging Face Hub")
parser.add_argument('model_id', type=str, help='The model ID from Hugging Face Hub')
args = parser.parse_args()
model_id = args.model_id
local_dir = parse_model_name(model_id)
snapshot_download(repo_id=model_id, local_dir=f'models/{local_dir}', local_dir_use_symlinks=False, revision="main")
if __name__ == "__main__":
main()
Run the script to download the models:
python download.py Maykeye/TinyLLama-v0
This will download each model into its own directory:
ls -lash models/TinyLlama-v0
Directory structure after downloading:
$ tree .
.
├── download.py
└── models
├── TinyLLama-v0
│ ├── README.md
│ ├── backup
│ │ ├── do_backup.py
│ │ └── step-1-10300.bin
│ ├── config.json
│ ├── demo.py
│ ├── generation_config.json
│ ├── model.safetensors
│ ├── pytorch_model.bin
│ ├── special_tokens_map.json
│ ├── tokenizer.json
│ ├── tokenizer.model
│ ├── tokenizer_config.json
│ ├── train.ipynb
│ └── valid.
Convert Models to GGUF
Now it’s time to convert the downloaded HuggingFace model to a GGUF model.
Llama.cpp comes with a converter script to do this.
Get the script by cloning the llama.cpp repo:
git clone https://github.com/ggerganov/llama.cpp.git
Install the required python libraries:
pip install -r llama.cpp/requirements.txt
Verify the script is there and understand the various options:
python llama.cpp/convert_hf_to_gguf.py -h
Convert the HF model to GGUF model:
cd hf-models
python llama.cpp/convert_hf_to_gguf.py \
models/TinyLLama-v0 \
--outfile TinyLLama-v0.gguf \
--bigendian \
--outtype f16
In this case we’re also quantizing the model using default f16.
Verify the GGUF model was created:
ls -lash TinyLLama-v0.gguf
Transfer GGUF model to IBM i
You can use scp to transfer the model to IBM i host:
scp TinyLLama-v0.gguf USER@pHOST:~/models
Run Model
Back on IBM i, you can then run the `llama-cli` to test the model
./llama.cpp/build/bin/llama-cli -m models/TinyLlama-v0.gguf --prompt "once upon a time," -n 20
Here is the output
build: 4537 (f7fb43cd) with gcc-10 (IBM i) 10.5.0 20230707 (IBM 10.5.0-2) for powerpc-ibm-aix6.1.0.0
main: llama backend init
main: load the model and apply lora adapter, if any
llama_model_loader: loaded meta data with 26 key-value pairs and 75 tensors from models/TinyLlama-v0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = TinyLlama v0
llama_model_loader: - kv 3: general.version str = v0
llama_model_loader: - kv 4: general.basename str = TinyLlama
llama_model_loader: - kv 5: general.size_label str = 4.6M
llama_model_loader: - kv 6: general.license str = apache-2.0
llama_model_loader: - kv 7: llama.block_count u32 = 8
llama_model_loader: - kv 8: llama.context_length u32 = 2048
llama_model_loader: - kv 9: llama.embedding_length u32 = 64
llama_model_loader: - kv 10: llama.feed_forward_length u32 = 256
llama_model_loader: - kv 11: llama.attention.head_count u32 = 16
llama_model_loader: - kv 12: llama.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 13: general.file_type u32 = 1
llama_model_loader: - kv 14: llama.vocab_size u32 = 32000
llama_model_loader: - kv 15: llama.rope.dimension_count u32 = 4
llama_model_loader: - kv 16: tokenizer.ggml.model str = llama
llama_model_loader: - kv 17: tokenizer.ggml.pre str = default
llama_model_loader: - kv 18: tokenizer.ggml.tokens arr[str,32000] = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv 19: tokenizer.ggml.scores arr[f32,32000] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 20: tokenizer.ggml.token_type arr[i32,32000] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv 21: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 22: tokenizer.ggml.eos_token_id u32 = 2
llama_model_loader: - kv 23: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 24: tokenizer.ggml.padding_token_id u32 = 0
llama_model_loader: - kv 25: general.quantization_version u32 = 2
llama_model_loader: - type f32: 17 tensors
llama_model_loader: - type f16: 58 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = F16
print_info: file size = 8.82 MiB (16.00 BPW)
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: special tokens cache size = 3
load: token to piece cache size = 0.1914 MB
print_info: arch = llama
print_info: vocab_only = 0
print_info: n_ctx_train = 2048
print_info: n_embd = 64
print_info: n_layer = 8
print_info: n_head = 16
print_info: n_head_kv = 16
print_info: n_rot = 4
print_info: n_swa = 0
print_info: n_embd_head_k = 4
print_info: n_embd_head_v = 4
print_info: n_gqa = 1
print_info: n_embd_k_gqa = 64
print_info: n_embd_v_gqa = 64
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-06
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: n_ff = 256
print_info: n_expert = 0
print_info: n_expert_used = 0
print_info: causal attn = 1
print_info: pooling type = 0
print_info: rope type = 0
print_info: rope scaling = linear
print_info: freq_base_train = 10000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn = 2048
print_info: rope_finetuned = unknown
print_info: ssm_d_conv = 0
print_info: ssm_d_inner = 0
print_info: ssm_d_state = 0
print_info: ssm_dt_rank = 0
print_info: ssm_dt_b_c_rms = 0
print_info: model type = ?B
print_info: model params = 4.62 M
print_info: general.name = TinyLlama v0
print_info: vocab type = SPM
print_info: n_vocab = 32000
print_info: n_merges = 0
print_info: BOS token = 1 '<s>'
print_info: EOS token = 2 '</s>'
print_info: UNK token = 0 '<unk>'
print_info: PAD token = 0 '<unk>'
print_info: LF token = 13 '<0x0A>'
print_info: EOG token = 2 '</s>'
print_info: max token length = 18
load_tensors: CPU_Mapped model buffer size = 8.82 MiB
llama_init_from_model: n_seq_max = 1
llama_init_from_model: n_ctx = 4096
llama_init_from_model: n_ctx_per_seq = 4096
llama_init_from_model: n_batch = 2048
llama_init_from_model: n_ubatch = 512
llama_init_from_model: flash_attn = 0
llama_init_from_model: freq_base = 10000.0
llama_init_from_model: freq_scale = 1
llama_init_from_model: n_ctx_pre_seq (4096) > n_ctx_train (2048) -- possible training context overflow
llama_kv_cache_init: kv_size = 4096, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 8, can_shift = 1
llama_kv_cache_init: CPU KV buffer size = 8.00 MiB
llama_init_from_model: KV self size = 8.00 MiB, K (f16): 4.00 MiB, V (f16): 4.00 MiB
llama_init_from_model: CPU output buffer size = 0.12 MiB
llama_init_from_model: CPU compute buffer size = 136.51 MiB
llama_init_from_model: graph nodes = 262
llama_init_from_model: graph splits = 114 (with bs=512), 1 (with bs=1)
common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
main: llama threadpool init, n_threads = 4
main: model was trained on only 2048 context tokens (4096 specified)
system_info: n_threads = 4 (n_threads_batch = 4) / 8 | CPU : LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 |
sampler seed: 1830645824
sampler params:
repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 4096
top_k = 40, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, temp = 0.800
mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampler chain: logits -> logit-bias -> penalties -> dry -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
generate: n_ctx = 4096, n_batch = 2048, n_predict = 20, n_keep = 1
once upon a time,####################
llama_perf_sampler_print: sampling time = 0.80 ms / 26 runs ( 0.03 ms per token, 32459.43 tokens per second)
llama_perf_context_print: load time = 123.39 ms
llama_perf_context_print: prompt eval time = 12.79 ms / 6 tokens ( 2.13 ms per token, 469.01 tokens per second)
llama_perf_context_print: eval time = 160.09 ms / 19 runs ( 8.43 ms per token, 118.69 tokens per second)
llama_perf_context_print: total time = 179.67 ms / 25 tokens
Outputs
Now you might notice that the model results are not very useful:
once upon a time,####################
mainly, the#token is repeatedly sampled. I tried multiple different parameter combinations—adjusting the repetition penalty, temperature, and sampling parameters—but I could not get the model to generate anything besides repeated ‘#’.
Test llama-server
To investigate I started the llama-server to test the model
./llama.cpp/build/bin/llama-server -m models/TinyLlama-v0.gguf
I sent the following request to the server:
curl --request POST \
--url http://localhost:8080/completion \
--header "Content-Type: application/json" \
--data '{
"prompt": "once upon a time,",
"n_predict": 3,
"n_probs": 3
}'
with the n_probs set to 3, the response will contain the probabilities of the top 3 tokens for each generated token given in the sampling. Here was part of the response from the server
{
"index": 0,
"content": "###",
...,
"completion_probabilities": [
{
"id": 38,
"token": "#",
"bytes": [35],
"logprob": null,
"top_logprobs": [
{
"id": 21328,
"token": "anticipate",
"bytes": [97, 110, 116, 105, 99, 105, 112, 97, 116, 101],
"logprob": null
},
{
"id": 21343,
"token": "Erin",
"bytes": [69, 114, 105, 110],
"logprob": null
},
{
"id": 21342,
"token": "ochemistry",
"bytes": [111, 99, 104, 101, 109, 105, 115, 116, 114, 121],
"logprob": null
}
]
},
{
"id": 38,
"token": "#",
"bytes": [35],
"logprob": null,
"top_logprobs": [
{
"id": 21328,
"token": "anticipate",
"bytes": [97, 110, 116, 105, 99, 105, 112, 97, 116, 101],
"logprob": null
},
{
"id": 21343,
"token": "Erin",
"bytes": [69, 114, 105, 110],
"logprob": null
},
{
"id": 21342,
"token": "ochemistry",
"bytes": [111, 99, 104, 101, 109, 105, 115, 116, 114, 121],
"logprob": null
}
]
},
{
"id": 38,
"token": "#",
"bytes": [35],
"logprob": null,
"top_logprobs": [
{
"id": 21328,
"token": "anticipate",
"bytes": [97, 110, 116, 105, 99, 105, 112, 97, 116, 101],
"logprob": null
},
{
"id": 21343,
"token": "Erin",
"bytes": [69, 114, 105, 110],
"logprob": null
},
{
"id": 21342,
"token": "ochemistry",
"bytes": [111, 99, 104, 101, 109, 105, 115, 116, 114, 121],
"logprob": null
}
]
}
]
}
Observations:
- The ids are the same (38 for token #)
- the
top_logprobsare the same for each token - the
logprobis null for all entries
After investigating this issue, it appears there’s a problem with the logprob calculations. The issue likely stems from either a failure in the calculation process or an incompatibility between the GGUF model, IBM i, and the Llama.cpp build.
Another possibility is the precision we used earlier (f16) is not compatible. To test this, I ran the same conversion script as above, but using the f32 precision:
cd hf-models
python llama.cpp/convert_hf_to_gguf.py \
models/TinyLLama-v0 \
--outfile TinyLLama-v0-f32.gguf \
--bigendian \
--outtype f32
Then I transferred this model version to IBM i and ran the same server test. With some better luck:
{
"index": 0,
"content": "\u0019\u000f",
...,
"completion_probabilities": [
{
"id": 28,
"token": "\u0019",
"bytes": [25],
"logprob": -10.373491287231445,
"top_logprobs": [
{
"id": 21328,
"token": "anticipate",
"bytes": [97, 110, 116, 105, 99, 105, 112, 97, 116, 101],
"logprob": -10.373491287231445
},
{
"id": 21343,
"token": "Erin",
"bytes": [69, 114, 105, 110],
"logprob": -10.373491287231445
},
{
"id": 21342,
"token": "ochemistry",
"bytes": [111, 99, 104, 101, 109, 105, 115, 116, 114, 121],
"logprob": -10.373491287231445
}
]
},
{
"id": 18,
"token": "\u000f",
"bytes": [15],
"logprob": -10.373491287231445,
"top_logprobs": [
{
"id": 21328,
"token": "anticipate",
"bytes": [97, 110, 116, 105, 99, 105, 112, 97, 116, 101],
"logprob": -10.373491287231445
},
{
"id": 21343,
"token": "Erin",
"bytes": [69, 114, 105, 110],
"logprob": -10.373491287231445
},
{
"id": 21342,
"token": "ochemistry",
"bytes": [111, 99, 104, 101, 109, 105, 115, 116, 114, 121],
"logprob": -10.373491287231445
}
]
},
{
"id": 2,
"token": "",
"bytes": [],
"logprob": -10.373491287231445,
"top_logprobs": [
{
"id": 21328,
"token": "anticipate",
"bytes": [97, 110, 116, 105, 99, 105, 112, 97, 116, 101],
"logprob": -10.373491287231445
},
{
"id": 21343,
"token": "Erin",
"bytes": [69, 114, 105, 110],
"logprob": -10.373491287231445
},
{
"id": 21342,
"token": "ochemistry",
"bytes": [111, 99, 104, 101, 109, 105, 115, 116, 114, 121],
"logprob": -10.373491287231445
}
]
}
]
}
There are our probabilities! But they are the same value… -10.373491287231445
lets try with n_probs 10
{
"index": 0,
"content": "\u0018",
"completion_probabilities": [
{
"id": 27,
"token": "\u0018",
"bytes": [24],
"logprob": -10.373491287231445,
"top_logprobs": [
{
"id": 21328,
"token": "anticipate",
"bytes": [97, 110, 116, 105, 99, 105, 112, 97, 116, 101],
"logprob": -10.373491287231445
},
{
"id": 21343,
"token": "Erin",
"bytes": [69, 114, 105, 110],
"logprob": -10.373491287231445
},
{
"id": 21342,
"token": "ochemistry",
"bytes": [111, 99, 104, 101, 109, 105, 115, 116, 114, 121],
"logprob": -10.373491287231445
},
{
"id": 21341,
"token": "Azerbai",
"bytes": [65, 122, 101, 114, 98, 97, 105],
"logprob": -10.373491287231445
},
{
"id": 21340,
"token": "Effective",
"bytes": [69, 102, 102, 101, 99, 116, 105, 118, 101],
"logprob": -10.373491287231445
},
{
"id": 21339,
"token": "Cay",
"bytes": [67, 97, 121],
"logprob": -10.373491287231445
},
{
"id": 21338,
"token": "Electronics",
"bytes": [69, 108, 101, 99, 116, 114, 111, 110, 105, 99, 115],
"logprob": -10.373491287231445
},
{
"id": 21337,
"token": "canal",
"bytes": [99, 97, 110, 97, 108],
"logprob": -10.373491287231445
},
{
"id": 21336,
"token": "glucose",
"bytes": [103, 108, 117, 99, 111, 115, 101],
"logprob": -10.373491287231445
},
{
"id": 21335,
"token": "pixels",
"bytes": [112, 105, 120, 101, 108, 115],
"logprob": -10.373491287231445
}
]
}
]
}
All the same logprob !!
What could be going on
- Flat or Malformed Logits: The raw scores (logits) before applying softmax might be identical or nearly identical, leading to uniform probabilities.
- Endianness Issues: On IBM i’s big-endian architecture, improperly handled floating-point values could corrupt logits.
- Improper Model Conversion: Errors during model conversion (e.g.,
f16tof32) might corrupt weights, resulting in invalid logits. - Softmax or Sampling Bug: Faulty implementation of softmax or token sampling could produce uniform probabilities regardless of logits.
Given the uniform distribution, It appears that the model is selecting next tokens randomly, as shown in this demo
Expected output on Mac
I ran the same model with llama.cpp on my mac and the results are much better:
{
"index": 0,
"content": " there",
...,
"completion_probabilities": [
{
"id": 635,
"token": " there",
"bytes": [32, 116, 104, 101, 114, 101],
"logprob": -0.22087286412715912,
"top_logprobs": [
{
"id": 635,
"token": "there",
"bytes": [116, 104, 101, 114, 101],
"logprob": -0.22087286412715912
},
{
"id": 260,
"token": "a",
"bytes": [97],
"logprob": -2.958108901977539
},
{
"id": 288,
"token": "in",
"bytes": [105, 110],
"logprob": -2.986186981201172
},
{
"id": 266,
"token": "the",
"bytes": [116, 104, 101],
"logprob": -5.346122741699219
},
{
"id": 357,
"token": "it",
"bytes": [105, 116],
"logprob": -5.935047149658203
},
{
"id": 1419,
"token": "something",
"bytes": [115, 111, 109, 101, 116, 104, 105, 110, 103],
"logprob": -6.294443130493164
},
{
"id": 31844,
"token": ",",
"bytes": [44],
"logprob": -6.348053932189941
},
{
"id": 350,
"token": "he",
"bytes": [104, 101],
"logprob": -6.7392497062683105
},
{
"id": 328,
"token": "on",
"bytes": [111, 110],
"logprob": -6.799753665924072
},
{
"id": 484,
"token": "all",
"bytes": [97, 108, 108],
"logprob": -7.111557960510254
}
]
}
]
}
Here is the full response on Mac:
once upon a time, there was a little girl named Lily. She was very alert and loved to play outside. One day, she saw a big box in the park. It was very tall and had many colors. She wanted to see what was inside.
Lily asked her mom for help. Her mom said, "We can help the box open, but you can't go far." Lily was happy and took a nap. She ran to the box and picked it up. The box was very soft and pretty.
When Lily opened the box, she saw a big, mean bear. The bear had a long tail. Lily said, "Don't worry, dog. I will help you." The bear took the ball back to Lily and said, "Thank you for being so brave."
Lily was happy. She thanked the bear and thanked the bear. From that day on, Lily and the bear became best friends. They played together and became the best of friends.
Summary
Here are the key findings from running llama.cpp on IBM i:
- Successfully built llama.cpp on IBM i system with minimal modifications
- Faced challenges with model inference, specifically in logprob calculations
- Tested both f16 and f32 precision models, with f32 achieving limited success
- Several potential root causes: endianness issues, model conversion errors, and sampling implementation bugs
- Proved that while LLM deployment is feasible on IBM i, substantial technical challenges persist for usability and actual use cases