RuntimeError: llama-server embedding request returned 500: {"error":{"code":500,"message":"input (596 tokens) is too large to process. increase the physical batch size (current batch size: 512)","type":"server_error"}}
Observed on the same docs set: chunks measured <=400 (cl100k) arriving as 750 tokens
(HTML page with a long navigation link list) and 596 tokens (Markdown with many long
URLs / inline code). Plain prose rarely triggers it, which makes the failure look
random across doc sets.
Embedding chunks via llama.cpp...
Embedding 5/216 chunks...Traceback (most recent call last):
File "/usr/bin/doc2rag", line 553, in
run_pipeline(args)
File "/usr/bin/doc2rag", line 526, in run_pipeline
store_in_qdrant(chunks, ids, doc_ids, chunk_indices, args.output, embedder, embedding_model=embedding_model)
File "/usr/bin/doc2rag", line 340, in store_in_qdrant
vectors = embedder.embed(chunks)
File "/usr/bin/doc2rag", line 298, in embed
all_embeddings.extend(self._embed_batch(batch))
File "/usr/bin/doc2rag", line 317, in _embed_batch
raise RuntimeError(f"llama-server embedding request returned {e.code}: {body}") from None
RuntimeError: llama-server embedding request returned 500: {"error":{"code":500,"message":"input (596 tokens) is too large to process. increase the physical batch size (current batch size: 512)","type":"server_error"}}
The whole build aborts (exit status 1); no image is produced.
The build completes. The chunker should never emit a chunk the embedding server
rejects. Possible fixes, in rough order of correctness:
a) doc2rag counts chunk tokens with the embedding model's own tokenizer instead of
cl100k_base (e.g. via the embedding llama-server's /tokenize endpoint, since the
server is already running during chunking);
b) or RamaLama starts the embedding llama-server with a physical batch size safely
above the effective chunk limit (e.g. -ub 2048);
c) or doc2rag splits/truncates any chunk that exceeds the server limit before
POSTing to /v1/embeddings, instead of aborting the whole build.
{
"Accelerator": "none",
"Config": {
"benchmarks": {},
"provider": {
"openai": {}
},
"runtimes": {
"llama_cpp": {},
"mlx": {}
},
"settings": {
"config_files": [
"/home/ramalama/infrapilot-venv/share/ramalama/ramalama.conf"
]
},
"user": {}
},
"Engine": {
"Info": {
"host": {
"arch": "amd64",
"buildahVersion": "1.33.7",
"cgroupControllers": [
"cpu",
"memory",
"pids"
],
"cgroupManager": "systemd",
"cgroupVersion": "v2",
"conmon": {
"package": "conmon_2.1.10+ds1-1build2_amd64",
"path": "/usr/bin/conmon",
"version": "conmon version 2.1.10, commit: unknown"
},
"cpuUtilization": {
"idlePercent": 90.55,
"systemPercent": 0.37,
"userPercent": 9.08
},
"cpus": 8,
"databaseBackend": "sqlite",
"distribution": {
"codename": "noble",
"distribution": "ubuntu",
"version": "24.04"
},
"eventLogger": "journald",
"freeLocks": 2043,
"hostname": "DESKTOP-RV5R09V",
"idMappings": {
"gidmap": [
{
"container_id": 0,
"host_id": 1000,
"size": 1
},
{
"container_id": 1,
"host_id": 100000,
"size": 65536
}
],
"uidmap": [
{
"container_id": 0,
"host_id": 1000,
"size": 1
},
{
"container_id": 1,
"host_id": 100000,
"size": 65536
}
]
},
"kernel": "6.6.87.2-microsoft-standard-WSL2",
"linkmode": "dynamic",
"logDriver": "journald",
"memFree": 114073600,
"memTotal": 12543287296,
"networkBackend": "netavark",
"networkBackendInfo": {
"backend": "netavark",
"dns": {
"package": "aardvark-dns_1.4.0-5_amd64",
"path": "/usr/lib/podman/aardvark-dns",
"version": "aardvark-dns 1.4.0"
},
"package": "netavark_1.4.0-4_amd64",
"path": "/usr/lib/podman/netavark",
"version": "netavark 1.4.0"
},
"ociRuntime": {
"name": "crun",
"package": "crun_1.14.1-1_amd64",
"path": "/usr/bin/crun",
"version": "crun version 1.14.1\ncommit: de537a7965bfbe9992e2cfae0baeb56a08128171\nrundir: /run/user/1000/crun\nspec: 1.0.0\n+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +WASM:wasmedge +YAJL"
},
"os": "linux",
"pasta": {
"executable": "/usr/bin/pasta",
"package": "passt_0.0~git20240220.1e6f92b-1_amd64",
"version": "pasta unknown version\nCopyright Red Hat\nGNU General Public License, version 2 or later\n <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>\nThis is free software: you are free to change and redistribute it.\nThere is NO WARRANTY, to the extent permitted by law.\n"
},
"remoteSocket": {
"exists": false,
"path": "/run/user/1000/podman/podman.sock"
},
"security": {
"apparmorEnabled": false,
"capabilities": "CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT",
"rootless": true,
"seccompEnabled": true,
"seccompProfilePath": "/usr/share/containers/seccomp.json",
"selinuxEnabled": false
},
"serviceIsRemote": false,
"slirp4netns": {
"executable": "/usr/bin/slirp4netns",
"package": "slirp4netns_1.2.1-1build2_amd64",
"version": "slirp4netns version 1.2.1\ncommit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194\nlibslirp: 4.7.0\nSLIRP_CONFIG_VERSION_MAX: 4\nlibseccomp: 2.5.5"
},
"swapFree": 3219103744,
"swapTotal": 3221225472,
"uptime": "1h 22m 43.00s (Approximately 0.04 days)",
"variant": ""
},
"plugins": {
"authorization": null,
"log": [
"k8s-file",
"none",
"passthrough",
"journald"
],
"network": [
"bridge",
"macvlan",
"ipvlan"
],
"volume": [
"local"
]
},
"registries": {},
"store": {
"configFile": "/home/ramalama/.config/containers/storage.conf",
"containerStore": {
"number": 4,
"paused": 0,
"running": 4,
"stopped": 0
},
"graphDriverName": "overlay",
"graphOptions": {},
"graphRoot": "/home/ramalama/.local/share/containers/storage",
"graphRootAllocated": 1081101176832,
"graphRootUsed": 14642786304,
"graphStatus": {
"Backing Filesystem": "extfs",
"Native Overlay Diff": "true",
"Supports d_type": "true",
"Supports shifting": "false",
"Supports volatile": "true",
"Using metacopy": "false"
},
"imageCopyTmpDir": "/var/tmp",
"imageStore": {
"number": 4
},
"runRoot": "/run/user/1000/containers",
"transientStore": false,
"volumePath": "/home/ramalama/.local/share/containers/storage/volumes"
},
"version": {
"APIVersion": "4.9.3",
"Built": 0,
"BuiltTime": "Thu Jan 1 01:00:00 1970",
"GitCommit": "",
"GoVersion": "go1.22.2",
"Os": "linux",
"OsArch": "linux/amd64",
"Version": "4.9.3"
}
},
"Name": "podman"
},
"Image": "quay.io/ramalama/ramalama:0.23",
"RagImage": "quay.io/ramalama/ramalama-rag:0.23",
"Runtimes": {
"Available": [
"llama.cpp",
"mlx",
"vllm"
],
"Default": "llama.cpp"
},
"Selinux": false,
"Shortnames": {
"Files": [
"/home/ramalama/infrapilot-venv/share/ramalama/shortnames.conf"
],
"Names": {
"cerebrum": "hf://bartowski/Cerebrum-1.0-7b-GGUF",
"codellama": "hf://TheBloke/CodeLlama-7B-Instruct-GGUF",
"codellama:13b": "hf://TheBloke/CodeLlama-13B-Instruct-GGUF",
"codellama:34b": "hf://TheBloke/CodeLlama-34B-Instruct-GGUF",
"codellama:7b": "hf://TheBloke/CodeLlama-7B-Instruct-GGUF",
"deepseek": "hf://bartowski/DeepSeek-R1-Distill-Qwen-7B-GGUF",
"deepseek-coder": "hf://TheBloke/deepseek-coder-6.7b-instruct-GGUF",
"deepseek-coder:33b": "hf://TheBloke/deepseek-coder-33b-instruct-GGUF",
"deepseek-coder:6.7b": "hf://TheBloke/deepseek-coder-6.7b-instruct-GGUF",
"deepseek-r1": "hf://bartowski/DeepSeek-R1-GGUF",
"deepseek-v3": "hf://unsloth/DeepSeek-V3-GGUF",
"devstral": "hf://bartowski/mistralai_Devstral-Small-2-24B-Instruct-2512-GGUF",
"devstral:123b": "hf://bartowski/mistralai_Devstral-2-123B-Instruct-2512-GGUF",
"devstral:24b": "hf://bartowski/mistralai_Devstral-Small-2-24B-Instruct-2512-GGUF",
"dragon": "hf://llmware/dragon-mistral-7b-v0",
"embeddinggemma": "hf://ggml-org/embeddinggemma-300M-GGUF",
"falcon3": "hf://tiiuae/Falcon3-10B-Instruct-GGUF",
"falcon3-moe:2x7b": "hf://mradermacher/Falcon3-MoE-2x7B-Insruct-GGUF",
"falcon3:10b": "hf://tiiuae/Falcon3-10B-Instruct-GGUF",
"flan-t5-large": "hf://Felladrin/gguf-flan-t5-large",
"gemma-4": "hf://bartowski/google_gemma-4-E4B-it-GGUF",
"gemma-4:26b": "hf://bartowski/google_gemma-4-26B-A4B-it-GGUF",
"gemma-4:31b": "hf://lmstudio-community/gemma-4-31B-it-GGUF",
"gemma-4:e2b": "hf://bartowski/google_gemma-4-E2B-it-GGUF",
"gemma-4:e4b": "hf://bartowski/google_gemma-4-E4B-it-GGUF",
"gemma2": "hf://bartowski/gemma-2-9b-it-GGUF",
"gemma2:2b": "hf://bartowski/gemma-2-2b-it-GGUF",
"gemma2:9b": "hf://bartowski/gemma-2-9b-it-GGUF",
"gemma3": "hf://ggml-org/gemma-3-4b-it-GGUF",
"gemma3:12b": "hf://ggml-org/gemma-3-12b-it-GGUF",
"gemma3:1b": "hf://ggml-org/gemma-3-1b-it-GGUF",
"gemma3:27b": "hf://ggml-org/gemma-3-27b-it-GGUF",
"gemma3:4b": "hf://ggml-org/gemma-3-4b-it-GGUF",
"gemma3n": "hf://ggml-org/gemma-3n-E4B-it-GGUF",
"gemma3n:e2b": "hf://ggml-org/gemma-3n-E2B-it-GGUF",
"gemma3n:e2b-it-f16": "hf://ggml-org/gemma-3n-E2B-it-GGUF:F16",
"gemma3n:e2b-it-q8_0": "hf://ggml-org/gemma-3n-E2B-it-GGUF:Q8_0",
"gemma3n:e4b": "hf://ggml-org/gemma-3n-E4B-it-GGUF",
"gemma3n:e4b-it-f16": "hf://ggml-org/gemma-3n-E4B-it-GGUF:F16",
"gemma3n:e4b-it-q8_0": "hf://ggml-org/gemma-3n-E4B-it-GGUF:Q8_0",
"gemma4": "hf://bartowski/google_gemma-4-E4B-it-GGUF",
"gemma4:26b": "hf://bartowski/google_gemma-4-26B-A4B-it-GGUF",
"gemma4:31b": "hf://lmstudio-community/gemma-4-31B-it-GGUF",
"gemma4:e2b": "hf://bartowski/google_gemma-4-E2B-it-GGUF",
"gemma4:e4b": "hf://bartowski/google_gemma-4-E4B-it-GGUF",
"glm-4.6": "hf://unsloth/GLM-4.6-GGUF",
"glm-4.6-vl": "hf://ggml-org/GLM-4.6V-GGUF",
"glm-4.6v-flash": "hf://bartowski/zai-org_GLM-4.6V-Flash-GGUF",
"gpt-5.1": "openai://gpt-5.1-2025-11-13",
"gpt-oss": "hf://ggml-org/gpt-oss-20b-GGUF",
"gpt-oss:120b": "hf://ggml-org/gpt-oss-120b-GGUF",
"gpt-oss:20b": "hf://ggml-org/gpt-oss-20b-GGUF",
"granite": "hf://ibm-granite/granite-3.3-8b-instruct-GGUF",
"granite-be-3.0:1b": "hf://taronaeo/Granite-3.0-1B-A400M-Instruct-BE-GGUF",
"granite-be-3.3:2b": "hf://taronaeo/Granite-3.3-2B-Instruct-BE-GGUF",
"granite-lab-7b": "hf://instructlab/granite-7b-lab-GGUF",
"granite-lab-8b": "hf://ibm-granite/granite-3.3-8b-instruct-GGUF",
"granite-lab:7b": "hf://instructlab/granite-7b-lab-GGUF",
"granite:2b": "hf://bartowski/ibm-granite_granite-3.3-2b-instruct-GGUF",
"granite:7b": "hf://instructlab/granite-7b-lab-GGUF",
"granite:8b": "hf://ibm-granite/granite-3.3-8b-instruct-GGUF",
"hermes": "hf://bartowski/Hermes-2-Pro-Mistral-7B-GGUF",
"ibm/granite": "hf://ibm-granite/granite-3.3-8b-instruct-GGUF",
"ibm/granite:2b": "hf://bartowski/ibm-granite_granite-3.3-2b-instruct-GGUF",
"ibm/granite:7b": "hf://instructlab/granite-7b-lab-GGUF",
"ibm/granite:8b": "hf://ibm-granite/granite-3.3-8b-instruct-GGUF",
"internlm2.5": "hf://bartowski/internlm2_5-20b-chat-GGUF",
"internlm2.5:20b": "hf://bartowski/internlm2_5-20b-chat-GGUF",
"kimi-k2": "hf://bartowski/moonshotai_Kimi-K2-Instruct-0905-GGUF",
"kimi-k2.5": "hf://bartowski/moonshotai_Kimi-K2.5-GGUF",
"kimi-k2.6": "hf://bartowski/moonshotai_Kimi-K2.6-GGUF",
"llama2": "hf://TheBloke/Llama-2-7B-Chat-GGUF",
"llama2:70b": "hf://TheBloke/Llama-2-70B-Chat-GGUF",
"llama2:7b": "hf://TheBloke/Llama-2-7B-Chat-GGUF",
"llama3": "hf://bartowski/Meta-Llama-3-8B-Instruct-GGUF",
"llama3.1": "hf://bartowski/Meta-Llama-3.1-8B-Instruct-GGUF",
"llama3.1:70b": "hf://bartowski/Meta-Llama-3.1-70B-Instruct-GGUF",
"llama3.1:8b": "hf://bartowski/Meta-Llama-3.1-8B-Instruct-GGUF",
"llama3.2:1b": "hf://bartowski/Llama-3.2-1B-Instruct-GGUF",
"llama3.2:3b": "hf://bartowski/Llama-3.2-3B-Instruct-GGUF",
"llama3.3": "hf://bartowski/Llama-3.3-70B-Instruct-GGUF",
"llama3.3:70b": "hf://bartowski/Llama-3.3-70B-Instruct-GGUF",
"llama3:70b": "hf://bartowski/Meta-Llama-3-70B-Instruct-GGUF",
"llama3:8b": "hf://bartowski/Meta-Llama-3-8B-Instruct-GGUF",
"magistral": "hf://bartowski/mistralai_Magistral-Small-2509-GGUF",
"magistral:2509": "hf://bartowski/mistralai_Magistral-Small-2509-GGUF",
"merlinite": "hf://instructlab/merlinite-7b-lab-GGUF",
"merlinite-lab-7b": "hf://instructlab/merlinite-7b-lab-GGUF",
"merlinite-lab:7b": "hf://instructlab/merlinite-7b-lab-GGUF",
"merlinite:7b": "hf://instructlab/merlinite-7b-lab-GGUF",
"minimax-m2": "hf://bartowski/MiniMaxAI_MiniMax-M2.7-GGUF",
"mistral": "hf://bartowski/Mistral-7B-Instruct-v0.3-GGUF",
"mistral-large": "hf://bartowski/Mistral-Large-Instruct-2411-GGUF",
"mistral-large:2411": "hf://bartowski/Mistral-Large-Instruct-2411-GGUF",
"mistral-nemo": "hf://bartowski/Mistral-Nemo-Instruct-2407-GGUF",
"mistral-small3.1": "hf://bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF",
"mistral-small3.1:24b": "hf://bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF",
"mistral:7b": "hf://bartowski/Mistral-7B-Instruct-v0.3-GGUF",
"mistral:7b-v1": "hf://TheBloke/Mistral-7B-Instruct-v0.1-GGUF",
"mistral:7b-v2": "hf://TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
"mistral:7b-v3": "hf://bartowski/Mistral-7B-Instruct-v0.3-GGUF",
"mistral_code_16k": "hf://TheBloke/Mistral-7B-Code-16K-qlora-GGUF",
"mistral_codealpaca": "hf://TheBloke/Mistral-7B-codealpaca-lora-GGUF",
"mixtao": "hf://MaziyarPanahi/MixTAO-7Bx2-MoE-Instruct-v7.0-GGUF",
"mixtral": "hf://TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF",
"mixtral:8x7b": "hf://TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF",
"nemotron": "hf://lmstudio-community/NVIDIA-Nemotron-3-Nano-30B-A3B-GGUF",
"nemotron:120b": "hf://lmstudio-community/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF",
"nemotron:30b": "hf://lmstudio-community/NVIDIA-Nemotron-3-Nano-30B-A3B-GGUF",
"nemotron:49b": "hf://bartowski/nvidia_Llama-3_3-Nemotron-Super-49B-v1-GGUF",
"nemotron:4b": "hf://lmstudio-community/NVIDIA-Nemotron-3-Nano-4B-GGUF",
"nemotron:51b": "hf://bartowski/Llama-3_1-Nemotron-51B-Instruct-GGUF",
"nemotron:70b": "hf://bartowski/Llama-3.1-Nemotron-70B-Instruct-HF-GGUF",
"nemotron:omni-30b": "hf://lmstudio-community/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF",
"olmo3": "hf://bartowski/allenai_Olmo-3-7B-Instruct-GGUF",
"olmo3.1:32b": "hf://bartowski/allenai_Olmo-3.1-32B-Instruct-GGUF",
"olmo3.1:32b-think": "hf://bartowski/allenai_Olmo-3.1-32B-Think-GGUF",
"olmo3:32b-think": "hf://bartowski/allenai_Olmo-3-32B-Think-GGUF",
"olmo3:7b": "hf://bartowski/allenai_Olmo-3-7B-Instruct-GGUF",
"openchat": "hf://TheBloke/openchat-3.5-0106-GGUF",
"openorca": "hf://TheBloke/Mistral-7B-OpenOrca-GGUF",
"phi2": "hf://MaziyarPanahi/phi-2-GGUF",
"phi3": "hf://microsoft/Phi-3-mini-4k-instruct-gguf",
"phi3:mini": "hf://microsoft/Phi-3-mini-4k-instruct-gguf",
"phi4": "hf://microsoft/phi-4-GGUF",
"qwen-3.6": "hf://bartowski/Qwen_Qwen3.6-27B-GGUF",
"qwen-3.6:27b": "hf://bartowski/Qwen_Qwen3.6-27B-GGUF",
"qwen-3.6:35b": "hf://lmstudio-community/Qwen3.6-35B-A3B-GGUF",
"qwen1.5:110b": "hf://bartowski/Qwen1.5-110B-Chat-GGUF",
"qwen2": "hf://bartowski/Qwen2-7B-Instruct-GGUF",
"qwen2-vl": "hf://bartowski/Qwen2-VL-7B-Instruct-GGUF",
"qwen2-vl:2b": "hf://bartowski/Qwen2-VL-2B-Instruct-GGUF",
"qwen2-vl:72b": "hf://bartowski/Qwen2-VL-72B-Instruct-GGUF",
"qwen2-vl:7b": "hf://bartowski/Qwen2-VL-7B-Instruct-GGUF",
"qwen2.5": "hf://bartowski/Qwen2.5-7B-Instruct-GGUF",
"qwen2.5-coder": "hf://bartowski/Qwen2.5-Coder-7B-Instruct-GGUF",
"qwen2.5-coder:1.5b": "hf://bartowski/Qwen2.5-Coder-1.5B-Instruct-GGUF",
"qwen2.5-coder:14b": "hf://bartowski/Qwen2.5-Coder-14B-Instruct-GGUF",
"qwen2.5-coder:32b": "hf://bartowski/Qwen2.5-Coder-32B-Instruct-GGUF",
"qwen2.5-coder:7b": "hf://bartowski/Qwen2.5-Coder-7B-Instruct-GGUF",
"qwen2.5:0.5b": "hf://bartowski/Qwen2.5-0.5B-Instruct-GGUF",
"qwen2.5:1.5b": "hf://bartowski/Qwen2.5-1.5B-Instruct-GGUF",
"qwen2.5:14b": "hf://bartowski/Qwen2.5-14B-Instruct-GGUF",
"qwen2.5:32b": "hf://bartowski/Qwen2.5-32B-Instruct-GGUF",
"qwen2.5:3b": "hf://bartowski/Qwen2.5-3B-Instruct-GGUF",
"qwen2.5:72b": "hf://bartowski/Qwen2.5-72B-Instruct-GGUF",
"qwen2.5:7b": "hf://bartowski/Qwen2.5-7B-Instruct-GGUF",
"qwen2.5vl": "hf://bartowski/Qwen_Qwen2.5-VL-32B-Instruct-GGUF",
"qwen2.5vl:2b": "hf://ggml-org/Qwen2.5-VL-2B-Instruct-GGUF",
"qwen2.5vl:32b": "hf://bartowski/Qwen_Qwen2.5-VL-32B-Instruct-GGUF",
"qwen2.5vl:3b": "hf://ggml-org/Qwen2.5-VL-3B-Instruct-GGUF",
"qwen2.5vl:7b": "hf://bartowski/Qwen_Qwen2.5-VL-7B-Instruct-GGUF",
"qwen2:7b": "hf://bartowski/Qwen2-7B-Instruct-GGUF",
"qwen3-coder-next": "hf://unsloth/Qwen3-Coder-Next-GGUF",
"qwen3-coder:30b": "hf://unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF",
"qwen3-moe:30b": "hf://bartowski/Qwen_Qwen3-30B-A3B-Instruct-2507-GGUF",
"qwen3-omni:30b": "hf://ggml-org/Qwen3-Omni-30B-A3B-Instruct-GGUF",
"qwen3-vl:30b": "hf://bartowski/Qwen_Qwen3-VL-30B-A3B-Instruct-GGUF",
"qwen3-vl:8b": "hf://bartowski/Qwen_Qwen3-VL-8B-Instruct-GGUF",
"qwen3.5:0.8b": "hf://bartowski/Qwen_Qwen3.5-0.8B-GGUF",
"qwen3.5:27b": "hf://bartowski/Qwen_Qwen3.5-27B-GGUF",
"qwen3.5:2b": "hf://bartowski/Qwen_Qwen3.5-2B-GGUF",
"qwen3.5:35b": "hf://bartowski/Qwen_Qwen3.5-35B-A3B-GGUF",
"qwen3.5:4b": "hf://bartowski/Qwen_Qwen3.5-4B-GGUF",
"qwen3.5:9b": "hf://bartowski/Qwen_Qwen3.5-9B-GGUF",
"qwen3.6": "hf://bartowski/Qwen_Qwen3.6-27B-GGUF",
"qwen3.6:27b": "hf://bartowski/Qwen_Qwen3.6-27B-GGUF",
"qwen3.6:35b": "hf://lmstudio-community/Qwen3.6-35B-A3B-GGUF",
"qwen3:0.6b": "hf://bartowski/Qwen_Qwen3-0.6B-GGUF",
"qwen3:1.7b": "hf://bartowski/Qwen_Qwen3-1.7B-GGUF",
"qwen3:14b": "hf://bartowski/Qwen_Qwen3-14B-GGUF",
"qwen3:30b": "hf://bartowski/Qwen_Qwen3-30B-A3B-Instruct-2507-GGUF",
"qwen3:32b": "hf://bartowski/Qwen_Qwen3-32B-GGUF",
"qwen3:4b": "hf://bartowski/Qwen_Qwen3-4B-GGUF",
"qwen3:8b": "hf://bartowski/Qwen_Qwen3-8B-GGUF",
"smollm2": "hf://bartowski/SmolLM2-1.7B-Instruct-GGUF",
"smollm2:1.7b": "hf://bartowski/SmolLM2-1.7B-Instruct-GGUF",
"smollm2:360m": "hf://bartowski/SmolLM2-360M-Instruct-GGUF",
"smollm:135m": "hf://HuggingFaceTB/smollm-135M-instruct-v0.2-Q8_0-GGUF",
"smolvlm": "hf://ggml-org/SmolVLM-500M-Instruct-GGUF",
"smolvlm:256m": "hf://ggml-org/SmolVLM-256M-Instruct-GGUF",
"smolvlm:2b": "hf://ggml-org/SmolVLM-Instruct-GGUF",
"smolvlm:500m": "hf://ggml-org/SmolVLM-500M-Instruct-GGUF",
"solar-pro": "hf://MaziyarPanahi/solar-pro-preview-instruct-GGUF",
"stories-be:260k": "hf://taronaeo/tinyllamas-BE/stories260K-be.gguf",
"tiny": "hf://TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF",
"tinyllama": "hf://TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF"
},
"Sources": [
"/home/ramalama/infrapilot-venv/share/ramalama/shortnames.conf"
]
},
"Store": "/home/ramalama/.local/share/ramalama",
"ToolsImage": "quay.io/ramalama/ramalama-tools:0.23",
"UseContainer": true,
"Version": "0.23.0"
}WSL2 (Ubuntu 24.04) on Windows 11, podman 4.9.3 (Ubuntu archive), rootless, CPU-only.
Nothing WSL-specific about the bug itself — the mismatch is in doc2rag's counting.
Issue Description
ramalama ragfails during the embedding phase on real-world documentation folders with:RuntimeError: llama-server embedding request returned 500: {"error":{"code":500,"message":"input (596 tokens) is too large to process. increase the physical batch size (current batch size: 512)","type":"server_error"}}
Root cause (verified against
container-images/scripts/doc2raginramalama-rag:0.23):the chunker (semchunk) measures chunks with the tiktoken
cl100k_baseencoding(default
--chunk-size 400), but the embedding llama-server counts the same textwith the embedding model's own tokenizer (embeddinggemma-300M). On URL-, code- and
YAML-dense content the embeddinggemma count comes out up to ~1.9x higher than the
cl100k_base count, so a chunk measured as <=400 tokens arrives at the server as
500-750 real tokens — over llama-server's physical batch limit (
-ub, default 512),which the embedding request must fit entirely. RamaLama starts the embedding server
without raising
-ub, and exposes no flag to do so.Observed on the same docs set: chunks measured <=400 (cl100k) arriving as 750 tokens
(HTML page with a long navigation link list) and 596 tokens (Markdown with many long
URLs / inline code). Plain prose rarely triggers it, which makes the failure look
random across doc sets.
Steps to reproduce the issue
paragraph is ~50 long GitHub PR URLs (a typical auto-generated release-notes page),
or any page with a long list glued together with
tags.
"Embedding chunks via llama.cpp...".
Describe the results you received
Embedding chunks via llama.cpp...
Embedding 5/216 chunks...Traceback (most recent call last):
File "/usr/bin/doc2rag", line 553, in
run_pipeline(args)
File "/usr/bin/doc2rag", line 526, in run_pipeline
store_in_qdrant(chunks, ids, doc_ids, chunk_indices, args.output, embedder, embedding_model=embedding_model)
File "/usr/bin/doc2rag", line 340, in store_in_qdrant
vectors = embedder.embed(chunks)
File "/usr/bin/doc2rag", line 298, in embed
all_embeddings.extend(self._embed_batch(batch))
File "/usr/bin/doc2rag", line 317, in _embed_batch
raise RuntimeError(f"llama-server embedding request returned {e.code}: {body}") from None
RuntimeError: llama-server embedding request returned 500: {"error":{"code":500,"message":"input (596 tokens) is too large to process. increase the physical batch size (current batch size: 512)","type":"server_error"}}
The whole build aborts (exit status 1); no image is produced.
Describe the results you expected
The build completes. The chunker should never emit a chunk the embedding server
rejects. Possible fixes, in rough order of correctness:
a) doc2rag counts chunk tokens with the embedding model's own tokenizer instead of
cl100k_base (e.g. via the embedding llama-server's /tokenize endpoint, since the
server is already running during chunking);
b) or RamaLama starts the embedding llama-server with a physical batch size safely
above the effective chunk limit (e.g. -ub 2048);
c) or doc2rag splits/truncates any chunk that exceeds the server limit before
POSTing to /v1/embeddings, instead of aborting the whole build.
Workaround that works today:
ramalama rag --chunk-size 256(256 x ~1.9 < 512).ramalama info output
Upstream Latest Release
Yes
Additional environment details
WSL2 (Ubuntu 24.04) on Windows 11, podman 4.9.3 (Ubuntu archive), rootless, CPU-only.
Nothing WSL-specific about the bug itself — the mismatch is in doc2rag's counting.
Additional information
per-chunk size issue, not request batching.
works on one docs folder and fails on another — easy to misdiagnose as a doc
problem. The error message ("increase the physical batch size") points at a
server setting the user has no way to change through the ramalama CLI.