TokenSmith is a Retrieval-Augmented Generation (RAG) application that enables intelligent document search and question answering using local LLMs. Built with llama.cpp for efficient inference and FAISS for high-performance vector search, TokenSmith allows you to index PDF documents and chat with them using natural language queries.
- ๐ PDF Document Processing: Extract and index content from PDF documents
- ๐ Intelligent Retrieval: Fast semantic search using FAISS vector database
- ๐ค Local LLM Integration: Powered by llama.cpp for privacy-focused inference
- โก Hardware Acceleration: Supports Metal (Apple Silicon), CUDA (NVIDIA), and CPU inference
- ๐ฏ Flexible Chunking: Token-based or character-based document segmentation
- ๐ Visualization Support: Optional indexing progress visualization
- ๐ ๏ธ Production-Ready: Conda-based environment management with automated builds
- ๐ง Configurable: YAML-based configuration system
- Python: 3.9+
- Conda/Miniconda: For environment management
- System Requirements:
- macOS: Xcode Command Line Tools
- Linux: GCC, make, cmake
- Windows: Visual Studio Build Tools (for compilation)
git clone https://github.com/georgia-tech-db/TokenSmith.git
cd tokensmithmake buildThis will:
- Create a conda environment named
tokensmith - Install all Python dependencies
- Detect or build llama.cpp with platform-specific optimizations
- Install TokenSmith in development mode
conda activate tokensmithPlace your PDF files in the data directory
mkdir -p data/chapters
cp your-documents.pdf data/chapters/Index with default settings
make run-indexOr with custom parameters, eg.
make run-index ARGS="--pdf_range 1-10 --chunk_mode chars --visualize"Activate environment first (required for interactive mode)
conda activate tokensmith
python -m src.main chatYou might have to download
qwen2.5-0.5b-instruct-q5_k_m.ggufinto yourllama.cpp/modelsif you get an error about a missing model.
conda deactivateTokenSmith uses YAML configuration files with the following priority order:
- Command-line
--configargument - User config (
~/.config/tokensmith/config.yaml) - Default config (
config/config.yaml)
# config/config.yaml
embed_model: "sentence-transformers/all-MiniLM-L6-v2"
top_k: 5
max_gen_tokens: 400
halo_mode: "none"
seg_filter: null
# Model settings
model_path: "models/qwen2.5-0.5b-instruct-q5_k_m.gguf"
# Indexing settings
chunk_mode: "tokens" # or "chars"
chunk_tokens: 500
chunk_size_char: 20000make run-indexmake run-index ARGS="--pdf_range <start_page_number>-<end_page_number> --chunk_mode <tokens_or_chars>"make run-index ARGS="--keep_tables --visualize --chunk_tokens <number_of_chunk_tokens>"make run-index ARGS="--pdf_dir <path_to_pdf> --index_prefix book_index --config <path_to_yaml_config_file>"python -m src.main chat --config <path_to_yaml_config_file> --model_path <path_to_llm_model>export LLAMA_CPP_BINARY=/usr/local/bin/llama-cli
make buildmake update-envmake export-envmake show-depsmode: Operation mode (indexorchat)--config: Configuration file path--pdf_dir: Directory containing PDF files--index_prefix: Prefix for index files--model_path: Path to GGUF model file
--pdf_range: Process specific page range (e.g., "1-10")--chunk_mode: Chunking strategy (tokensorchars)--chunk_tokens: Tokens per chunk (default: 500)--chunk_size_char: Characters per chunk (default: 20000)--keep_tables: Preserve table formatting--visualize: Show indexing progress visualization
make help # Show all available commands
make env # Create conda environment
make build-llama # Build llama.cpp from source
make install # Install package in development mode
make build # Full build process
make test # Run tests
make clean # Clean build artifacts
make show-deps # Show installed packages
make update-env # Update environment
make export-env # Export environment with exact versions# Add new conda package
conda activate tokensmith
conda install new-packageAdd to environment.yml for persistence. Edit environment.yml, then:
make update-envTokenSmith includes a comprehensive testing framework for evaluating RAG performance. The framework is fully integrated with the main pipeline, ensuring tests use the same code path as production.
# Run all benchmarks
pytest tests/
# Run with terminal output to see detailed results
pytest tests/ -s
# Run specific benchmark
pytest tests/ --benchmark-ids="test" -s- โ
Integrated Pipeline: Tests use the same
get_answer()function as chat - โ Multiple Metrics: Semantic similarity, BLEU, keyword matching, text similarity
- โ Flexible Output: Terminal (detailed) or HTML (reports)
- โ System Prompts: Four modes (baseline, tutor, concise, detailed)
- โ Component Isolation: Test with/without chunks, or use golden chunks
- โ Beautiful Reports: Interactive HTML reports with metric breakdowns
Test results are saved in:
tests/results/benchmark_results.json- Detailed JSON datatests/results/benchmark_summary.html- Interactive HTML reporttests/results/failed_tests.log- Failure details
For complete testing documentation, usage examples, and configuration options:
๐ tests/README.md - Complete testing guide with all CLI options and examples