essay · 2026-09-05

How to implement dynamic runtime profiling in multi-llm merges

Research into large language model evaluation highlights that effective merging depends heavily on accurate profiling before applying combination strategies like SLERP or linear interpolation, which determine the final embedding for tokens present in multiple models. The provided tools address this by offering a structured way to profile specific model types using parameters such as --model_type and --persistent_dir, ensuring data is stored consistently across evaluations. A key script identified for this purpose is located at llm-profiling-toolkit/run_profiling.py, which serves as the central mechanism for generating the metrics needed to optimize runtime composition in dynamic systems of software-defined hardware. These tools are critical because modern environments require on-the-fly discovery and integration of constituent models without compromising programmability or introducing significant down-time during reconfiguration. By utilizing these specific scripts, researchers can avoid designing compilers that fail to leverage different types of hardware at runtime due to a lack of precise performance data.

Sources

• Design and Implementation of Knowledge Base for Runtime Management of Software Defined Hardware — 2022-03-28T16:40:26Z
• Runtime Composition in Dynamic System of Systems: A Systematic Review of Challenges, Solutions, Tools, and Evaluation Methods — 2025-10-14T15:14:57Z
• Runtime Reconfiguration of J2EE Applications — 2004-11-17T13:34:06Z
• GitHub - Cohere-Labs-Community/llm-profiling-toolkit · GitHub
• GitHub - arcee-ai/mergekit: Tools for merging pretrained large language models. · GitHub
• llm-profiling-toolkit/run_profiling.py at main · Cohere-Labs-Community/llm-profiling-toolkit
• GitHub - signerless/llm-checker: Advanced CLI tool that scans your hardware and tells you exactly which LLM or sLLM models you can run locally, with full Ollama
• GitHub - llm-profiling/llm-profiling.github.io · GitHub