Llama 2 on DOS: Retro AI

In a remarkable feat of engineering and nostalgia, a developer has successfully ported the Llama 2 Large Language Model (LLM) to the venerable DOS operating system. This unexpected achievement brings modern AI capabilities to vintage computing environments, showcasing the enduring power of open-source software and the ingenuity of dedicated programmers. The project, completed over a single weekend, demonstrates the adaptability of Llama 2 and opens up intriguing possibilities for exploring AI on older hardware.

The Genesis of DOS LLM

The mastermind behind this ambitious project is Yeo Kheng Meng, a programmer known for his previous work on a DOS client for ChatGPT. Driven by a passion for both AI and retro computing, Meng set out to create a fully functional LLM for DOS that could perform inference tasks offline. This endeavor required overcoming significant technical hurdles, given the limitations of the DOS environment compared to modern operating systems. Meng's project leverages the readily available open-source code of Meta's Llama 2 model, released in 2023, and builds upon the foundation provided by llama2.c, a project designed to facilitate the porting of Llama 2 models to diverse systems.

The FreeDOS Project community has enthusiastically embraced Meng's work, recognizing its potential to breathe new life into older hardware and provide a unique platform for exploring AI applications. The project highlights the continued relevance of DOS in niche computing circles and the power of collaborative development in pushing the boundaries of what's possible.

Technical Challenges and Solutions

Porting Llama 2 to DOS was far from a straightforward task. Despite the portability focus of llama2.c, Meng encountered numerous challenges specific to the vintage computing environment. One of the primary obstacles was the limited memory available in DOS systems. The traditional 640KB of conventional memory is insufficient to load and run a large language model. To address this, Meng had to carefully select a suitable DOS extender to access extended memory beyond the 640KB barrier. This allowed the LLM to utilize a larger RAM pool, albeit still significantly smaller than what modern systems offer.

Another hurdle was the compatibility of the llama2.c code with older C compilers. The project required a compiler with relatively modern features, which were not readily available in the DOS ecosystem. Meng had to modify the OWC compiler to ensure it could handle the code's requirements. This involved adjusting the compilation process and resolving compatibility issues to create a fully working llama2 binary that could run on DOS systems. The entire process demonstrates a deep understanding of both modern AI models and the intricacies of vintage computing.

Meng's ingenuity extended to optimizing the performance of the LLM on limited hardware. He meticulously tweaked the code and experimented with different configurations to achieve the best possible inference speeds. This involved balancing memory usage, processing power, and code efficiency to maximize the capabilities of the DOS environment. The resulting DOS LLM client is a testament to the power of optimization and the resourcefulness of developers in overcoming technical constraints.

Performance Benchmarks and System Comparisons 💻

To evaluate the performance of the DOS LLM, Meng conducted a series of benchmarks on various systems running MS-DOS 6.22. The test configurations ranged from a vintage 486 DX2 66MHz PC to a modern Ryzen-based desktop, providing a comprehensive comparison across different generations of hardware. As expected, the newer systems delivered significantly faster inference speeds, showcasing the advancements in processing power over the years. However, the results also revealed some surprises.

Notably, a ThinkPad T42 with a 1.7GHz Pentium M processor outperformed a ThinkPad X13G1 released 16 years later. This unexpected outcome highlights the importance of factors beyond raw processing speed, such as memory architecture and software optimization. The T42's optimized memory management and efficient code execution likely contributed to its superior performance in this specific task. These benchmarks provide valuable insights into the capabilities and limitations of different hardware configurations for running AI models.

The side-by-side performance comparison between a Thinkpad T42 (2004) and a Toshiba Satellite 315CDT (1996) vividly illustrates the progress made in computing technology over the past few decades. While the older Toshiba system struggled to run the LLM at a reasonable speed, the T42 demonstrated the feasibility of running AI models on relatively older hardware. These comparisons underscore the importance of continuous innovation and the potential for even older systems to contribute to the evolving landscape of artificial intelligence.

The Legacy of Open Source and Community Collaboration

Meng openly acknowledged the critical role of open-source software in making this project possible. He specifically thanked Andrej Karpathy, the creator of llama2.c, for open-sourcing the code and providing a foundation for others to build upon. Without Karpathy's contribution, the task of porting Llama 2 to DOS would have been significantly more challenging, if not impossible. This project exemplifies the power of open-source collaboration in democratizing access to AI technology and fostering innovation across diverse platforms.

The success of this project also underscores the importance of community support and knowledge sharing. Meng benefited from the collective expertise of the FreeDOS Project community, who provided valuable insights and assistance in navigating the intricacies of the DOS environment. This collaborative spirit is essential for driving innovation and pushing the boundaries of what's possible in both the AI and retro computing domains. The project serves as an inspiring example of how individuals and communities can come together to create something truly remarkable.

In conclusion, the successful porting of Llama 2 to DOS is a testament to the ingenuity of developers, the power of open-source software, and the enduring appeal of retro computing. Yeo Kheng Meng's weekend project has not only demonstrated the adaptability of modern AI models but has also opened up new avenues for exploring AI on vintage hardware. This achievement is a reminder that innovation can occur in unexpected places and that the legacy of older technologies can continue to inspire and contribute to the future of computing. The project is a win for the retro computing community, and a fascinating glimpse into the potential for AI on DOS .