Open Source Today (2024-08-13): 1.5-Pints - Efficient LLM Pre-Trained in 9 Days, Outperforms Small Models
Open Source Today (2024-08-13): 1.5-Pints - Efficient LLM Pre-Trained in 9 Days, Outperforms Small Models
Explore cutting-edge AI open-source projects like 1.5-Pints, strwythura, and Deep-Live-Cam. Discover efficient LLMs, knowledge graphs, and more.
Here are some interesting AI open-source models and frameworks I wanted to share today:
Project: 1.5-Pints
1.5-Pints is an LLM pre-trained on high-quality data in just 9 days. It uses a modified Mistral optimizer and the Llama-2 architecture, with a training dataset containing 57 billion tokens.
The dataset prioritizes text with explanatory and "textbook-like" content to enhance the model's reasoning abilities.
This model outperforms other small-parameter models like Apple's OpenELM and Microsoft's Phi in following instructions.
https://arxiv.org/abs/2408.03506
https://github.com/Pints-AI/1.5-Pints
Project: strwythura
strwythura is an open-source project aimed at building knowledge graphs from unstructured data sources.
It uses deep learning models and open-source libraries to create nodes, edges, and attributes of the graph through a series of steps.
The project offers detailed tutorials and examples on how to parse text, build vocabularies, perform entity recognition and relationship extraction, and finally generate a visual knowledge graph.
https://github.com/DerwenAI/strwythura
Project: Transformer Explainer
Transformer Explainer is an interactive visualization tool developed by a research team at Georgia Tech. It helps users understand how Transformer-based models work.
The tool runs a real-time GPT-2 model in the browser, allowing users to experiment with their own text and see how the Transformer's internal components work together to predict the next token.
https://github.com/poloclub/transformer-explainer
https://arxiv.org/abs/2408.04619
Project: RagBuilder
RagBuilder is a toolkit designed for developing optimal production-level RAG systems. It fine-tunes various RAG parameters, like chunking strategies and sizes, by evaluating these configurations on test datasets to identify the best setup.
RagBuilder also includes several pre-defined RAG templates that perform well across different datasets. Users simply provide the data, and RagBuilder generates a production-ready RAG setup in minutes.
https://github.com/KruxAI/ragbuilder
Project: LLM-Distillery
LLM-Distillery is a pipeline for distilling one or more teacher models into a student model.
It supports distillation for both instruction and completion text, offering offline distillation after collecting the dataset.
The project supports Windows and Linux operating systems and includes an automatic hdf5 dataset synchronization feature, allowing data collection to resume after forced termination.
https://github.com/golololologol/LLM-Distillery
Project: Deep-Live-Cam
Deep Live Cam is an open-source tool for real-time face swapping and deepfake video creation. Users can upload an image to replace the face in a video or live stream in real-time, useful for video production, animation, and other fields.
