This project is inspired by the Deep Red film trilogy from the creators of "Iron Sky". It aims to create a custom LLM model that has only knowledge up to July 1969 - the launch date of the fictional Mars mission in the movie - and responds in a style that aligns with a "Soviet utopia" setting controlled by the fictional "chess playing" AI.
These steps document how to perform a Model Training from Scratch procedure using a continued pre-training approach for temporal and thematic alignment.
- How to setup an AMD "Strix Halo" device (Fedora) — primary development and training system; includes automated setup script
- How to setup an optional second NVIDIA device (A4000) — dedicated GPU for training/inference; includes automated setup script
- How to set up the Wikipedia MCP server and data pipeline — extraction, indexing, search, and MCP server for Wikipedia content
- How to extract year-based historical topics from Wikipedia — enriched event data for temporal training
- How to augment Wikipedia with temporal metadata — YAGO/Wikidata parsing, normalization, and database augmentation for time-period filtering
- How to retrieve Project Gutenberg literature — thematically relevant books for training data
- How to prepare the chess training corpus — chess content retrieval, PGN conversion, and corpus preparation for the Deep Red persona
- How to tokenize and prepare the training corpus — tokenization, shuffling, and train/val splitting for continued pre-training
- How to train the Deep Red model — continued pre-training on the temporally-filtered pre-1969 corpus using dev (SmolLM2-360M) or prod (TinyLlama-1.1B) profiles
- How to use the generated GGUF models with LM Studio — loading, testing, and comparing trained model checkpoints in LM Studio (sample models linked here)
These steps document a failed fine-tuning approach to modify an existing model with "temporal knowledge cutoff" and "theme alignment".
- How to setup an AMD "Strix Halo" device (Ubuntu, legacy)
- How to setup LMStudio as server for "headless" operation
- How to create a Wikipedia database, enable vector database searches for articles, and provide an MCP server for the data
- How to extract temporal information from YAGO about Wikipedia articles
- How to normalize YAGO output to match local English Wikipedia database
- How to extract temporal information from Wikidata
- How to augment the Wikipedia database with temporal information
- How to fine-tune an existing LLM model with a Temporal Knowledge Cutoff, restating its knowledge base into the past
- Temporal Finetuning Phased Plan - Content retrieval, analysis, and finetuning for temporal alignment
- How to generate training datasets
- How to finetune model with temporal knowledge cutoff using these datasets
- How to fine-tune the model further for "Soviet utopia" theme alignment
- Theme Finetuning Phased Plan - Content retrieval, analysis, and finetuning for stylistic alignment
- Phase 1: How to retrieve training content from Project Gutenberg
- Phase 2: How to chunk and filter content for theme alignment
- Phase 3: How to generate ChatML training dataset from filtered content
- Phase 4: How to finetune model with themed dataset
/documentation- Setup guides and planning documents for the project/scripts- Python scripts for Wikipedia extraction/indexing, temporal augmentation (YAGO/Wikidata), Gutenberg and chess content retrieval, MCP server, and system setup/services- Systemd service files for automated startup (inference servers, MCP server, OpenSearch, web GUI)/webapp- React-based web interface for Wikipedia search with Vite configuration/notebooks- Jupyter notebooks for testing embeddings and OpenSearch functionality/patches- System patches (network driver fix for AMD Strix Halo for older kernels)