Distributed Systems · AI · Full Stack
Course
Copilot
A distributed AI-powered learning platform that converts entire video playlists into multilingual transcripts, generates searchable embeddings, and enables users to interact with course content through a RAG-powered chatbot.
Type
AI / Distributed Systems
Architecture
Event-Driven Microservices
Focus
Kafka · RAG · Video Processing
01 / Overview
Turning long-form video content into an interactive knowledge system.
Course-Copilot allows users to process an entire video playlist and convert its content into searchable, multilingual learning material.
Videos are processed asynchronously through an event-driven architecture. Transcription, translation, embedding generation and storage are handled by independent services.
Once the course content has been processed, users can interact with the material through an AI-powered chatbot using Retrieval-Augmented Generation.
02 / Workflow
From playlist to AI-powered knowledge.
Import
A user submits a video playlist through the web application.
Queue
The application publishes processing events to Apache Kafka.
Process
Video services transcribe, translate and process the course content.
Embed
Course content is transformed into vector embeddings and stored for retrieval.
Interact
The RAG-powered AI assistant retrieves relevant context and answers user questions.
03 / Architecture
Event-driven microservices.
Course-Copilot separates video processing, embedding generation, background jobs and AI inference into independent services. Apache Kafka acts as the communication backbone between asynchronous workloads.
04 / Services
Independent services for independent workloads.
Web App
Next.js · TypeScript
Provides the user interface for creating projects, managing playlists and interacting with the AI assistant.
Background Job Service
TypeScript · Kafka
Consumes Kafka events and distributes processing tasks to the appropriate microservices.
Video Processing Service
Python · Whisper · FFmpeg
Handles transcription, translation and video processing workloads.
Embedding Service
Python · Vector Embeddings
Processes course content, generates embeddings and stores the resulting knowledge representation.
Intelligence Service
Python · LangChain · RAG
Provides the AI assistant that retrieves relevant course context and generates answers.
05 / Distributed Systems
Kafka as the backbone for asynchronous workloads.
Why Kafka?
Video processing and embedding generation are long-running workloads that should not block the user-facing application. Kafka allows these operations to be handled asynchronously.
Decoupled Services
Services communicate through events instead of tightly coupling every processing step to the web application.
Independent Scaling
Resource-intensive workloads such as video processing can be scaled independently from the frontend and chatbot services.
06 / AI
RAG over the entire course.
Once course content has been processed and converted into embeddings, the resulting vector representation becomes the knowledge layer for the AI assistant.
When a user asks a question, the intelligence service retrieves the most relevant content from the vector database before passing the retrieved context to the language model.
This allows the chatbot to answer questions based on the actual course material instead of relying entirely on the model's general knowledge.
07 / Infrastructure
Containerized and reproducible.
Docker
Every microservice contains its own Dockerfile, allowing the complete application to be orchestrated consistently.
Docker Compose
The development environment can start the complete collection of services through a single Compose configuration.
Terraform
Infrastructure is defined as code, including AWS S3 resources used for course and processing data.
08 / Technology
09 / Engineering
Engineering decisions.
Event-Driven Processing
Long-running video and embedding workloads are decoupled from the web application through Kafka.
Polyglot Architecture
TypeScript is used for the web and orchestration layers, while Python is used where the ecosystem provides stronger support for AI, transcription and embedding workloads.
Asynchronous Workflows
Processing an entire playlist can involve many independent jobs. Asynchronous workers allow those operations to execute without blocking user requests.
Infrastructure as Code
Terraform makes infrastructure reproducible and version-controlled instead of relying entirely on manually configured cloud resources.