Technologies Used in building YouTube Website and Application

- Frontend: JavaScript, HTML5, and Polymer
- Backend: C++, Java, and Python
- Database: MySQL, Oracle, and Bigtable
- Package Manager: npm (Node.js)
- Frameworks: AngularJS, Polymer, and TensorFlow
- Tools: Jenkins, Git, and Docker

To help you get this environment running locally, we’ll use Docker Compose. This setup simulates a modern microservices architecture by separating the Frontend (React/Vite), the API Gateway (Go), and a Media Transcoder (Python/FFmpeg).

1. Project Directory Setup

First, create this folder structure on your machine:

yt-clone-local/
├── docker-compose.yml
├── gateway/
│   ├── main.go
│   └── go.mod
├── transcoder/
│   ├── processor.py
│   └── requirements.txt
└── web/
    ├── src/
    └── package.json

2. The docker-compose.yml

This file acts as the "orchestrator," linking your services, networks, and volumes together.

YAML

version: '3.8'

services:
  # 1. Frontend: Serves the UI
  web-client:
    image: node:20-alpine
    working_dir: /app
    volumes:
      - ./web:/app
    ports:
      - "3000:3000"
    command: sh -c "npm install && npm run dev"

  # 2. API Gateway: Routes requests (Go)
  api-gateway:
    image: golang:1.21-alpine
    working_dir: /app
    volumes:
      - ./gateway:/app
    ports:
      - "8080:8080"
    command: go run main.go

  # 3. Transcoder: Handles video processing (Python + FFmpeg)
  transcoder:
    build:
      context: ./transcoder
      dockerfile: Inline
    volumes:
      - ./media:/videos
    depends_on:
      - redis-cache

  # 4. Infrastructure: Shared Cache
  redis-cache:
    image: redis:7-alpine
    ports:
      - "6379:6379"

  # 5. Database: Sharded Metadata
  metadata-db:
    image: mysql:8.0
    environment:
      MYSQL_ROOT_PASSWORD: password
      MYSQL_DATABASE: youtube_meta

3. The Transcoder (Dockerfile Logic)

YouTube's secret sauce is video compression. For the transcoder service, you'll need a custom image that includes FFmpeg. Create a Dockerfile inside the /transcoder folder:

Dockerfile

FROM python:3.11-slim

# Install FFmpeg (The industry standard for video)
RUN apt-get update && apt-get install -y ffmpeg

WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . .
CMD ["python", "processor.py"]

4. Logical Interaction Flow

When a user interacts with this system:

1. Frontend (Web): Sends a POST request with a video file to the API Gateway.

2. Gateway (Go): Authenticates the user and pushes the raw file into an "Uploads" bucket. It then sends a message to Redis.

3. Transcoder (Python): Watches the Redis queue. When it sees a new job, it pulls the video, uses FFmpeg to create 720p/1080p versions, and updates the MySQL database when finished.

To wrap up this architecture tour, let’s look at the complete "Big Picture" of the YouTube stack. We’ve covered the project structure, the individual services, and the local orchestration.

Here is the definitive summary of the technologies that make YouTube function at a global scale.

The Full Technology Inventory

The Operational Workflow

1. Ingestion: The Frontend (TypeScript/Lit) sends video chunks to the API Gateway (Go).

2. Orchestration: Borg (Google's K8s) spins up Transcoder nodes.

3. Processing: FFmpeg (C++) converts the video into multiple bitrates.

4. Metadata: Video title, tags, and creator info are saved in Vitess.

5. Delivery: The Google Global Cache (GGC) pushes the video to an edge server near you.

The "Package Manager" Reality

In a project of this scale, "Package Management" is tiered:

• System Level: Bazel manages the entire dependency graph.

• Language Level: Go Modules (go.mod), Python Wheels (requirements.txt), and NPM/PNPM (package.json) handle individual library versions.

• Infrastructure Level: Docker and Terraform (or Google-internal equivalents) manage the environment consistency.

Final Project Structure Recap

/yt-monorepo
 ├── /api            # Shared Protobufs (The "Source of Truth")
 ├── /build          # Bazel configurations
 ├── /services       # Backend (Go/Python/C++)
 ├── /web            # Frontend (Lit/TS)
 ├── /infra          # Vitess/K8s configs
 └── /ml             # TensorFlow models for RecSys

Wrapping Up

You now have the conceptual blueprint, the project structure, and the docker-compose logic needed to build a "Mini-YouTube." This architecture is designed to fail gracefully; if the recommendation service goes down, the video player still works. If one database shard fails, only a tiny fraction of videos are unavailable.

To finalize your YouTube-style environment, we need the API Gateway. In a real-world scenario, this service acts as the entry point for millions of concurrent uploads.

Below is a robust Go implementation for the gateway/main.go file. It handles multipart/form-data uploads, which is the standard way to send large video files from a browser to a server.

The API Gateway (gateway/main.go)

This code sets up a high-performance HTTP server that streams the video file directly to a local "storage" directory to avoid exhausting the RAM.

Go

package main

import (
	"fmt"
	"io"
	"log"
	"net/http"
	"os"
	"path/filepath"
)

const uploadPath = "./uploads"

func main() {
	// Ensure the upload directory exists
	os.MkdirAll(uploadPath, os.ModePerm)

	http.HandleFunc("/upload", uploadHandler)

	fmt.Println("YouTube Clone Gateway running on :8080...")
	log.Fatal(http.ListenAndServe(":8080", nil))
}

func uploadHandler(w http.ResponseWriter, r *http.Request) {
	if r.Method != "POST" {
		http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
		return
	}

	// 1. Limit upload size (e.g., 500MB)
	r.Body = http.MaxBytesReader(w, r.Body, 500*1024*1024)

	// 2. Parse the multipart form
	file, handler, err := r.FormFile("video")
	if err != nil {
		http.Error(w, "Error retrieving the file", http.StatusBadRequest)
		return
	}
	defer file.Close()

	// 3. Create a destination file in our 'storage'
	dst, err := os.Create(filepath.Join(uploadPath, handler.Filename))
	if err != nil {
		http.Error(w, "Internal error", http.StatusInternalServerError)
		return
	}
	defer dst.Close()

	// 4. Stream the file from the request to the disk
	_, err = io.Copy(dst, file)
	if err != nil {
		http.Error(w, "Failed to save file", http.StatusInternalServerError)
		return
	}

	fmt.Fprintf(w, "Successfully Uploaded: %s\n", handler.Filename)
	log.Printf("Received upload: %s", handler.Filename)
	
	// NEXT LOGICAL STEP: Trigger Transcoder via Redis/gRPC
}

Infrastructure Summary Table

Now that you have the code, here is how all the pieces we discussed interact within your docker-compose environment:

Final Wrap-Up

You now have a complete conceptual and practical foundation:

1. The Stack: A mix of Go, Python, C++, and TypeScript.

2. The Structure: A Monorepo managed by Bazel logic.

3. The Environment: A Dockerized microservices mesh.

4. The Logic: A streaming API Gateway that handles heavy lifting without crashing.

How to run it:

1. Place the main.go above in your /gateway folder.

2. Run go mod init gateway inside that folder.

3. Run docker-compose up --build from the root.