Skip to content

Commit 953a438

Browse files
ritchiengritchieng
committed
feat: image tokenization
1 parent 3112593 commit 953a438

File tree

4 files changed

+390
-0
lines changed

4 files changed

+390
-0
lines changed

‎aigc/app.js‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ async function loadPosts() {
3131
// Try to fetch a list of posts - we'll use a fallback approach
3232
const postFiles = [
3333
'hallucination-mitigation',
34+
'image-tokenization',
3435
'intro-to-aigc',
3536
'tokenization-embeddings'
3637
];

‎aigc/posts/image-tokenization.md‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
---
2+
title: "Image Tokenization"
3+
date: "2026-01-22"
4+
category: "AIGC"
5+
pinned: false
6+
excerpt: "How CLIP Bridges Pixels and Prose"
7+
---
8+
9+
## How CLIP Bridges Pixels and Prose
10+
This post focuses specifically on **CLIP** (Contrastive Language-Image Pre-training), the breakthrough architecture from OpenAI that taught AI to "see" by reading and "read" by looking.
11+
12+
Unlike standard LLMs that only understand text, CLIP creates a bridge between vision and language. Here's how it tokenizes images to match the language of text.
13+
14+
For years, Computer Vision and Natural Language Processing (NLP) were two different worlds. Vision models looked for edges and textures; text models looked for grammar and syntax.
15+
16+
CLIP changed everything. It proved that if you "tokenize" an image correctly, you can force it into the same mathematical space as text. This is the foundation of tools like DALL-E, Midjourney, and advanced semantic image search.
17+
18+
## The "Two Towers" Architecture
19+
CLIP operates using two parallel encoders—often called the "Two Towers":
20+
21+
- **The Text Tower**: A standard Transformer (like GPT) that tokenizes text using Byte Pair Encoding (BPE).
22+
- **The Image Tower**: A Vision Transformer (ViT) that "tokenizes" an image into a sequence of patches.
23+
24+
The goal isn't just to understand them separately, but to project both into a shared **Multimodal Embedding Space**.
25+
26+
## Tokenizing the Image: The Patching Strategy
27+
To make an image "look" like a sentence, CLIP doesn't look at individual pixels. Instead, it treats the image like a jigsaw puzzle:
28+
29+
- **Patching**: The image (e.g., $224 \times 224$ pixels) is chopped into a grid of squares, usually $16 \times 16$ pixels each.
30+
- **Linear Projection**: Each patch is flattened into a vector. If a patch has $16 \times 16$ pixels with 3 color channels (RGB), it starts as 768 numbers.
31+
- **The "Class" Token**: Just as a text model adds a special token to represent the "entire sentence," CLIP adds a `[CLS]` token to the image sequence. This token eventually learns to represent the summary of the entire image.
32+
- **Position Embeddings**: Because the model needs to know that the "cloud" patch is above the "grass" patch, a unique "coordinate" vector is added to each patch.
33+
34+
## The "Aha!" Moment: Contrastive Learning
35+
How do we make the vector for the word "Golden Retriever" look like the vector for a patch-grid of a dog? CLIP uses a training method called **Contrastive Learning**.
36+
37+
Imagine a batch of $N$ images and $N$ captions:
38+
39+
- **The Goal**: Maximize the cosine similarity between the correct pairs (Image A + Caption A).
40+
- **The Constraint**: Minimize the similarity between the incorrect pairs (Image A + Caption B).
41+
42+
Mathematically, CLIP calculates the dot product of the image and text vectors. The model is essentially playing a massive game of "Match the Caption to the Photo."
43+
44+
## Why This Matters: Zero-Shot Intelligence
45+
Traditional vision models were rigid. If you trained a model on "cats" and "dogs," it would fail if it saw a "panda."
46+
47+
Because CLIP uses textual descriptions as its labels, it has **Zero-Shot capabilities**. You can give it an image of a "Cyberpunk-style neon city" (something it was never specifically trained to categorize) and because it understands the tokens for "neon," "city," and "cyberpunk," it can find that image in the vector space with incredible accuracy.
48+
49+
## Practical Implementation: The Latent Space
50+
When you build an application using CLIP, you aren't storing images; you are storing **Image Embeddings** (typically 512 or 768 dimensions).
51+
52+
- **Image Search**: You embed your entire photo library. When a user types "sunset at the beach," you embed that text and find the image vectors with the highest cosine similarity.
53+
- **Content Moderation**: You can check if an image embedding is mathematically close to the embedding of "prohibited content" tokens.
54+
55+
## Summary: The Unified Language of Vectors
56+
| Feature | Text Tokenization | Image "Tokenization" (ViT) |
57+
|---------|-------------------|---------------------------|
58+
| Basic Unit | Sub-word (BPE) | $16 \times 16$ Pixel Patch |
59+
| Sequence | List of word IDs | Grid of patch vectors |
60+
| Final Output | Text Embedding (d-dim) | Visual Embedding (d-dim) |
61+
| The Bridge | Contrastive Loss forces them to match in the Latent Space. | |
62+
63+
## What's Next?
64+
By bridging the gap between sight and language, CLIP allows machines to navigate the world more like we do—through concepts rather than just raw data.
65+

‎aigc/posts/image-tokenization/app.js‎

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
// Parse YAML frontmatter from markdown
2+
function parseFrontmatter(content) {
3+
const frontmatterRegex = /^---\n([\s\S]*?)\n---/;
4+
const match = content.match(frontmatterRegex);
5+
6+
if (!match) {
7+
return { metadata: {}, content: content };
8+
}
9+
10+
const frontmatterStr = match[1];
11+
const metadata = {};
12+
13+
// Simple YAML parser
14+
frontmatterStr.split('\n').forEach(line => {
15+
const [key, ...valueParts] = line.split(':');
16+
if (key && valueParts.length > 0) {
17+
let value = valueParts.join(':').trim();
18+
value = value.replace(/^["']|["']$/g, '');
19+
metadata[key.trim()] = value;
20+
}
21+
});
22+
23+
const bodyContent = content.replace(frontmatterRegex, '').trim();
24+
return { metadata, content: bodyContent };
25+
}
26+
27+
// Get the post filename from URL
28+
function getPostFilename() {
29+
const path = window.location.pathname;
30+
const parts = path.split('/');
31+
// Should be something like /aigc/posts/post-name/
32+
for (let i = 0; i < parts.length; i++) {
33+
if (parts[i] === 'posts' && i + 1 < parts.length) {
34+
return parts[i + 1];
35+
}
36+
}
37+
return null;
38+
}
39+
40+
// Format date
41+
function formatDate(dateStr) {
42+
try {
43+
const [year, month, day] = dateStr.split('-');
44+
const date = new Date(year, month - 1, day);
45+
return date.toLocaleDateString('en-US', { day: 'numeric', month: 'short', year: 'numeric' });
46+
} catch (e) {
47+
return dateStr;
48+
}
49+
}
50+
51+
// Load and render the post
52+
async function loadPost() {
53+
const filename = getPostFilename();
54+
if (!filename) {
55+
document.getElementById('post-container').innerHTML = '<p>Post not found</p>';
56+
return;
57+
}
58+
59+
console.log('Loading post:', filename);
60+
61+
try {
62+
// Fetch the markdown file from the posts directory (one level up from current post dir)
63+
const response = await fetch(`/aigc/posts/${filename}.md`);
64+
65+
if (!response.ok) {
66+
console.error('Failed to fetch markdown:', response.status, response.statusText);
67+
document.getElementById('post-container').innerHTML = `<p>Could not load post (${response.status})</p>`;
68+
return;
69+
}
70+
71+
const markdownContent = await response.text();
72+
console.log('Markdown loaded, length:', markdownContent.length);
73+
74+
const { metadata, content } = parseFrontmatter(markdownContent);
75+
console.log('Metadata:', metadata);
76+
77+
// Update page title
78+
if (metadata.title) {
79+
document.title = `ritchie@singapore~$ ${metadata.title}`;
80+
}
81+
82+
// Convert markdown to HTML - ensure marked is available
83+
let htmlContent;
84+
console.log('marked available:', typeof marked !== 'undefined');
85+
86+
if (typeof marked !== 'undefined' && marked.parse) {
87+
try {
88+
htmlContent = marked.parse(content);
89+
console.log('Markdown parsed successfully, HTML length:', htmlContent.length);
90+
} catch (e) {
91+
console.error('Error parsing markdown:', e);
92+
htmlContent = `<pre>${content}</pre>`;
93+
}
94+
} else {
95+
console.warn('marked.js not loaded, showing raw markdown');
96+
htmlContent = `<pre>${content}</pre>`;
97+
}
98+
99+
// Build the post header
100+
const headerHTML = `
101+
<div class="post-header">
102+
<h1 class="post-title">${metadata.title || 'Untitled'}</h1>
103+
<div class="post-meta">
104+
<div class="post-meta-item">
105+
<span class="post-meta-label">Published:</span> ${formatDate(metadata.date || 'Unknown')}
106+
</div>
107+
<div class="post-meta-item">
108+
<span class="post-meta-label">Category:</span> <span style="color: #00ff88;">${metadata.category || 'AIGC'}</span>
109+
</div>
110+
</div>
111+
</div>
112+
`;
113+
114+
// Insert the header and content
115+
const container = document.getElementById('post-container');
116+
container.innerHTML = headerHTML + htmlContent;
117+
118+
// Re-render MathJax if it's loaded
119+
if (typeof MathJax !== 'undefined' && MathJax.typesetPromise) {
120+
MathJax.typesetPromise([container]).catch(err => console.log('MathJax error:', err));
121+
}
122+
123+
} catch (error) {
124+
console.error('Error loading post:', error);
125+
document.getElementById('post-container').innerHTML = `<p>Error loading post: ${error.message}</p>`;
126+
}
127+
}
128+
129+
document.addEventListener('DOMContentLoaded', loadPost);

‎aigc/posts/image-tokenization/index.html‎

Lines changed: 195 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,195 @@
1+
<!DOCTYPE html>
2+
<html lang="en">
3+
<head>
4+
<meta charset="UTF-8">
5+
<meta name="viewport" content="width=device-width, initial-scale=1.0">
6+
<title>ritchie@singapore~$ AIGC Post</title>
7+
8+
<!-- Preload critical resources -->
9+
<link rel="preload" href="https://fonts.googleapis.com/css2?family=Fira+Code&display=swap" as="style" onload="this.onload=null;this.rel='stylesheet'">
10+
<noscript><link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Fira+Code&display=swap"></noscript>
11+
12+
<!-- Critical CSS loaded synchronously -->
13+
<link rel="stylesheet" href="/css/styles.css">
14+
15+
<!-- Load Marked.js for markdown rendering - load before app.js -->
16+
<script src="/lib/marked/marked.min.js"></script>
17+
18+
<!-- Defer heavy scripts to prevent render blocking -->
19+
<script src="/lib/tailwind/tailwind-cdn.js" defer></script>
20+
21+
<!-- MathJax Configuration - defer loading -->
22+
<script>
23+
window.MathJax = {
24+
tex: { inlineMath: [['$', '$'], ['\\(', '\\)']] },
25+
svg: { fontCache: 'global' }
26+
};
27+
</script>
28+
<script src="/lib/mathjax/tex-mml-chtml.js" defer></script>
29+
30+
<style>
31+
.post-content {
32+
color: #efefef;
33+
line-height: 1.8;
34+
}
35+
36+
.post-content h1,
37+
.post-content h2,
38+
.post-content h3,
39+
.post-content h4,
40+
.post-content h5,
41+
.post-content h6 {
42+
color: #f1c40f;
43+
margin-top: 1.5rem;
44+
margin-bottom: 0.5rem;
45+
font-weight: bold;
46+
}
47+
48+
.post-content h1 { font-size: 2rem; }
49+
.post-content h2 { font-size: 1.5rem; border-bottom: 1px solid #333; padding-bottom: 0.5rem; }
50+
.post-content h3 { font-size: 1.25rem; }
51+
52+
.post-content p {
53+
margin-bottom: 1rem;
54+
}
55+
56+
.post-content code {
57+
background-color: #1a1a1a;
58+
color: #00ff88;
59+
padding: 0.2rem 0.4rem;
60+
border-radius: 3px;
61+
font-size: 0.9em;
62+
font-family: 'Fira Code', monospace;
63+
}
64+
65+
.post-content pre {
66+
background-color: #1a1a1a;
67+
border-left: 3px solid #00ff88;
68+
padding: 1rem;
69+
border-radius: 4px;
70+
overflow-x: auto;
71+
margin: 1rem 0;
72+
font-family: 'Fira Code', monospace;
73+
}
74+
75+
.post-content pre code {
76+
background-color: transparent;
77+
color: #00ff88;
78+
padding: 0;
79+
border-radius: 0;
80+
}
81+
82+
.post-content blockquote {
83+
border-left: 4px solid #00ccff;
84+
padding-left: 1rem;
85+
color: #b0b0b0;
86+
margin: 1rem 0;
87+
font-style: italic;
88+
}
89+
90+
.post-content ul,
91+
.post-content ol {
92+
margin-left: 2rem;
93+
margin-bottom: 1rem;
94+
}
95+
96+
.post-content li {
97+
margin-bottom: 0.5rem;
98+
}
99+
100+
.post-content a {
101+
color: #00ccff;
102+
text-decoration: underline;
103+
}
104+
105+
.post-content a:hover {
106+
color: #00ff88;
107+
}
108+
109+
.post-content table {
110+
border-collapse: collapse;
111+
width: 100%;
112+
margin: 1rem 0;
113+
border: 1px solid #333;
114+
}
115+
116+
.post-content table th,
117+
.post-content table td {
118+
border: 1px solid #333;
119+
padding: 0.75rem;
120+
text-align: left;
121+
}
122+
123+
.post-content table th {
124+
background-color: #1a1a1a;
125+
color: #f1c40f;
126+
font-weight: bold;
127+
}
128+
129+
.post-content table tr:hover {
130+
background-color: #0a0a0a;
131+
}
132+
133+
.post-header {
134+
padding-bottom: 1.5rem;
135+
margin-bottom: 2rem;
136+
}
137+
138+
.post-header::after {
139+
content: '';
140+
display: block;
141+
width: 40px;
142+
height: 1px;
143+
background: #00ff88;
144+
margin-top: 1rem;
145+
}
146+
147+
.post-title {
148+
color: #f1c40f;
149+
font-size: 2.5rem;
150+
margin-bottom: 0;
151+
margin-top: 0;
152+
font-weight: 600;
153+
letter-spacing: -0.5px;
154+
}
155+
156+
.post-meta {
157+
display: flex;
158+
gap: 2rem;
159+
flex-wrap: wrap;
160+
margin-top: 1rem;
161+
font-size: 0.95rem;
162+
}
163+
164+
.post-meta-item {
165+
color: #00ccff;
166+
font-size: 0.9rem;
167+
font-family: 'Fira Code', monospace;
168+
}
169+
170+
.post-meta-label {
171+
color: #989898;
172+
font-weight: 500;
173+
}
174+
</style>
175+
</head>
176+
177+
<body>
178+
<div class="container">
179+
<div id="menu"></div>
180+
181+
<div id="post-container" class="post-content">
182+
<!-- Post content will be loaded here -->
183+
</div>
184+
185+
<div class="bash-line" style="margin-top: 2rem;">
186+
<span class="prompt">ritchie@singapore</span>:<span class="command">~/aigc/posts</span>$ <span class="cursor">|</span>
187+
</div>
188+
</div>
189+
190+
<!-- Load marked.js synchronously before app.js -->
191+
<script src="/lib/marked/marked.min.js"></script>
192+
<script src="/js/menu.js" defer></script>
193+
<script src="./app.js" defer></script>
194+
</body>
195+
</html>

0 commit comments

Comments
 (0)