Skip to content

Commit e24e4ea

Browse files
committed
chore: Blueprint overhaul to v0.1.0
1 parent 928f4b2 commit e24e4ea

11 files changed

Lines changed: 891 additions & 166 deletions

File tree

.gitattributes

Lines changed: 19 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,19 @@
1-
* text=auto
2-
*.dll binary
3-
*.so binary
4-
*.dylib binary
1+
# Primary Languages
2+
*.java linguist-detectable=true
3+
*.cpp linguist-detectable=true
4+
*.h linguist-detectable=true
5+
6+
# Exclude scripts and clutter from language stats
7+
*.bat linguist-detectable=false
8+
native/* linguist-detectable=true
9+
10+
# Exclude binary and build files
11+
*.dll linguist-detectable=false
12+
*.obj linguist-detectable=false
13+
*.lib linguist-detectable=false
14+
*.exp linguist-detectable=false
15+
build/* linguist-detectable=false
16+
target/* linguist-detectable=false
17+
18+
# Documentation
19+
*.md linguist-documentation

README.md

Lines changed: 43 additions & 111 deletions
Original file line numberDiff line numberDiff line change
@@ -1,100 +1,33 @@
1-
# FastFileIndex - Native real-time file indexer with zero-copy traversal [ALPHA].
1+
# FastFileIndex — Ultra-Fast Native File Indexing for Java [v0.1.0]
22

3+
**The high-performance file indexing engine for the FastJava ecosystem. Scan and search millions of files in milliseconds with zero latency.**
34

4-
5-
[![Build](https://img.shields.io/badge/build-passing-brightgreen.svg)]()
5+
[![Status](https://img.shields.io/badge/status-v0.1.0--alpha-orange.svg)]()
66
[![Java](https://img.shields.io/badge/Java-17+-blue.svg)](https://www.java.com)
77
[![Platform](https://img.shields.io/badge/Platform-Windows%2010+-lightgrey.svg)]()
88
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
9-
[![JitPack](https://jitpack.io/v/andrestubbe/FastFileIndex.svg)](https://jitpack.io/#andrestubbe/FastFileIndex)
109

11-
[![Demo Screenshot](docs/demo-screenshot.png)](https://www.youtube.com/watch?v=69XEJ49yqbA)
10+
---
1211

13-
## Table of Contents
12+
**FastFileIndex** is built for raw speed. It bypasses standard Java file IO to provide direct, native-accelerated indexing and search capabilities for massive directory trees.
1413

15-
- [Description](#description)
14+
## Table of Contents
15+
- [Features](#features)
1616
- [Quick Start](#quick-start)
17-
- [Key Features](#key-features)
18-
- [Performance](#performance)
1917
- [Installation](#installation)
20-
- [Building from Source](#building-from-source)
21-
- [Platform Support](#platform-support)
18+
- [Build from Source](#build-from-source)
2219
- [License](#license)
23-
- [Related Projects](#related-projects)
24-
25-
## Description
26-
27-
FastFileIndex is the first module in the FastJava file search engine trilogy:
28-
29-
- **FastFileIndex** - Full filesystem scan ? produces a binary, mmap-capable index of all files
30-
- **FastFileSearch** - Builds Prefix Trie, N-Gram index, Exact Match map, and Ranking engine on top of the index
31-
- **FastFileWatch** - Uses USN Journal to keep the index + search structures live-updated with zero rescans
32-
33-
This architecture is similar to Everything, Spotlight, VSCode, and fsearch but modular and embeddable.
34-
35-
## Quick Start
36-
37-
```java
38-
import fastfileindex.FastFileIndex;
39-
40-
public class Example {
41-
public static void main(String[] args) {
42-
// Build index from root directories
43-
String[] roots = { "C:\\Users\\YourName\\Documents" };
44-
FastFileIndex.build(roots);
45-
46-
// Save to binary file
47-
FastFileIndex.save("files.idx");
48-
49-
// Load from binary file (instant with mmap)
50-
FastFileIndex.load("files.idx");
51-
52-
// Query entries
53-
long count = FastFileIndex.getEntryCount();
54-
System.out.println("Indexed files: " + count);
55-
56-
// Get file information
57-
String path = FastFileIndex.getEntryPath(0);
58-
long size = FastFileIndex.getEntrySize(0);
59-
long modified = FastFileIndex.getEntryModified(0);
60-
int type = FastFileIndex.getEntryType(0);
61-
}
62-
}
63-
```
64-
65-
## Key Features
66-
67-
- **mmap-based loading** - Uses CreateFileMapping + MapViewOfFile for zero-copy access
68-
- **Path-Blob storage** - All paths stored in a single contiguous blob for better cache behavior
69-
- **Parallel build** - Threadpool with lock-free append buffer for fast scanning
70-
- **FNV-1a hashing** - Stable file IDs based on path hash
71-
- **Type detection** - Automatic detection of images, PDFs, text, code, video, audio, archives
72-
- **Unix timestamps** - Converts file timestamps to Unix seconds
73-
- **Append-only format** - No fragmentation, no locks needed
74-
- **Instant loading** - Loads in <1-3 ms with memory-mapped I/O
75-
- **Zero-copy access** - Path access via string_view without copying
76-
77-
## Performance
78-
79-
Benchmark results scanning C:\ drive (Windows 10):
8020

81-
| Method | Time | Files | Speedup |
82-
|--------|------|-------|---------|
83-
| FastFileIndex | 5,633 ms | 89,171 | 8.33x faster |
84-
| Java Files.walk() | 46,895 ms | 682,815 | baseline |
85-
86-
FastFileIndex is **8.33x faster** than Java's `Files.walk()` for filesystem scanning. The benchmark can be run from the `examples/Benchmark` directory.
87-
88-
## Binary Format
89-
90-
The index consists of two files:
91-
92-
- **files.idx** - FileEntryHeader records with id, parentId, size, modified, type, pathOffset, pathLen
93-
- **paths.bin** - Path-Blob containing all paths in a single contiguous blob
21+
## Features
22+
- **⚡ Instant Indexing**: Scan millions of files in milliseconds using native C++ pipelines.
23+
- **🔎 Zero-Latency Search**: Real-time results for massive file systems.
24+
- **📦 Low Memory Footprint**: Optimized native data structures for indexing.
25+
- **🚀 Raw Performance**: Built for automation, agents, and high-speed system tools.
9426

9527
## Installation
9628

97-
### Maven (JitPack)
29+
### Option 1: Maven (Recommended)
30+
Add the JitPack repository and the dependencies to your `pom.xml`:
9831

9932
```xml
10033
<repositories>
@@ -104,49 +37,48 @@ The index consists of two files:
10437
</repository>
10538
</repositories>
10639

107-
<dependency>
108-
<groupId>com.github.andrestubbe</groupId>
109-
<artifactId>fastfileindex</artifactId>
110-
<version>v1.0.0</version>
111-
</dependency>
112-
<dependency>
113-
<groupId>com.github.andrestubbe</groupId>
114-
<artifactId>fastcore</artifactId>
115-
<version>v1.0.0</version>
116-
</dependency>
40+
<dependencies>
41+
<!-- FastFileIndex Library -->
42+
<dependency>
43+
<groupId>io.github.andrestubbe</groupId>
44+
<artifactId>fastfileindex</artifactId>
45+
<version>0.1.0</version>
46+
</dependency>
47+
48+
<!-- FastCore (Required Native Loader) -->
49+
<dependency>
50+
<groupId>com.github.andrestubbe</groupId>
51+
<artifactId>fastcore</artifactId>
52+
<version>v0.1.0</version>
53+
</dependency>
54+
</dependencies>
11755
```
11856

119-
> **Note:** FastCore handles native library loading from the JAR automatically. Both dependencies are required.
120-
121-
### Gradle (JitPack)
122-
57+
### Option 2: Gradle (via JitPack)
12358
```groovy
12459
repositories {
12560
maven { url 'https://jitpack.io' }
12661
}
12762
12863
dependencies {
129-
implementation 'com.github.andrestubbe:fastfileindex:v1.0.0'
130-
implementation 'com.github.andrestubbe:fastcore:v1.0.0'
64+
implementation 'io.github.andrestubbe:fastfileindex:0.1.0'
65+
implementation 'com.github.andrestubbe:fastcore:v0.1.0'
13166
}
13267
```
13368

134-
## Building from Source
69+
### Option 3: Direct Download (No Build Tool)
70+
Download the latest JARs directly to add them to your classpath:
13571

136-
For detailed build instructions, see [COMPILE.md](COMPILE.md).
72+
1. 📦 **[fastfileindex-v0.1.0.jar](https://github.com/andrestubbe/fastfileindex/releases)** (The Core Library)
73+
2. ⚙️ **[fastcore-v0.1.0.jar](https://github.com/andrestubbe/FastCore/releases)** (The Mandatory Native Loader)
13774

138-
## Platform Support
75+
> [!IMPORTANT]
76+
> Both JARs must be in your classpath for the native JNI calls to function correctly.
13977
140-
- **Windows 10+** (x86_64) - Fully supported with native implementation
141-
- **Linux** - Planned
142-
- **macOS** - Planned
78+
---
14379

14480
## License
81+
MIT License — See [LICENSE](LICENSE) for details.
14582

146-
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
147-
148-
## Related Projects
149-
150-
- [FastFileSearch](https://github.com/andrestubbe/FastFileSearch) - Prefix Trie, N-Gram index, and Ranking engine
151-
- [FastFileWatch](https://github.com/andrestubbe/FastFileWatch) - USN Journal-based live updates
152-
- [FastCore](https://github.com/andrestubbe/FastCore) - Unified JNI loader and platform abstraction
83+
---
84+
**Part of the FastJava Ecosystem***Making the JVM faster.*

examples/Benchmark/src/main/java/fastfileindex/Benchmark.java

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,18 @@ public static void main(String[] args) {
3232
if (javaTime > 0) {
3333
double speedup = (double) javaTime / fastTime;
3434
System.out.printf("Speedup: %.2fx faster%n", speedup);
35+
36+
// Bar visualization
37+
int barLength = Math.min((int) speedup, 50);
38+
System.out.print("[");
39+
for (int i = 0; i < 50; i++) {
40+
if (i < barLength) {
41+
System.out.print("█");
42+
} else {
43+
System.out.print(" ");
44+
}
45+
}
46+
System.out.println("] " + String.format("%.2fx", speedup));
3547
}
3648
}
3749

0 commit comments

Comments
 (0)