Skip to content

Commit 7735d85

Browse files
andrestubbeandrestubbe
committed
Initial commit - FastFileIndex v1.0.0
0 parents  commit 7735d85

14 files changed

Lines changed: 885 additions & 0 deletions

File tree

‎.gitattributes‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
* text=auto
2+
*.dll binary
3+
*.so binary
4+
*.dylib binary

‎.github/workflows/maven.yml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
name: Maven CI
2+
3+
on:
4+
push:
5+
branches: [ main ]
6+
pull_request:
7+
branches: [ main ]
8+
9+
jobs:
10+
build:
11+
runs-on: windows-latest
12+
13+
steps:
14+
- uses: actions/checkout@v4
15+
16+
- name: Set up JDK 17
17+
uses: actions/setup-java@v4
18+
with:
19+
java-version: '17'
20+
distribution: 'temurin'
21+
22+
- name: Build with Maven
23+
run: mvn -B verify

‎.gitignore‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Build outputs
2+
target/
3+
build/
4+
*.dll
5+
*.obj
6+
*.exp
7+
*.lib
8+
9+
# IDE
10+
.vscode/
11+
.idea/
12+
*.iml
13+
14+
# OS
15+
.DS_Store
16+
Thumbs.db
17+
18+
# Maven
19+
.mvn/
20+
mvnw
21+
mvnw.cmd

‎CHANGELOG.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# Changelog
2+
3+
All notable changes to this project will be documented in this file.
4+
5+
## [v1.0.0] - 2026-04-25
6+
7+
### Added
8+
- Initial release
9+
- Native JNI implementation for FastFileIndex
10+
- Binary file indexing with memory-mapped I/O support
11+
- FNV-1a hashing for stable file IDs
12+
- Type detection for common file types
13+
- Full filesystem scanning with parallel support
14+
- Maven build configuration
15+
- GitHub Actions CI/CD

‎COMPILE.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# Compile Guide
2+
3+
## Requirements
4+
5+
- Java JDK 17+
6+
- Visual Studio Build Tools (for C++ compilation)
7+
- Maven 3.6+
8+
9+
## Native Compilation
10+
11+
### Windows
12+
13+
```bash
14+
compile.bat
15+
```
16+
17+
This script:
18+
1. Loads Visual Studio Build Tools environment
19+
2. Compiles C++ source to DLL
20+
3. Copies DLL to resources directory
21+
22+
### Manual Compilation
23+
24+
```bash
25+
call "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\Build\vcvars64.bat"
26+
cl.exe /LD /Fe:build\fastfileindex.dll native\FastFileIndex.cpp /I"C:\Program Files\Java\jdk-25\include" /I"C:\Program Files\Java\jdk-25\include\win32" /std:c++17 /DEF:native\FastFileIndex.def
27+
copy build\fastfileindex.dll src\main\resources\native\
28+
```
29+
30+
## Maven Build
31+
32+
```bash
33+
mvn clean package
34+
```
35+
36+
This creates:
37+
- `target/fastfileindex-v1.0.0.jar` - Standard JAR
38+
- `target/fastfileindex-v1.0.0-jar-with-dependencies.jar` - FatJAR with all dependencies and native DLL
39+
40+
## Running
41+
42+
```bash
43+
java -jar target/fastfileindex-v1.0.0-jar-with-dependencies.jar
44+
```
45+
46+
**Note:** When running the demo, ensure you are in the `build` directory for DLL loading:
47+
```bash
48+
cd build
49+
java -cp ..\target\fastfileindex-v1.0.0-jar-with-dependencies.jar;..\examples\Demo\src\main\java fastfileindex.Demo
50+
```

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2026 Andre Stubbe
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 176 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,176 @@
1+
# FastFileIndex
2+
3+
[![Build](https://img.shields.io/badge/build-passing-brightgreen.svg)]()
4+
[![Java](https://img.shields.io/badge/Java-17+-blue.svg)](https://www.java.com)
5+
[![Platform](https://img.shields.io/badge/Platform-Windows%2010+-lightgrey.svg)]()
6+
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
7+
[![JitPack](https://jitpack.io/v/andrestubbe/FastFileIndex.svg)](https://jitpack.io/#andrestubbe/FastFileIndex)
8+
9+
## Table of Contents
10+
11+
- [Description](#description)
12+
- [Quick Start](#quick-start)
13+
- [Key Features](#key-features)
14+
- [Installation](#installation)
15+
- [Building from Source](#building-from-source)
16+
- [Platform Support](#platform-support)
17+
- [License](#license)
18+
- [Related Projects](#related-projects)
19+
20+
## Description
21+
22+
FastFileIndex creates a compact, append-only, mmap-friendly binary index of all files. It is the first module in the FastJava file search engine trilogy:
23+
24+
- **FastFileIndex** - Full filesystem scan ? produces a binary, mmap-capable index of all files
25+
- **FastFileSearch** - Builds Prefix Trie, N-Gram index, Exact Match map, and Ranking engine on top of the index
26+
- **FastFileWatch** - Uses USN Journal to keep the index + search structures live-updated with zero rescans
27+
28+
This architecture is similar to Everything, Spotlight, VSCode, and fsearch but modular and embeddable.
29+
30+
## Quick Start
31+
32+
```java
33+
import fastfileindex.FastFileIndex;
34+
35+
public class Example {
36+
public static void main(String[] args) {
37+
// Build index from root directories
38+
String[] roots = { "C:\\Users\\YourName\\Documents" };
39+
FastFileIndex.build(roots);
40+
41+
// Save to binary file
42+
FastFileIndex.save("files.idx");
43+
44+
// Load from binary file (instant with mmap)
45+
FastFileIndex.load("files.idx");
46+
47+
// Query entries
48+
long count = FastFileIndex.getEntryCount();
49+
System.out.println("Indexed files: " + count);
50+
51+
// Get file information
52+
String path = FastFileIndex.getEntryPath(0);
53+
long size = FastFileIndex.getEntrySize(0);
54+
long modified = FastFileIndex.getEntryModified(0);
55+
int type = FastFileIndex.getEntryType(0);
56+
}
57+
}
58+
```
59+
60+
## Key Features
61+
62+
- **mmap-based loading** - Uses CreateFileMapping + MapViewOfFile for zero-copy access
63+
- **Path-Blob storage** - All paths stored in a single contiguous blob for better cache behavior
64+
- **Parallel build** - Threadpool with lock-free append buffer for fast scanning
65+
- **FNV-1a hashing** - Stable file IDs based on path hash
66+
- **Type detection** - Automatic detection of images, PDFs, text, code, video, audio, archives
67+
- **Unix timestamps** - Converts file timestamps to Unix seconds
68+
- **Append-only format** - No fragmentation, no locks needed
69+
- **Instant loading** - Loads in <1-3 ms with memory-mapped I/O
70+
- **Zero-copy access** - Path access via string_view without copying
71+
72+
## Binary Format
73+
74+
The index consists of two files:
75+
76+
- **files.idx** - FileEntryHeader records with id, parentId, size, modified, type, pathOffset, pathLen
77+
- **paths.bin** - Path-Blob containing all paths in a single contiguous blob
78+
79+
## Installation
80+
81+
### Maven
82+
83+
Add JitPack repository to your `pom.xml`:
84+
85+
```xml
86+
<repositories>
87+
<repository>
88+
<id>jitpack.io</id>
89+
<url>https://jitpack.io</url>
90+
</repository>
91+
</repositories>
92+
```
93+
94+
Add dependency:
95+
96+
```xml
97+
<dependency>
98+
<groupId>com.github.andrestubbe</groupId>
99+
<artifactId>fastfileindex</artifactId>
100+
<version>v1.0.0</version>
101+
</dependency>
102+
```
103+
104+
### Gradle
105+
106+
Add JitPack repository:
107+
108+
```groovy
109+
repositories {
110+
maven { url 'https://jitpack.io' }
111+
}
112+
```
113+
114+
Add dependency:
115+
116+
```groovy
117+
implementation 'com.github.andrestubbe:fastfileindex:v1.0.0'
118+
version>'
119+
```
120+
121+
## Building from Source
122+
123+
### Requirements
124+
125+
- Java JDK 17+
126+
- Visual Studio Build Tools (for C++ compilation on Windows)
127+
- Maven 3.6+
128+
129+
### Native Compilation
130+
131+
```bash
132+
compile.bat
133+
```
134+
135+
This script:
136+
1. Loads Visual Studio Build Tools environment
137+
2. Compiles C++ source to DLL
138+
3. Copies DLL to resources directory
139+
140+
### Maven Build
141+
142+
```bash
143+
mvn clean package
144+
```
145+
146+
This creates:
147+
- `target/fastfileindex-v1.0.0.jar` - Standard JAR
148+
- `target/fastfileindex-v1.0.0-jar-with-dependencies.jar` - FatJAR with all dependencies and native DLL
149+
150+
### Running
151+
152+
```bash
153+
java -jar target/fastfileindex-v1.0.0-jar-with-dependencies.jar
154+
```
155+
156+
**Note:** When running the demo, ensure you are in the `build` directory for DLL loading:
157+
```bash
158+
cd build
159+
java -cp ..\target\fastfileindex-v1.0.0-jar-with-dependencies.jar;..\examples\Demo\src\main\java fastfileindex.Demo
160+
```
161+
162+
## Platform Support
163+
164+
- **Windows 10+** (x86_64) - Fully supported with native implementation
165+
- **Linux** - Planned
166+
- **macOS** - Planned
167+
168+
## License
169+
170+
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
171+
172+
## Related Projects
173+
174+
- [FastFileSearch](https://github.com/andrestubbe/FastFileSearch) - Prefix Trie, N-Gram index, and Ranking engine
175+
- [FastFileWatch](https://github.com/andrestubbe/FastFileWatch) - USN Journal-based live updates
176+
- [FastCore](https://github.com/andrestubbe/FastCore) - Unified JNI loader and platform abstraction

‎compile.bat‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
@echo off
2+
call "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\Build\vcvars64.bat"
3+
cl.exe /LD /EHsc /Fe:build\fastfileindex.dll native\FastFileIndex.cpp /I"C:\Program Files\Java\jdk-25\include" /I"C:\Program Files\Java\jdk-25\include\win32" /std:c++17 /DEF:native\FastFileIndex.def
4+
copy build\fastfileindex.dll src\main\resources\native\
5+
echo Compilation complete

‎examples/Demo/src/main/java/fastfileindex/Demo.class‎

1.52 KB
Binary file not shown.

‎examples/Demo/src/main/java/fastfileindex/Demo.java‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
package fastfileindex;
2+
3+
import fastfileindex.FastFileIndex;
4+
5+
/**
6+
* Demo class for testing FastFileIndex functionality.
7+
* This class demonstrates usage and validates the implementation.
8+
*/
9+
public class Demo {
10+
public static void main(String[] args) {
11+
System.out.println("=== FastFileIndex Demo ===");
12+
13+
// Test basic functionality
14+
String[] roots = { "C:\\Users\\andre\\Documents" };
15+
FastFileIndex.build(roots);
16+
17+
String indexPath = "test-index.idx";
18+
FastFileIndex.save(indexPath);
19+
20+
FastFileIndex.load(indexPath);
21+
22+
long count = FastFileIndex.getEntryCount();
23+
System.out.println("Indexed files: " + count);
24+
25+
if (count > 0) {
26+
System.out.println("First file: " + FastFileIndex.getEntryPath(0));
27+
System.out.println("Size: " + FastFileIndex.getEntrySize(0) + " bytes");
28+
System.out.println("Type: " + FastFileIndex.getEntryType(0));
29+
}
30+
31+
System.out.println("=== Demo Complete ===");
32+
}
33+
}

0 commit comments

Comments
 (0)