Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
5a4a043
Keep track of the number of messages in the writeFile and the number …
kev1n80 May 24, 2021
15339eb
Update tests to reflect changes made to the metadata file. Metadata f…
kev1n80 May 24, 2021
ddadcb9
Test thtat write messages resets when a new file is created.
kev1n80 May 24, 2021
609f5c9
Add a test that checks if meta data file is correct after writing a c…
kev1n80 May 24, 2021
cd45da2
Allocate 8 bytes instead of 4 since writeMessages is in64, and reset …
kev1n80 May 27, 2021
eeaa65a
Update comment about number of bytes reserved for writeMessages.
kev1n80 May 27, 2021
2496eff
Allow user to choose whether to use new feature or not.
kev1n80 May 27, 2021
b35c029
Separate testing from the original with the implementation of depth f…
kev1n80 May 27, 2021
5cdd0ae
Revert to original code style.
kev1n80 May 27, 2021
fcb31cc
Add comment for functions.
kev1n80 May 27, 2021
c90f17f
Explain that maxBytesDiskSpace is 0 when user is not using disk space…
kev1n80 May 27, 2021
a28e6d5
Add explicit disk size limit feature flag.
kev1n80 Jun 1, 2021
293a774
Add a new line after if block.
kev1n80 Jun 1, 2021
4c956aa
Track the disk space the files tracked by DiskQueue takes up, and tes…
kev1n80 Jun 1, 2021
48c7a85
Update totalBytes and writePos when writing number of messages, and r…
kev1n80 Jun 1, 2021
652b548
Remove the additional 8 bytes to totalBytes.
kev1n80 Jun 1, 2021
b07cec4
Revert "Remove the additional 8 bytes to totalBytes."
kev1n80 Jun 2, 2021
5968b13
Revert "Update totalBytes and writePos when writing number of message…
kev1n80 Jun 2, 2021
53c0e47
Revert changes that removed the additional 8 bytes.
kev1n80 Jun 2, 2021
a3ce86a
Add comment to make code more readable.
kev1n80 Jun 2, 2021
3ed7338
Add test that completes the write file by meeting the file size limit…
kev1n80 Jun 2, 2021
30c6ef6
Add extra testing to validate the increment/decrement of bytes in cor…
kev1n80 Jun 2, 2021
76a0ddd
Merge branch 'TrackDiskSize' into DepthImpl
kev1n80 Jun 2, 2021
e9b23ac
Update variable names.
kev1n80 Jun 2, 2021
7acba5f
Revert "Update variable names."
kev1n80 Jun 2, 2021
7032086
Merge branch 'DepthImpl' of https://github.com/kev1n80/go-diskqueue i…
kev1n80 Jun 2, 2021
92819e1
Revert "Merge branch 'TrackDiskSize' into DepthImpl"
kev1n80 Jun 2, 2021
2616c7e
Update variable names.
kev1n80 Jun 2, 2021
5cd6288
Replace 8 with a constant to improve readability.
kev1n80 Jun 3, 2021
24fe56a
Reset Diskqueue data during readError.
kev1n80 Jun 9, 2021
5d83e9b
Merge pull request #1 from kev1n80/DepthImpl
kev1n80 Jun 9, 2021
6525bb0
Track disk size (#5)
kev1n80 Jun 14, 2021
5c1f588
Update go.mod
CatherineF-dev Jun 16, 2021
78955ac
Disk size limit (#6)
kev1n80 Jul 8, 2021
c7bc524
Update README.md
kev1n80 Jul 22, 2021
90e0163
Update README.md
kev1n80 Jul 23, 2021
adc5610
Read correct number of messages from end of file (#7)
leonzz Jul 23, 2021
6f62ded
Update Readme and add description for the fork.
leonzz Jul 26, 2021
649245b
Update \d\d\d\d\d\d to \d+
CatherineF-dev Aug 4, 2021
bb13b3d
Merge pull request #8 from CatherineF-dev/master
CatherineF-dev Aug 4, 2021
9ac83ef
Add PeekChan() (#14)
kev1n80 Sep 22, 2023
2bab43a
Fix diskqueue_test.go (#15)
wentingg2023 Sep 26, 2023
71f60e7
Fixed issue with shared regexps (#10)
moredure Nov 30, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 41 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,47 @@
# go-diskqueue

[![Build Status](https://secure.travis-ci.org/nsqio/go-diskqueue.png?branch=master)](http://travis-ci.org/nsqio/go-diskqueue) [![GoDoc](https://godoc.org/github.com/nsqio/go-diskqueue?status.svg)](https://godoc.org/github.com/nsqio/go-diskqueue) [![GitHub release](https://img.shields.io/github/release/nsqio/go-diskqueue.svg)](https://github.com/nsqio/go-diskqueue/releases/latest)
This is a fork of https://github.com/nsqio/go-diskqueue with the additional support of total disk space limit.

[![Build Status](https://travis-ci.com/kev1n80/go-diskqueue.svg?branch=master)](https://travis-ci.com/github/kev1n80/go-diskqueue) [![Go Reference](https://pkg.go.dev/badge/github.com/kev1n80/go-diskqueue.svg)](https://pkg.go.dev/github.com/kev1n80/go-diskqueue) [![GitHub release](https://img.shields.io/github/release/kev1n80/go-diskqueue.svg)](https://github.com/kev1n80/go-diskqueue/releases/latest)

A Go package providing a filesystem-backed FIFO queue

Pulled out of https://github.com/nsqio/nsq

# Description
Diskqueue is a synchronized "filesystem-backed FIFO queue” meaning it will store data you pass in by writing them to file.

Diskqueue writes messages and their message length to files in the order: message length in binary and then message. This allows Diskqueue to know how much of the file to read in order to get the next message. Once Diskqueue reads a file completely (when the number of bytes read surpasses the size of the file), it deletes the file.

In terms of threads, creating a Diskqueue object starts a “worker thread” by calling the private function ioLoop, which is a continuous loop that accepts requests to read, write, empty, get depth, or exit. This worker thread DOES NOT create other worker threads to handle tasks asynchronously. It is important to note that Diskqueue will sync if needed (i.e. set by sync flag after user retrieves read data) before handling a new request. Using a public function can be seen as creating a request to the Diskqueue object’s “worker thread” which is implemented by using Channels.

# Disk Space limit Feature
The original DiskQueue package did not contain a disk size limit feature; however, this forked repo does! By using a separate constructor `NewWithDiskSpace`, the user can use this disk space limit feature which will delete the oldest files in order to create space for new, incoming data.

In order to accurately adjust `depth` when a file is deleted, DiskQueue will store the number of messages in a file by writing this number to the end of the file. That way we can access this number and decrement `depth` accordingly.

Note: The disk size limit must be greater than 56 bytes which is reserved for the meta data file.

# Public Functions

## Put([]byte) error
Add data to the queue, and if a failure occurs none of the data will be written.

## ReadChan() <-chan []byte
This is expected to be an *unbuffered* channel that will not close until `Close()` or `Delete()` is called.

## Close() error
Cleans up the queue and persists the current state to metadata.

## Delete() error
Cleans up the queue, but does not save the current state to metadata.

## Depth() int64
Returns the number of data in the queue; however, this number can become inaccurate if a file becomes corrupted or unaccessible.
Although there are times when this number can be inaccurate, this number will always be 0 when there is nothing in the queue due to the `checkTailCorruption(depth int64)` private function.

## Empty() error
Empties out the queue by deleting all of the files containing data.

## TotalBytesFolderSize() int64
Returns the total number of bytes the content in the targeted folder take up.
Loading