Skip to content

Commit c8444e8

Browse files
committed
Update README.md
1 parent 92d7836 commit c8444e8

File tree

1 file changed

+26
-3
lines changed

1 file changed

+26
-3
lines changed

README.md

Lines changed: 26 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,13 +6,37 @@ Robots.txt checker
66
[![Test Coverage](https://codeclimate.com/github/tomverran/robots/badges/coverage.svg)](https://codeclimate.com/github/tomverran/robots/coverage)
77
[![Code Climate](https://codeclimate.com/github/tomverran/robots/badges/gpa.svg)](https://codeclimate.com/github/tomverran/robots)
88

9+
910
Given a robots.txt file this library will give you a straight forward yes/no as to whether you're allowed to access
1011
a given resource with a given user agent.
1112

12-
Wildcards are supported.
13+
This library has been built with reference to both [Google's robots.txt specification](https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt) and the draft [norobots RFC](http://www.robotstxt.org/norobots-rfc.txt) from 1996(!). As such all the following features are supported:
14+
15+
- Wildcards in paths, including across directory boundaries
16+
- The '$' suffix to anchor a path match to the end of a URL
17+
- Sorting of multiple matching user-agent/path blocks by user-agent length
18+
- Decoding of URL encoded paths, with the exception of forward slashes
19+
- A trailing slash on a path indicating that all files under that directory should match
1320

1421
If you find any bugs with this library please don't hesitate to let me know, either create an issue on GitHub or submit a pull request.
1522

23+
Installation
24+
------------
25+
26+
Either type the following into your terminal:
27+
28+
```bash
29+
composer require tomverran/robots-txt-checker
30+
```
31+
32+
or add the following to your composer.json:
33+
34+
```json
35+
"require": {
36+
"tomverran/robots-txt-checker": "^1.12"
37+
}
38+
```
39+
1640
Example Usage
1741
-------------
1842

@@ -28,7 +52,7 @@ License
2852

2953
tl;dr MIT license
3054

31-
Copyright (c) 2014 Tom Verran
55+
Copyright (c) 2014, 2016 Tom Verran
3256

3357
Permission is hereby granted, free of charge, to any person obtaining a copy
3458
of this software and associated documentation files (the "Software"), to deal
@@ -47,4 +71,3 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
4771
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
4872
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
4973
THE SOFTWARE.
50-

0 commit comments

Comments
 (0)