You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Given a robots.txt file this library will give you a straight forward yes/no as to whether you're allowed to access
10
11
a given resource with a given user agent.
11
12
12
-
Wildcards are supported.
13
+
This library has been built with reference to both [Google's robots.txt specification](https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt) and the draft [norobots RFC](http://www.robotstxt.org/norobots-rfc.txt) from 1996(!). As such all the following features are supported:
14
+
15
+
- Wildcards in paths, including across directory boundaries
16
+
- The '$' suffix to anchor a path match to the end of a URL
17
+
- Sorting of multiple matching user-agent/path blocks by user-agent length
18
+
- Decoding of URL encoded paths, with the exception of forward slashes
19
+
- A trailing slash on a path indicating that all files under that directory should match
13
20
14
21
If you find any bugs with this library please don't hesitate to let me know, either create an issue on GitHub or submit a pull request.
15
22
23
+
Installation
24
+
------------
25
+
26
+
Either type the following into your terminal:
27
+
28
+
```bash
29
+
composer require tomverran/robots-txt-checker
30
+
```
31
+
32
+
or add the following to your composer.json:
33
+
34
+
```json
35
+
"require": {
36
+
"tomverran/robots-txt-checker": "^1.12"
37
+
}
38
+
```
39
+
16
40
Example Usage
17
41
-------------
18
42
@@ -28,7 +52,7 @@ License
28
52
29
53
tl;dr MIT license
30
54
31
-
Copyright (c) 2014 Tom Verran
55
+
Copyright (c) 2014, 2016 Tom Verran
32
56
33
57
Permission is hereby granted, free of charge, to any person obtaining a copy
34
58
of this software and associated documentation files (the "Software"), to deal
@@ -47,4 +71,3 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
47
71
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
48
72
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
0 commit comments