Merge pull request #39 from brootware/dev

Dev

Author: brootware

.flake8

@@ -0,0 +1,2 @@
+[flake8]
+max-line-length = 300

.github/workflows/ci.yml

@@ -5,6 +5,7 @@ on:
     branches: [dev]
   pull_request:
     branches: [main]
+    types: [opened, synchronize, reopened]
 
 permissions:
   contents: read
@@ -17,6 +18,8 @@ jobs:
         python-version: ["3.8", "3.9", "3.10"]
     steps:
       - uses: actions/checkout@v2
+        with:
+          fetch-depth: 0 # Shallow clones should be disabled for a better relevancy of analysis
 
       - name: Set up Python ${{ matrix.python-version }}
         uses: actions/setup-python@v2
@@ -25,22 +28,24 @@ jobs:
 
       - name: Install dependencies
         run: |
-          python -m pip install --upgrade pip
           python -m pip install --upgrade poetry
-          pip install flake8 pytest nltk numpy
           poetry install
-          python tools/install_nltk_popular.py
 
       - name: Lint with flake8
         run: |
           # stop the build if there are Python syntax errors or undefined names
-          flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
+          poetry run flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
           # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
-          flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
+          poetry run flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
 
-      - name: Test with pytest
-        run: |
-          pytest
+      - name: Run tox with pytest
+        run: poetry run tox -e py
+
+      - name: SonarCloud Scan
+        uses: SonarSource/sonarcloud-github-action@master
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # Needed to get PR information, if any
+          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
 
       # - name: Build and push Docker Image
       #   uses: mr-smithers-excellent/docker-build-push@v5

README.md

@@ -21,7 +21,6 @@
 
 Redacts and Unredacts the following from your text files. 📄 ✍️
 
-- names 👤
 - sg nric 🆔
 - credit cards 🏧
 - domain names 🌐
@@ -46,6 +45,12 @@ Quick install
 python -m pip install pyredactkit
 ```
 
+Redact from terminal
+
+```bash
+pyredactkit 'this is my ip:127.0.0.1. my email is [email protected]. secret link is github.com'
+```
+
 Redact a single file
 
 ```bash
@@ -55,10 +60,16 @@ pyredactkit test.txt
 Unredact the file
 
 ```bash
-pyredactkit redacted_test.txt -u .hashshadow_test.txt.json 
+pyredactkit -f redacted_test.txt -u .hashshadow_test.txt.json 
 ```
 
-Install nltk data for redacting names
+Redact using custom regex pattern
+
+```bash
+pyredactkit -f file -c custom.json
+```
+
+<!-- Install nltk data for redacting names
 
 ```bash
 python -c "import nltk
@@ -74,11 +85,11 @@ else:
 nltk.download('popular')"
 ```
 
-Redact names from text file
+Redact names from a text file
 
 ```bash
 pyredactkit test.txt -t name
-```
+``` -->
 
 ### Use from github source
 
@@ -190,12 +201,6 @@ e0b66cbd-6174-4491-b938-408a47d38fb9,Platinum,142000,CC90518
 24f31233-cba6-4f6a-a2d6-0ce49952b2cb,Premium,781000,CC66746
 ```
 
-To redact specific type of data. E.g (name)
-
-```bash
-poetry run pyredactkit test.txt -t name
-```
-
 Sample result:
 
 ```txt
@@ -217,7 +222,7 @@ My router is: 10.10.10.1
 71.159.188.33
 ```
 
-To redact multiple files from a directory and place it in a new directory
+To redact multiple files from a directory and place them in a new directory
 
 ```bash
 poetry run pyredactkit dir_test -d redacted_dir
@@ -226,28 +231,28 @@ poetry run pyredactkit dir_test -d redacted_dir
 ## Optional Help Menu as below
 
 ```bash
-usage: pyredactkit [-h] [-u UNREDACT] [-t REDACTIONTYPE] [-d DIROUT] [-r] [-e EXTENSION] file [file ...]
+usage: pyredactkit [-h] [-f FILE [FILE ...]] [-u UNREDACT] [-d DIROUT] [-r] [-e EXTENSION] [text ...]
 
-Read in a file or set of files, and return the result.
+Supply a sentence or paragraph to redact sensitive data from it. Or read in a file or set of files with -f , and return the result.
 
 positional arguments:
-  file                  Path of a file or a directory of files. Usage: pyredactkit [file/filestoredact]
+  text                  Redact sensitive data of a sentence from command prompt. (default: None)
 
 optional arguments:
   -h, --help            show this help message and exit
+  -f FILE [FILE ...], --file FILE [FILE ...]
+                        Path of a file or a directory of files. Usage: pyredactkit [file/filestoredact] (default: None)
   -u UNREDACT, --unredact UNREDACT
-                        Option to unredact masked data. Usage: pyredactkit [redacted_file] -u [.hashshadow.json] (default: None)
-  -t REDACTIONTYPE, --redactiontype REDACTIONTYPE
-                        Type of data to redact. names, nric, dns, emails, ipv4, ipv6, base64. Usage: pyredactkit [file/filestoredact] -t ip (default: None)
+                        Option to unredact masked data. Usage: pyredactkit -f [redacted_file] -u [.hashshadow.json] (default: None)
   -d DIROUT, --dirout DIROUT
-                        Output directory of the file. Usage: pyredactkit [file/filestoredact] -d [redacted_dir] (default: None)
+                        Output directory of the file. Usage: pyredactkit -f [file/filestoredact] -d [redacted_dir] (default: None)
   -r, --recursive       Search through subfolders (default: True)
   -e EXTENSION, --extension EXTENSION
                         File extension to filter by. (default: )
 ```
 
 ## Sample files
 
-- [All types of data](https://raw.githubusercontent.com/brootware/PyRedactKit/main/test/test.txt)
-- [itcont.txt - 4GB uncompressed](https://sanitizationbq.s3.ap-southeast-1.amazonaws.com/itcont.tar.gz)
+- [All types of data](./logdata/test.txt)
+- [Differnt log file types](./logdata/)
 - [test_sample2.txt - 10002 lines of IP addresses](https://sanitizationbq.s3.ap-southeast-1.amazonaws.com/test_sample2.txt)

custom.json

@@ -0,0 +1,14 @@
+[
+  {
+    "pattern": "(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)",
+    "type": ["ip", "ipv4"]
+  },
+  {
+    "pattern": "([a-z0-9!#$%&'*+/=?^_`{|.}~-]+@(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?)",
+    "type": ["email", "emails"]
+  },
+  {
+    "pattern": "^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$",
+    "type": ["base64", "b64"]
+  }
+]

logdata/emailsecrets.log

@@ -0,0 +1,27 @@
+this is my IP: 102.23.5.1
+My router is : 10.10.10.1
+71.159.188.33
+81.141.167.45
+165.65.59.139
+64.248.67.225
+
+https://tech.gov.sg
+
+My emails are
+[email protected]
+[email protected]
+[email protected]
+[email protected]
+
+this is my IP: 102.23.5.1
+My router is: 10.10.10.1
+71.159.188.33
+81.141.167.45
+165.65.59.139
+64.248.67.225
+
+Base64 data
+QVBJX1RPS0VO
+UzNjcjN0UGFzc3dvcmQ=
+U3VwM3JTM2NyZXRQQHNzd29yZA==
+WwogICAgICAgIHsKICAgICAgICAgICAgInBhdHRlcm4iOiAiKFthLXowLTkhIyQlJicqK1wvPT9eX2B7fC59fi1dK0AoPzpbYS16MC05XSg/OlthLXowLTktXSpbYS16MC05XSk/XC4pK1thLXowLTldKD86W2EtejAtOS1dKlthLXowLTldKT8pIiwKICAgICAgICAgICAgInR5cGUiOiAiZW1haWxzIiwKICAgICAgICAgICAgInBvc2l0aW9uIjogImlkX29iamVjdC5yZWdleGVzWzBdWydwYXR0ZXJuJ10iCiAgICAgICAgfSwKICAgICAgICB7CiAgICAgICAgICAgICJwYXR0ZXJuIjogJyg/aSkoKD86aHR0cHM/Oi8vfHd3d1xkezAsM31bLl0pP1thLXowLTkuXC1dK1suXSg/Oig/OmludGVybmF0aW9uYWwpfCg/OmNvbnN0cnVjdGlvbil8KD86Y29udHJhY3RvcnMpfCg/OmVudGVycHJpc2VzKXwoPzpwaG90b2dyYXBoeSl8KD86aW1tb2JpbGllbil8KD86bWFuYWdlbWVudCl8KD86dGVjaG5vbG9neSl8KD86ZGlyZWN0b3J5KXwoPzplZHVjYXRpb24pfCg/OmVxdWlwbWVudCl8KD86aW5zdGl0dXRlKXwoPzptYXJrZXRpbmcpfCg/OnNvbHV0aW9ucyl8KD86YnVpbGRlcnMpfCg/OmNsb3RoaW5nKXwoPzpjb21wdXRlcil8KD86ZGVtb2NyYXQpfCg/OmRpYW1vbmRzKXwoPzpncmFwaGljcyl8KD86aG9sZGluZ3MpfCg/OmxpZ2h0aW5nKXwoPzpwbHVtYmluZyl8KD86dHJhaW5pbmcpfCg/OnZlbnR1cmVzKXwoPzphY2FkZW15KXwoPzpjYXJlZXJzKXwoPzpjb21wYW55KXwoPzpkb21haW5zKXwoPzpmbG9yaXN0KXwoPzpnYWxsZXJ5KXwoPzpndWl0YXJzKXwoPzpob2xpZGF5KXwoPzpraXRjaGVuKXwoPzpyZWNpcGVzKXwoPzpzaGlrc2hhKXwoPzpzaW5nbGVzKXwoPzpzdXBwb3J0KXwoPzpzeXN0ZW1zKXwoPzphZ2VuY3kpfCg/OmJlcmxpbil8KD86Y2FtZXJhKXwoPzpjZW50ZXIpfCg/OmNvZmZlZSl8KD86ZXN0YXRlKXwoPzprYXVmZW4pfCg/Omx1eHVyeSl8KD86bW9uYXNoKXwoPzptdXNldW0pfCg/OnBob3Rvcyl8KD86cmVwYWlyKXwoPzpzb2NpYWwpfCg/OnRhdHRvbyl8KD86dHJhdmVsKXwoPzp2aWFqZXMpfCg/OnZveWFnZSl8KD86YnVpbGQpfCg/OmNoZWFwKXwoPzpjb2Rlcyl8KD86ZGFuY2UpfCg/OmVtYWlsKXwoPzpnbGFzcyl8KD86aG91c2UpfCg/Om5pbmphKXwoPzpwaG90byl8KD86c2hvZXMpfCg/OnNvbGFyKXwoPzp0b2RheSl8KD86YWVybyl8KD86YXJwYSl8KD86YXNpYSl8KD86YmlrZSl8KD86YnV6eil8KD86Y2FtcCl8KD86Y2x1Yil8KD86Y29vcCl8KD86ZmFybSl8KD86Z2lmdCl8KD86Z3VydSl8KD86aW5mbyl8KD86am9icyl8KD86a2l3aSl8KD86bGFuZCl8KD86bGltbyl8KD86bGluayl8KD86bWVudSl8KD86bW9iaSl8KD86bW9kYSl8KD86bmFtZSl8KD86cGljcyl8KD86cGluayl8KD86cG9zdCl8KD86cmljaCl8KD86cnVocil8KD86c2V4eSl8KD86dGlwcyl8KD86d2FuZyl8KD86d2llbil8KD86em9uZSl8KD86Yml6KXwoPzpjYWIpfCg/OmNhdCl8KD86Y2VvKXwoPzpjb20pfCg/OmVkdSl8KD86Z292KXwoPzppbnQpfCg/Om1pbCl8KD86bmV0KXwoPzpvbmwpfCg/Om9yZyl8KD86cHJvKXwoPzpyZWQpfCg/OnRlbCl8KD86dW5vKXwoPzp4eHgpfCg/OmFjKXwoPzphZCl8KD86YWUpfCg/OmFmKXwoPzphZyl8KD86YWkpfCg/OmFsKXwoPzphbSl8KD86YW4pfCg/OmFvKXwoPzphcSl8KD86YXIpfCg/OmFzKXwoPzphdCl8KD86YXUpfCg/OmF3KXwoPzpheCl8KD86YXopfCg/OmJhKXwoPzpiYil8KD86YmQpfCg/OmJlKXwoPzpiZil8KD86YmcpfCg/OmJoKXwoPzpiaSl8KD86YmopfCg/OmJtKXwoPzpibil8KD86Ym8pfCg/OmJyKXwoPzpicyl8KD86YnQpfCg/OmJ2KXwoPzpidyl8KD86YnkpfCg/OmJ6KXwoPzpjYSl8KD86Y2MpfCg/OmNkKXwoPzpjZil8KD86Y2cpfCg/OmNoKXwoPzpjaSl8KD86Y2spfCg/OmNsKXwoPzpjbSl8KD86Y24pfCg/OmNvKXwoPzpjcil8KD86Y3UpfCg/OmN2KXwoPzpjdyl8KD86Y3gpfCg/OmN5KXwoPzpjeil8KD86ZGUpfCg/OmRqKXwoPzpkayl8KD86ZG0pfCg/OmRvKXwoPzpkeil8KD86ZWMpfCg/OmVlKXwoPzplZyl8KD86ZXIpfCg/OmVzKXwoPzpldCl8KD86ZXUpfCg/OmZpKXwoPzpmail8KD86ZmspfCg/OmZtKXwoPzpmbyl8KD86ZnIpfCg/OmdhKXwoPzpnYil8KD86Z2QpfCg/OmdlKXwoPzpnZil8KD86Z2cpfCg/OmdoKXwoPzpnaSl8KD86Z2wpfCg/OmdtKXwoPzpnbil8KD86Z3ApfCg/OmdxKXwoPzpncil8KD86Z3MpfCg/Omd0KXwoPzpndSl8KD86Z3cpfCg/Omd5KXwoPzpoayl8KD86aG0pfCg/OmhuKXwoPzpocil8KD86aHQpfCg/Omh1KXwoPzppZCl8KD86aWUpfCg/OmlsKXwoPzppbSl8KD86aW4pfCg/OmlvKXwoPzppcSl8KD86aXIpfCg/OmlzKXwoPzppdCl8KD86amUpfCg/OmptKXwoPzpqbyl8KD86anApfCg/OmtlKXwoPzprZyl8KD86a2gpfCg/OmtpKXwoPzprbSl8KD86a24pfCg/OmtwKXwoPzprcil8KD86a3cpfCg/Omt5KXwoPzpreil8KD86bGEpfCg/OmxiKXwoPzpsYyl8KD86bGkpfCg/OmxrKXwoPzpscil8KD86bHMpfCg/Omx0KXwoPzpsdSl8KD86bHYpfCg/Omx5KXwoPzptYSl8KD86bWMpfCg/Om1kKXwoPzptZSl8KD86bWcpfCg/Om1oKXwoPzptayl8KD86bWwpfCg/Om1tKXwoPzptbil8KD86bW8pfCg/Om1wKXwoPzptcSl8KD86bXIpfCg/Om1zKXwoPzptdCl8KD86bXUpfCg/Om12KXwoPzptdyl8KD86bXgpfCg/Om15KXwoPzpteil8KD86bmEpfCg/Om5jKXwoPzpuZSl8KD86bmYpfCg/Om5nKXwoPzpuaSl8KD86bmwpfCg/Om5vKXwoPzpucCl8KD86bnIpfCg/Om51KXwoPzpueil8KD86b20pfCg/OnBhKXwoPzpwZSl8KD86cGYpfCg/OnBnKXwoPzpwaCl8KD86cGspfCg/OnBsKXwoPzpwbSl8KD86cG4pfCg/OnByKXwoPzpwcyl8KD86cHQpfCg/OnB3KXwoPzpweSl8KD86cWEpfCg/OnJlKXwoPzpybyl8KD86cnMpfCg/OnJ1KXwoPzpydyl8KD86c2EpfCg/OnNiKXwoPzpzYyl8KD86c2QpfCg/OnNlKXwoPzpzZyl8KD86c2gpfCg/OnNpKXwoPzpzail8KD86c2spfCg/OnNsKXwoPzpzbSl8KD86c24pfCg/OnNvKXwoPzpzcil8KD86c3QpfCg/OnN1KXwoPzpzdil8KD86c3gpfCg/OnN5KXwoPzpzeil8KD86dGMpfCg/OnRkKXwoPzp0Zil8KD86dGcpfCg/OnRoKXwoPzp0ail8KD86dGspfCg/OnRsKXwoPzp0bSl8KD86dG4pfCg/OnRvKXwoPzp0cCl8KD86dHIpfCg/OnR0KXwoPzp0dil8KD86dHcpfCg/OnR6KXwoPzp1YSl8KD86dWcpfCg/OnVrKXwoPzp1cyl8KD86dXkpfCg/OnV6KXwoPzp2YSl8KD86dmMpfCg/OnZlKXwoPzp2Zyl8KD86dmkpfCg/OnZuKXwoPzp2dSl8KD86d2YpfCg/OndzKXwoPzp5ZSl8KD86eXQpfCg/OnphKXwoPzp6bSl8KD86encpKSg/Oi9bXlxzKCk8Pl0rW15cc2AhKClcW1xde307OlwnIi4sPD4/XHhhYlx4YmJcdTIwMWNcdTIwMWRcdTIwMThcdTIwMTldKT8pJywKICAgICAgICAgICAgInR5cGUiOiAiZG5zIiwKICAgICAgICAgICAgInBvc2l0aW9uIjogImlkX29iamVjdC5yZWdleGVzWzFdWydwYXR0ZXJuJ10iCiAgICAgICAgfSwKICAgICAgICB7CiAgICAgICAgICAgICJwYXR0ZXJuIjogIig/OjI1WzAtNV18MlswLTRdWzAtOV18WzAxXT9bMC05XVswLTldPylcLig/OjI1WzAtNV18MlswLTRdWzAtOV18WzAxXT9bMC05XVswLTldPylcLig/OjI1WzAtNV18MlswLTRdWzAtOV18WzAxXT9bMC05XVswLTldPylcLig/OjI1WzAtNV18MlswLTRdWzAtOV18WzAxXT9bMC05XVswLTldPykiLAogICAgICAgICAgICAidHlwZSI6ICJpcHY0IiwKICAgICAgICAgICAgInBvc2l0aW9uIjogImlkX29iamVjdC5yZWdleGVzWzJdWydwYXR0ZXJuJ10iCiAgICAgICAgfSwKICAgICAgICB7CiAgICAgICAgICAgICJwYXR0ZXJuIjogJygoPzooPzpcXGR7NH1bLSBdPyl7M31cXGR7NH18XFxkezE1LDE2fSkpKD8hW1xcZF0pJywKICAgICAgICAgICAgInR5cGUiOiAiY3JlZGl0X2NhcmRzIiwKICAgICAgICAgICAgInBvc2l0aW9uIjogImlkX29iamVjdC5yZWdleGVzWzNdWydwYXR0ZXJuJ10iCiAgICAgICAgfSwKICAgICAgICB7CiAgICAgICAgICAgICJwYXR0ZXJuIjogIltTVEZHXVxkezd9W0EtWl0iLAogICAgICAgICAgICAidHlwZSI6ICJTaW5nYXBvcmVfTlJJQyIsCiAgICAgICAgICAgICJwb3NpdGlvbiI6ICJpZF9vYmplY3QucmVnZXhlc1s0XVsncGF0dGVybiddIgogICAgICAgIH0sCiAgICAgICAgewogICAgICAgICAgICAicGF0dGVybiI6ICdccyooPyEuKjo6Lio6OikoPzooPyE6KXw6KD89OikpKD86WzAtOWEtZl17MCw0fSg/Oig/PD06Oil8KD88ITo6KTopKXs2fSg/OlswLTlhLWZdezAsNH0oPzooPzw9OjopfCg/PCE6Oik6KVswLTlhLWZdezAsNH0oPzooPzw9OjopfCg/PCE6KXwoPzw9OikoPzwhOjopOil8KD86MjVbMC00XXwyWzAtNF1cZHwxXGRcZHxbMS05XT9cZCkoPzpcLig/OjI1WzAtNF18MlswLTRdXGR8MVxkXGR8WzEtOV0/XGQpKXszfSlccyonLAogICAgICAgICAgICAidHlwZSI6ICJpcHY2IiwKICAgICAgICAgICAgInBvc2l0aW9uIjogImlkX29iamVjdC5yZWdleGVzWzVdWydwYXR0ZXJuJ10iCiAgICAgICAgfSwKICAgICAgICB7CiAgICAgICAgICAgICJwYXR0ZXJuIjogIi9eKD86W2EtekEtWjAtOStcL117NH0pKig/OnwoPzpbYS16QS1aMC05K1wvXXszfT0pfCg/OlthLXpBLVowLTkrXC9dezJ9PT0pfCg/OlthLXpBLVowLTkrXC9dezF9PT09KSkkLyIsCiAgICAgICAgICAgICJ0eXBlIjogImJhc2U2NCIsCiAgICAgICAgICAgICJwb3NpdGlvbiI6ICJpZF9vYmplY3QucmVnZXhlc1s2XVsncGF0dGVybiddIgogICAgICAgIH0KICAgIF0=