v0.16.0

Author: vladkens

.github/workflows/ci.yml

@@ -46,5 +46,7 @@ jobs:
 
       - name: Create Github Release
         uses: softprops/action-gh-release@v2
+        with:
+          generate_release_notes: true
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.vscode/settings.json

@@ -7,6 +7,10 @@
   "[toml]": {
     "editor.defaultFormatter": "tamasfe.even-better-toml"
   },
+  "[markdown]": {
+    "editor.wordWrap": "wordWrapColumn",
+    "editor.wordWrapColumn": 99
+  },  
   "files.exclude": {
     ".ruff_cache": true,
     ".pytest_cache": true,

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "twscrape"
-version = "0.15.0"
+version = "0.16.0"
 authors = [{ name = "vladkens", email = "v.pronsky@gmail.com" }]
 description = "Twitter GraphQL and Search API implementation with SNScrape data models"
 readme = "readme.md"

readme.md

@@ -36,31 +36,37 @@ pip install git+https://github.com/vladkens/twscrape.git
 
 ## Usage
 
-Since this project works through an authorized API, accounts need to be added. You can register and add an account yourself. You can also google sites that provide these things.
+This project requires authorized X/Twitter accounts to work with the API. You have two options:
 
-The email password is needed to get the code to log in to the account automatically (via imap protocol).
+1. **Create Your Own Account**: While you can register a new account on X/Twitter yourself, it's can be difficult due to strict verification processes and high ban rates.
 
-Data models:
-- [User](https://github.com/vladkens/twscrape/blob/main/twscrape/models.py#L87)
-- [Tweet](https://github.com/vladkens/twscrape/blob/main/twscrape/models.py#L136)
+2. **Use Ready Accounts**: For immediate access, you can get ready-to-use accounts with cookies from [our recommended provider](https://kutt.it/ueeM5f). Cookie-based accounts typically have fewer login issues.
+
+For optimal performance and to avoid IP-based restrictions, we also recommend using proxies from [our provider](https://kutt.it/eb3rXk).
+
+**Disclaimer**: While X/Twitter's Terms of Service discourage using multiple accounts, this is a common practice for data collection and research purposes. Use responsibly and at your own discretion.
 
 ```python
 import asyncio
 from twscrape import API, gather
 from twscrape.logger import set_log_level
 
 async def main():
-    api = API()  # or API("path-to.db") - default is `accounts.db`
+    api = API()  # or API("path-to.db") – default is `accounts.db`
 
-    # ADD ACCOUNTS (for CLI usage see BELOW)
-    await api.pool.add_account("user1", "pass1", "u1@example.com", "mail_pass1")
-    await api.pool.add_account("user2", "pass2", "u2@example.com", "mail_pass2")
-    await api.pool.login_all()
+    # ADD ACCOUNTS (for CLI usage see next readme section)
 
-    # or add account with COOKIES (with cookies login not required)
+    # Option 1. Adding account with cookies (more stable)
     cookies = "abc=12; ct0=xyz"  # or '{"abc": "12", "ct0": "xyz"}'
     await api.pool.add_account("user3", "pass3", "u3@mail.com", "mail_pass3", cookies=cookies)
 
+    # Option2. Adding account with login / password (less stable)
+    # email login / password required to receive the verification code via IMAP protocol
+    # (not all email providers are supported, e.g. ProtonMail)
+    await api.pool.add_account("user1", "pass1", "u1@example.com", "mail_pass1")
+    await api.pool.add_account("user2", "pass2", "u2@example.com", "mail_pass2")
+    await api.pool.login_all() # try to login to receive account cookies
+
     # API USAGE
 
     # search (latest tab)
@@ -311,21 +317,21 @@ So if you want to use proxy PER ACCOUNT, do NOT override proxy with env variable
 
 _Note:_ If proxy not working, exception will be raised from API class.
 
-## Environment variables
+## Environment Variables
 
-- `TWS_WAIT_EMAIL_CODE` – timeout for email verification code during login (default: `30`, in seconds)
-- `TWS_RAISE_WHEN_NO_ACCOUNT` – raise `NoAccountError` exception when no available accounts right now, instead of waiting for availability (default: `false`, possible value: `false` / `0` / `true` / `1`)
+- `TWS_PROXY` - global proxy for all accounts (e.g. `socks5://user:pass@127.0.0.1:1080`)
+- `TWS_WAIT_EMAIL_CODE` - timeout for email verification code during login (default: `30`, in seconds)
+- `TWS_RAISE_WHEN_NO_ACCOUNT` - raise `NoAccountError` exception when no available accounts, instead of waiting (default: `false`, values: `false`/`0`/`true`/`1`)
 
 ## Limitations
 
-After 1 July 2023 Twitter [introduced new limits](https://x.com/elonmusk/status/1675187969420828672) and still continue to update it periodically.
-
-The basic behaviour is as follows:
-- the request limit is updated every 15 minutes for each endpoint individually
-- e.g. each account have 50 search requests / 15 min, 50 profile requests / 15 min, etc.
+X/Twitter regularly [updates](https://x.com/elonmusk/status/1675187969420828672) their rate limits. Current basic behavior:
+- Request limits reset every 15 minutes for each endpoint individually
+- Each account has separate limits for different operations (search, profile views, etc.)
 
-API data limits:
-- `user_tweets` & `user_tweets_and_replies` – can return ~3200 tweets maximum
+API data limitations:
+- `user_tweets` & `user_tweets_and_replies` - can return ~3200 tweets maximum
+- Rate limits may vary based on account age and status
 
 ## Articles
 - [How to still scrape millions of tweets in 2023](https://medium.com/@vladkens/how-to-still-scrape-millions-of-tweets-in-2023-using-twscrape-97f5d3881434)