|
| 1 | +--- |
| 2 | +title: Persistent HTTP Sessions with Python Requests |
| 3 | +lang: en |
| 4 | +layout: post |
| 5 | +audio: false |
| 6 | +translated: false |
| 7 | +generated: true |
| 8 | +image: false |
| 9 | +--- |
| 10 | + |
| 11 | +A `Session` in Python's `requests` library is a way to persist certain parameters, such as cookies, headers, and authentication, across multiple HTTP requests. It provides a convenient interface for making repeated requests to the same server or related endpoints while maintaining state, improving efficiency, and simplifying code. Below, I’ll explain how `requests.Session` works, its key features, and provide examples to illustrate its usage. |
| 12 | + |
| 13 | +## What is a `requests.Session`? |
| 14 | + |
| 15 | +The `requests` library is widely used for making HTTP requests in Python. While you can use standalone functions like `requests.get()` or `requests.post()`, a `Session` object allows you to create a persistent session that retains configurations (e.g., cookies, headers, or authentication credentials) across multiple requests. This is particularly useful for interacting with websites or APIs that require stateful interactions, such as maintaining a login session or reusing TCP connections. |
| 16 | + |
| 17 | +A `Session` object: |
| 18 | +- Persists cookies across requests. |
| 19 | +- Reuses underlying TCP connections (via connection pooling) for better performance when making multiple requests to the same host. |
| 20 | +- Allows you to set default parameters (e.g., headers, timeouts) that apply to all requests made with the session. |
| 21 | +- Supports authentication and custom configurations. |
| 22 | + |
| 23 | +## How Does `Session` Work? |
| 24 | + |
| 25 | +When you create a `Session` object, it acts as a container for your HTTP requests. Here’s a breakdown of how it functions: |
| 26 | + |
| 27 | +1. **Persistent Cookies**: When you make a request with a `Session`, any cookies set by the server (e.g., session cookies after logging in) are stored in the session and automatically sent in subsequent requests. This is key for maintaining state, such as staying logged in. |
| 28 | + |
| 29 | +2. **Connection Pooling**: For requests to the same host, the `Session` reuses the same TCP connection, reducing latency and overhead compared to creating new connections for each request. |
| 30 | + |
| 31 | +3. **Default Parameters**: You can set attributes like headers, authentication, or timeouts on the `Session` object, and they will apply to all requests made with that session unless overridden. |
| 32 | + |
| 33 | +4. **Customizable**: You can configure proxies, SSL verification, or even mount custom adapters (e.g., for retries or custom transport) to control how requests are handled. |
| 34 | + |
| 35 | +## Basic Usage |
| 36 | + |
| 37 | +Here’s a simple example of how to use `requests.Session`: |
| 38 | + |
| 39 | +```python |
| 40 | +import requests |
| 41 | + |
| 42 | +# Create a session |
| 43 | +session = requests.Session() |
| 44 | + |
| 45 | +# Set default headers for all requests in this session |
| 46 | +session.headers.update({'User-Agent': 'MyApp/1.0'}) |
| 47 | + |
| 48 | +# Make a GET request |
| 49 | +response1 = session.get('https://api.example.com/data') |
| 50 | +print(response1.json()) |
| 51 | + |
| 52 | +# Make another request; cookies and headers are reused |
| 53 | +response2 = session.post('https://api.example.com/submit', data={'key': 'value'}) |
| 54 | +print(response2.json()) |
| 55 | + |
| 56 | +# Close the session to release resources |
| 57 | +session.close() |
| 58 | +``` |
| 59 | + |
| 60 | +In this example: |
| 61 | +- A `Session` is created, and a custom `User-Agent` header is set for all requests. |
| 62 | +- The session handles cookies automatically, so if `response1` sets a cookie, it’s sent with `response2`. |
| 63 | +- The session reuses the connection to `api.example.com`, improving performance. |
| 64 | + |
| 65 | +## Key Features and Examples |
| 66 | + |
| 67 | +### 1. **Persisting Cookies** |
| 68 | +Sessions are particularly useful for websites that use cookies to maintain state, such as login sessions. |
| 69 | + |
| 70 | +```python |
| 71 | +import requests |
| 72 | + |
| 73 | +# Create a session |
| 74 | +session = requests.Session() |
| 75 | + |
| 76 | +# Log in to a website |
| 77 | +login_data = {'username': 'user', 'password': 'pass'} |
| 78 | +response = session.post('https://example.com/login', data=login_data) |
| 79 | + |
| 80 | +# Access a protected page; the session automatically sends the login cookie |
| 81 | +protected_page = session.get('https://example.com/protected') |
| 82 | +print(protected_page.text) |
| 83 | + |
| 84 | +# Close the session |
| 85 | +session.close() |
| 86 | +``` |
| 87 | + |
| 88 | +Here, the session stores the authentication cookie from the login request and sends it with the subsequent request to the protected page. |
| 89 | + |
| 90 | +### 2. **Setting Default Parameters** |
| 91 | +You can set default headers, authentication, or other parameters for all requests in the session. |
| 92 | + |
| 93 | +```python |
| 94 | +import requests |
| 95 | + |
| 96 | +session = requests.Session() |
| 97 | + |
| 98 | +# Set default headers |
| 99 | +session.headers.update({ |
| 100 | + 'Authorization': 'Bearer my_token', |
| 101 | + 'Accept': 'application/json' |
| 102 | +}) |
| 103 | + |
| 104 | +# Set default timeout |
| 105 | +session.request = functools.partial(session.request, timeout=5) |
| 106 | + |
| 107 | +# Make requests; headers and timeout are automatically applied |
| 108 | +response1 = session.get('https://api.example.com/endpoint1') |
| 109 | +response2 = session.get('https://api.example.com/endpoint2') |
| 110 | + |
| 111 | +session.close() |
| 112 | +``` |
| 113 | + |
| 114 | +### 3. **Connection Pooling** |
| 115 | +When making multiple requests to the same host, `Session` reuses connections, which is more efficient than standalone requests. |
| 116 | + |
| 117 | +```python |
| 118 | +import requests |
| 119 | +import time |
| 120 | + |
| 121 | +# Without session |
| 122 | +start = time.time() |
| 123 | +for _ in range(5): |
| 124 | + requests.get('https://api.example.com/data') |
| 125 | +print(f"Without session: {time.time() - start} seconds") |
| 126 | + |
| 127 | +# With session |
| 128 | +session = requests.Session() |
| 129 | +start = time.time() |
| 130 | +for _ in range(5): |
| 131 | + session.get('https://api.example.com/data') |
| 132 | +print(f"With session: {time.time() - start} seconds") |
| 133 | +session.close() |
| 134 | +``` |
| 135 | + |
| 136 | +The session-based requests are typically faster because they reuse the TCP connection. |
| 137 | + |
| 138 | +### 4. **Authentication** |
| 139 | +Sessions simplify handling authentication, such as HTTP Basic Auth or custom token-based authentication. |
| 140 | + |
| 141 | +```python |
| 142 | +import requests |
| 143 | +from requests.auth import HTTPBasicAuth |
| 144 | + |
| 145 | +session = requests.Session() |
| 146 | +session.auth = HTTPBasicAuth('user', 'pass') |
| 147 | + |
| 148 | +# All requests will include Basic Auth |
| 149 | +response = session.get('https://api.example.com/protected') |
| 150 | +print(response.json()) |
| 151 | + |
| 152 | +session.close() |
| 153 | +``` |
| 154 | + |
| 155 | +### 5. **Custom Adapters** |
| 156 | +You can mount custom adapters to control things like retries or connection pooling behavior. |
| 157 | + |
| 158 | +```python |
| 159 | +import requests |
| 160 | +from requests.adapters import HTTPAdapter |
| 161 | +from urllib3.util.retry import Retry |
| 162 | + |
| 163 | +session = requests.Session() |
| 164 | + |
| 165 | +# Configure retries |
| 166 | +retries = Retry(total=3, backoff_factor=0.1, status_forcelist=[500, 502, 503, 504]) |
| 167 | +session.mount('https://', HTTPAdapter(max_retries=retries)) |
| 168 | + |
| 169 | +# Make a request with retry logic |
| 170 | +response = session.get('https://api.example.com/unstable_endpoint') |
| 171 | +print(response.json()) |
| 172 | + |
| 173 | +session.close() |
| 174 | +``` |
| 175 | + |
| 176 | +This example sets up automatic retries for specific HTTP error codes. |
| 177 | + |
| 178 | +## When to Use `Session` |
| 179 | + |
| 180 | +Use `requests.Session` when: |
| 181 | +- You need to maintain state (e.g., cookies) across multiple requests, such as for authentication. |
| 182 | +- You’re making multiple requests to the same host and want to benefit from connection pooling. |
| 183 | +- You want to apply consistent configurations (headers, timeouts, etc.) across requests. |
| 184 | +- You’re interacting with APIs or websites that require session-based interactions. |
| 185 | + |
| 186 | +If you’re making a single request or don’t need to persist state, standalone `requests.get()` or `requests.post()` is sufficient. |
| 187 | + |
| 188 | +## Closing a Session |
| 189 | + |
| 190 | +Always close a session when you’re done to release resources (e.g., connections): |
| 191 | + |
| 192 | +```python |
| 193 | +session.close() |
| 194 | +``` |
| 195 | + |
| 196 | +Alternatively, use a session as a context manager to ensure it’s closed automatically: |
| 197 | + |
| 198 | +```python |
| 199 | +with requests.Session() as session: |
| 200 | + session.headers.update({'User-Agent': 'MyApp/1.0'}) |
| 201 | + response = session.get('https://api.example.com/data') |
| 202 | + print(response.json()) |
| 203 | +# Session is automatically closed when the block exits |
| 204 | +``` |
| 205 | + |
| 206 | +## Advanced Notes |
| 207 | +- **Thread Safety**: `Session` objects are not thread-safe. If you’re using multithreading, create a separate `Session` for each thread. |
| 208 | +- **Proxies**: You can set proxies on a session with `session.proxies = {'http': 'http://proxy.com', 'https': 'https://proxy.com'}`. |
| 209 | +- **SSL Verification**: By default, `Session` verifies SSL certificates. To disable (not recommended for production), use `session.verify = False`. |
| 210 | + |
| 211 | +## References |
| 212 | +[Requests Documentation](https://requests.readthedocs.io/en/latest/) |
| 213 | +[Requests Session Objects](https://requests.readthedocs.io/en/latest/user/advanced/#session-objects) |
0 commit comments