@@ -784,24 +784,25 @@ def _create_smarter_in_clause(self, column, values_list):
784784 The 500k batch threshold is chosen to balance several factors:
785785
786786 - **Parameter limits**: Each batch uses 1 parameter. With SQLite's minimum limit of 999
787- parameters, this allows up to ~500M items (999 x 500k). PostgreSQL's limit of ~65k
788- parameters allows up to ~33B items (65,535 x 500k).
787+ parameters, this allows up to ~500M items (999 x 500k). PostgreSQL's limit of ~65k
788+ parameters allows up to ~33B items (65,535 x 500k).
789789 - **Memory constraints**: In practice, Python memory becomes the bottleneck before
790- database limits. A list of 500M items would require 4-20GB RAM before even reaching
791- the database.
790+ database limits. A list of 500M items would require 4-20GB RAM before even reaching
791+ the database.
792792 - **Database performance**: Modern databases handle 500k-item arrays/JSON easily on
793- typical workstations and servers.
793+ typical workstations and servers.
794794
795- For example:
796- Small list (50k items):
797- WHERE column IN (SELECT unnest(:array)) -- 1 parameter
795+ For example, small list (50k items)::
798796
799- Large list (1.5M items):
800- WHERE (
801- column IN (SELECT unnest(:array_1)) -- First 500k
802- OR column IN (SELECT unnest(:array_2)) -- Second 500k
803- OR column IN (SELECT unnest(:array_3)) -- Remaining 500k
804- )
797+ WHERE column IN (SELECT unnest(:array)) -- 1 parameter
798+
799+ Large list (1.5M items)::
800+
801+ WHERE (
802+ column IN (SELECT unnest(:array_1)) -- First 500k
803+ OR column IN (SELECT unnest(:array_2)) -- Second 500k
804+ OR column IN (SELECT unnest(:array_3)) -- Remaining 500k
805+ )
805806 """
806807 import json
807808
0 commit comments