Skip to content

Commit 653ebee

Browse files
committed
docs: Add Couchbase index management instructions to README
This update introduces a new section in the README detailing the required N1QL indexes for the `travel-sample` bucket, including automatic and manual index setup instructions. Additionally, it updates the CI workflow to automatically set up Couchbase indexes before running tests and modifies the setup script to include index creation. Changes: - Added Couchbase index management section to README.md - Updated CI workflow to include Couchbase index setup - Modified bin/setup to create Couchbase indexes during setup
1 parent 31dd41e commit 653ebee

File tree

6 files changed

+304
-5
lines changed

6 files changed

+304
-5
lines changed

.github/workflows/ci.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,9 @@ jobs:
3838
fi
3939
echo "Couchbase configuration validated successfully"
4040
41+
- name: Setup Couchbase indexes
42+
run: bundle exec rake couchbase:setup_indexes
43+
4144
- name: Run integration tests
4245
run: bundle exec rspec spec/requests/api/v1
4346

README.md

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,91 @@ production:
9191
9292
> Note: The connection string expects the `couchbases://` or `couchbase://` part.
9393

94+
## Couchbase Index Management
95+
96+
This application requires specific N1QL indexes on the `travel-sample` bucket to function correctly. These indexes optimize the SQL++ queries used by the application for filtering and joining documents.
97+
98+
### Automatic Index Setup
99+
100+
Indexes are automatically created when you:
101+
102+
- Run `bin/setup` for local development setup
103+
- Run tests in CI/CD (GitHub Actions automatically creates indexes before running tests)
104+
105+
The application uses idempotent index creation (using `CREATE INDEX IF NOT EXISTS`), so it's safe to run the setup multiple times.
106+
107+
### Required Indexes
108+
109+
The application requires the following indexes on the `travel-sample` bucket:
110+
111+
1. **`idx_type`** - General index on the `type` field for all document queries
112+
2. **`idx_type_country`** - Index for airline queries filtered by country (`Airline.list_by_country_or_all`)
113+
3. **`idx_type_destinationairport`** - Index for route queries by destination airport (`Airline.to_airport`)
114+
4. **`idx_type_sourceairport_stops`** - Index for route queries by source airport and stops (`Route.direct_connections`)
115+
5. **`idx_type_airlineid`** - Index for airline queries by airline ID (used in joins with routes)
116+
117+
### Manual Index Management
118+
119+
You can manually manage indexes using the following Rake tasks:
120+
121+
#### Create All Required Indexes
122+
123+
```sh
124+
bundle exec rake couchbase:setup_indexes
125+
```
126+
127+
This command creates all required indexes on the `travel-sample` bucket. It's idempotent and safe to run multiple times.
128+
129+
#### List All Indexes
130+
131+
```sh
132+
bundle exec rake couchbase:list_indexes
133+
```
134+
135+
This command lists all indexes currently present in the `travel-sample` bucket, including their state, type, and indexed fields.
136+
137+
#### Drop Application Indexes
138+
139+
```sh
140+
bundle exec rake couchbase:drop_indexes
141+
```
142+
143+
This command drops all application-managed indexes. It requires confirmation before executing. For automated scripts, you can force the operation:
144+
145+
```sh
146+
FORCE_DROP=true bundle exec rake couchbase:drop_indexes
147+
```
148+
149+
> **Warning**: Use with caution! Dropping indexes will cause queries to fail until indexes are recreated.
150+
151+
### Troubleshooting Index Issues
152+
153+
If you encounter index-related errors:
154+
155+
1. **Verify indexes exist**:
156+
```sh
157+
bundle exec rake couchbase:list_indexes
158+
```
159+
160+
2. **Check index state**: Indexes should be in "online" state. If they're "building" or "pending", wait for them to complete.
161+
162+
3. **Recreate indexes**:
163+
```sh
164+
bundle exec rake couchbase:drop_indexes
165+
bundle exec rake couchbase:setup_indexes
166+
```
167+
168+
4. **Check permissions**: Ensure your Couchbase user has "Query Manage Index" permission to create and drop indexes.
169+
170+
### Index Creation in CI/CD
171+
172+
The GitHub Actions workflow automatically creates indexes before running tests. If index creation fails in CI:
173+
174+
1. Check the "Setup Couchbase indexes" step in the GitHub Actions log
175+
2. Verify that `DB_CONN_STR`, `DB_USERNAME`, and `DB_PASSWORD` secrets/variables are correctly set
176+
3. Ensure the Couchbase user has "Query Manage Index" permission
177+
4. Check that the Couchbase cluster is accessible from GitHub Actions runners
178+
94179
## Running The Application
95180

96181
### Directly on machine

bin/setup

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,9 @@ FileUtils.chdir APP_ROOT do
2525
puts "\n== Preparing database =="
2626
system! "bin/rails db:prepare"
2727

28+
puts "\n== Setting up Couchbase indexes =="
29+
system! "bin/rails couchbase:setup_indexes"
30+
2831
puts "\n== Removing old logs and tempfiles =="
2932
system! "bin/rails log:clear tmp:clear"
3033

config/environments/development.rb

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -51,16 +51,16 @@
5151
config.active_support.disallowed_deprecation_warnings = []
5252

5353
# Raise an error on page load if there are pending migrations.
54-
config.active_record.migration_error = :page_load
54+
# config.active_record.migration_error = :page_load # Commented out - not using ActiveRecord
5555

5656
# Highlight code that triggered database queries in logs.
57-
config.active_record.verbose_query_logs = true
57+
# config.active_record.verbose_query_logs = true # Commented out - not using ActiveRecord
5858

5959
# Highlight code that enqueued background job in logs.
6060
config.active_job.verbose_enqueue_logs = true
6161

6262
# Suppress logger output for asset requests.
63-
config.assets.quiet = true
63+
# config.assets.quiet = true # Commented out - not using asset pipeline
6464

6565
# Raises error for missing translations.
6666
# config.i18n.raise_on_missing_translations = true

config/environments/production.rb

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@
2727
# config.assets.css_compressor = :sass
2828

2929
# Do not fall back to assets pipeline if a precompiled asset is missed.
30-
config.assets.compile = false
30+
# config.assets.compile = false # Commented out - not using asset pipeline
3131

3232
# Enable serving of images, stylesheets, and JavaScripts from an asset server.
3333
# config.asset_host = "http://assets.example.com"
@@ -85,7 +85,7 @@
8585
config.active_support.report_deprecations = false
8686

8787
# Do not dump schema after migrations.
88-
config.active_record.dump_schema_after_migration = false
88+
# config.active_record.dump_schema_after_migration = false # Commented out - not using ActiveRecord
8989

9090
# Enable DNS rebinding protection and other `Host` header attacks.
9191
# config.hosts = [

lib/tasks/couchbase.rake

Lines changed: 208 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,208 @@
1+
# frozen_string_literal: true
2+
3+
namespace :couchbase do
4+
desc 'Setup required Couchbase indexes for the application'
5+
task setup_indexes: :environment do
6+
puts '=== Setting up Couchbase indexes ==='
7+
8+
# Get cluster connection from any model (they all share the same cluster)
9+
cluster = Airline.cluster
10+
bucket_name = Airline.bucket.name
11+
12+
puts "Target bucket: #{bucket_name}"
13+
14+
# Define all required indexes based on N1QL queries in models
15+
indexes = [
16+
{
17+
name: 'idx_type',
18+
fields: ['type'],
19+
description: 'Index on type field for all document queries'
20+
},
21+
{
22+
name: 'idx_type_country',
23+
fields: ['type', 'country'],
24+
where: "type = 'airline'",
25+
description: 'Index for airline queries by country (Airline.list_by_country_or_all)'
26+
},
27+
{
28+
name: 'idx_type_destinationairport',
29+
fields: ['type', 'destinationairport'],
30+
where: "type = 'route'",
31+
description: 'Index for route queries by destination airport (Airline.to_airport)'
32+
},
33+
{
34+
name: 'idx_type_sourceairport_stops',
35+
fields: ['type', 'sourceairport', 'stops'],
36+
where: "type = 'route'",
37+
description: 'Index for route queries by source airport and stops (Route.direct_connections)'
38+
},
39+
{
40+
name: 'idx_type_airlineid',
41+
fields: ['type', 'airlineid'],
42+
where: "type = 'airline'",
43+
description: 'Index for airline queries by airline ID (Airline.to_airport join)'
44+
}
45+
]
46+
47+
created_count = 0
48+
skipped_count = 0
49+
failed_indexes = []
50+
51+
indexes.each do |index_def|
52+
begin
53+
puts "\nCreating index: #{index_def[:name]}"
54+
puts " Description: #{index_def[:description]}"
55+
56+
# Build the CREATE INDEX query with IF NOT EXISTS for idempotency
57+
query = build_create_index_query(bucket_name, index_def)
58+
puts " Query: #{query}"
59+
60+
# Execute the query
61+
cluster.query(query)
62+
63+
# If no error, index is ready
64+
created_count += 1
65+
puts " \u2713 Index created or already exists"
66+
67+
rescue Couchbase::Error::IndexExists => e
68+
# This shouldn't happen with IF NOT EXISTS, but handle it anyway
69+
puts " \u2299 Index already exists (skipped)"
70+
skipped_count += 1
71+
rescue Couchbase::Error::CouchbaseError => e
72+
puts " \u2717 Failed: #{e.message}"
73+
failed_indexes << { name: index_def[:name], error: e.message }
74+
rescue StandardError => e
75+
puts " \u2717 Unexpected error: #{e.message}"
76+
failed_indexes << { name: index_def[:name], error: e.message }
77+
end
78+
end
79+
80+
# Print summary
81+
puts "\n=== Index Setup Summary ==="
82+
puts "Total indexes: #{indexes.count}"
83+
puts "Successfully processed: #{created_count}"
84+
puts "Skipped (already existed): #{skipped_count}"
85+
puts "Failed: #{failed_indexes.count}"
86+
87+
if failed_indexes.any?
88+
puts "\n=== Failed Indexes ==="
89+
failed_indexes.each do |failed|
90+
puts " - #{failed[:name]}: #{failed[:error]}"
91+
end
92+
93+
# Exit with error code for CI
94+
exit 1
95+
else
96+
puts "\n\u2713 All indexes are ready!"
97+
end
98+
rescue Couchbase::Error::AuthenticationFailure => e
99+
puts "\n\u2717 Authentication failed: #{e.message}"
100+
puts "Please verify your Couchbase credentials:"
101+
puts " - DB_USERNAME environment variable"
102+
puts " - DB_PASSWORD environment variable"
103+
exit 1
104+
rescue Couchbase::Error::CouchbaseError => e
105+
puts "\n\u2717 Couchbase connection error: #{e.message}"
106+
puts "Please check your connection settings:"
107+
puts " - DB_CONN_STR environment variable"
108+
puts " - DB_USERNAME environment variable"
109+
puts " - DB_PASSWORD environment variable"
110+
puts " - Network connectivity to Couchbase cluster"
111+
exit 1
112+
rescue StandardError => e
113+
puts "\n\u2717 Unexpected error: #{e.class} - #{e.message}"
114+
puts e.backtrace.first(5).join("\n")
115+
exit 1
116+
end
117+
118+
desc 'Drop all application indexes (use with caution!)'
119+
task drop_indexes: :environment do
120+
puts '=== Dropping Couchbase indexes ==='
121+
puts 'WARNING: This will drop all application indexes!'
122+
123+
unless ENV['FORCE_DROP'] == 'true'
124+
print 'Are you sure? (yes/no): '
125+
confirmation = $stdin.gets.chomp
126+
unless confirmation.downcase == 'yes'
127+
puts 'Aborted.'
128+
exit 0
129+
end
130+
end
131+
132+
cluster = Airline.cluster
133+
bucket_name = Airline.bucket.name
134+
135+
index_names = [
136+
'idx_type',
137+
'idx_type_country',
138+
'idx_type_destinationairport',
139+
'idx_type_sourceairport_stops',
140+
'idx_type_airlineid'
141+
]
142+
143+
dropped_count = 0
144+
not_found_count = 0
145+
146+
index_names.each do |index_name|
147+
begin
148+
query = "DROP INDEX `#{bucket_name}`.`#{index_name}`"
149+
cluster.query(query)
150+
puts " \u2713 Dropped index: #{index_name}"
151+
dropped_count += 1
152+
rescue Couchbase::Error::IndexNotFound
153+
puts " \u2299 Index not found (already dropped): #{index_name}"
154+
not_found_count += 1
155+
rescue StandardError => e
156+
puts " \u2717 Failed to drop #{index_name}: #{e.message}"
157+
end
158+
end
159+
160+
puts "\n=== Drop Summary ==="
161+
puts "Dropped: #{dropped_count}"
162+
puts "Not found: #{not_found_count}"
163+
puts "\n\u2713 Index cleanup complete!"
164+
end
165+
166+
desc 'List all indexes in the bucket'
167+
task list_indexes: :environment do
168+
puts '=== Couchbase Indexes ==='
169+
170+
cluster = Airline.cluster
171+
bucket_name = Airline.bucket.name
172+
173+
query = "SELECT idx.* FROM system:indexes AS idx WHERE idx.keyspace_id = '#{bucket_name}' ORDER BY idx.name"
174+
result = cluster.query(query)
175+
176+
if result.rows.empty?
177+
puts "No indexes found in bucket '#{bucket_name}'"
178+
else
179+
puts "Indexes in bucket '#{bucket_name}':\n"
180+
result.rows.each do |row|
181+
puts " Name: #{row['name']}"
182+
puts " State: #{row['state']}"
183+
puts " Type: #{row['using']}"
184+
puts " Keys: #{row['index_key']}"
185+
puts " Condition: #{row['condition']}" if row['condition']
186+
puts ""
187+
end
188+
puts "Total: #{result.rows.count} indexes"
189+
end
190+
rescue StandardError => e
191+
puts "\u2717 Error listing indexes: #{e.message}"
192+
exit 1
193+
end
194+
195+
# Private helper method to build CREATE INDEX query
196+
def build_create_index_query(bucket_name, index_def)
197+
query = "CREATE INDEX IF NOT EXISTS `#{index_def[:name]}` ON `#{bucket_name}`"
198+
199+
# Add fields
200+
fields = index_def[:fields].map { |f| "`#{f}`" }.join(', ')
201+
query += "(#{fields})"
202+
203+
# Add WHERE clause if present
204+
query += " WHERE #{index_def[:where]}" if index_def[:where]
205+
206+
query
207+
end
208+
end

0 commit comments

Comments
 (0)