Skip to content

Claude/fix portal rollout timeout e t hy i #532

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aurelianware merged 5 commits into from
Mar 21, 2026
Merged

Claude/fix portal rollout timeout e t hy i #532

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
64ef1e5
Reduce encounter-service replicas from 3 to 1 for free Cosmos tier
claude Mar 21, 2026
027b040
Reduce reference-data-service and portal replicas to 1
claude Mar 21, 2026
d68a2d3
Migrate encounter-service and claims-scrubbing-service to MongoDB
claude Mar 21, 2026
a81c940
Fix stale Cosmos DB references in claims-scrubber comments
claude Mar 21, 2026
caa67e7
Merge branch 'main' into claude/fix-portal-rollout-timeout-eTHyI
aurelianware Mar 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/portal/CloudHealthOffice.Portal/k8s/portal-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -244,7 +244,7 @@ spec:
apiVersion: apps/v1
kind: Deployment
name: portal
minReplicas: 2 # Safe with Redis-backed session + token cache
minReplicas: 1 # Safe with Redis-backed session + token cache
Copy link

Copilot AI Mar 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reducing portal HPA minReplicas from 2 to 1 removes redundancy; a single replica means any rollout/node disruption causes full portal downtime. If this change is to mitigate rollout timeouts due to cluster capacity, consider keeping minReplicas >= 2 for production and addressing the scheduling issue via resources/cluster sizing/rollout strategy, or document that this manifest is intended for smaller non-prod clusters.

Suggested change
minReplicas: 1 # Safe with Redis-backed session + token cache
minReplicas: 2 # Production redundancy; use separate non-prod manifest if running a single replica

Copilot uses AI. Check for mistakes.
maxReplicas: 5
metrics:
- type: Resource
Expand Down
12 changes: 6 additions & 6 deletions src/services/claims-scrubbing-service/k8s/claims-scrubbing-service-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@ data:
CLEAN_CLAIMS_TOPIC: "claims.clean"
FLAGGED_CLAIMS_TOPIC: "claims.flagged"
REJECTED_CLAIMS_TOPIC: "claims.rejected"
COSMOS_DATABASE: "CloudHealthOffice"
COSMOS_RULES_CONTAINER: "ScrubRules"
COSMOS_AUDIT_CONTAINER: "ScrubAudit"
MONGODB_DATABASE: "CloudHealthOffice"
Copy link

Copilot AI Mar 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MONGODB_DATABASE is set to "CloudHealthOffice" here, but most other services use "cloudhealthoffice". If these services are expected to share a single Mongo database, this will create/use a different DB and lead to missing data. Align the configured DB name with the rest of the stack unless the separation is intentional and documented.

Suggested change
MONGODB_DATABASE: "CloudHealthOffice"
MONGODB_DATABASE: "cloudhealthoffice"

Copilot uses AI. Check for mistakes.
MONGODB_RULES_COLLECTION: "ScrubRules"
MONGODB_AUDIT_COLLECTION: "ScrubAudit"
CLAIMS_CONTAINER: "claims-archive"
PARALLEL_RULES: "true"
MAX_RULE_CONCURRENCY: "10"
Expand Down Expand Up @@ -72,11 +72,11 @@ spec:
name: kafka-secret
key: saslPassword
optional: true
- name: COSMOS_ENDPOINT
- name: MONGODB_CONNECTION_STRING
valueFrom:
secretKeyRef:
name: cosmos-db-secret
key: endpoint
name: mongodb-secret
key: connectionString
- name: STORAGE_CONNECTION_STRING
valueFrom:
secretKeyRef:
Expand Down
2 changes: 1 addition & 1 deletion src/services/claims-scrubbing-service/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,10 +33,10 @@
"author": "Cloud Health Office Team",
"license": "Apache-2.0",
"dependencies": {
"@azure/cosmos": "^4.2.0",
"@azure/identity": "^4.13.0",
"@azure/keyvault-secrets": "^4.9.0",
"@azure/storage-blob": "^12.30.0",
"mongodb": "^6.12.0",
Copy link

Copilot AI Mar 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

package.json adds mongodb and removes @azure/cosmos, but the repo’s package-lock.json still contains @azure/cosmos and does not include mongodb. This will likely break npm ci/reproducible builds. Regenerate and commit the updated lockfile so dependencies match package.json.

Suggested change
"mongodb": "^6.12.0",
"@azure/cosmos": "^4.0.0",

Copilot uses AI. Check for mistakes.
"kafkajs": "^2.2.4",
"uuid": "^13.0.0"
},
Expand Down
71 changes: 33 additions & 38 deletions src/services/claims-scrubbing-service/src/claims-scrubber.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,12 @@
* - Configurable validation rule engine
* - Standard and custom rule support
* - Kafka integration for claim routing
* - Cosmos DB for rule storage and audit
* - MongoDB for rule storage and audit
* - First-pass rate metrics tracking
*/

import { Kafka, Producer, Consumer, logLevel } from 'kafkajs';
import { CosmosClient, Container, Database } from '@azure/cosmos';
import { MongoClient, Db, Collection } from 'mongodb';
import { BlobServiceClient, ContainerClient } from '@azure/storage-blob';
import { DefaultAzureCredential } from '@azure/identity';
import { v4 as uuidv4 } from 'uuid';
Expand Down Expand Up @@ -43,10 +43,10 @@ export class ClaimsScrubberService {
private kafka: Kafka | null = null;
private producer: Producer | null = null;
private consumer: Consumer | null = null;
private cosmosClient: CosmosClient | null = null;
private database: Database | null = null;
private rulesContainer: Container | null = null;
private auditContainer: Container | null = null;
private mongoClient: MongoClient | null = null;
private database: Db | null = null;
private rulesCollection: Collection | null = null;
private auditCollection: Collection | null = null;
private blobServiceClient: BlobServiceClient | null = null;
private archiveContainer: ContainerClient | null = null;
private startTime: number;
Expand Down Expand Up @@ -104,14 +104,12 @@ export class ClaimsScrubberService {
});
}

// Initialize Cosmos DB
this.cosmosClient = new CosmosClient({
endpoint: this.config.cosmosDb.endpoint,
aadCredentials: credential,
});
this.database = this.cosmosClient.database(this.config.cosmosDb.databaseName);
this.rulesContainer = this.database.container(this.config.cosmosDb.rulesContainerName);
this.auditContainer = this.database.container(this.config.cosmosDb.auditContainerName);
// Initialize MongoDB
this.mongoClient = new MongoClient(this.config.mongoDb.connectionString);
await this.mongoClient.connect();
this.database = this.mongoClient.db(this.config.mongoDb.databaseName);
this.rulesCollection = this.database.collection(this.config.mongoDb.rulesCollectionName);
this.auditCollection = this.database.collection(this.config.mongoDb.auditCollectionName);

// Initialize Blob Storage
if (this.config.storage.connectionString) {
Expand All @@ -127,29 +125,26 @@ export class ClaimsScrubberService {
this.archiveContainer = this.blobServiceClient.getContainerClient(this.config.storage.containerName);
}

// Load custom rules from Cosmos DB
// Load custom rules from MongoDB
await this.loadCustomRules();
}

/**
* Load custom rules from Cosmos DB
* Load custom rules from MongoDB
*/
private async loadCustomRules(): Promise<void> {
if (!this.rulesContainer) return;
if (!this.rulesCollection) return;

try {
const query = {
query: 'SELECT * FROM c WHERE c.type = @type AND c.enabled = true',
parameters: [{ name: '@type', value: 'custom' }],
};

const { resources } = await this.rulesContainer.items.query<CustomRule>(query).fetchAll();
const resources = await this.rulesCollection
.find<CustomRule>({ type: 'custom', enabled: true })
.toArray();

for (const rule of resources) {
this.ruleEngine.addCustomRule(rule);
}

console.log(`Loaded ${resources.length} custom rules from Cosmos DB`);
console.log(`Loaded ${resources.length} custom rules from MongoDB`);
} catch (error) {
console.error('Failed to load custom rules:', error);
}
Expand Down Expand Up @@ -376,18 +371,17 @@ export class ClaimsScrubberService {
}

/**
* Audit the validation to Cosmos DB
* Audit the validation to MongoDB
*/
private async auditValidation(
claim: X12_837_Claim,
result: ClaimValidationResult,
correlationId: string
): Promise<void> {
if (!this.auditContainer) return;
if (!this.auditCollection) return;

try {
const auditRecord = {
id: uuidv4(),
claimId: claim.claimId,
claimType: claim.claimType,
patientControlNumber: claim.claimHeader.patientControlNumber,
Expand All @@ -405,11 +399,11 @@ export class ClaimsScrubberService {
.map(r => r.editCode),
validationTimeMs: result.totalValidationTimeMs,
correlationId,
timestamp: new Date().toISOString(),
ttl: 90 * 24 * 60 * 60, // 90 days TTL
timestamp: new Date(),
expireAt: new Date(Date.now() + 90 * 24 * 60 * 60 * 1000), // 90 days TTL
};

await this.auditContainer.items.create(auditRecord);
await this.auditCollection.insertOne(auditRecord);
Comment on lines +402 to +406
Copy link

Copilot AI Mar 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

expireAt is being written for a 90-day TTL, but MongoDB will only delete documents automatically if the collection has a TTL index on that field. Consider creating a TTL index on expireAt during initialization (or provisioning it explicitly), otherwise audit records will accumulate indefinitely.

Copilot uses AI. Check for mistakes.
} catch (error) {
console.error('Failed to audit claim', { claimId: claim.claimId }, error);
}
Expand All @@ -419,9 +413,9 @@ export class ClaimsScrubberService {
* Add a custom validation rule
*/
async addCustomRule(rule: CustomRule): Promise<void> {
// Save to Cosmos DB
if (this.rulesContainer) {
await this.rulesContainer.items.create(rule);
// Save to MongoDB
if (this.rulesCollection) {
await this.rulesCollection.insertOne(rule);
}

// Add to in-memory rule engine
Expand All @@ -448,7 +442,7 @@ export class ClaimsScrubberService {
async getHealth(): Promise<HealthStatus> {
const checks: HealthStatus['checks'] = {
kafka: await this.checkKafkaHealth(),
cosmosDb: await this.checkCosmosDbHealth(),
mongoDb: await this.checkMongoDbHealth(),
storage: await this.checkStorageHealth(),
ruleEngine: this.checkRuleEngineHealth(),
};
Expand Down Expand Up @@ -515,19 +509,19 @@ export class ClaimsScrubberService {
}

/**
* Check Cosmos DB health
* Check MongoDB health
*/
private async checkCosmosDbHealth(): Promise<ComponentHealth> {
private async checkMongoDbHealth(): Promise<ComponentHealth> {
const start = Date.now();
try {
if (!this.database) {
return {
status: 'unhealthy',
lastCheck: new Date().toISOString(),
error: 'Cosmos DB client not initialized',
error: 'MongoDB client not initialized',
};
}
await this.database.read();
await this.database.command({ ping: 1 });
return {
status: 'healthy',
latencyMs: Date.now() - start,
Expand Down Expand Up @@ -629,6 +623,7 @@ export class ClaimsScrubberService {
async close(): Promise<void> {
if (this.consumer) await this.consumer.disconnect();
if (this.producer) await this.producer.disconnect();
if (this.mongoClient) await this.mongoClient.close();
}
}

Expand Down
14 changes: 7 additions & 7 deletions src/services/claims-scrubbing-service/src/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,11 +49,11 @@ function getConfig(): ClaimsScrubberConfig {
containerName: process.env.CLAIMS_CONTAINER || 'claims-archive',
archivePathPattern: '{claimType}/{status}/{yyyy}/{MM}/{dd}',
},
cosmosDb: {
endpoint: process.env.COSMOS_ENDPOINT || '',
databaseName: process.env.COSMOS_DATABASE || 'claims-scrubbing',
rulesContainerName: process.env.COSMOS_RULES_CONTAINER || 'validation-rules',
auditContainerName: process.env.COSMOS_AUDIT_CONTAINER || 'validation-audit',
mongoDb: {
connectionString: process.env.MONGODB_CONNECTION_STRING || '',
databaseName: process.env.MONGODB_DATABASE || 'CloudHealthOffice',
Copy link

Copilot AI Mar 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Defaulting mongoDb.databaseName to "CloudHealthOffice" (and setting the ConfigMap to the same) is inconsistent with most other service manifests, which use "cloudhealthoffice". MongoDB treats database names as distinct strings, so this can split data across two DBs. Consider standardizing the default/configured DB name to the shared value used elsewhere.

Suggested change
databaseName: process.env.MONGODB_DATABASE || 'CloudHealthOffice',
databaseName: process.env.MONGODB_DATABASE || 'cloudhealthoffice',

Copilot uses AI. Check for mistakes.
rulesCollectionName: process.env.MONGODB_RULES_COLLECTION || 'ScrubRules',
auditCollectionName: process.env.MONGODB_AUDIT_COLLECTION || 'ScrubAudit',
},
ruleEngine: {
parallelExecution: process.env.PARALLEL_RULES === 'true',
Expand Down Expand Up @@ -357,11 +357,11 @@ async function start(): Promise<void> {
// Initialize service (connect to Azure resources)
// Skip initialization in development mode without Azure resources
const config = getConfig();
if (config.cosmosDb.endpoint) {
if (config.mongoDb.connectionString) {
await svc.initialize();
console.log('Claims Scrubbing Service initialized');
} else {
console.log('Running in development mode (no Azure resources)');
console.log('Running in development mode (no database configured)');
}

server.listen(PORT, () => {
Expand Down
12 changes: 6 additions & 6 deletions src/services/claims-scrubbing-service/src/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -652,12 +652,12 @@ export interface ClaimsScrubberConfig {
/** Archive path pattern */
archivePathPattern: string;
};
/** Cosmos DB for rule storage and audit */
cosmosDb: {
endpoint: string;
/** MongoDB for rule storage and audit */
mongoDb: {
connectionString: string;
databaseName: string;
rulesContainerName: string;
auditContainerName: string;
rulesCollectionName: string;
auditCollectionName: string;
};
/** Rule engine configuration */
ruleEngine: {
Expand Down Expand Up @@ -770,7 +770,7 @@ export interface HealthStatus {
timestamp: string;
checks: {
kafka: ComponentHealth;
cosmosDb: ComponentHealth;
mongoDb: ComponentHealth;
storage: ComponentHealth;
ruleEngine: ComponentHealth;
dapr?: ComponentHealth;
Expand Down
31 changes: 7 additions & 24 deletions src/services/encounter-service/k8s/encounter-service-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,7 @@ metadata:
namespace: cloudhealthoffice
data:
ASPNETCORE_ENVIRONMENT: "Production"
CosmosDb__DatabaseName: "CloudHealthOffice"
CosmosDb__ContainerName: "Encounters"
MongoDb__DatabaseName: "CloudHealthOffice"
Copy link

Copilot AI Mar 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MongoDb__DatabaseName is set to "CloudHealthOffice" here, while most other service deployments use "cloudhealthoffice". Because MongoDB database names are distinct by string, this can cause the encounter-service to read/write from a different database than the rest of the platform. Align the DB name with the shared convention unless intentional.

Suggested change
MongoDb__DatabaseName: "CloudHealthOffice"
MongoDb__DatabaseName: "cloudhealthoffice"

Copilot uses AI. Check for mistakes.
---
apiVersion: apps/v1
kind: Deployment
Expand All @@ -18,7 +17,7 @@ metadata:
tier: backend
component: encounter-processing
spec:
replicas: 3
replicas: 1
selector:
Comment on lines 19 to 21
Copy link

Copilot AI Mar 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reduces the Deployment replica count from 3 to 1 (and HPA minReplicas to 1). If this manifest is used for production, a single replica reduces availability during node drain/rollouts and can cause brief downtime. If the goal is to avoid rollout timeouts due to capacity, consider keeping minReplicas >= 2 and adjusting resources/rollout strategy instead, or document that this manifest is for dev/staging.

Copilot uses AI. Check for mistakes.
matchLabels:
app: encounter-service
Expand All @@ -40,34 +39,18 @@ spec:
- name: MongoDb__ConnectionString
valueFrom:
secretKeyRef:
name: encounter-service-secret
key: MongoDb__ConnectionString
optional: true
name: mongodb-secret
key: connectionString
- name: ASPNETCORE_ENVIRONMENT
valueFrom:
configMapKeyRef:
name: encounter-service-config
key: ASPNETCORE_ENVIRONMENT
- name: CosmosDb__DatabaseName
- name: MongoDb__DatabaseName
valueFrom:
configMapKeyRef:
name: encounter-service-config
key: CosmosDb__DatabaseName
- name: CosmosDb__ContainerName
valueFrom:
configMapKeyRef:
name: encounter-service-config
key: CosmosDb__ContainerName
- name: CosmosDb__Endpoint
valueFrom:
secretKeyRef:
name: cosmos-db-secret
key: endpoint
- name: CosmosDb__Key
valueFrom:
secretKeyRef:
name: cosmos-db-secret
key: key
key: MongoDb__DatabaseName
resources:
requests:
cpu: 200m
Expand Down Expand Up @@ -115,7 +98,7 @@ spec:
apiVersion: apps/v1
kind: Deployment
name: encounter-service
minReplicas: 3
minReplicas: 1
maxReplicas: 20
Comment on lines +101 to 102
Copy link

Copilot AI Mar 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lowering HPA minReplicas to 1 reduces baseline availability and can introduce downtime during maintenance/rollouts. If this is intended as a capacity/rollout-timeout mitigation, consider adding a note about environment scope (dev/staging) or keeping minReplicas >= 2 for production HA.

Copilot uses AI. Check for mistakes.
metrics:
- type: Resource
Expand Down
4 changes: 2 additions & 2 deletions src/services/reference-data-service/k8s/reference-data-service-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ metadata:
name: reference-data-service
namespace: cloudhealthoffice
spec:
replicas: 2
replicas: 1
selector:
Comment on lines 92 to 94
Copy link

Copilot AI Mar 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This lowers replicas and HPA minReplicas from 2 to 1. For a production deployment this removes redundancy and increases risk of downtime during rollouts or node maintenance. If the goal is to address rollout timeouts/capacity, consider keeping minReplicas at 2 and adjusting resource requests/limits or rollout parameters instead, or document that this manifest targets non-prod environments.

Copilot uses AI. Check for mistakes.
matchLabels:
app: reference-data-service
Expand Down Expand Up @@ -155,7 +155,7 @@ spec:
apiVersion: apps/v1
kind: Deployment
name: reference-data-service
minReplicas: 2
minReplicas: 1
maxReplicas: 10
metrics:
- type: Resource
Expand Down
Loading