feat(database): add PostgreSQL database support

[itsmostafa]

Summary

This PR migrates the CSET backend from SQL Server to PostgreSQL, enabling cross-platform database compatibility and improving deployment flexibility. The migration includes comprehensive updates to the data layer, business logic, and database I/O components while maintaining backward compatibility with existing functionality.

Changes Made

Backend (.NET Core API)

NuGet Package Migration:

• ✅ Removed Microsoft.Data.SqlClient and all SQL Server-specific EF Core packages
• ✅ Removed Snickler.EFCore (deprecated stored procedure library)
• ✅ Added Npgsql.EntityFrameworkCore.PostgreSQL v8.0.11 across all backend projects

Data Layer (Entity Framework Core):

• CSETContext.cs - Switched provider from UseSqlServer to UseNpgsql
• CsetwebContextProcedures.cs - Regenerated stored procedure scaffolding for PostgreSQL
• Computed Columns - Converted SQL Server computed column syntax to PostgreSQL format (SHA1 hashing)
• Stored Procedures - Migrated all stored procedure calls to async/await pattern using EF Core scaffolded methods

Database I/O Layer (DBIO.cs):

• Replaced SqlConnection, SqlCommand, SqlDataAdapter with NpgsqlConnection, NpgsqlCommand, NpgsqlDataAdapter
• Implemented PostgreSQL COPY command for bulk inserts (replacing SqlBulkCopy)
• Updated identity retrieval from SCOPE_IDENTITY() to LASTVAL()
• Enhanced binary import for improved bulk insert performance

Business Layer:

• AggregationBusiness.cs - Converted stored procedure calls to async
• AnalyticsBusiness.cs - Converted stored procedure calls to async
• AssessmentBusiness.cs - Converted stored procedure calls to async
• MaturityBusiness.cs - Converted stored procedure calls to async
• ComponentQuestionBusiness.cs - Converted stored procedure calls to async
• QuestionRequirementManager.cs - Converted stored procedure calls to async
• ReportsDataBusiness.cs - Converted stored procedure calls to async
• TrendDataProcessor.cs - Converted stored procedure calls to async

Interfaces:

• Updated all business interfaces to support Task<T> async return types
• Maintained method signatures for backward compatibility

Controllers:

• AnalysisController.cs - Updated to handle async business layer calls
• AnalyticsDashboardController.cs - Updated to handle async business layer calls
• CieController.cs - Updated to handle async business layer calls
• QuestionsController.cs - Updated to handle async business layer calls
• ReportsController.cs - Updated to handle async business layer calls

Database Migration Scripts

• convert-mssql-bak-to-postgres.sh - Fixed PostgreSQL port configuration (55432 → 5432)
• pgloader-mssql-to-pg.load - Updated migration configuration

Configuration

• appsettings.json - Updated default connection string to PostgreSQL format
• Startup.cs - No changes required (EF Core handles provider abstraction)

Files Modified

• 34 files changed
• 1,055 insertions
• 1,236 deletions
• Net reduction of 181 lines (improved code efficiency with async patterns)

Testing Notes

Prerequisites

• PostgreSQL 17 installed and running
• Database migrated using DatabaseScripts/Migration/convert-mssql-bak-to-postgres.sh

Testing Checklist

• Verify backend builds successfully (make build-backend or dotnet build)
• Verify connection to PostgreSQL database
• Test stored procedure calls across all business layers
• Verify bulk insert operations (BulkSqlUpload functionality)
• Test assessment creation and retrieval
• Test analytics and reporting functionality
• Verify async operations complete successfully
• Run existing unit tests (if available)
• Test data export functionality

Database Connection

Default PostgreSQL connection (from appsettings.json):

Host=postgres;Database=csetweb;Username=cset;Password=password;Port=5432

Update this connection string to match your PostgreSQL instance.

Technical Notes

Async/Await Pattern

All stored procedure calls now use async/await for improved scalability:

// Old (SQL Server with Snickler.EFCore)
var result = context.LoadStoredProc("usp_ProcName")
    .WithSqlParam("param", value)
    .ExecuteStoredProc();

// New (PostgreSQL with EF Core scaffolded procedures)
var result = await context.Procedures.usp_ProcNameAsync(value);

Bulk Insert Implementation

PostgreSQL COPY command provides better performance than row-by-row inserts:

using (var writer = connection.BeginBinaryImport(copyCommand))
{
    foreach (DataRow row in dataTable.Rows)
    {
        writer.StartRow();
        foreach (var item in row.ItemArray)
        {
            writer.Write(item ?? DBNull.Value);
        }
    }
    writer.Complete();
}

Breaking Changes

⚠️ Connection String Format Change

• SQL Server: Server=...;Database=...;User Id=...;Password=...
• PostgreSQL: Host=...;Database=...;Username=...;Password=...;Port=5432

Update your deployment configuration files accordingly.

Checklist

• Code follows conventional commits standard
• All stored procedure calls converted to async
• NuGet packages updated for PostgreSQL
• Database I/O layer migrated
• Business layer interfaces updated
• Controllers updated to handle async calls
• Migration scripts updated
• Tests added/updated (pending backend test suite review)
• Documentation updated (pending verification of migration process)
• Breaking changes documented (connection string format)

Next Steps

After merging this PR:

1. Update deployment documentation with PostgreSQL setup instructions
2. Update CI/CD pipelines to use PostgreSQL for testing
3. Create PostgreSQL database initialization scripts
4. Test migration script with production database backup
5. Update developer onboarding documentation