Copilot Context - Matrikkel Java Project

This file provides additional context for AI assistants working with this codebase.

Project Purpose

Integration with Norwegian Matrikkel (cadastral register) SOAP API to download and store property data in PostgreSQL.

Architecture Decisions

Why Java over PHP?

Automatic SOAP Serialization: JAX-WS handles complex nested objects (MatrikkelBubbleId with snapshotVersion) without manual XML construction
Type Safety: Compile-time checking prevents runtime errors
Better Tooling: wsimport generates perfect client classes from WSDL
Official Support: Matrikkel API documentation uses Java examples

Why Spring Boot?

Dependency injection for clean architecture
Spring Data JPA for database abstraction
Excellent PostgreSQL integration
Built-in transaction management
Production-ready features (health checks, metrics)

Why Cursor-based Pagination?

API supports up to 5000 objects per batch
More efficient than offset-based pagination for large datasets
Uses MatrikkelBubbleId as cursor for next batch
Automatically stops when receiving fewer objects than requested

API Specifics

MatrikkelBubbleId Structure

public class MatrikkelBubbleId {
    private Long value;              // Object ID
    private SnapshotVersion version; // Temporal version
}

SnapshotVersion Critical Setup

⚠️ MUST set to far future date to avoid permission errors:

SnapshotVersion snapshotVersion = new SnapshotVersion();
ZonedDateTime futureDate = ZonedDateTime.of(9999, 1, 1, 0, 0, 0, 0, ZoneId.of("Europe/Oslo"));
snapshotVersion.setTimestamp(futureDate);

Why future date?

Matrikkel API has temporal versioning
Historical data requires special permissions
Using far future date (9999-01-01) ensures we get current data
This matches working PHP implementation

Batch Processing Pattern

MatrikkelBubbleId cursor = null;
do {
    List<Matrikkelenhet> batch = nedlastningService.findObjekterEtterId(
        cursor,                           // null for first batch
        "Matrikkelenhet",                // Object type
        "{\"kommunefilter\": [\"4601\"]}", // JSON filter
        5000,                            // Max batch size
        context                          // With snapshotVersion
    );
    
    // Process batch...
    
    // Get last object's ID as cursor
    cursor = batch.get(batch.size() - 1).getId();
    
    // Stop if batch is smaller than requested (last batch)
    if (batch.size() < 5000) break;
    
} while (true);

Database Schema

Core Tables

matrikkel_matrikkelenheter - Property units (main table)
matrikkel_eiere - Owners (1:many with matrikkelenheter)
matrikkel_adresser - Addresses (1:many with matrikkelenheter)

Key Relationships

Matrikkelenhet (1) ←→ (many) Eier
Matrikkelenhet (1) ←→ (many) Adresse

Important Indexes

kommunenummer - Primary query filter
gardsnummer, bruksnummer - Property identification
matrikkelenhet_id - API object ID (unique)

Common Code Patterns

Service Layer Transaction

@Service
@Transactional
public class MatrikkelenhetImportService {
    public int importMatrikkelenheterForKommune(String kommunenummer) {
        // 1. Fetch from API
        List<ApiMatrikkelenhet> apiObjects = 
            nedlastningClient.findAllMatrikkelenheterForKommune(kommunenummer);
        
        // 2. Map to entities
        List<Matrikkelenhet> entities = apiObjects.stream()
            .map(mapper::toEntity)
            .collect(Collectors.toList());
        
        // 3. Save (batch insert)
        return repository.saveAll(entities).size();
    }
}

SOAP Client Wrapper

@Component
public class NedlastningClientWrapper {
    private final NedlastningService nedlastningService; // Auto-configured
    
    public List<Matrikkelenhet> findAllMatrikkelenheterForKommune(String kommunenummer) {
        // Pagination loop
        // Error handling
        // Logging
    }
}

Repository Pattern

@Repository
public interface MatrikkelenhetRepository extends JpaRepository<Matrikkelenhet, Long> {
    Optional<Matrikkelenhet> findByMatrikkelenhetId(Long matrikkelenhetId);
    List<Matrikkelenhet> findByKommunenummer(String kommunenummer);
    
    @Query("SELECT m FROM Matrikkelenhet m WHERE ...")
    Optional<Matrikkelenhet> customQuery(...);
}

Development Workflow

1. Adding New WSDL Service

# 1. Add WSDL file
cp ServiceXYZ.wsdl src/main/resources/wsdl/

# 2. Add execution block in pom.xml
<execution>
    <id>wsimport-xyz</id>
    <configuration>
        <wsdlFiles><wsdlFile>ServiceXYZ.wsdl</wsdlFile></wsdlFiles>
        <packageName>no.matrikkel.client.generated.xyz</packageName>
    </configuration>
</execution>

# 3. Generate classes
mvn clean compile

# 4. Create wrapper
@Component
public class XyzClientWrapper {
    private final XyzService xyzService;
    // ...
}

# 5. Register bean
@Bean
public XyzService xyzService() throws Exception {
    // ...
}

2. Adding New Entity

# 1. Create entity class
@Entity
@Table(name = "matrikkel_xyz")
public class XyzEntity { ... }

# 2. Create repository
public interface XyzRepository extends JpaRepository<XyzEntity, Long> { ... }

# 3. Create Flyway migration
# V{version}__add_xyz_table.sql

# 4. Create service
@Service
public class XyzImportService { ... }

3. Testing Strategy

// Unit test - Mock dependencies
@ExtendWith(MockitoExtension.class)
class ServiceTest {
    @Mock private SoapClient client;
    @Mock private Repository repository;
    @InjectMocks private Service service;
    
    @Test
    void testLogic() { ... }
}

// Integration test - Real database
@SpringBootTest
@Testcontainers
class IntegrationTest {
    @Container
    static PostgreSQLContainer<?> postgres = ...;
    
    @Test
    void testFullFlow() { ... }
}

Performance Considerations

Batch Inserts

Hibernate batch size: 100 (configured in application.yml)
Use saveAll() instead of multiple save() calls
Consider @Transactional(propagation = Propagation.REQUIRES_NEW) for large batches

Connection Pooling

spring.datasource.hikari:
  maximum-pool-size: 10
  minimum-idle: 2
  connection-timeout: 30000

Memory Management

Don't load all objects into memory at once
Process batches incrementally
Consider streaming for very large datasets

Security Considerations

Credentials Management

✅ Use environment variables
✅ Use Spring Boot's encrypted properties
❌ Never commit credentials to git
❌ Never hardcode passwords in code

API Authentication

// Basic Auth via JAX-WS
BindingProvider bp = (BindingProvider) port;
bp.getRequestContext().put(BindingProvider.USERNAME_PROPERTY, username);
bp.getRequestContext().put(BindingProvider.PASSWORD_PROPERTY, password);

Monitoring and Logging

Log Levels

INFO - Batch progress, import summaries
DEBUG - API calls, SQL queries, detailed flow
WARN - Recoverable errors, retries
ERROR - Fatal errors, data corruption

Key Metrics to Track

Objects imported per batch
Total import time
API response times
Database insert performance
Error rates

Known Issues and Solutions

Issue: "No historical data permission"

Solution: Set snapshotVersion to far future date (9999-01-01)

Issue: "Batch stops prematurely"

Solution: Check if batch.size() < maxBatchSize condition is correct

Issue: "OutOfMemoryError"

Solution: Process batches incrementally, don't accumulate all objects

Issue: "Connection timeout"

Solution: Increase timeout in application.yml or implement retry logic

Related Projects

PHP Implementation: /opt/matrikkel/ (reference for API behavior)
WSDL Documentation: /opt/matrikkel/doc/wsdl/

Useful Commands

# Build
mvn clean package

# Run with specific profile
mvn spring-boot:run -Dspring-boot.run.profiles=dev

# Generate classes from WSDL only
mvn generate-sources

# Run specific test
mvn test -Dtest=MatrikkelenhetImportServiceTest

# Database migration
mvn flyway:migrate
mvn flyway:info
mvn flyway:clean  # CAUTION: Drops all objects!

# Docker
docker-compose up -d postgres
docker-compose logs -f matrikkel-app

Quick Reference

Bergen Kommune Number

4601 - Use for all testing/examples

API Environments

Test: https://wsweb-test.matrikkel.no/matrikkel-ws-v1.0/
Prod: https://wsweb.matrikkel.no/matrikkel-ws-v1.0/

Database Connection (Local)

Host: 10.0.2.15
Port: 5435
Database: matrikkel
Schema: public

This context file should help AI assistants understand the project structure, patterns, and important implementation details when assisting with code generation and problem-solving.

FilesExpand file tree

.copilot-context.md

Latest commit

History