Feature/beeper connector

PR.md

@@ -0,0 +1,51 @@
+# Add Beeper Connector Service for MetaState Integration
+
+## Description
+
+This PR adds a new service for extracting messages from the Beeper messaging platform and converting them to Resource Description Framework (RDF) format. This enables semantic integration with the MetaState ecosystem, particularly the eVault and Ontology Service, while providing visualization tools for analyzing communication patterns.
+
+## Features
+
+- Extract messages from the Beeper SQLite database
+- Convert messages to RDF triples with semantic relationships compatible with MetaState ontology
+- Generate visualization tools for data analysis:
+  - Network graph showing connections between senders and rooms
+  - Message activity timeline
+  - Word cloud of common terms
+  - Sender activity chart
+- NPM scripts for easy integration with the monorepo structure
+
+## Implementation
+
+- New service under `services/beeper-connector/`
+- Python-based implementation with clear CLI interface
+- RDF output compatible with semantic web standards and MetaState ontology
+- Comprehensive documentation for integration with other MetaState services
+
+## Integration with MetaState Architecture
+
+This connector enhances the MetaState ecosystem by:
+
+1. **Data Ingestion**: Providing a way to import real-world messaging data into the MetaState eVault
+2. **Semantic Representation**: Converting messages to RDF triples that can be processed by the Ontology Service
+3. **Identity Integration**: Supporting connections with the W3ID system for identity verification
+4. **Visualization**: Offering tools to analyze communication patterns and relationships
+
+## How to Test
+
+1. Install the required packages: `pip install -r services/beeper-connector/requirements.txt`
+2. Run the extraction: `cd services/beeper-connector && python beeper_to_rdf.py --visualize`
+3. Check the output RDF file (`beeper_messages.ttl`) and visualizations folder
+
+## Future Enhancements
+
+- Direct integration with eVault API for seamless data import
+- Support for additional messaging platforms
+- Enhanced ontology mapping for richer semantic relationships
+- Real-time data synchronization
+
+## Notes
+
+- This tool respects user privacy by only accessing local database files
+- RDF output follows standard Turtle format compatible with semantic web tools
+- Visualizations require matplotlib, networkx, and wordcloud libraries 

README.md

@@ -92,14 +92,15 @@ Learn more about the power of Turborepo:
 | [W3ID](./infrastructure/w3id/)             | In Progress |
 | [eID Wallet](./infrastructure/eid-wallet/) | In Progress |
 | EVault Core                                | Planned     |
-| Web3 Adapter                               | Planned     |
+| [Web3 Adapter](./infrastructure/web3-adapter/) | ✅ Complete |
 
 ## Documentation Links
 
 | Documentation                | Description                                 | Link                                                                       |
 | ---------------------------- | ------------------------------------------- | -------------------------------------------------------------------------- |
 | MetaState Prototype          | Main project README                         | [README.md](./README.md)                                                   |
 | W3ID                         | Web 3 Identity System documentation         | [W3ID README](./infrastructure/w3id/README.md)                             |
+| Web3 Adapter                 | Platform data synchronization adapter       | [Web3 Adapter README](./infrastructure/web3-adapter/README.md)             |
 | eVault Core                  | Core eVault system documentation            | [eVault Core README](./infrastructure/evault-core/README.md)               |
 | eVault Core W3ID Integration | W3ID integration details for eVault Core    | [W3ID Integration](./infrastructure/evault-core/docs/w3id-integration.md)  |
 | eVault Provisioner           | Provisioning eVault instances documentation | [eVault Provisioner README](./infrastructure/evault-provisioner/README.md) |
@@ -114,7 +115,14 @@ prototype/
 ├─ infrastructure/
 │  ├─ evault-core/
 │  │  └─ package.json
-│  └─ w3id/
+│  ├─ w3id/
+│  │  └─ package.json
+│  └─ web3-adapter/
+│     ├─ src/
+│     │  ├─ adapter.ts
+│     │  ├─ types.ts
+│     │  └─ index.ts
+│     ├─ examples/
 │     └─ package.json
 ├─ packages/
 │  ├─ eslint-config/
@@ -143,3 +151,23 @@ prototype/
 ├─ README.md (This File)
 └─ turbo.json (Configures TurboRepo)
 ```
+
+## Web3 Adapter
+
+The Web3 Adapter is a critical infrastructure component that enables seamless data exchange between different social media platforms through the W3DS infrastructure. It provides:
+
+### Key Features
+- **Schema Mapping**: Maps platform-specific data models to universal ontology schemas
+- **ID Translation**: Maintains bidirectional mapping between W3IDs and platform-specific identifiers
+- **ACL Management**: Handles access control lists for read/write permissions
+- **MetaEnvelope Support**: Converts data to/from eVault's envelope-based storage format
+- **Cross-Platform Exchange**: Enables data sharing between Twitter, Instagram, and other platforms
+
+### How It Works
+1. Platform data is converted to universal ontology format using schema mappings
+2. Data is broken down into atomic Envelopes with ontology references
+3. MetaEnvelopes group related envelopes as logical entities
+4. W3IDs are mapped to local platform IDs for seamless integration
+5. ACLs control data access across platforms
+
+For detailed implementation and usage examples, see the [Web3 Adapter documentation](./infrastructure/web3-adapter/README.md).

infrastructure/README.md

@@ -0,0 +1,227 @@
+# MetaState Infrastructure Components
+
+## Overview
+
+The infrastructure layer provides the core building blocks for the MetaState Prototype ecosystem, enabling decentralized identity, data storage, and cross-platform interoperability.
+
+## Components
+
+### 1. W3ID - Web3 Identity System
+**Status:** In Progress
+
+A decentralized identity management system that provides:
+- Unique global identifiers (W3IDs)
+- Identity verification and authentication
+- Cross-platform identity resolution
+- Integration with eVaults for secure data storage
+
+[📖 Documentation](./w3id/README.md)
+
+### 2. Web3 Adapter
+**Status:** ✅ Complete
+
+Enables seamless data exchange between different platforms through the W3DS infrastructure:
+- Schema mapping between platform-specific and universal formats
+- W3ID to local ID bidirectional mapping
+- Access Control List (ACL) management
+- MetaEnvelope creation and parsing
+- Cross-platform data transformation
+
+[📖 Documentation](./web3-adapter/README.md) | [📋 Schemas](./web3-adapter/docs/schemas.md)
+
+### 3. eVault Core
+**Status:** Planned
+
+Personal data vaults that serve as the source of truth for user data:
+- Envelope-based data storage
+- Graph database structure
+- W3ID integration
+- Access control enforcement
+- Web3 Protocol support
+
+[📖 Documentation](./evault-core/README.md)
+
+### 4. eVault Provisioner
+**Status:** Planned
+
+Manages the lifecycle of eVault instances:
+- eVault creation and initialization
+- Resource allocation
+- Backup and recovery
+- Multi-vault management
+
+[📖 Documentation](./evault-provisioner/README.md)
+
+### 5. eID Wallet
+**Status:** In Progress
+
+Digital wallet for managing electronic identities:
+- Credential storage
+- Identity verification
+- Signature generation
+- Integration with W3ID
+
+## Architecture
+
+```
+┌──────────────────────────────────────────────────────────┐
+│                     Applications Layer                     │
+│          (Twitter, Instagram, Chat Platforms)              │
+└──────────────────────────────────────────────────────────┘
+                              │
+                              ▼
+┌──────────────────────────────────────────────────────────┐
+│                      Web3 Adapter                          │
+│  • Schema Mapping  • ID Translation  • ACL Management      │
+└──────────────────────────────────────────────────────────┘
+                              │
+                              ▼
+┌──────────────────────────────────────────────────────────┐
+│                    Infrastructure Layer                    │
+│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐ │
+│  │   W3ID   │  │  eVault  │  │Provisioner│  │eID Wallet│ │
+│  └──────────┘  └──────────┘  └──────────┘  └──────────┘ │
+└──────────────────────────────────────────────────────────┘
+                              │
+                              ▼
+┌──────────────────────────────────────────────────────────┐
+│                      Services Layer                        │
+│         (Ontology Service, Registry, PKI, etc.)           │
+└──────────────────────────────────────────────────────────┘
+```
+
+## Data Flow
+
+### Writing Data (Platform → eVault)
+
+1. **Platform generates data** (e.g., a tweet, post, message)
+2. **Web3 Adapter converts** using schema mappings
+3. **Data broken into Envelopes** with ontology references
+4. **MetaEnvelope created** with W3ID and ACL
+5. **Stored in user's eVault** as graph nodes
+
+### Reading Data (eVault → Platform)
+
+1. **Platform requests data** using W3ID or query
+2. **eVault retrieves MetaEnvelope** with access control check
+3. **Web3 Adapter converts** to platform-specific format
+4. **ID mapping applied** (W3ID → local ID)
+5. **Data delivered to platform** in native format
+
+## Key Concepts
+
+### Envelopes
+Atomic units of data with:
+- Unique identifier
+- Ontology reference
+- Value and type
+- Access control
+
+### MetaEnvelopes
+Logical containers that group related envelopes:
+- Represent complete entities (posts, profiles, etc.)
+- Have unique W3IDs
+- Include ACL for access control
+
+### Schema Mappings
+Define relationships between:
+- Platform-specific fields
+- Universal ontology fields
+- Transformation functions
+
+### W3IDs
+Global unique identifiers that:
+- Are platform-agnostic
+- Enable cross-platform references
+- Map to local platform IDs
+
+## Development
+
+### Prerequisites
+
+- Node.js 20+ and pnpm
+- TypeScript 5.0+
+- Docker (for services)
+
+### Setup
+
+```bash
+# Install dependencies
+pnpm install
+
+# Build all infrastructure components
+pnpm build
+
+# Run tests
+pnpm test
+```
+
+### Testing Individual Components
+
+```bash
+# Test Web3 Adapter
+cd infrastructure/web3-adapter
+pnpm test
+
+# Test W3ID
+cd infrastructure/w3id
+pnpm test
+```
+
+## Integration Points
+
+### With Services
+
+- **Ontology Service**: Schema definitions and validation
+- **Registry**: Service discovery and metadata
+- **PKI**: Certificate management and verification
+- **Search**: Content indexing and discovery
+
+### With Platforms
+
+- **Social Media**: Twitter, Instagram, Facebook
+- **Messaging**: Chat applications, forums
+- **Content**: Blogs, media platforms
+- **Enterprise**: Business applications
+
+## Security Considerations
+
+1. **Access Control**: All data protected by ACLs
+2. **Identity Verification**: W3ID system ensures authenticity
+3. **Data Encryption**: Sensitive data encrypted at rest
+4. **Audit Logging**: All operations logged for compliance
+5. **Privacy**: Users control their data through eVaults
+
+## Roadmap
+
+### Phase 1: Foundation (Current)
+- ✅ Web3 Adapter implementation
+- 🔄 W3ID system development
+- 🔄 eID Wallet implementation
+
+### Phase 2: Core Infrastructure
+- [ ] eVault Core implementation
+- [ ] eVault Provisioner
+- [ ] Web3 Protocol integration
+
+### Phase 3: Platform Integration
+- [ ] Platform SDKs
+- [ ] Migration tools
+- [ ] Performance optimization
+
+### Phase 4: Advanced Features
+- [ ] AI-powered schema mapping
+- [ ] Real-time synchronization
+- [ ] Conflict resolution
+- [ ] Advanced analytics
+
+## Contributing
+
+See the main [project README](../README.md) for contribution guidelines.
+
+## Resources
+
+- [MetaState Prototype Documentation](../README.md)
+- [Web3 Adapter Documentation](./web3-adapter/README.md)
+- [W3ID Documentation](./w3id/README.md)
+- [Ontology Service](../services/ontology/README.md)