Technical Deep Dive

EOS Core follows a layered architecture designed for modularity, extensibility, and auditable environmental impact calculations.

System Overview

Architectural Layers

API Layer

The API layer provides two interfaces:

REST API v2 (/v2/* on port 8040)

• FastAPI-based modern REST API
• JWT token authentication with access groups
• Batch calculation support
• Structured request logging
• Graceful shutdown with drain file detection

Legacy API v1 (/api/* on port 8050)

• Backward-compatible endpoints
• Wraps v2 API functionality
• Supports legacy authentication methods

Core Layer

CalcGraph - The central calculation structure:

# CalcGraph manages the entire calculation as a directed graph
class CalcGraph:
    root_node_uid: str           # Entry point for calculation
    nodes: dict[str, Node]       # All nodes in the graph
    mutations: list[Mutation]    # Auditable change log

    def add_graph_observer(observer)  # GFM triggering
    def apply_mutation(mutation)       # Transparent changes

Orchestrator - Coordinates GFM execution:

class Orchestrator(AbstractGraphObserver):
    async def run():
        # 1. Initialize nodes from root or linked sub-nodes
        # 2. Spawn GFM workers on initial nodes
        # 3. Scheduler loop:
        while gfms_pending:
            for gfm in scheduled_gfms:
                if gfm.should_be_scheduled():
                    status = gfm.can_run_now()
                    if status == READY:
                        await gfm.run(calc_graph)
                    elif status == WAITING:
                        reschedule(gfm)

Service Provider - Dependency injection container:

class ServiceProvider:
    postgres_db: PostgresDb
    glossary_service: GlossaryService
    matching_service: MatchingService
    node_service: NodeService
    calc_service: CalcService
    gap_filling_module_loader: GapFillingModuleLoader
    # ... additional services

Module Layer

Modules follow the Factory/Worker pattern:

Factory (Singleton per service):

• Initialized once at service startup
• Holds database connections and caches
• Spawns workers for individual nodes

Worker (Per node):

• should_be_scheduled() - Is this GFM relevant for this node?
• can_run_now() - Are dependencies satisfied?
• run() - Execute the gap-filling logic

Data Layer

PostgreSQL with asyncpg:

• Connection pooling for async operations
• JSONB fields for flexible node property storage
• Schema in database/postgres/schema.sql

Manager Classes (DAO pattern):

• PostgresGraphMgr - Node/edge persistence
• PgTermMgr - Glossary operations
• PgMatchingMgr - Ingredient matching data
• PostgresAccessMgr - User/group access control

Node Types

EOS uses a rich type system for graph nodes:

Node Type	Purpose
FoodProductFlowNode	Food product with composition
AggregationFoodProductFlowNode	Daily aggregation caches for performance optimization
ElementaryResourceEmissionNode	Environmental emissions
FoodProcessingActivityNode	Processing operations
TransportActivityNode	Transportation
ModeledActivityNode	Brightway LCA inventory

Property System

Nodes store data through typed properties:

Property Type	Description
QuantityProp	Measurements with units
LocationProp	Geographic data
GlossaryTermProp	Links to terminology
EnvironmentalFlowsProp	Impact results
NamesProp	Multi-language naming
GfmStateProp	GFM execution state

Data Flow

Multi-Tenancy

EOS supports multi-tenant isolation:

• Namespace - Organization/ecosystem isolation
• Access Group - Team/department within namespace
• User - Individual with OAuth2/email authentication
• Permissions - Per-node access control

Messaging Architecture

RabbitMQ enables distributed processing:

• High priority queue - Interactive requests
• Low priority queue - Batch processing
• Prefetch management - Resource-aware scheduling

Error Handling

Structured error model for domain errors:

@dataclass
class DataError:
    node_uid: str
    gfm_name: str
    message: str
    classification: ErrorClassification  # missing_matching, missing_lca_inventory, etc.
    log_level: LogLevel  # INFO, WARNING, ERROR

Graph Mutations

All graph changes are transparent and auditable:

Mutation Type	Purpose
AddNodeMutation	Insert new node
PropMutation	Update single property
AddEdgeMutation	Create node relationship
RemoveEdgeMutation	Delete relationship
DuplicateNodeMutation	Clone node

Benefits:

• Auditability - Full mutation log
• Determinism - Reproducible calculations
• Transparency - Non-technical review possible

Performance Architecture

EOS is optimized for real-time environmental impact calculations, targeting sub-2-second response times for interactive restaurant applications.

Performance Optimizations

Optimization	Impact
Matrix Calculations	Accelerated from 5s to 1s per calculation (80% improvement)
Concurrent Processing	Parallel operation across multiple pods with intelligent request queuing
GADM Service	Rust implementation achieving 3x memory reduction and 10x speed improvement
Multi-level Caching	In-memory callbacks at each LCA level within the same instance
Load Balancing	RabbitMQ-based request distribution across worker pods

Scalability

The system uses Elastic Kubernetes Service (EKS) with automatic scaling:

• Dynamic Scaling: Up to 1024 pods based on workload
• Karpenter: Automatic node provisioning and scaling
• Resource Optimization: Separated legacy and core deployments for efficient RAM usage
• Microservice Architecture: Streamlined in-memory processing to minimize messaging complexity

Caching Strategy

Multi-level caching reduces computational load:

┌─────────────────────────────────────────────────┐
│  Request Level   │ Cached calculation results  │
├─────────────────────────────────────────────────┤
│  GFM Level       │ Factory caches (emissions,  │
│                  │ activities, glossary terms) │
├─────────────────────────────────────────────────┤
│  Database Level  │ Connection pooling,         │
│                  │ query result caching        │
└─────────────────────────────────────────────────┘

• Intelligent cache invalidation based on data changes
• Selective invalidation to minimize recomputation
• Distributed caching for cloud deployments

Security Architecture

EOS implements a multi-layered security approach:

Authentication

Token Types:

• Staff tokens - Administrative access with elevated privileges
• Namespace tokens - Organization-level access
• User tokens - Individual user access with limited validity periods

Network Security

• Private Subnets: Worker nodes operate in private subnets
• Bastion Host: SSH access to VPC through bastion
• Access Groups: IAM roles for fine-grained access control

Monitoring

• CloudWatch: Continuous system monitoring and logging
• Structured Logging: Request tracking and performance metrics
• Health Checks: Automated system health monitoring

Brightway Integration

EOS leverages Brightway, an open-source LCA framework:

Why Brightway?

Capability	Benefit
Flexibility	Custom-tailored LCA calculations for diverse food products
Database Integration	Easy integration with ecoinvent, Agribalyse via import routines
Uncertainty Analysis	Robust methods for sensitivity analysis
Scalability	Handles complex calculations for detailed assessments
Community	Open-source with continuous methodology updates

Matrix Transformation

The Matrix Transform is the computational core for LCA calculations:

1. Graph to Matrix: Convert supply chain networks into mathematical matrices
2. Input-Output Modeling: Represent relationships between processes
3. Matrix Algebra: Calculate cumulative environmental impacts
4. Impact Assessment: Apply LCIA methods (e.g., IPCC GWP100)

# Conceptual matrix calculation
# A = technology matrix (process inputs/outputs)
# B = intervention matrix (environmental exchanges)
# f = final demand vector
# s = scaling vector = A^(-1) * f
# g = inventory result = B * s

Infrastructure

Container Orchestration

• Kubernetes: Docker container orchestration and deployment
• Karpenter: Automatic node provisioning based on demand
• Helm Charts: Standardized deployment configurations

Data Storage

Component	Technology	Purpose
Primary Database	PostgreSQL	Structured data (products, recipes, users)
Async Driver	asyncpg	High-performance async database access
Message Queue	RabbitMQ	Request distribution and load balancing

External ID System (Xid)

The Xid system ensures data integrity with external systems:

• Unique Identifiers: Each entity has a namespace-scoped external ID
• UUID Mapping: 1-1 relationship between Xid and internal UUID
• Cross-referencing: Easy lookup across namespaces
• Versioning: Change tracking over time

Mutation Log (Rewind/Replay)

All graph changes are logged for auditability and debugging:

# Every mutation is recorded
mutation = PropMutation(
    node_uid="abc123",
    property_type="OriginProp",
    old_value=None,
    new_value=origin_data,
    gfm_name="origin_gfm",
    timestamp=datetime.now()
)
calc_graph.apply_mutation(mutation)

Capabilities:

• Rewind: Reverse mutations to restore previous state
• Replay: Reconstruct system state or apply changes to different environments
• Debugging: Trace calculation steps for error investigation
• Transparency: Complete audit trail for all calculations

Next Steps

• How It Works - Step-by-step calculation flow
• Gap Filling Modules - Module system details
• GFM SDK (coming soon) - Build custom modules