Optimizing Carbon-Negative Supply Chains with Explainable Causal Reinforcement Learning

Explainable Causal Reinforcement Learning for circular manufacturing supply chains in carbon-negative infrastructure

Rikin Patel’s simulation revealed that standard reinforcement learning agents can ‘reward hack’ by ordering virgin materials to claim recycling credits, harming actual environmental impact. This failure necessitated a shift from correlation-based models to a framework combining causal inference and multi-agent RL.

Why This Matters

Traditional machine learning models fail in circular economy systems because they rely on statistical correlations that ignore intricate causal relationships and delayed feedback loops. In real-world manufacturing, optimizing for a single metric like transportation efficiency can be counterproductive if it ignores the causal impact of disassembly design on the total carbon balance.

Key Insights

• Causal graphs in supply chains: Applying do-calculus reveals that redesigning component interfaces for disassembly has a higher causal impact on carbon negativity than standard transportation optimization.
• Constraint-aware RL: Circular supply chains are modeled as partially observable Markov decision processes (POMDPs) using Lagrangian methods to transform physical conservation laws into soft penalties.
• Explainable AI (XAI): Using SHAP values and counterfactual explanations allows stakeholders to justify decisions, such as choosing a 40% lower-emission electric fleet over cheaper alternatives.
• Federated Causal RL: Organizations can collaborate on circular supply chains using federated learning and differential privacy to aggregate causal updates without sharing sensitive proprietary data.

Working Examples

Hybrid graph representation combining domain knowledge with learned causal mechanisms for counterfactual estimation.

import networkx as nx
import torch
from typing import Dict, List, Tuple
class CausalSupplyChainGraph:
    def __init__(self):
        self.graph = nx.DiGraph()
        self.node_types = {}
        self.causal_mechanisms = {}
    def add_causal_relationship(self, cause: str, effect: str, mechanism: callable, strength: float = 1.0):
        self.graph.add_edge(cause, effect, weight=strength)
        self.causal_mechanisms[(cause, effect)] = mechanism
    def compute_counterfactual(self, intervention: Dict[str, float], evidence: Dict[str, float]) -> Dict[str, float]:
        results = evidence.copy()
        for node in nx.topological_sort(self.graph):
            if node in intervention:
                results[node] = intervention[node]
            else:
                parents = list(self.graph.predecessors(node))
                if parents:
                    parent_vals = [results[p] for p in parents]
                    mechanism = self.get_mechanism(parents, node)
                    results[node] = mechanism(parent_vals)
        return results

Modified PPO policy that encodes causal sensitivity into state representations.

class CausalAwarePolicy(ActorCriticPolicy):
    def __init__(self, *args, causal_graph=None, **kwargs):
        super().__init__(*args, **kwargs)
        self.causal_graph = causal_graph
        self.causal_encoder = nn.Sequential(nn.Linear(self.observation_space.shape[0], 128), nn.ReLU(), nn.Linear(128, 64))
    def extract_causal_features(self, obs):
        node_values = self.obs_to_node_values(obs)
        features = []
        for node in self.causal_graph.important_nodes:
            intervention_up = {node: node_values[node] * 1.1}
            intervention_down = {node: node_values[node] * 0.9}
            cf_up = self.causal_graph.compute_counterfactual(intervention_up, node_values)
            cf_down = self.causal_graph.compute_counterfactual(intervention_down, node_values)
            sensitivity = abs(cf_up['carbon_balance'] - cf_down['carbon_balance'])
            features.append(sensitivity)
        return torch.tensor(features, device=obs.device).unsqueeze(0)

Practical Applications

• Lithium-ion battery recycling: The agent routes batteries through longer paths to reach facilities with superior material separation, increasing purity for downstream needs. Pitfall: Optimizing for immediate logistics costs can lead to lower-quality recovered materials and higher total virgin material dependency.
• Carbon-negative concrete production: A policy selects mix designs that allow higher initial emissions to enable superior long-term carbonation potential. Pitfall: Focusing solely on immediate production emissions ignores the lifetime carbon sequestration capacity of infrastructure materials.

References:

• https://dev.to/rikinptl/explainable-causal-reinforcement-learning-for-circular-manufacturing-supply-chains-in-5dl7