The Observability Service (token/services/observability) provides non-intrusive, injectable decorators, Prometheus metrics collection, and circuit breaker back-pressure protection for Token-API entry points (driver.WalletService, driver.OwnerWallet, driver.IssuerWallet, driver.AuditorWallet, and driver.CertifierWallet).
The Observability Service is responsible for:
maxWalletBreakers, default 1000) so that errors in one wallet method (e.g. token store outages during ListTokens) do not impact unrelated methods (e.g. identity store lookups) or other wallets, and memory usage remains bounded.token.tms.<tms-name>.circuitBreaker).The decorator wraps token driver services during Token Manager Service (TMS) initialization:
graph TD
Client[Token API Client] --> Decorator[WalletServiceDecorator]
Decorator --> Metrics[WalletMetrics]
Decorator --> CB[Circuit Breakers]
Decorator --> Driver[Concrete Driver WalletService]
Decorator --> OwnerW[OwnerWalletDecorator]
Decorator --> IssuerW[IssuerWalletDecorator]
Decorator --> AuditorW[AuditorWalletDecorator]
Decorator --> CertifierW[CertifierWalletDecorator]
OwnerWallet, IssuerWallet, AuditorWallet, CertifierWallet, GetAuditInfo, GetEnrollmentID, GetRevocationHandle, GetEIDAndRH) operate outside the circuit breaker. This guarantees that benign lookup misses (such as foreign transaction participant identities) never trip a circuit breaker and authorization checks like IsMine do not misclassify ownership due to transient trips.RegisterRecipientIdentity, RegisterOwnerIdentity, RegisterIssuerIdentity) and per-wallet operations (ListTokens, Balance, Sign, SpendIDs) are protected by circuit breakers.CircuitBreaker instances cached in an LRU map per wallet ID and method (walletType:walletID:method). Outages in the token store (affecting ListTokens or Balance) do not fast-fail identity store methods (GetRecipientIdentity or RegisterRecipient), and failures within one wallet only trip the breaker for that specific wallet.Each circuit breaker implements a three-state machine:
stateDiagram-v2
[*] --> Closed
Closed --> Open: Consecutive Failures >= MaxFailures
Open --> HalfOpen: Cooldown Timeout Elapsed & Probe Request
HalfOpen --> Closed: Trial Probe Succeeded
HalfOpen --> Open: Trial Probe Failed
ErrCircuitOpen. The base service is not invoked.halfOpenInFlight token). Concurrent callers during the probe are rejected with ErrCircuitOpen to prevent stampeding a recovering backend. If an admitted probe hangs or exceeds probeTimeout (default: 10s), subsequent requests admit a fresh probe. If the probe succeeds, the breaker resets to Closed. If the probe fails with an operational error, it transitions back to Open and restarts cooldown. If the probe returns an inconclusive cancellation (context.Canceled or context.DeadlineExceeded), the in-flight token is released and the breaker remains in HalfOpen for the next caller.Routine, expected, or benign conditions are filtered out using typed sentinel errors and do not increment the failure count or trip the circuit breaker:
identity.ErrUnresolvableIdentitydriver.ErrTokenNotFound (routine query miss for unspent or deleted tokens)context.Canceled (client cancelled request)context.DeadlineExceeded (client timeout)Arbitrary string matching is avoided so that genuine storage, network, or deserialization failures are never masked as routine query misses.
Circuit breakers are configured via the TMS configuration provider under the token.tms.<tms-name>.circuitBreaker key (accessed as circuitBreaker relative to the TMS configuration root). Circuit breaking is disabled by default (opt-in):
token:
tms:
<tms-name>:
circuitBreaker:
enabled: true # Circuit breaker is disabled by default; set to true to enable
maxConsecutiveFailures: 5 # Number of consecutive failures before tripping to Open
cooldownTimeout: 30s # Cooldown period before transitioning from Open to HalfOpen
probeTimeout: 10s # Timeout before admitting a replacement probe in HalfOpen state
maxWalletBreakers: 1000 # Maximum number of per-wallet circuit breakers in LRU cache
The following metrics are exposed when the observability decorator is enabled.
Exported metric names are assembled from the calling package. In production token driver paths (both FabToken and ZKAT-DLog), the decorator receives a TMS-scoped provider, so metrics are exported with the panurus_core_common_metrics_ prefix (e.g., panurus_core_common_metrics_wallet_requests_total). If instantiated through the standalone WalletServiceFactory path with a container provider, they would be exported under panurus_services_observability_ (e.g., panurus_services_observability_wallet_requests_total).
All metrics include wallet_id to prevent metric series collisions across wallets:
| Metric Name | Type | Labels | Description |
|---|---|---|---|
panurus_core_common_metrics_wallet_requests_total |
Counter | network, channel, namespace, method, wallet_id |
Total number of wallet service requests |
panurus_core_common_metrics_wallet_errors_total |
Counter | network, channel, namespace, method, wallet_id |
Total number of wallet service errors (excludes circuit breaker rejections) |
panurus_core_common_metrics_wallet_request_duration_seconds |
Histogram | network, channel, namespace, method, wallet_id |
Execution duration in seconds (fast-failed rejections are not recorded to avoid skewing latency) |
panurus_core_common_metrics_wallet_inflight_requests |
Gauge | network, channel, namespace, method, wallet_id |
Current number of in-flight wallet service requests |
panurus_core_common_metrics_wallet_circuit_breaker_rejections_total |
Counter | network, channel, namespace, method, wallet_id |
Total number of calls fast-failed by circuit breaker |
panurus_core_common_metrics_wallet_circuit_breaker_state |
Gauge | network, channel, namespace, method, wallet_id |
Current circuit breaker state (0=Closed, 1=Open, 2=HalfOpen) |
The wallet metrics include wallet_id as a label dimension to enable granular debugging and isolate high error rates, latencies, or circuit breaker trips to specific wallets:
wallet_id values is small and bounded.wallet_id label) if cardinality exceeds monitoring capacity in high-churn environments.