panurus

Storage Service

The Storage Service (token/services/storage) encapsulates all data persistence mechanisms required by Panurus. It provides a robust, SQL-based management system to ensure that token states, transaction history, and cryptographic identities are securely tracked and retrievable.

Architecture

The storage layer is built on a provider-based architecture that supports multiple SQL backends, primarily SQLite (for local development and edge nodes) and PostgreSQL (for production-grade scalability).

Database Schema

Schema Description

Panurus storage is organized into logical databases, each serving a specific role in the transaction lifecycle and identity management.

Amount Storage Strategy

All token amounts across the storage layer are stored as NUMERIC(78, 0) in the database, supporting arbitrary precision integers up to 78 digits. This design choice enables:

Panurus uses Go’s *big.Int type throughout the codebase to handle these large values, with automatic conversion between database NUMERIC values and in-memory big.Int representations via the BigInt scanner type.

Transaction & Audit Store (TTXDB / AuditDB)

The ttxdb/auditdb manages the token request lifecycle and transactions records.

The following tables track the lifecycle of token requests from assembly to finality. AuditDB uses the same schema but is isolated for compliance reporting.

Endorser Store (EndorserDB)

The endorserdb manages validation records created during the token request endorsement process. It shares the physical database with TTXDB but owns its own, self-contained table — it does not write to the Requests table.

Token Store (TokenDB)

This store serves as the authoritative registry for all tokens (UTXOs) known to the node.

Wallet & Identity Store (WalletDB / IdentityDB)

Manages the cryptographic identities and logical wallet groupings used by the node.

Generic Store

Table Name Customisation

Each SQL table name is derived from a canonical short code (e.g. id_signers) that is wrapped by the FSC formatter with the configured TablePrefix and TableNameParams. Two independent options let you control how the final name is composed:

Option Config key Default
Override individual short codes token.storage.tableNames (none)
Skip the FSC-generated prefix entirely token.storage.skipPrefix false

Allowed characters

The TableNameParams are the TMS identity — network, channel and namespace — and they become part of every table name, so they must be reducible to a legal SQL identifier. The characters that are common in Fabric names but illegal in an identifier are escaped rather than rejected:

Character in a param Becomes
_ __
- _d
. _f

Letters, digits and underscores pass through as-is, so a channel called channel1 or mychannel01 is fine. The final composed name — prefix included — must still be a valid identifier: only letters, digits and underscores, and it cannot start with a digit (unquoted identifiers in both SQLite and PostgreSQL must start with a letter or an underscore). Since the prefix comes first and can never start with a digit, this only bites when skipPrefix is enabled and the network name starts with a digit.

Anything else — a space, !, /, ;, … — is a configuration error: store construction returns an error naming the offending value, it does not crash the node. A bad prefix or param breaks every table name in the same way and is reported once; invalid short-code overrides are per-key, so all of them are reported together and you do not have to fix them one restart at a time.

The TablePrefix is stricter: only letters and underscores, at most 100 characters. It is lower-cased before use.

Overriding short codes (tableNames)

You can replace any short code globally (for all TMS instances on the node) via the token.storage.tableNames key. The override value replaces the short code before the FSC formatter runs, so the final table name still has the FSC-generated prefix and params applied around it.

Unknown keys are warned about in the log and silently ignored — they do not cause an error.

token:
  storage:
    tableNames:
      id_signers: identity_signers   # was: fsc_id_signers_<prefix>_<params>
      tokens: my_tokens              # was: fsc_tokens_<prefix>_<params>

Skipping the prefix (skipPrefix)

When set to true, the FSC-generated prefix is omitted from all table names. This is useful when connecting to an existing database whose tables were created without a prefix, or when the target database already enforces schema-level isolation.

Default: false.

token:
  storage:
    skipPrefix: true

With skipPrefix: true the name pattern changes from fsc_<short_code>_<params> to <params>_<short_code>. Short-code overrides from tableNames are still respected.

Caution: Enabling skipPrefix on a node that previously ran with the default prefix will cause the node to look for tables under different names. Ensure the underlying tables already exist under the unprefixed names before enabling this flag.

Both options can be combined:

token:
  storage:
    skipPrefix: true
    tableNames:
      tokens: my_tokens   # final name: <params>_my_tokens  (no prefix)

Short Code Reference

The table below lists every short code, the TableNames struct field it controls, and the default table name pattern it produces (before any override is applied).

Short Code TableNames field Default pattern
movements Movements fsc_movements_<prefix>_<params>
txs Transactions fsc_txs_<prefix>_<params>
tx_ends TransactionEndorseAck fsc_tx_ends_<prefix>_<params>
requests Requests fsc_requests_<prefix>_<params>
req_vals Validations fsc_req_vals_<prefix>_<params>
tokens Tokens fsc_tokens_<prefix>_<params>
tkn_own Ownership fsc_tkn_own_<prefix>_<params>
tkn_crts Certifications fsc_tkn_crts_<prefix>_<params>
tkn_locks TokenLocks fsc_tkn_locks_<prefix>_<params>
public_params PublicParams fsc_public_params_<prefix>_<params>
wallets Wallets fsc_wallets_<prefix>_<params>
id_cfgs IdentityConfigurations fsc_id_cfgs_<prefix>_<params>
id_info IdentityInfo fsc_id_info_<prefix>_<params>
id_signers Signers fsc_id_signers_<prefix>_<params>
key_store KeyStore fsc_key_store_<prefix>_<params>
eid_leases EIDLeases fsc_eid_leases_<prefix>_<params>
tkn_ski_cleanups TokenSKICleanups fsc_tkn_ski_cleanups_<prefix>_<params>

Note: When no TablePrefix is configured (empty string) and no TableNameParams are present, the pattern simplifies to fsc_<short_code> (e.g. fsc_id_signers).

Internal Databases

Panurus partitions its data into several specialized databases to maintain a clean separation of concerns.

Transaction Store (TTXDB)

The ttxdb serves as the central repository for the lifecycle of token requests. It is used by the TTX Service to track:

Endorser Store (EndorserDB)

The endorserdb manages validation records for token requests during the endorsement process. It is used by the Endorsement Service to:

The endorserdb shares the same physical database as ttxdb but owns its own table and interface for validation-specific operations, improving modularity and separation of concerns.

Token Store (TokenDB)

The tokendb is the registry for the current state of all tokens (UTXOs) known to the node. It is used by the Selector Service and Vault Service to:

Audit Transactions Store (AuditDB)

For nodes acting in an Auditor role, the auditdb provides a specialized repository for audit-related records. While its schema is identical to ttxdb, it is isolated to ensure that auditing activities do not interfere with standard transaction processing and to support enhanced compliance reporting.

Wallet and Identity Store (WalletDB)

The walletdb and associated stores (IdentityDB, KeyStore) manage the cryptographic identities used by the node. They track:

Data Persistence Strategy

The Storage Service follows a “Finality-Driven” update strategy. While transactions are being assembled, they are stored in a Pending state. The TokenDB and Movements tables are typically updated only when the Network Service confirms that a transaction has reached finality on the ledger. This ensures that the local view of the “Token Landscape” always reflects the ground truth of the distributed ledger.

Transaction Recovery Service

The Storage Service includes a Transaction Recovery Service that provides the core recovery mechanism for handling pending transactions that may have lost their finality listeners due to node restarts, network interruptions, or other failures.

For detailed documentation on the recovery service architecture, configuration, and usage, see Transaction Recovery Service.

Architecture

The recovery service is instantiated by the Network Service (both Fabric and FabricX implementations) and operates on either the TTXDB (for regular transactions) or AuditDB (for auditor nodes). It provides a generic recovery mechanism that is independent of the specific network backend.

Recovery Manager

The recovery manager runs in the background and periodically scans for pending transactions that are eligible for recovery. It uses a distributed locking mechanism (PostgreSQL advisory locks) to ensure only one replica in a multi-instance deployment performs recovery at a time.

Key Features:

Recovery Process

The recovery service follows this workflow:

  1. Scan Phase: The recovery manager periodically scans the transaction database for pending transactions older than the configured TTL. The claim query reads only the minimum projection needed (tx_id, stored_at) and returns a lightweight RecoveryClaim for each row, avoiding the cost of materialising the full transaction record on every sweep.
  2. Claim Phase: Eligible transactions are claimed by the current instance using a lease mechanism. On PostgreSQL this is an atomic UPDATE ... RETURNING; SQLite uses a non-atomic but functionally equivalent permissive claim.
  3. Recovery Phase: For each claimed transaction, the recovery handler:
    • Queries the network for the transaction’s current status
    • Based on the status (Valid, Invalid, NotFound, or Busy), applies the appropriate finality logic
    • For valid transactions, verifies the token request hash and commits to the local database
    • For invalid transactions, marks them as Deleted
    • For transactions whose status keeps returning NotFound past the configured notFoundGracePeriod, marks them as Orphan. This indicates the transaction never reached the ledger (e.g. broadcast failure, mempool drop) and prevents long-stuck rows from blocking the head of the recovery queue, while keeping them distinguishable from ledger-rejected transactions that are marked Deleted.
    • For busy (pending) transactions, returns an error and the transaction remains eligible for future recovery
  4. Lease Management: Claimed transactions are leased to the instance for a configured duration to prevent stuck transactions from blocking recovery

Integration with Network Service

The Network Service (Fabric and FabricX implementations) instantiates the recovery service during initialization:

This design ensures that recovery is handled consistently across different network backends while maintaining the separation of concerns between storage and network layers.

Database Backend Considerations

The recovery service supports both PostgreSQL and SQLite backends, with different characteristics:

PostgreSQL (Recommended for Production):

SQLite (Development & Single-Node Deployments):

Important: When using SQLite, ensure that only one node instance accesses the database file. Running multiple replicas with SQLite will result in undefined behavior and potential data corruption.

Configuration

Recovery behavior is controlled by the token.tms.<name>.services.network.fabric.recovery configuration section.

Keystore Cleanup Service

The Storage Service includes a Keystore Cleanup Service that provides automatic deletion of cryptographic keys from the keystore for tokens that have been deleted (spent, expired, or invalidated). This ensures that the keystore doesn’t accumulate stale keys indefinitely, improving security and reducing storage overhead.

For detailed documentation on the cleanup service architecture, configuration, and usage, see Keystore Cleanup Service.

Architecture

The cleanup service operates on the token database and keystore, scanning for deleted tokens that are eligible for key cleanup. It uses a distributed locking mechanism (PostgreSQL advisory locks) to ensure only one replica in a multi-instance deployment performs cleanup at a time.

Cleanup Manager

The cleanup manager runs in the background and periodically scans for deleted tokens whose cryptographic keys can be safely removed from the keystore.

Key Features:

Cleanup Process

The cleanup service follows this workflow:

  1. Leadership Acquisition: The cleanup manager acquires an advisory lock to become the leader
  2. Scan Phase: Scans the token database for deleted tokens older than the configured TTL (default: 24 hours) that haven’t had their keys cleaned
  3. SKI Derivation: For each eligible token, derives SKIs from the owner identity
  4. Key Deletion: Deletes the derived keys from the keystore using parallel workers
  5. Tracking: Marks tokens as cleaned in the database to prevent reprocessing

Database Backend Considerations

The cleanup service supports both PostgreSQL and SQLite backends, with different characteristics:

PostgreSQL:

SQLite:

Configuration

Cleanup behavior is controlled by the configuration section. See the Configuration Guide for detailed parameter descriptions and tuning recommendations.

See the Configuration Guide, Section Optional: token.tms.<name>.services.network.fabric.recovery, for detailed parameter descriptions and tuning recommendations.

This site is open source. Improve this page.