The Keystore Cleanup Service provides automatic deletion of cryptographic keys from the keystore for tokens that have been deleted (spent, expired, or invalidated). This ensures that the keystore doesn’t accumulate stale keys indefinitely, improving security and reducing storage overhead.
The cleanup system consists of four main components:
The Manager runs in the background and periodically scans for deleted tokens that are eligible for key cleanup. It uses distributed locking (PostgreSQL advisory locks) to ensure only one replica in a multi-instance deployment performs cleanup at a time.
Key Features:
Lifecycle:
The SKI (Subject Key Identifier) extraction system uses a pluggable provider architecture to support different identity types. This allows identity-type-specific logic to be encapsulated in separate providers.
Components:
type TypedSKIProvider interface {
// GetSKIsFromIdentity derives one or more SKIs from an identity's raw bytes
GetSKIsFromIdentity(ctx context.Context, identity []byte) ([]string, error)
}
Provider Registration:
extractor := NewSKIExtractor()
extractor.RegisterProvider("idemix", idemix.NewSKIProvider())
extractor.RegisterProvider("idemixnym", idemixnym.NewSKIProvider(identityStore))
// X.509 keys belong to the wallet, not to individual tokens: deliberately never cleaned up
extractor.RegisterProvider("x509", NewNoopSKIProvider())
// Fallback provider is used for any unregistered types
SKI Derivation Process:
X.509 is registered with NoopSKIProvider, so no keystore key is ever deleted for an
X.509-owned token. This is deliberate, not an unimplemented provider.
An X.509 owner identity is a long-lived, non-anonymous certificate: x509.KeyManager reports
Anonymous() == false and always serves the same identity descriptor. Its private key therefore
belongs to the wallet, not to any individual token — the same key signs every token that
wallet ever owns, and it remains in use long after those tokens are spent. A real X.509 SKI
provider would make the cleanup sweep delete the wallet’s own signing key as soon as the first of
its tokens aged past the TTL, permanently breaking the wallet.
The Idemix providers are the opposite case: their SKIs identify a one-shot pseudonym key created for a single recipient identity. That key is dead once its token is deleted, which is exactly what makes it safe to remove.
Consequence to be aware of: a deleted X.509-owned token still gets a row in
token_ski_cleanups, because the cleanup manager records “no key material to delete” the same way
it records a completed deletion. For X.509 that row means nothing to clean, not keys were
removed. Do not read the token_ski_cleanups table as evidence that X.509 key material was
purged.
The Storage interface abstracts database operations needed for cleanup:
type Storage interface {
// AcquireCleanupLeadership obtains distributed lock for leader election
AcquireCleanupLeadership(ctx context.Context) (Leadership, bool, error)
// GetDeletedTokensPendingSKICleanup queries deleted, owned tokens older than TTL that haven't been cleaned.
// Tokens for which this node is only an auditor or issuer are excluded, since this node
// never holds the secret keys for those tokens.
GetDeletedTokensPendingSKICleanup(ctx context.Context, olderThan time.Duration, limit int) ([]DeletedToken, error)
// MarkTokenCleaned records successful key cleanup to prevent reprocessing
MarkTokenCleaned(ctx context.Context, txID string, index uint64, cleanedBy string) error
}
type Leadership interface {
// Close releases the leadership lock
Close() error
}
type DeletedToken struct {
TxID string
Index uint64
OwnerIdentity []byte
OwnerType string
DeletedAt time.Time
}
type SKIProvider interface {
// GetSKIsFromIdentity derives one or more SKIs from an owner identity
GetSKIsFromIdentity(ctx context.Context, identity []byte, identityType string) ([]string, error)
}
type Keystore interface {
// Delete removes the key with the given identifier
Delete(id string) error
// Close closes the keystore
Close() error
}
type KeystoreProvider interface {
// Keystore returns the keystore for the given TMS
Keystore(tmsID token.TMSID) (Keystore, error)
}
Cleanup behavior is controlled via configuration (see Configuration):
services:
storage:
cleanup:
enabled: false # Disabled by default - must be explicitly enabled
ttl: 24h # Minimum age before cleanup
scanInterval: 1h # How often to scan
batchSize: 100 # Max tokens per scan
workerCount: 1 # Parallel workers (default: 1)
instanceID: "cleanup-1" # Instance identifier (auto-generated if empty)
Configuration Details:
true to activate cleanup (conservative default)cleanup-<pointer> if not providedLeader election uses a PostgreSQL advisory lock whose identifier is not configurable. It is derived from the TMS’s own token SKI cleanup table name, which already carries network, channel and namespace, so it is unique per TMS automatically and several TMSes sharing one database no longer compete for a single lock.
Configuration Loading:
The configuration is loaded from the TMS configuration using the key services.storage.cleanup. The LoadConfig() function merges provided values with defaults, preserving defaults for any unset values.
import (
"github.com/LFDT-Panurus/panurus/token"
"github.com/LFDT-Panurus/panurus/token/services/storage/services/cleanup"
)
// Create configuration
config := cleanup.Config{
Enabled: true,
TTL: 24 * time.Hour,
ScanInterval: 1 * time.Hour,
BatchSize: 100,
WorkerCount: 4,
InstanceID: "cleanup-instance-1",
}
// Create TMS ID
tmsID := token.TMSID{
Network: "fabric",
Channel: "mychannel",
Namespace: "token-chaincode",
}
// Create manager
manager := cleanup.NewManager(
logger, // logging.Logger
storage, // Storage interface implementation
skiProvider, // SKIProvider interface implementation
keystoreProvider, // KeystoreProvider interface implementation
tmsID, // Token Management System ID
config, // Configuration
)
// Start cleanup (runs initial sweep immediately)
if err := manager.Start(); err != nil {
return err
}
// Stop cleanup gracefully
defer manager.Stop()
To add support for a custom identity type:
// Implement TypedSKIProvider interface
type CustomSKIProvider struct {
// your fields
}
func (p *CustomSKIProvider) GetSKIsFromIdentity(ctx context.Context, identity []byte) ([]string, error) {
// Parse identity and extract SKIs
// Return SKIs as hexadecimal strings
return []string{"ski1", "ski2"}, nil
}
// Register with extractor
extractor := cleanup.NewSKIExtractor()
extractor.RegisterProvider("custom-type", &CustomSKIProvider{})
The cleanup service is typically integrated automatically through the service manager:
import (
"github.com/LFDT-Panurus/panurus/token/services/storage/services/cleanup"
)
// Create service manager (manages cleanup instances per TMS)
cleanupManager := cleanup.NewServiceManager(
configuration, // Configuration interface
identityStorageProvider, // Identity storage provider
tokensProvider, // Tokens service manager
)
// Service manager automatically:
// - Creates cleanup manager per TMS
// - Registers built-in SKI providers (idemix, idemixnym, x509)
// - Sets up storage and keystore adapters
// - Starts the cleanup manager
Timing Details:
Start(), before first intervalThe cleanup service handles errors gracefully with specific retry behavior:
Key deletion is all-or-nothing per token:
errors.Is/errors.AsMarking a token cleaned while some of its key material is still in the keystore would turn a
transient database error into a permanent, un-retriable key-retention hole: the token would never
be selected again, so the surviving key would never be deleted. Retrying the whole token is safe
and cheap because Keystore.Delete is idempotent — re-deleting the keys that already succeeded
costs one no-op call each.
Empty SKI cases are still marked complete, otherwise every sweep would rescan the same tokens forever.
A token transitions through the following states related to cleanup:
is_deleted=true, spent_at set)owner=true) older than TTL without a cleanup record in token_ski_cleanupstoken_ski_cleanups)The cleanup service only processes tokens in the “Eligible for Cleanup” state. Tokens for which
this node is only an auditor or issuer (owner=false) are never selected, since this node
never holds the secret keys for tokens it does not own.
The cleanup service uses a dedicated tracking table to record cleanup operations:
Token SKI Cleanups Table:
CREATE TABLE IF NOT EXISTS token_ski_cleanups (
tx_id TEXT NOT NULL,
idx INT NOT NULL,
cleaned_at TIMESTAMP NOT NULL,
cleaned_by TEXT NOT NULL,
PRIMARY KEY (tx_id, idx),
FOREIGN KEY (tx_id, idx) REFERENCES tokens
);
CREATE INDEX IF NOT EXISTS idx_cleaned_at_token_ski_cleanups ON token_ski_cleanups ( cleaned_at );
This table tracks when each token’s cryptographic keys were cleaned from the keystore, preventing reprocessing. The cleaned_by field records which cleanup instance performed the operation, useful for debugging in multi-instance deployments.
The token_ski_cleanups table is automatically created by the schema initialization and does not require manual database alterations.
false (must be explicitly enabled)cleanup-<pointer> if not providedbatchSize to 200-500 for more tokens per sweepworkerCount to 4-16 for faster parallel processingscanInterval to 30m for more frequent cleanupworkerCount at 1 to minimize CPU usagescanInterval to 2-4h to reduce database loadbatchSize to limit memory usagettl to 12h for faster key removalscanInterval to 30m for more frequent cleanupinstanceID values for easier debugging and monitoringscanInterval directly affects database loadworkerCount affects CPU utilization during cleanup sweepsbatchSize affects memory usage and the duration of each cleanup sweepscanInterval < ttl ensures timely cleanup without premature processingKey metrics to monitor:
Log Levels:
INFO: Successful cleanup operations, manager start/stopWARN: Failed key deletions, key not found, leadership issuesDEBUG: Detailed sweep information, SKI derivation, leadership acquisitiontoken_ski_cleanups table provides cleanup history with timestamps and instance trackingtoken_ski_cleanups row
for an X.509-owned token means “nothing to clean”. See
X.509 is intentionally out of scopecleaned_by field records which instance performed cleanup| Feature | Recovery Service | Cleanup Service |
|---|---|---|
| Purpose | Re-register finality listeners | Delete stale cryptographic keys |
| Frequency | Every 5 seconds | Every 1 hour |
| TTL | 30 seconds | 24 hours |
| Target | Pending transactions | Deleted tokens |
| Urgency | High (affects finality) | Low (housekeeping) |
| Batch Size | 100 | 100 |
| Workers | 4 | 1 (default) |
| Initial Sweep | Immediate | Immediate |
| Leadership | Per sweep | Per sweep |