@earth-app/collegedb
    Preparing search index...

    Class KVShardMapper

    The KVShardMapper class provides a persistent storage layer for mapping primary keys to their assigned D1 database shards. It uses Cloudflare KV for global, eventually consistent storage with low latency reads.

    Features:

    • CRUD operations for shard mappings
    • Multiple lookup keys for the same shard (username, email, id, etc.)
    • SHA-256 hashing of keys for security and privacy
    • Atomic updates with timestamp tracking
    • Efficient bulk operations and statistics
    • Prefix-based key organization for performance
    const mapper = new KVShardMapper(env.KV, { hashShardMappings: true });

    // Create a new mapping with multiple lookup keys
    await mapper.setShardMapping('primary-key-123', 'db-central', ['username:john', 'email:john@example.com']);

    // Update an existing mapping
    await mapper.updateShardMapping('username:john', 'db-west');

    // Query mapping by any key
    const mapping = await mapper.getShardMapping('email:john@example.com');
    if (mapping) {
      console.log(`User is on ${mapping.shard}`);
    }
    Index

    Constructors

    constructor

    Methods

    addKnownShard addLookupKeys clearAllMappings deleteShardMapping getAllLookupKeys getKeysForShard getKnownShards getShardKeyCounts getShardMapping hashKey setKnownShards setShardMapping updateShardMapping

    Constructors

    • Creates a new KVShardMapper instance

      Parameters

      • kv: KVNamespace

        Cloudflare KV namespace

      • config: Partial<Pick<CollegeDBConfig, "hashShardMappings">> = {}

        Configuration options including hashing preference

      Returns KVShardMapper

    Methods

    • Appends a new shard to the list of known shards if it's not already present. This operation is idempotent - adding the same shard multiple times has no effect.

      Parameters

      • shard: string

        The shard binding name to add

      Returns Promise<void>

      Promise that resolves when the shard is added

      If KV operations fail

      // Register a new shard when it comes online
      await mapper.addKnownShard('db-europe');
      console.log('European shard registered');

    • Adds additional lookup keys to an existing shard mapping. This allows you to query the same shard mapping using multiple identifiers (e.g., username, email, id).

      Parameters

      • primaryKey: string

        An existing key in the mapping

      • additionalKeys: string[]

        New keys to add for lookup

      Returns Promise<void>

      Promise that resolves when the additional keys are added

      If no existing mapping found or KV operations fail

      // Add email lookup to an existing user mapping
      await mapper.addLookupKeys('user-123', ['email:user@example.com']);

      1.0.3

    • Deletes ALL shard mappings from KV storage. This is a destructive operation that removes all primary key assignments. After this operation, all keys will be treated as new and may be assigned to different shards.

      DANGER: This operation is irreversible and will cause data routing issues if used in production. Only use during development, testing, or complete system resets.

      Returns Promise<void>

      Promise that resolves when all mappings are deleted

      If KV operations fail

      // Only use in development/testing!
      if (process.env.NODE_ENV === 'development') {
        await mapper.clearAllMappings();
        console.log('All mappings cleared for testing');
      }

    • Completely removes the shard assignment for a primary key from KV storage. This is typically used when data is being permanently deleted or when cleaning up orphaned mappings. Handles both single-key and multi-key mappings.

      WARNING: After deletion, the primary key will be treated as new and may be assigned to a different shard on next access.

      Parameters

      • primaryKey: string

        The primary key mapping to remove

      Returns Promise<void>

      Promise that resolves when the mapping is deleted

      If KV delete operation fails

      // Remove mapping for deleted user
      await mapper.deleteShardMapping('email:deleted@example.com');
      console.log('Mapping removed for deleted user');

    • Gets all lookup keys associated with a shard mapping. This is useful for understanding what keys resolve to the same shard.

      Parameters

      • primaryKey: string

        Any key in the mapping

      Returns Promise<string[]>

      Promise resolving to array of all keys in the mapping

      If no existing mapping found

      const allKeys = await mapper.getAllLookupKeys('email:user@example.com');
      console.log(allKeys); // ['user-123', 'username:john', 'email:user@example.com']

      1.0.3

    • Scans all shard mappings to find primary keys assigned to the specified shard. This operation requires reading all mappings and can be expensive for large datasets. Consider caching results or using getShardKeyCounts() for statistics.

      Parameters

      • shard: string

        The shard binding name to search for

      Returns Promise<string[]>

      Promise resolving to array of primary keys assigned to the shard

      If KV operations fail

      // Find all users on the east coast shard
      const eastCoastUsers = await mapper.getKeysForShard('db-east');
      console.log(`East coast has ${eastCoastUsers.length} users`);

    • Retrieves the list of all shard binding names that have been registered with the system. This is maintained separately from the individual mappings for efficient shard discovery.

      Returns Promise<string[]>

      Promise resolving to array of shard binding names

      If KV read operation fails

      const shards = await mapper.getKnownShards();
      console.log('Available shards:', shards);
      // Output: ['db-east', 'db-west', 'db-central']

    • Scans all shard mappings to count how many primary keys are assigned to each shard. Returns a mapping of shard names to their key counts. This is useful for load balancing and monitoring shard utilization.

      Performance Note: This operation scans all mappings and can be expensive for large datasets. Consider implementing caching for frequently accessed statistics.

      Returns Promise<Record<string, number>>

      Promise resolving to object mapping shard names to key counts

      If KV operations fail

      const counts = await mapper.getShardKeyCounts();
      console.log('Shard utilization:', counts);
      // Output: { 'db-east': 1247, 'db-west': 982, 'db-central': 1156 }

      // Find the least loaded shard
      const leastLoaded = Object.entries(counts)
        .sort(([,a], [,b]) => a - b)[0][0];
      console.log('Least loaded shard:', leastLoaded);

    • Retrieves the shard assignment for a given primary key from KV storage. Returns null if no mapping exists, indicating the key has not been assigned to any shard yet. Supports both single-key and multi-key lookups.

      Parameters

      • primaryKey: string

        The primary key to look up (will be hashed if hashing is enabled)

      Returns Promise<null | ShardMapping>

      Promise resolving to the shard mapping or null if not found

      If KV read operation fails

      const mapping = await mapper.getShardMapping('email:user@example.com');
      if (mapping) {
        console.log(`User is on shard: ${mapping.shard}`);
        console.log(`Created: ${new Date(mapping.createdAt)}`);
      } else {
        console.log('User not yet assigned to any shard');
      }

    • Hashes a key using SHA-256 if hashing is enabled

      Parameters

      • key: string

        The key to hash

      Returns Promise<string>

      The hashed key or original key if hashing is disabled

    • Replaces the entire list of known shards with a new list. This is typically used during system initialization or when shards are added/removed in bulk.

      Parameters

      • shards: string[]

        Array of shard binding names to store

      Returns Promise<void>

      Promise that resolves when the list is updated

      If KV write operation fails

      // Update shard list after adding new regions
      await mapper.setKnownShards(['db-east', 'db-west', 'db-central', 'db-asia']);

    • Creates a new shard assignment for a primary key. This is typically used when a new primary key is first encountered and needs to be assigned to a shard. Sets both created and updated timestamps to the current time.

      Note: This will overwrite any existing mapping for the same key. Use updateShardMapping() if you want to preserve creation timestamp.

      Parameters

      • primaryKey: string

        The primary key to map

      • shard: string

        The shard binding name to assign

      • additionalKeys: string[] = []

        Optional array of additional lookup keys for the same mapping

      Returns Promise<void>

      Promise that resolves when the mapping is stored

      If KV write operation fails

      // Assign a new user to the west coast shard with multiple lookup keys
      await mapper.setShardMapping('user-123', 'db-west', ['username:john', 'email:john@example.com']);

    • Changes the shard assignment for a primary key that already has a mapping. Preserves the original creation timestamp while updating the modified timestamp. Throws an error if no existing mapping is found.

      This is typically used during shard rebalancing or data migration operations. Works with both single-key and multi-key mappings.

      Parameters

      • primaryKey: string

        The primary key to update

      • newShard: string

        The new shard binding name to assign

      Returns Promise<void>

      Promise that resolves when the mapping is updated

      If no existing mapping found or KV operation fails

      try {
        // Move user to a different shard for rebalancing
        await mapper.updateShardMapping('email:user@example.com', 'db-central');
        console.log('User successfully moved to central shard');
      } catch (error) {
        console.error('Failed to update mapping:', error.message);
      }

    Settings

    Member Visibility

    On This Page

    Constructors
    Methods