@earth-app/collegedb
    Preparing search index...

    Class KVShardMapper

    The KVShardMapper class provides a persistent storage layer for mapping primary keys to their assigned D1 database shards. It uses Cloudflare KV for global, eventually consistent storage with low latency reads.

    Features:

    • CRUD operations for shard mappings
    • Atomic updates with timestamp tracking
    • Efficient bulk operations and statistics
    • Prefix-based key organization for performance
    const mapper = new KVShardMapper(env.KV);

    // Create a new mapping
    await mapper.setShardMapping('order-456', 'db-central');

    // Update an existing mapping
    await mapper.updateShardMapping('order-456', 'db-west');

    // Query mapping
    const mapping = await mapper.getShardMapping('order-456');
    if (mapping) {
      console.log(`Order 456 is on ${mapping.shard}`);
    }
    Index

    Constructors

    constructor

    Methods

    addKnownShard clearAllMappings deleteShardMapping getKeysForShard getKnownShards getShardKeyCounts getShardMapping setKnownShards setShardMapping updateShardMapping

    Constructors

    Methods

    • Appends a new shard to the list of known shards if it's not already present. This operation is idempotent - adding the same shard multiple times has no effect.

      Parameters

      • shard: string

        The shard binding name to add

      Returns Promise<void>

      Promise that resolves when the shard is added

      If KV operations fail

      // Register a new shard when it comes online
      await mapper.addKnownShard('db-europe');
      console.log('European shard registered');

    • Deletes ALL shard mappings from KV storage. This is a destructive operation that removes all primary key assignments. After this operation, all keys will be treated as new and may be assigned to different shards.

      DANGER: This operation is irreversible and will cause data routing issues if used in production. Only use during development, testing, or complete system resets.

      Returns Promise<void>

      Promise that resolves when all mappings are deleted

      If KV operations fail

      // Only use in development/testing!
      if (process.env.NODE_ENV === 'development') {
        await mapper.clearAllMappings();
        console.log('All mappings cleared for testing');
      }

    • Completely removes the shard assignment for a primary key from KV storage. This is typically used when data is being permanently deleted or when cleaning up orphaned mappings.

      WARNING: After deletion, the primary key will be treated as new and may be assigned to a different shard on next access.

      Parameters

      • primaryKey: string

        The primary key mapping to remove

      Returns Promise<void>

      Promise that resolves when the mapping is deleted

      If KV delete operation fails

      // Remove mapping for deleted user
      await mapper.deleteShardMapping('user-deleted-789');
      console.log('Mapping removed for deleted user');

    • Scans all shard mappings to find primary keys assigned to the specified shard. This operation requires reading all mappings and can be expensive for large datasets. Consider caching results or using getShardKeyCounts() for statistics.

      Parameters

      • shard: string

        The shard binding name to search for

      Returns Promise<string[]>

      Promise resolving to array of primary keys assigned to the shard

      If KV operations fail

      // Find all users on the east coast shard
      const eastCoastUsers = await mapper.getKeysForShard('db-east');
      console.log(`East coast has ${eastCoastUsers.length} users`);

    • Retrieves the list of all shard binding names that have been registered with the system. This is maintained separately from the individual mappings for efficient shard discovery.

      Returns Promise<string[]>

      Promise resolving to array of shard binding names

      If KV read operation fails

      const shards = await mapper.getKnownShards();
      console.log('Available shards:', shards);
      // Output: ['db-east', 'db-west', 'db-central']

    • Scans all shard mappings to count how many primary keys are assigned to each shard. Returns a mapping of shard names to their key counts. This is useful for load balancing and monitoring shard utilization.

      Performance Note: This operation scans all mappings and can be expensive for large datasets. Consider implementing caching for frequently accessed statistics.

      Returns Promise<Record<string, number>>

      Promise resolving to object mapping shard names to key counts

      If KV operations fail

      const counts = await mapper.getShardKeyCounts();
      console.log('Shard utilization:', counts);
      // Output: { 'db-east': 1247, 'db-west': 982, 'db-central': 1156 }

      // Find the least loaded shard
      const leastLoaded = Object.entries(counts)
        .sort(([,a], [,b]) => a - b)[0][0];
      console.log('Least loaded shard:', leastLoaded);

    • Retrieves the shard assignment for a given primary key from KV storage. Returns null if no mapping exists, indicating the key has not been assigned to any shard yet.

      Parameters

      • primaryKey: string

        The primary key to look up

      Returns Promise<null | ShardMapping>

      Promise resolving to the shard mapping or null if not found

      If KV read operation fails

      const mapping = await mapper.getShardMapping('user-789');
      if (mapping) {
        console.log(`User 789 is on shard: ${mapping.shard}`);
        console.log(`Created: ${new Date(mapping.createdAt)}`);
      } else {
        console.log('User 789 not yet assigned to any shard');
      }

    • Replaces the entire list of known shards with a new list. This is typically used during system initialization or when shards are added/removed in bulk.

      Parameters

      • shards: string[]

        Array of shard binding names to store

      Returns Promise<void>

      Promise that resolves when the list is updated

      If KV write operation fails

      // Update shard list after adding new regions
      await mapper.setKnownShards(['db-east', 'db-west', 'db-central', 'db-asia']);

    • Creates a new shard assignment for a primary key. This is typically used when a new primary key is first encountered and needs to be assigned to a shard. Sets both created and updated timestamps to the current time.

      Note: This will overwrite any existing mapping for the same key. Use updateShardMapping() if you want to preserve creation timestamp.

      Parameters

      • primaryKey: string

        The primary key to map

      • shard: string

        The shard binding name to assign

      Returns Promise<void>

      Promise that resolves when the mapping is stored

      If KV write operation fails

      // Assign a new user to the west coast shard
      await mapper.setShardMapping('user-california-123', 'db-west');

    • Changes the shard assignment for a primary key that already has a mapping. Preserves the original creation timestamp while updating the modified timestamp. Throws an error if no existing mapping is found.

      This is typically used during shard rebalancing or data migration operations.

      Parameters

      • primaryKey: string

        The primary key to update

      • newShard: string

        The new shard binding name to assign

      Returns Promise<void>

      Promise that resolves when the mapping is updated

      If no existing mapping found or KV operation fails

      try {
        // Move user to a different shard for rebalancing
        await mapper.updateShardMapping('user-456', 'db-central');
        console.log('User successfully moved to central shard');
      } catch (error) {
        console.error('Failed to update mapping:', error.message);
      }

    Settings

    Member Visibility

    On This Page

    Constructors
    Methods