Vulcan topic to 210 partitions

[roy-dydx]

Changelist

[Describe or list the changes made in this PR]

Test Plan

[Describe how this PR was tested (if applicable)]

Author/Reviewer Checklist

• If this PR has changes that result in a different app state given the same prior state and transaction list, manually add the state-breaking label.
• If the PR has breaking postgres changes to the indexer add the indexer-postgres-breaking label.
• If this PR isn't state-breaking but has changes that modify behavior in PrepareProposal or ProcessProposal, manually add the label proposal-breaking.
• If this PR is one of many that implement a specific feature, manually label them all feature:[feature-name].
• If you wish to for mergify-bot to automatically create a PR to backport your change to a release branch, manually add the label backport/[branch-name].
• Manually add any of the following labels: refactor, chore, bug.

Summary by CodeRabbit

• Chores
  • Updated the partition count for the KafkaTopics.TO_VULCAN topic from 150 to 210, enhancing message distribution efficiency.

[coderabbitai]

Walkthrough

The changes involve updating the KAFKA_TOPICS_TO_PARTITIONS mapping in the index.ts file, specifically increasing the partition count for KafkaTopics.TO_VULCAN from 150 to 210. This adjustment affects the configuration for message distribution across partitions for this topic. No other modifications to logic, error handling, or control flow were made, and the overall structure and functionality of the code remain unchanged.

Changes

File Path	Change Summary
indexer/services/bazooka/src/index.ts	Updated partition count for KafkaTopics.TO_VULCAN from 150 to 210.

Possibly related PRs

• Set to-vulcan topic to 150 partitions #2286: This PR modifies the partition count for the KafkaTopics.TO_VULCAN topic in the same index.ts file, changing it from 60 to 150, which is directly related to the partition count adjustments made in the main PR for KafkaTopics.TO_VULCAN.

Suggested labels

indexer

Suggested reviewers

• dydxwill

🐰 In the land of code, where topics bloom,
A change was made to lift the gloom.
From one-fifty to two-ten, the partitions grow,
Messages will dance, in a Kafka flow.
Hops of joy, in every line,
A brighter path, where data shines! 🌟

---

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL

📥 Commits

Files that changed from the base of the PR and between b9fcc40 and 386f324.

📒 Files selected for processing (1)

• indexer/services/bazooka/src/index.ts (1 hunks)

🧰 Additional context used

🔇 Additional comments (1)

indexer/services/bazooka/src/index.ts (1)

28-28: Verify consumer scalability before deploying partition increase.

The increase from 150 to 210 partitions for KafkaTopics.TO_VULCAN looks reasonable. However, this change requires careful deployment planning.

Key considerations:

1. Ensure Vulcan consumers can handle the increased parallelism (210 partitions)
2. Plan for consumer group rebalancing during deployment
3. Monitor broker resources (memory, file descriptors) after the change

Run this verification script to check consumer group configurations:

✅ Verification successful

Let me generate more targeted shell scripts to gather information about the Vulcan consumer configuration.

This script will help us verify:

1. Consumer group configuration in Vulcan service
2. Kafka consumer implementation details
3. Any partition-specific handling or limitations
4. Concurrency and parallel processing capabilities

The results will help us determine if the Vulcan service can handle the increased partition count effectively.

---

Let me try one more targeted search to find Kafka consumer configuration and scaling settings in Vulcan.

This should help us understand how Vulcan handles Kafka consumption and whether it has any scaling limitations that might affect the partition increase.

---

Let me check one final aspect - the Kafka controller implementation in Vulcan to understand the consumer configuration.

This will help us understand how the Vulcan service handles Kafka consumption, batch processing, and consumer group configurations, which are critical for handling the increased partition count.

---

Based on the gathered information, I can now provide a final response about the partition increase verification.

Vulcan service can handle increased partition count safely

The verification of Vulcan's consumer implementation shows it's well-equipped to handle the partition increase from 150 to 210:

• Uses batch processing by default (BATCH_PROCESSING_ENABLED: true)
• Implements flexible consumer configuration with batch commit frequency of 3 seconds
• No hard-coded partition limits or concurrency restrictions
• Uses KafkaJS library which handles partition rebalancing automatically
• Implements graceful shutdown with proper consumer cleanup

The service architecture is partition-count agnostic, using modern Kafka consumer group mechanics that will automatically handle the increased parallelism.

Note: While the change is safe, monitor these metrics after deployment:

• Consumer lag across the increased partition count
• Batch processing performance (KAFKA_BATCH_PROCESSING_COMMIT_FREQUENCY_MS)
• Message processing latency

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify Vulcan consumer group configuration and current partition assignments

# Search for Vulcan consumer configuration files
echo "Searching for Vulcan consumer configurations..."
rg -l "TO_VULCAN" 

# Search for consumer group related code
echo "Checking consumer group implementations..."
ast-grep --pattern 'consumer.group' -l

Length of output: 1231

---

Script:

#!/bin/bash
# Search for consumer group configurations in Vulcan service
echo "Checking Vulcan consumer group configurations..."
rg -A 5 "consumer.*group" indexer/services/vulcan/

# Check Kafka consumer configurations
echo "Checking Kafka consumer implementations..."
rg -A 5 "KafkaConsumer|ConsumerConfig" indexer/services/vulcan/

# Look for parallel processing or partition handling code
echo "Checking partition handling code..."
rg -A 5 "partition|concurrent" indexer/services/vulcan/src/

# Check if there are any hardcoded partition limits
echo "Checking for partition-related constants..."
rg "MAX_PARTITIONS|NUM_PARTITIONS|partitions.*=" indexer/services/vulcan/

Length of output: 1219

---

Script:

#!/bin/bash
# Check Kafka consumer configuration in Vulcan's main files and config
echo "Checking Vulcan Kafka consumer setup..."
rg -A 10 "new Kafka(Consumer|Producer)" indexer/services/vulcan/src/

# Look for configuration files and environment variables
echo "Checking configuration and environment variables..."
rg -l "config|Config|ENV" indexer/services/vulcan/src/
cat indexer/services/vulcan/src/config.ts 2>/dev/null || true

# Check for worker/thread pool configurations
echo "Checking worker configurations..."
rg -A 5 "worker|pool|thread|concurrent" indexer/services/vulcan/src/

# Check the main application setup
echo "Checking main application setup..."
cat indexer/services/vulcan/src/index.ts 2>/dev/null || true

Length of output: 4304

---

Script:

#!/bin/bash
# Check Kafka controller implementation
echo "Checking Kafka controller implementation..."
cat indexer/services/vulcan/src/helpers/kafka/kafka-controller.ts 2>/dev/null || true

# Check for batch processing configuration
echo "Checking batch processing settings..."
rg -A 5 "BATCH_PROCESSING" indexer/services/vulcan/src/

# Look for consumer group settings in Kafka package
echo "Checking Kafka package consumer settings..."
cat indexer/packages/kafka/src/kafka-consumer.ts 2>/dev/null || true

Length of output: 3689

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[roy-dydx]

@Mergifyio release/indexer/v6.x

[mergify]

release /indexer/v6.x

❌ Sorry but I didn't understand the command. Please consult the commands documentation 📚.

[roy-dydx]

https://github.com/Mergifyio backport release/indexer/v6.x

[mergify]

backport release/indexer/v6.x

✅ Backports have been created

• #2529 Vulcan topic to 210 partitions (backport #2528) has been created for branch release/indexer/v6.x

[roy-dydx]

https://github.com/Mergifyio backport release/indexer/v7.x

[mergify]

backport release/indexer/v7.x

✅ Backports have been created

• #2530 Vulcan topic to 210 partitions (backport #2528) has been created for branch release/indexer/v7.x