Skip to content

BigQuery Source Connector Code #599

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

raheelkumarpersistent
Copy link

Created a Source Connector for BigQuery which takes a SELECT query from the user and a Service Account credentials file to fetch data from BigQuery and store in Quix Topics.

@tomas-quix tomas-quix requested a review from Copilot April 1, 2025 09:21
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements a new BigQuery Source Connector that extracts data from BigQuery using a user-defined SQL query and publishes it to a Kafka topic using Quix Streams.

  • Implements custom JSON encoding and serialization for handling various BigQuery data types
  • Integrates with Quix Streams to publish data to Kafka
  • Adds a README with detailed usage, configuration, and setup instructions

Reviewed Changes

Copilot reviewed 2 out of 5 changed files in this pull request and generated no comments.

File Description
python/sources/BigQuery_source/main.py New implementation for BigQuery data extraction and publishing
python/sources/BigQuery_source/README.md Documentation detailing connector usage and setup
Files not reviewed (3)
  • python/sources/BigQuery_source/dockerfile: Language not supported
  • python/sources/BigQuery_source/library.json: Language not supported
  • python/sources/BigQuery_source/requirements.txt: Language not supported
Comments suppressed due to low confidence (2)

python/sources/BigQuery_source/main.py:138

  • The check 'header not in df.columns' is redundant since 'headers' is derived from df.columns. Consider removing this condition.
if pd.isna(value) and header not in df.columns:

python/sources/BigQuery_source/main.py:35

  • [nitpick] For consistency with the rest of the code, consider using the imported 'time' type (i.e., 'if isinstance(obj, time):') instead of 'datetime.time'.
if isinstance(obj, datetime.time):

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant