Join the Teckiy Community
Sign up for Free !!
World's first open source developer community with
Ticketing System
Team,
Currently, I have Datasphere transmitting transactional data from Azure to BigQuery. Separately, I am ingesting master data from an API that is triggered by a Pub/Sub Topic and orchestrated through Logic Apps.
At the destination (BigQuery), I perform Change Data Capture (CDC) on the transactional data. For the master data, the process involves unnesting in the staging area, appending a hash value, and then creating a nested table structure in BigQuery.
I'm exploring if there’s a more efficient approach to eliminate the need for CDC while seamlessly integrating both data sources.
