Airbyte Postgres to Snowflake sync failed for 10M rows table

Submitted 1 year, 8 months ago
Ticket #441
Views 271
Language/Framework Other
Priority Low
Status Closed

Hi,

I am trying to sync the data from POSTGRES to SNOWFLAKE of 10M rows table. But after pulled 600k rows got stuck on below error,
2022-08-02 03:54:44 INFO i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):325 - Records read: 698000 (6 GB)
2022-08-02 03:54:45 destination > 2022-08-02 03:54:45 INFO i.a.i.d.r.SerializedBufferingStrategy(flushWriter):93 - Flushing buffer of stream MTL_SYSTEM_ITEMS (200 MB)
2022-08-02 03:54:45 destination > 2022-08-02 03:54:45 INFO i.a.i.d.s.StagingConsumerFactory(lambda$flushBufferFunction$3):158 - Flushing buffer for stream MTL_SYSTEM_ITEMS (200 MB) to staging
2022-08-02 03:54:45 destination > 2022-08-02 03:54:45 INFO i.a.i.d.r.BaseSerializedBuffer(flush):131 - Wrapping up compression and write GZIP trailer data.
2022-08-02 03:54:45 destination > 2022-08-02 03:54:45 INFO i.a.i.d.r.BaseSerializedBuffer(flush):138 - Finished writing data to 4802c05e-271e-4b8c-b4d2-953dda97865a16960591700195388958.csv.gz (200 MB)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.internal.SdkFilterInputStream.close(SdkFilterInputStream.java:99)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.event.ProgressInputStream.close(ProgressInputStream.java:211)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.util.IOUtils.closeQuietly(IOUtils.java:70)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.closeQuietlyForRuntimeExceptions(AmazonHttpClient.java:767)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:760)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:719)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:701)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:669)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:651)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:515)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4443)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4390)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.services.s3.AmazonS3Client.doUploadPart(AmazonS3Client.java:3395)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.services.s3.AmazonS3Client.uploadPart(AmazonS3Client.java:3380)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.services.s3.transfer.internal.UploadPartCallable.call(UploadPartCallable.java:33)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.services.s3.transfer.internal.UploadPartCallable.call(UploadPartCallable.java:23)
2022-08-02 03:57:53 destination > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2022-08-02 03:57:53 destination > at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2022-08-02 03:57:53 destination > at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2022-08-02 03:57:53 destination > at java.base/java.lang.Thread.run(Thread.java:833)
2022-08-02 03:57:53 destination >

Configuration below
Service: Docker
Hardware: 32 GB RAM (Local Laptop)
OS: Windows 10 Enterprise

Submitted on Aug 03, 22
add a comment

1 Answer

Verified

this can happen if the resources are limiting could you help me with information on resources(CPU, RAM). Also see if you can increase them and try again

Submitted 11 months, 1 week ago


Latest Blogs