Airbyte Postgres to Snowflake sync failed for 10M rows table

Submitted 2 years, 2 months ago
Ticket #441
Views 382
Language/Framework Other
Priority Low
Status Closed

Hi,

I am trying to sync the data from POSTGRES to SNOWFLAKE of 10M rows table. But after pulled 600k rows got stuck on below error,
2022-08-02 03:54:44 INFO i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):325 - Records read: 698000 (6 GB)
2022-08-02 03:54:45 destination > 2022-08-02 03:54:45 INFO i.a.i.d.r.SerializedBufferingStrategy(flushWriter):93 - Flushing buffer of stream MTL_SYSTEM_ITEMS (200 MB)
2022-08-02 03:54:45 destination > 2022-08-02 03:54:45 INFO i.a.i.d.s.StagingConsumerFactory(lambda$flushBufferFunction$3):158 - Flushing buffer for stream MTL_SYSTEM_ITEMS (200 MB) to staging
2022-08-02 03:54:45 destination > 2022-08-02 03:54:45 INFO i.a.i.d.r.BaseSerializedBuffer(flush):131 - Wrapping up compression and write GZIP trailer data.
2022-08-02 03:54:45 destination > 2022-08-02 03:54:45 INFO i.a.i.d.r.BaseSerializedBuffer(flush):138 - Finished writing data to 4802c05e-271e-4b8c-b4d2-953dda97865a16960591700195388958.csv.gz (200 MB)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.internal.SdkFilterInputStream.close(SdkFilterInputStream.java:99)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.event.ProgressInputStream.close(ProgressInputStream.java:211)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.util.IOUtils.closeQuietly(IOUtils.java:70)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.closeQuietlyForRuntimeExceptions(AmazonHttpClient.java:767)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:760)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:719)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:701)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:669)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:651)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:515)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4443)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4390)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.services.s3.AmazonS3Client.doUploadPart(AmazonS3Client.java:3395)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.services.s3.AmazonS3Client.uploadPart(AmazonS3Client.java:3380)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.services.s3.transfer.internal.UploadPartCallable.call(UploadPartCallable.java:33)
2022-08-02 03:57:53 destination > at net.snowflake.client.jdbc.internal.amazonaws.services.s3.transfer.internal.UploadPartCallable.call(UploadPartCallable.java:23)
2022-08-02 03:57:53 destination > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2022-08-02 03:57:53 destination > at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2022-08-02 03:57:53 destination > at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2022-08-02 03:57:53 destination > at java.base/java.lang.Thread.run(Thread.java:833)
2022-08-02 03:57:53 destination >

Configuration below
Service: Docker
Hardware: 32 GB RAM (Local Laptop)
OS: Windows 10 Enterprise

Submitted on Aug 03, 22
add a comment

1 Answer

Verified

this can happen if the resources are limiting could you help me with information on resources(CPU, RAM). Also see if you can increase them and try again

Submitted 1 year, 4 months ago


Latest Blogs