How to use MYSQL/PostgreSQL in multi thread python program

Submitted 2 years, 11 months ago

Ticket #431

Views 1234

Language/Framework MySQL

Priority High

Status Closed

I have a requirement to get the data from MYSQL table, The table has date column.

Based on date filter condition i need to do the multi threading and insert into other tables.

For example,

Source MYSQL Table(Order)

I need to select * from Order where last_date >= '?'(this date should be passed via from mult threading loops)

From this output ,load/insert into another table lets say(Order1)

Please help me with some sample code.

Submitted on May 04, 21

Vengat

Our team is working on your ticket, we will try to provide solution as soon as possible, thanks! - Jaanvi 2 years, 11 months ago

add a comment

1 Answer

1

Verified

How about this?

import psycopg2
import random
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from psycopg2.pool import ThreadedConnectionPool
DSN = "postgresql://user:pass@localhost/database"
tcp = ThreadedConnectionPool(1, 10, DSN)

def write_sim_to_db(all_ids2):
    conn = tcp.getconn()
    c = conn.cursor()
    c.execute("""SELECT * FROM TABLE A WHERE DATA_FILTER= %s""", ("test_data",))
    conn.commit()
    tcp.putconn(conn)

if __name__ == "__main__":
    start_time = time.time()
    all_ids1 = list(n for n in range(10))
    all_ids2_list = list(n for n in range(10))
    for i in range(len(all_ids1)):
        with ThreadPoolExecutor(max_workers=10) as pool:
            results = [pool.submit(write_sim_to_db, i) for i in all_ids2_list]
    elapsed_time = time.time() - start_time
    print(str(elapsed_time))

Submitted 2 years, 11 months ago

scott

Latest Blogs

Step by Step guide to deploy Django Web App on Heroku

Python XML - Read/Write and Create

Day 6-Mastering PySpark: A 100-Day Challenge

How to Implement the Django filters

Different way of processing the GET request in Javascript

Django Heroku Deployment Steps

How to setup the Notification in website using Javascript and Django

How to Build Desktop App Using Tkinter and Django Rest Framework Backend -Part 1

How to Build Desktop App Using Tkinter and Django Rest Framework Backend -Part 2

Python Enumerate in Django Choice Field

Creating A Popup Modal Login Form Using HTML5 And CSS and Bootstrap.

Simple Search App using Django API and Vanilla Javascript

Automation Testing Life Cycle

How to Integrate Django and Vue JS Part -1

Steps to deploy Django App into AWS

Building Live streaming web app with FastAPI and ReactJS

SOME STEPS TO FOLLOW BEFORE STARTING A PROJECT :

What is the secret behind SUCCESS?

Success

Django Static Files in Local

Creation of web pages for register,login ,logout (PART-2)

Data Analytics in a Nutshell

Python’s Generator and Yield

What is Mean? Why is it required?

IMPLEMENTATION OF SEARCH BAR USING DJANGO (in any website)

Creation of webpages for register ,login ,logout (PART-1)

Web designer or Front End Developer? Key Differences

Title: Unraveling the Wonders of Quantum Computing: A Leap into the Future

Python Desktop APP to Upload Image into AWS S3 with Progress Bar

Building a Gallery App with FastAPI and React in AWS S3

Tips for Optimizing Apache Spark Queries — Part 1 (Data Partitioning)

Building a Real-time Feedback System with Django, Kafka, Spark, and S3

Ongoing Knowledge - Keep Learning ang Keep Sharing

Custom Model Managers In Django

Python Merge PDF files

Python Challenge 1 - Capital indexes

How to Integrate Django and Vue JS Part -2( Creating Simple Todo App)

Program vs Process vs Thread

Python Multiprocessing with Real Time Example

Unlocking the Power of Caching with Python, Relational Database & Redis Unleashed

Stay Hydrated with a Python Script

Day 8 UDF -Mastering PySpark: A 100-Day Challenge

Pyspark Row : asDict(): A 100-Day PySpark Challenge

Snowflake Database Zero to Hero - Let's Learn Together