How to use MYSQL/PostgreSQL in multi thread python program

Submitted 3 years, 7 months ago
Ticket #431
Views 1358
Language/Framework MySQL
Priority High
Status Closed

I have a requirement to get the data from MYSQL table, The table has date column.

Based on date filter condition i need to do the multi threading and insert into other tables.

For example, 

Source MYSQL Table(Order)

I need to select * from Order where last_date >= '?'(this date should be passed via from mult threading loops)

From this output ,load/insert into another table lets say(Order1)

Please help me with some sample code.

Submitted on May 04, 21
Vengat
Our team is working on your ticket, we will try to provide solution as soon as possible, thanks! - Jaanvi 3 years, 7 months ago
add a comment
1 Answer
1


Verified

How about this?

import psycopg2
import random
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from psycopg2.pool import ThreadedConnectionPool
DSN = "postgresql://user:pass@localhost/database"
tcp = ThreadedConnectionPool(1, 10, DSN)

def write_sim_to_db(all_ids2):
    conn = tcp.getconn()
    c = conn.cursor()
    c.execute("""SELECT * FROM TABLE A WHERE DATA_FILTER= %s""", ("test_data",))
    conn.commit()
    tcp.putconn(conn)

if __name__ == "__main__":
    start_time = time.time()
    all_ids1 = list(n for n in range(10))
    all_ids2_list = list(n for n in range(10))
    for i in range(len(all_ids1)):
        with ThreadPoolExecutor(max_workers=10) as pool:
            results = [pool.submit(write_sim_to_db, i) for i in all_ids2_list]
    elapsed_time = time.time() - start_time
    print(str(elapsed_time))

Submitted 3 years, 6 months ago
scott

Latest Blogs

Step by Step guide to deploy Django Web App on Heroku
Python XML - Read/Write and Create
Day 6-Mastering PySpark: A 100-Day Challenge
How to Implement the Django filters
Different way of processing the GET request in Javascript
Django Heroku Deployment Steps
How to setup the Notification in website using Javascript and Django
How to Build Desktop App Using Tkinter and Django Rest Framework Backend -Part 1
How to Build Desktop App Using Tkinter and Django Rest Framework Backend -Part 2
Python Enumerate in Django Choice Field
Creating A Popup Modal Login Form Using HTML5 And CSS and Bootstrap.
Simple Search App using Django API and Vanilla Javascript
Automation Testing Life Cycle
How to Integrate Django and Vue JS Part -1
Steps to deploy Django App into AWS
Building Live streaming web app with FastAPI and ReactJS
SOME STEPS TO FOLLOW BEFORE STARTING A PROJECT :
What is the secret behind SUCCESS?
Success
Django Static Files in Local
Creation of web pages for register,login ,logout (PART-2)
Data Analytics in a Nutshell
Python’s Generator and Yield
What is Mean? Why is it required?
IMPLEMENTATION OF SEARCH BAR USING DJANGO (in any website)
Creation of webpages for register ,login ,logout (PART-1)
Web designer or Front End Developer? Key Differences
Title: Unraveling the Wonders of Quantum Computing: A Leap into the Future
Python Desktop APP to Upload Image into AWS S3 with Progress Bar
Building a Gallery App with FastAPI and React in AWS S3
Tips for Optimizing Apache Spark Queries — Part 1 (Data Partitioning)
Building a Real-time Feedback System with Django, Kafka, Spark, and S3
Ongoing Knowledge - Keep Learning ang Keep Sharing
Custom Model Managers In Django
Python Merge PDF files
Python Challenge 1 - Capital indexes
How to Integrate Django and Vue JS Part -2( Creating Simple Todo App)
Program vs Process vs Thread
Python Multiprocessing with Real Time Example
Unlocking the Power of Caching with Python, Relational Database & Redis Unleashed
Stay Hydrated with a Python Script
Day 8 UDF -Mastering PySpark: A 100-Day Challenge
Resilient in Spark— My Understanding
Pyspark Row : asDict(): A 100-Day PySpark Challenge
PySpark — Dataframe Operations CheatSheet
Snowflake Database Zero to Hero - Let's Learn Together