How to use MYSQL/PostgreSQL in multi thread python program

Submitted 4 months, 2 weeks ago
Ticket #431
Views 263
Language MYSQL
Priority High
Status Closed

I have a requirement to get the data from MYSQL table, The table has date column.

Based on date filter condition i need to do the multi threading and insert into other tables.

For example, 

Source MYSQL Table(Order)

I need to select * from Order where last_date >= '?'(this date should be passed via from mult threading loops)

From this output ,load/insert into another table lets say(Order1)

Please help me with some sample code.

Submitted on May 04, 21

Our team is working on your ticket, we will try to provide solution as soon as possible, thanks! - Jaanvi 4 months, 2 weeks ago
add a comment

1 Answer

Verified

How about this?

import psycopg2
import random
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from psycopg2.pool import ThreadedConnectionPool
DSN = "postgresql://user:pass@localhost/database"
tcp = ThreadedConnectionPool(1, 10, DSN)

def write_sim_to_db(all_ids2):
    conn = tcp.getconn()
    c = conn.cursor()
    c.execute("""SELECT * FROM TABLE A WHERE DATA_FILTER= %s""", ("test_data",))
    conn.commit()
    tcp.putconn(conn)

if __name__ == "__main__":
    start_time = time.time()
    all_ids1 = list(n for n in range(10))
    all_ids2_list = list(n for n in range(10))
    for i in range(len(all_ids1)):
        with ThreadPoolExecutor(max_workers=10) as pool:
            results = [pool.submit(write_sim_to_db, i) for i in all_ids2_list]
    elapsed_time = time.time() - start_time
    print(str(elapsed_time))

   

Submitted 4 months, 2 weeks ago


Wanna Post Ad?
Reach a large group of audience by posting an ad about your business here. For more details
Drop us a Message