Read a single byte from large file

Submitted 3 years, 7 months ago

Ticket #166

Views 248

Language/Framework Python

Priority Medium

Status Closed

I have this function that I want to read a single byte from a large file. The only problem is that after a certain amount of file reads the memory on the pc jumps up from a steady 1.5gb to 4gb and higher depending on how many file reads. (I break at 80 files because higher would crash my pc)

All i want is to get the 1 byte and not the whole file. Help please.

 def mem_test():
        count = 0
        for dirpath, dnames, fnames in scandir.walk(restorePaths[0]):
            for f in fnames:
                if 'DIR.EDB' in f or 'PRIV.EDB' in f:
                    f = open(os.path.join(dirpath, f), 'rb')
                    f.read(1) #Problem line!
                    f.close()
                    if count > 80:
                        print 'EXIT'
                        sys.exit(1)                    
                    count += 1
mem_test()

Submitted on Sep 13, 20

Anonymous

add a comment

1 Answer

Verified

The basic file operations in Python. This opens one file, reads the data into memory, then opens the second file and writes it out.

in_file = open("in-file", "rb") # opening for [r]eading as [b]inary
data = in_file.read() # if you only wanted to read 512 bytes, do .read(512)
in_file.close()

out_file = open("out-file", "wb") # open for [w]riting as [b]inary
out_file.write(data)
out_file.close()

We can do this more succinctly by using the with keyboard to handle closing the file.

with open("in-file", "rb") as in_file, open("out-file", "wb") as out_file:
    out_file.write(in_file.read())

If you don't want to store the entire file in memory, you can transfer it in pieces.

piece_size = 4096 # 4 KiB

with open("in-file", "rb") as in_file, open("out-file", "wb") as out_file:
    while True:
        piece = in_file.read(piece_size)

        if piece == "":
            break # end of file

        out_file.write(piece)

Submitted 3 years, 6 months ago