Deleting folders in python recursively

problem with deleting empty directories. Here is the code:

for dirpath, dirnames, filenames in os.walk(dir_to_search):
    //other codes

    try:
        os.rmdir(dirpath)
    except OSError as ex:
        print(ex)

The argument dir_to_search is where I'm passing the directory where the work needs to be done. That directory looks like this:

test/20/...
test/22/...
test/25/...
test/26/...

Note that all the above folders are empty. When I run this script the folders 20,25 alone gets deleted! But the folders 25 and 26 aren't deleted, even though they are empty folders.

Edit:

The exceptions are:

[Errno 39] Directory not empty: '/home/python-user/shell-scripts/s3logs/test'
[Errno 39] Directory not empty: '/home/python-user/shell-scripts/s3logs/test/2012'
[Errno 39] Directory not empty: '/home/python-user/shell-scripts/s3logs/test/2012/10'
[Errno 39] Directory not empty: '/home/python-user/shell-scripts/s3logs/test/2012/10/29'
[Errno 39] Directory not empty: '/home/python-user/shell-scripts/s3logs/test/2012/10/29/tmp'
[Errno 39] Directory not empty: '/home/python-user/shell-scripts/s3logs/test/2012/10/28'
[Errno 39] Directory not empty: '/home/python-user/shell-scripts/s3logs/test/2012/10/28/tmp'
[Errno 39] Directory not empty: '/home/python-user/shell-scripts/s3logs/test/2012/10/26'
[Errno 39] Directory not empty: '/home/python-user/shell-scripts/s3logs/test/2012/10/25'
[Errno 39] Directory not empty: '/home/python-user/shell-scripts/s3logs/test/2012/10/27'
[Errno 39] Directory not empty: '/home/python-user/shell-scripts/s3logs/test/2012/10/27/tmp'

What is the fix ?