CDR Tickets

Issue Number 3819
Summary FileSweeper Fails to remove Directory
Created 2014-10-23 17:30:45
Issue Type Bug
Submitted By Englisch, Volker (NIH/NCI) [C]
Assigned To alan
Status Closed
Resolved 2014-10-24 11:57:21
Resolution Fixed
Path /home/bkline/backups/jira/ocecdr/issue.140255
Description

It appears that our FileSweeper job failed to remove a directory after archiving it. This results the same archive file (with a different sequence number) to be created again the next time. When the maximum number of allowed sequences is reached, the FileSweeper exits because it's unable to create another archive file.

It may be possible that the removal of the directory fails because of spaces in the directory name. The directory in question on DEV is
"d:\cdr\Audio_from_CIPSFTP\Week_12 Revision1"

Comment entered 2014-10-23 17:36:13 by Englisch, Volker (NIH/NCI) [C]

I've created a tar file of the directory in question and will delete it now so that the FileSweeper can run again. The tar file located in the Audio_from_CIPSFTP directory with the name Audio.tar.bz2.

Comment entered 2014-10-23 17:36:59 by Englisch, Volker (NIH/NCI) [C]

Tarred Audio file directory attached.

Comment entered 2014-10-23 17:41:18 by Englisch, Volker (NIH/NCI) [C]

Issue was assigned to the wrong man but feel free, Bob, if you feel like sweeping.

Comment entered 2014-10-23 23:37:20 by alan

I tested several solutions to the problem but settled on replacing the shell out to "rm -r" with shutil.rmtree(). It provides clean handling of directory names with spaces and is consistent with the try / except method already in place for catching and reporting errors.

I've checked in the code. It's just a one line change.

FileSweeper can only run one instance at a time and an instance is already running, so I left that alone and thought I'd coordinate with you (Volker) about substituting it for the version running on Dev for a good test.

Comment entered 2014-10-24 11:56:40 by Englisch, Volker (NIH/NCI) [C]

I restored the directory that tripped up the FileSweeper.py before and this time it processed (and deleted the directory) without problems.

Alan's message that a FileSweeper process was already running surprised me, as I was under the impression that my earlier run of the program that I started at around 6pm had already been completed. It made me realize that the program prints out a statement when it's starting but not when it finishes.
I added that final message and versioned the FileSweeper:

  • R12939: FileSweeper

I also copied the new version to the bin-directory.

Attachments
File Name Posted User
Audio.tar.bz2 2014-10-23 17:36:59 Englisch, Volker (NIH/NCI) [C]

Elapsed: 0:00:00.001863