Issue Number | 3452 |
---|---|
Summary | Fix Program to FTP Vendor Data to FTP Server |
Created | 2011-11-14 15:11:50 |
Issue Type | Improvement |
Submitted By | Englisch, Volker (NIH/NCI) [C] |
Assigned To | Englisch, Volker (NIH/NCI) [C] |
Status | Closed |
Resolved | 2012-03-09 12:20:11 |
Resolution | Fixed |
Path | /home/bkline/backups/jira/ocecdr/issue.107780 |
BZISSUE::5146
BZDATETIME::2011-11-14 15:11:50
BZCREATOR::Volker Englisch
BZASSIGNEE::Volker Englisch
BZQACONTACT::William Osei-Poku
Last weekend the program that runs as part of the publishing job
which creates a compressed tar file and copies it to the CIPSFTP server
failed to copy the data.
As a result, last weeks PDQ XML data has been extracted and copied to
the public location.
We want to create a mechanism that submits a notification if this should happen again instead of silently failing.
BZDATETIME::2011-11-29 11:27:18
BZCOMMENTOR::Volker Englisch
BZCOMMENT::1
This same problem came up again this weekend. After we haven't had any problems with this for at least the beginning of the year this problem came up three times within the past eight weeks.
We still want to create a notification if the error occurs again but
the real issue is something else. Tar is reporting the error
message:
tar: file changed as we read it
Bob/Alan: I know there exists an option to ignore this error in
GNU-tar.
Are you aware of such an option with CYGWIN tar?
BZDATETIME::2011-11-29 12:37:20
BZCOMMENTOR::Alan Meyer
BZCOMMENT::2
(In reply to comment #1)
...
> Bob/Alan: I know there exists an option to ignore this error in
GNU-tar.
> Are you aware of such an option with CYGWIN tar?
The cygwin version is a port of the GNU tar, so I assume the option is the same. I think it would be:
--warning=no-file-changed
(see http://www.gnu.org/s/tar/manual/html_section/warnings.html)
I thought the idea of a "warning" was to issue the warning message but not stop processing, but I guess not if that's what's stopping us.
Do we know why the file is changing during the tar process?
BZDATETIME::2011-11-29 13:11:31
BZCOMMENTOR::Volker Englisch
BZCOMMENT::3
I'm actually getting an error and an exit code 1.
> Do we know why the file is changing during the tar process?
The only thing that I can imagine would be that a backup is running changing the access time of a file.
BZDATETIME::2012-02-24 15:17:03
BZCOMMENTOR::Volker Englisch
BZCOMMENT::4
So far, if the program failed I was writing a message to the log
file. Now I have modified the program
FtpExportData.py - R10333
to also submit an email about a failure of the program so that it
hopefully won't happen anymore that licensees are receiving the vendor
data from the last week because the current data failed to be
FTP'ed.
I've tested this change successfully on MAHLER by modifying a document that was in the process of being tar'ed. Since this was a fairly minor change (adding a few lines of text to be submitted by email) and it may take a long time before we "hit" this situation again to trigger the new code, I went ahead and copied the changes to BACH already.
I'll keep the issue open for a week or two before closing it.
BZDATETIME::2012-02-27 10:52:13
BZCOMMENTOR::Volker Englisch
BZCOMMENT::5
(In reply to comment #4)
> I'll keep the issue open for a week or two before closing it.
There were no problems during the weekly publishing job.
BZDATETIME::2012-03-04 11:40:20
BZCOMMENTOR::Volker Englisch
BZCOMMENT::6
Good news and bad news:
The good news is that the notification is working as implemented if the
job fails.
The bad news is that the FTP job failed.
I'm closing this issue since the change is working properly but I may
make additional changes if this job is going to fail more frequently due
to this 'file changed as we read it' failure.
We may even decide to completely ignore the message. The last few times
that we got this error and I recreated the tar file it turned out to be
identical to the file that had been created but finished with a
failure.
BZDATETIME::2012-03-09 12:20:11
BZCOMMENTOR::Volker Englisch
BZCOMMENT::7
Closing issue. The program ran correctly for the past two weeks.
Elapsed: 0:00:00.001215