CDR Tickets

Issue Number 3126
Summary Upgrade Bugzilla to version 3.6
Created 2010-04-13 19:25:09
Issue Type Improvement
Submitted By priced
Assigned To alan
Status Closed
Resolved 2010-04-29 18:25:42
Resolution Fixed
Path /home/bkline/backups/jira/ocecdr/issue.107454
Description

BZISSUE::4803
BZDATETIME::2010-04-13 19:25:09
BZCREATOR::Stas Firstov
BZASSIGNEE::Alan Meyer
BZQACONTACT::Alan Meyer

Bugzilla needs to be upgraded to version 3.6

From: Stas Firstov <sfirstov@firstov.com>
sender-time Sent at 6:34 PM (GMT-04:00). Current time there: 7:23 PM. ✆
to Alan Meyer <vrmeyer@comcast.net>
cc Volker Englisch <volker@englisch.us>,
Bob Kline <bkline@rksystems.com>,
John Rehmert <rehmertj@mail.nih.gov>,
"Lakshmi Grama (work)" <lgrama@mail.nih.gov>,
Margaret Beckwith <mbeckwit@mail.nih.gov>,
"Volker Englisch (work)" <volker@mail.nih.gov>
date Tue, Apr 13, 2010 at 6:34 PM
subject Re: Fwd: [ANN] Release of Bugzilla 3.6!
mailed-by firstov.com
hide details 6:34 PM (49 minutes ago)
Just a reminder Bugzilla will not be available today @ 6:30pm
Thanks,
Stas

  • Hide quoted text -

On Tue, Apr 13, 2010 at 12:03 PM, Stas Firstov <sfirstov@firstov.com> wrote:
> Alan,
> I plan to backup everything (including bugs DB) and convert current
> installation to the latest version then. During the conversion the old
> version will not be available.
> I'm sure there were some changes in the database structure since the
> current Bugzilla is so behind. The upgrade script will take care of
> the DB conversion.
> I'll send you an email tonight before the upgrade starts (estimated
> start time is 6pm)
> Thanks,
> Stas
>
>
>
> On Tue, Apr 13, 2010 at 11:53 AM, Alan Meyer <vrmeyer@comcast.net> wrote:
>> On 4/13/2010 11:16 AM, Stas Firstov wrote:
>>> I have two days left under the current contract to perform the
>>> upgrade. Please advise on the date/time when this needs to be done.
>>> I will need somebody on standby to to perform a QC to ensure there
>>> will be no unexpected issues after the upgrade.
>>>
>>> Thanks,
>>> Stas
>>
>> Hello Stas,
>>
>> I'll be here until very late, probably midnight.
>>
>> Assuming we go ahead with this tonight, please send me an email
>> if and when you need me to not use Bugzilla, and another telling
>> me when I can use it again and/or test the new version. I can do
>> some testing as soon as you need it.
>>
>> Do you know yet if the new version requires a database
>> conversion? Will the old version be available during testing? I
>> presume if the database is converted we'll want to shut down the
>> old version as soon as it's done so we don't have old and new
>> databases out of sync.
>>
>> Thanks.
>>
>> --
>> Alan Meyer
>> AM Systems, Inc.
>> Randallstown, MD, USA
>> vrmeyer@comcast.net
>>
>>
>

Comment entered 2010-04-13 19:33:26 by priced

BZDATETIME::2010-04-13 19:33:26
BZCOMMENTOR::Stas Firstov
BZCOMMENT::1

Bugzilla has been upgraded to version 3.6
During the upgrade a few extra perl modules needed to be installed and mysql config file (/etc/my.cnf) changed to increase the max_allowed_packet parameter to 12M (originally it was set to 4M)
Below is the log.

Please test and advise. The backup files are on the server and they could be restored if rollback is required.
Reassigning to Alan for the QC...

Thanks,
Stas

[root@otsavmlinux1 bugzilla]# ./checksetup.pl

  • This is Bugzilla 3.6 on perl 5.8.8

  • Running on Linux 2.6.18-164.9.1.el5 #1 SMP Wed Dec 9 03:29:54 EST 2009

Checking perl modules...
Checking for CGI.pm (v3.21) ok: found v3.49
Checking for Digest-SHA (any) ok: found v5.48
Checking for TimeDate (v2.21) ok: found v2.22
Checking for DateTime (v0.28) ok: found v0.55
Checking for DateTime-TimeZone (v0.71) ok: found v1.17
Checking for DBI (v1.41) ok: found v1.52
Checking for Template-Toolkit (v2.22) ok: found v2.22
Checking for Email-Send (v2.00) ok: found v2.185
Checking for Email-MIME (v1.861) ok: found v1.903
Checking for Email-MIME-Encodings (v1.313) ok: found v1.313
Checking for Email-MIME-Modifier (v1.442) ok: found v1.903
Checking for URI (any) ok: found v1.35

Checking available perl DBD modules...
Checking for DBD-Pg (v1.45) not found
Checking for DBD-mysql (v4.00) ok: found v4.013
Checking for DBD-Oracle (v1.19) not found

The following Perl modules are optional:
Checking for GD (v1.20) ok: found v2.35
Checking for Chart (v2.1) ok: found v2.3
Checking for Template-GD (any) ok: found v1.56
Checking for GDTextUtil (any) ok: found v0.86
Checking for GDGraph (any) ok: found v1.44
Checking for XML-Twig (any) ok: found v3.29
Checking for MIME-tools (v5.406) ok: found v5.420
Checking for libwww-perl (any) ok: found v2.033
Checking for PatchReader (v0.9.4) ok: found v0.9.5
Checking for perl-ldap (any) ok: found v0.34
Checking for Authen-SASL (any) not found
Checking for RadiusPerl (any) not found
Checking for SOAP-Lite (v0.710.06) found v0.69
Checking for JSON-RPC (any) not found
Checking for Test-Taint (any) not found
Checking for HTML-Parser (v3.40) ok: found v3.55
Checking for HTML-Scrubber (any) ok: found v0.08
Checking for Email-MIME-Attachment-Stripper (any) ok: found v1.313
Checking for Email-Reply (any) ok: found v1.202
Checking for TheSchwartz (any) not found
Checking for Daemon-Generic (any) not found
Checking for mod_perl (v1.999022) ok: found v2.000004
***********************************************************************

  • OPTIONAL MODULES *
    ***********************************************************************

  • Certain Perl modules are not required by Bugzilla, but by *

  • installing the latest version you gain access to additional *

  • features. *

  • *

  • The optional modules you do not have installed are listed below, *

  • with the name of the feature they enable. Below that table are the *

  • commands to install each module. *
    ***********************************************************************

  • MODULE NAME * ENABLES FEATURE(S) *
    ***********************************************************************

  • Authen-SASL * SMTP Authentication *

  • RadiusPerl * RADIUS Authentication *

  • SOAP-Lite * XML-RPC Interface *

  • JSON-RPC * JSON-RPC Interface *

  • Test-Taint * JSON-RPC Interface, XML-RPC Interface *

  • TheSchwartz * Mail Queueing *

  • Daemon-Generic * Mail Queueing *
    ***********************************************************************
    COMMANDS TO INSTALL OPTIONAL MODULES:

Authen-SASL: /usr/bin/perl install-module.pl Authen::SASL
RadiusPerl: /usr/bin/perl install-module.pl Authen::Radius
SOAP-Lite: /usr/bin/perl install-module.pl SOAP::Lite
JSON-RPC: /usr/bin/perl install-module.pl JSON::RPC
Test-Taint: /usr/bin/perl install-module.pl Test::Taint
TheSchwartz: /usr/bin/perl install-module.pl TheSchwartz
Daemon-Generic: /usr/bin/perl install-module.pl Daemon::Generic

To attempt an automatic install of every required and optional module
with one command, do:

/usr/bin/perl install-module.pl --all

Reading ./localconfig...

OPTIONAL NOTE: If you want to be able to use the 'difference between two
patches' feature of Bugzilla (which requires the PatchReader Perl module
as well), you should install patchutils from:

http://cyberelk.net/tim/patchutils/

Checking for DBD-mysql (v4.00) ok: found v4.013
Checking for MySQL (v4.1.2) ok: found v5.0.77

WARNING: You need to set the max_allowed_packet parameter in your MySQL
configuration to at least 12288000. Currently it is set to 4194304.
You can set this parameter in the [mysqld] section of your MySQL
configuration file.

Adding new table bug_see_also ...
Adding new table login_failure ...
Adding new table ts_error ...
Adding new table ts_exitstatus ...
Adding new table ts_funcmap ...
Adding new table ts_job ...
Adding new table ts_note ...
attachments.bug_id has incorrect DB default: 0
attachments.creation_ts has incorrect DB default: 0000-00-00 00:00:00
attachments.filename has incorrect DB default: ''
attachments.submitter_id has incorrect DB default: 0
bug_group_map.bug_id has incorrect DB default: 0
bug_group_map.group_id has incorrect DB default: 0
bug_severity.value has incorrect DB default: ''
bug_status.value has incorrect DB default: ''
bugs.priority has incorrect DB default: ''
bugs.reporter has incorrect DB default: 0
bugs.product_id has incorrect DB default: 0
bugs.rep_platform has incorrect DB default: ''
bugs.assigned_to has incorrect DB default: 0
bugs.everconfirmed has incorrect DB default: 0
bugs.bug_severity has incorrect DB default: ''
bugs.bug_status has incorrect DB default: ''
bugs.delta_ts has incorrect DB default: 0000-00-00 00:00:00
bugs.version has incorrect DB default: ''
bugs.component_id has incorrect DB default: 0
bugs.op_sys has incorrect DB default: ''
bugs_activity.bug_id has incorrect DB default: 0
bugs_activity.bug_when has incorrect DB default: 0000-00-00 00:00:00
bugs_activity.who has incorrect DB default: 0
bugs_activity.fieldid has incorrect DB default: 0
bz_schema.version has incorrect DB default: 0.00
category_group_map.category_id has incorrect DB default: 0
category_group_map.group_id has incorrect DB default: 0
cc.bug_id has incorrect DB default: 0
cc.who has incorrect DB default: 0
classifications.name has incorrect DB default: ''
components.initialowner has incorrect DB default: 0
components.product_id has incorrect DB default: 0
components.name has incorrect DB default: ''
dependencies.blocked has incorrect DB default: 0
dependencies.dependson has incorrect DB default: 0
duplicates.dupe_of has incorrect DB default: 0
duplicates.dupe has incorrect DB default: 0
email_setting.user_id has incorrect DB default: 0
email_setting.relationship has incorrect DB default: 0
email_setting.event has incorrect DB default: 0
fielddefs.sortkey has incorrect DB default: 0
fielddefs.name has incorrect DB default: ''
flagexclusions.type_id has incorrect DB default: 0
flaginclusions.type_id has incorrect DB default: 0
flags.status has incorrect DB default: ''
flags.bug_id has incorrect DB default: 0
flags.type_id has incorrect DB default: 0
flags.creation_date has incorrect DB default: 0000-00-00 00:00:00
flagtypes.name has incorrect DB default: ''
group_control_map.entry has incorrect DB default: 0
group_control_map.membercontrol has incorrect DB default: 0
group_control_map.canedit has incorrect DB default: 0
group_control_map.product_id has incorrect DB default: 0
group_control_map.othercontrol has incorrect DB default: 0
group_control_map.group_id has incorrect DB default: 0
group_group_map.grantor_id has incorrect DB default: 0
group_group_map.member_id has incorrect DB default: 0
groups.isbuggroup has incorrect DB default: 0
groups.name has incorrect DB default: ''
keyworddefs.name has incorrect DB default: ''
keywords.keywordid has incorrect DB default: 0
keywords.bug_id has incorrect DB default: 0
logincookies.lastused has incorrect DB default: 0000-00-00 00:00:00
logincookies.userid has incorrect DB default: 0
logincookies.ipaddr has incorrect DB default: ''
longdescs.bug_id has incorrect DB default: 0
longdescs.bug_when has incorrect DB default: 0000-00-00 00:00:00
longdescs.who has incorrect DB default: 0
milestones.product_id has incorrect DB default: 0
milestones.value has incorrect DB default: ''
namedqueries.userid has incorrect DB default: 0
namedqueries.name has incorrect DB default: ''
op_sys.value has incorrect DB default: ''
priority.value has incorrect DB default: ''
products.name has incorrect DB default: ''
profile_setting.user_id has incorrect DB default: 0
profile_setting.setting_name has incorrect DB default: ''
profile_setting.setting_value has incorrect DB default: ''
profiles.login_name has incorrect DB default: ''
profiles_activity.userid has incorrect DB default: 0
profiles_activity.profiles_when has incorrect DB default: 0000-00-00 00:00:00
profiles_activity.who has incorrect DB default: 0
profiles_activity.fieldid has incorrect DB default: 0
rep_platform.value has incorrect DB default: ''
resolution.value has incorrect DB default: ''
series.frequency has incorrect DB default: 0
series.category has incorrect DB default: 0
series.name has incorrect DB default: ''
series.subcategory has incorrect DB default: 0
series_categories.name has incorrect DB default: ''
series_data.series_value has incorrect DB default: 0
series_data.series_id has incorrect DB default: 0
series_data.series_date has incorrect DB default: 0000-00-00 00:00:00
setting.name has incorrect DB default: ''
setting.default_value has incorrect DB default: ''
setting_value.name has incorrect DB default: ''
setting_value.value has incorrect DB default: ''
setting_value.sortindex has incorrect DB default: 0
tokens.token has incorrect DB default: ''
tokens.tokentype has incorrect DB default: ''
tokens.issuedate has incorrect DB default: 0000-00-00 00:00:00
user_group_map.group_id has incorrect DB default: 0
user_group_map.user_id has incorrect DB default: 0
versions.product_id has incorrect DB default: 0
versions.value has incorrect DB default: ''
votes.bug_id has incorrect DB default: 0
votes.who has incorrect DB default: 0
votes.vote_count has incorrect DB default: 0
watch.watcher has incorrect DB default: 0
watch.watched has incorrect DB default: 0
whine_events.owner_userid has incorrect DB default: 0
whine_queries.eventid has incorrect DB default: 0
whine_schedules.eventid has incorrect DB default: 0
whine_schedules.mailto has incorrect DB default: 0
Fixing defaults...
Updating old chart storage format...
Creating ./data/extensions directory...
Creating ./skins/contrib/Dusk/yui directory...
Creating ./skins/custom directory...
Creating ./skins/custom/yui directory...
Creating graphs directory...
Creating ./data/extensions/additional...
Creating ./data/mailer.testfile...
Creating ./skins/contrib/Dusk/IE-fixes.css...
Creating ./skins/contrib/Dusk/admin.css...
Creating ./skins/contrib/Dusk/create_attachment.css...
Creating ./skins/contrib/Dusk/dependency-tree.css...
Creating ./skins/contrib/Dusk/duplicates.css...
Creating ./skins/contrib/Dusk/editusers.css...
Creating ./skins/contrib/Dusk/help.css...
Creating ./skins/contrib/Dusk/page.css...
Creating ./skins/contrib/Dusk/panel.css...
Creating ./skins/contrib/Dusk/params.css...
Creating ./skins/contrib/Dusk/reports.css...
Creating ./skins/contrib/Dusk/show_bug.css...
Creating ./skins/contrib/Dusk/show_multiple.css...
Creating ./skins/contrib/Dusk/summarize-time.css...
Creating ./skins/contrib/Dusk/voting.css...
Creating ./skins/contrib/Dusk/yui/calendar.css...
Creating ./skins/custom/IE-fixes.css...
Creating ./skins/custom/admin.css...
Creating ./skins/custom/buglist.css...
Creating ./skins/custom/create_attachment.css...
Creating ./skins/custom/dependency-tree.css...
Creating ./skins/custom/duplicates.css...
Creating ./skins/custom/editusers.css...
Creating ./skins/custom/global.css...
Creating ./skins/custom/help.css...
Creating ./skins/custom/index.css...
Creating ./skins/custom/page.css...
Creating ./skins/custom/panel.css...
Creating ./skins/custom/params.css...
Creating ./skins/custom/reports.css...
Creating ./skins/custom/show_bug.css...
Creating ./skins/custom/show_multiple.css...
Creating ./skins/custom/summarize-time.css...
Creating ./skins/custom/voting.css...
Creating ./skins/custom/yui/calendar.css...
Removing duplicates.rdf...
Removing duplicates directory...
Creating ./Bugzilla/.htaccess...
Creating ./lib/.htaccess...

Comment entered 2010-04-13 19:37:09 by priced

BZDATETIME::2010-04-13 19:37:09
BZCOMMENTOR::Stas Firstov
BZCOMMENT::2

Bugzilla has been upgraded to version 3.6
During the upgrade a few extra perl modules needed to be installed and mysql config file (/etc/my.cnf) changed to increase the max_allowed_packet parameter to 12M (originally it was set to 4M)
Below is the log.

Please test and advise. The backup files are on the server and they could be restored if rollback is required.
Reassigning to Alan for the QC...

Thanks,
Stas

[root@otsavmlinux1 bugzilla]# ./checksetup.pl

  • This is Bugzilla 3.6 on perl 5.8.8

  • Running on Linux 2.6.18-164.9.1.el5 #1 SMP Wed Dec 9 03:29:54 EST 2009

Checking perl modules...
Checking for CGI.pm (v3.21) ok: found v3.49
Checking for Digest-SHA (any) ok: found v5.48
Checking for TimeDate (v2.21) ok: found v2.22
Checking for DateTime (v0.28) ok: found v0.55
Checking for DateTime-TimeZone (v0.71) ok: found v1.17
Checking for DBI (v1.41) ok: found v1.52
Checking for Template-Toolkit (v2.22) ok: found v2.22
Checking for Email-Send (v2.00) ok: found v2.185
Checking for Email-MIME (v1.861) ok: found v1.903
Checking for Email-MIME-Encodings (v1.313) ok: found v1.313
Checking for Email-MIME-Modifier (v1.442) ok: found v1.903
Checking for URI (any) ok: found v1.35

Checking available perl DBD modules...
Checking for DBD-Pg (v1.45) not found
Checking for DBD-mysql (v4.00) ok: found v4.013
Checking for DBD-Oracle (v1.19) not found

The following Perl modules are optional:
Checking for GD (v1.20) ok: found v2.35
Checking for Chart (v2.1) ok: found v2.3
Checking for Template-GD (any) ok: found v1.56
Checking for GDTextUtil (any) ok: found v0.86
Checking for GDGraph (any) ok: found v1.44
Checking for XML-Twig (any) ok: found v3.29
Checking for MIME-tools (v5.406) ok: found v5.420
Checking for libwww-perl (any) ok: found v2.033
Checking for PatchReader (v0.9.4) ok: found v0.9.5
Checking for perl-ldap (any) ok: found v0.34
Checking for Authen-SASL (any) not found
Checking for RadiusPerl (any) not found
Checking for SOAP-Lite (v0.710.06) found v0.69
Checking for JSON-RPC (any) not found
Checking for Test-Taint (any) not found
Checking for HTML-Parser (v3.40) ok: found v3.55
Checking for HTML-Scrubber (any) ok: found v0.08
Checking for Email-MIME-Attachment-Stripper (any) ok: found v1.313
Checking for Email-Reply (any) ok: found v1.202
Checking for TheSchwartz (any) not found
Checking for Daemon-Generic (any) not found
Checking for mod_perl (v1.999022) ok: found v2.000004
***********************************************************************

  • OPTIONAL MODULES *
    ***********************************************************************

  • Certain Perl modules are not required by Bugzilla, but by *

  • installing the latest version you gain access to additional *

  • features. *

  • *

  • The optional modules you do not have installed are listed below, *

  • with the name of the feature they enable. Below that table are the *

  • commands to install each module. *
    ***********************************************************************

  • MODULE NAME * ENABLES FEATURE(S) *
    ***********************************************************************

  • Authen-SASL * SMTP Authentication *

  • RadiusPerl * RADIUS Authentication *

  • SOAP-Lite * XML-RPC Interface *

  • JSON-RPC * JSON-RPC Interface *

  • Test-Taint * JSON-RPC Interface, XML-RPC Interface *

  • TheSchwartz * Mail Queueing *

  • Daemon-Generic * Mail Queueing *
    ***********************************************************************
    COMMANDS TO INSTALL OPTIONAL MODULES:

Authen-SASL: /usr/bin/perl install-module.pl Authen::SASL
RadiusPerl: /usr/bin/perl install-module.pl Authen::Radius
SOAP-Lite: /usr/bin/perl install-module.pl SOAP::Lite
JSON-RPC: /usr/bin/perl install-module.pl JSON::RPC
Test-Taint: /usr/bin/perl install-module.pl Test::Taint
TheSchwartz: /usr/bin/perl install-module.pl TheSchwartz
Daemon-Generic: /usr/bin/perl install-module.pl Daemon::Generic

To attempt an automatic install of every required and optional module
with one command, do:

/usr/bin/perl install-module.pl --all

Reading ./localconfig...

OPTIONAL NOTE: If you want to be able to use the 'difference between two
patches' feature of Bugzilla (which requires the PatchReader Perl module
as well), you should install patchutils from:

http://cyberelk.net/tim/patchutils/

Checking for DBD-mysql (v4.00) ok: found v4.013
Checking for MySQL (v4.1.2) ok: found v5.0.77

WARNING: You need to set the max_allowed_packet parameter in your MySQL
configuration to at least 12288000. Currently it is set to 4194304.
You can set this parameter in the [mysqld] section of your MySQL
configuration file.

Adding new table bug_see_also ...
Adding new table login_failure ...
Adding new table ts_error ...
Adding new table ts_exitstatus ...
Adding new table ts_funcmap ...
Adding new table ts_job ...
Adding new table ts_note ...
attachments.bug_id has incorrect DB default: 0
attachments.creation_ts has incorrect DB default: 0000-00-00 00:00:00
attachments.filename has incorrect DB default: ''
attachments.submitter_id has incorrect DB default: 0
bug_group_map.bug_id has incorrect DB default: 0
bug_group_map.group_id has incorrect DB default: 0
bug_severity.value has incorrect DB default: ''
bug_status.value has incorrect DB default: ''
bugs.priority has incorrect DB default: ''
bugs.reporter has incorrect DB default: 0
bugs.product_id has incorrect DB default: 0
bugs.rep_platform has incorrect DB default: ''
bugs.assigned_to has incorrect DB default: 0
bugs.everconfirmed has incorrect DB default: 0
bugs.bug_severity has incorrect DB default: ''
bugs.bug_status has incorrect DB default: ''
bugs.delta_ts has incorrect DB default: 0000-00-00 00:00:00
bugs.version has incorrect DB default: ''
bugs.component_id has incorrect DB default: 0
bugs.op_sys has incorrect DB default: ''
bugs_activity.bug_id has incorrect DB default: 0
bugs_activity.bug_when has incorrect DB default: 0000-00-00 00:00:00
bugs_activity.who has incorrect DB default: 0
bugs_activity.fieldid has incorrect DB default: 0
bz_schema.version has incorrect DB default: 0.00
category_group_map.category_id has incorrect DB default: 0
category_group_map.group_id has incorrect DB default: 0
cc.bug_id has incorrect DB default: 0
cc.who has incorrect DB default: 0
classifications.name has incorrect DB default: ''
components.initialowner has incorrect DB default: 0
components.product_id has incorrect DB default: 0
components.name has incorrect DB default: ''
dependencies.blocked has incorrect DB default: 0
dependencies.dependson has incorrect DB default: 0
duplicates.dupe_of has incorrect DB default: 0
duplicates.dupe has incorrect DB default: 0
email_setting.user_id has incorrect DB default: 0
email_setting.relationship has incorrect DB default: 0
email_setting.event has incorrect DB default: 0
fielddefs.sortkey has incorrect DB default: 0
fielddefs.name has incorrect DB default: ''
flagexclusions.type_id has incorrect DB default: 0
flaginclusions.type_id has incorrect DB default: 0
flags.status has incorrect DB default: ''
flags.bug_id has incorrect DB default: 0
flags.type_id has incorrect DB default: 0
flags.creation_date has incorrect DB default: 0000-00-00 00:00:00
flagtypes.name has incorrect DB default: ''
group_control_map.entry has incorrect DB default: 0
group_control_map.membercontrol has incorrect DB default: 0
group_control_map.canedit has incorrect DB default: 0
group_control_map.product_id has incorrect DB default: 0
group_control_map.othercontrol has incorrect DB default: 0
group_control_map.group_id has incorrect DB default: 0
group_group_map.grantor_id has incorrect DB default: 0
group_group_map.member_id has incorrect DB default: 0
groups.isbuggroup has incorrect DB default: 0
groups.name has incorrect DB default: ''
keyworddefs.name has incorrect DB default: ''
keywords.keywordid has incorrect DB default: 0
keywords.bug_id has incorrect DB default: 0
logincookies.lastused has incorrect DB default: 0000-00-00 00:00:00
logincookies.userid has incorrect DB default: 0
logincookies.ipaddr has incorrect DB default: ''
longdescs.bug_id has incorrect DB default: 0
longdescs.bug_when has incorrect DB default: 0000-00-00 00:00:00
longdescs.who has incorrect DB default: 0
milestones.product_id has incorrect DB default: 0
milestones.value has incorrect DB default: ''
namedqueries.userid has incorrect DB default: 0
namedqueries.name has incorrect DB default: ''
op_sys.value has incorrect DB default: ''
priority.value has incorrect DB default: ''
products.name has incorrect DB default: ''
profile_setting.user_id has incorrect DB default: 0
profile_setting.setting_name has incorrect DB default: ''
profile_setting.setting_value has incorrect DB default: ''
profiles.login_name has incorrect DB default: ''
profiles_activity.userid has incorrect DB default: 0
profiles_activity.profiles_when has incorrect DB default: 0000-00-00 00:00:00
profiles_activity.who has incorrect DB default: 0
profiles_activity.fieldid has incorrect DB default: 0
rep_platform.value has incorrect DB default: ''
resolution.value has incorrect DB default: ''
series.frequency has incorrect DB default: 0
series.category has incorrect DB default: 0
series.name has incorrect DB default: ''
series.subcategory has incorrect DB default: 0
series_categories.name has incorrect DB default: ''
series_data.series_value has incorrect DB default: 0
series_data.series_id has incorrect DB default: 0
series_data.series_date has incorrect DB default: 0000-00-00 00:00:00
setting.name has incorrect DB default: ''
setting.default_value has incorrect DB default: ''
setting_value.name has incorrect DB default: ''
setting_value.value has incorrect DB default: ''
setting_value.sortindex has incorrect DB default: 0
tokens.token has incorrect DB default: ''
tokens.tokentype has incorrect DB default: ''
tokens.issuedate has incorrect DB default: 0000-00-00 00:00:00
user_group_map.group_id has incorrect DB default: 0
user_group_map.user_id has incorrect DB default: 0
versions.product_id has incorrect DB default: 0
versions.value has incorrect DB default: ''
votes.bug_id has incorrect DB default: 0
votes.who has incorrect DB default: 0
votes.vote_count has incorrect DB default: 0
watch.watcher has incorrect DB default: 0
watch.watched has incorrect DB default: 0
whine_events.owner_userid has incorrect DB default: 0
whine_queries.eventid has incorrect DB default: 0
whine_schedules.eventid has incorrect DB default: 0
whine_schedules.mailto has incorrect DB default: 0
Fixing defaults...
Updating old chart storage format...
Creating ./data/extensions directory...
Creating ./skins/contrib/Dusk/yui directory...
Creating ./skins/custom directory...
Creating ./skins/custom/yui directory...
Creating graphs directory...
Creating ./data/extensions/additional...
Creating ./data/mailer.testfile...
Creating ./skins/contrib/Dusk/IE-fixes.css...
Creating ./skins/contrib/Dusk/admin.css...
Creating ./skins/contrib/Dusk/create_attachment.css...
Creating ./skins/contrib/Dusk/dependency-tree.css...
Creating ./skins/contrib/Dusk/duplicates.css...
Creating ./skins/contrib/Dusk/editusers.css...
Creating ./skins/contrib/Dusk/help.css...
Creating ./skins/contrib/Dusk/page.css...
Creating ./skins/contrib/Dusk/panel.css...
Creating ./skins/contrib/Dusk/params.css...
Creating ./skins/contrib/Dusk/reports.css...
Creating ./skins/contrib/Dusk/show_bug.css...
Creating ./skins/contrib/Dusk/show_multiple.css...
Creating ./skins/contrib/Dusk/summarize-time.css...
Creating ./skins/contrib/Dusk/voting.css...
Creating ./skins/contrib/Dusk/yui/calendar.css...
Creating ./skins/custom/IE-fixes.css...
Creating ./skins/custom/admin.css...
Creating ./skins/custom/buglist.css...
Creating ./skins/custom/create_attachment.css...
Creating ./skins/custom/dependency-tree.css...
Creating ./skins/custom/duplicates.css...
Creating ./skins/custom/editusers.css...
Creating ./skins/custom/global.css...
Creating ./skins/custom/help.css...
Creating ./skins/custom/index.css...
Creating ./skins/custom/page.css...
Creating ./skins/custom/panel.css...
Creating ./skins/custom/params.css...
Creating ./skins/custom/reports.css...
Creating ./skins/custom/show_bug.css...
Creating ./skins/custom/show_multiple.css...
Creating ./skins/custom/summarize-time.css...
Creating ./skins/custom/voting.css...
Creating ./skins/custom/yui/calendar.css...
Removing duplicates.rdf...
Removing duplicates directory...
Creating ./Bugzilla/.htaccess...
Creating ./lib/.htaccess...%

Comment entered 2010-04-13 19:38:09 by alan

BZDATETIME::2010-04-13 19:38:09
BZCOMMENTOR::Alan Meyer
BZCOMMENT::3

Testing is under way.

I went to this bug, entered a comment, and got a mid-air collision - which was what should have happened since Stas had just entered a comment that I hadn't seen yet.

I discarded my change and am now entering this one. - Status is still NEW.

Comment entered 2010-04-13 19:38:38 by alan

BZDATETIME::2010-04-13 19:38:38
BZCOMMENTOR::Alan Meyer
BZCOMMENT::4

Another comment, status is still new.

Comment entered 2010-04-13 19:39:05 by alan

BZDATETIME::2010-04-13 19:39:05
BZCOMMENTOR::Alan Meyer
BZCOMMENT::5

Changing status to assigned.

Comment entered 2010-04-13 19:39:07 by priced

BZDATETIME::2010-04-13 19:39:07
BZCOMMENTOR::Stas Firstov
BZCOMMENT::6

Comment entered 2010-04-13 19:39:07 by priced

Attachment bugzilla_upgrade_log.txt has been added with description: checksetup.pl Output

Comment entered 2010-04-13 19:39:44 by priced

BZDATETIME::2010-04-13 19:39:44
BZCOMMENTOR::Stas Firstov
BZCOMMENT::7

Bugzilla has been upgraded to version 3.6
During the upgrade a few extra perl modules needed to be installed and mysql config file (/etc/my.cnf) changed to increase the max_allowed_packet parameter to 12M (originally it was set to 4M)

Please test and advise. The backup files are on the server and they could be restored if rollback is required.
Reassigning to Alan for the QC...

Thanks,
Stas

Comment entered 2010-04-13 19:41:44 by priced

BZDATETIME::2010-04-13 19:41:44
BZCOMMENTOR::Stas Firstov
BZCOMMENT::8

When adding comments I was timing out (and multiple comments were entered as a result). Not sure if this is my connection or a collision.

Alan, could you please test adding long comments.

Thanks!

(In reply to comment #7)
> Bugzilla has been upgraded to version 3.6
> During the upgrade a few extra perl modules needed to be installed and mysql
> config file (/etc/my.cnf) changed to increase the max_allowed_packet parameter
> to 12M (originally it was set to 4M)
>
> Please test and advise. The backup files are on the server and they could be
> restored if rollback is required.
> Reassigning to Alan for the QC...
>
> Thanks,
> Stas

Comment entered 2010-04-13 19:42:34 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2010-04-13 19:42:34
BZCOMMENTOR::Volker Englisch
BZCOMMENT::9

Our skins are lost. Has the new Bugzilla been installed in a new directory?

Comment entered 2010-04-13 19:44:09 by alan

BZDATETIME::2010-04-13 19:44:09
BZCOMMENTOR::Alan Meyer
BZCOMMENT::10

Read attachment fine.

I'll enter a long comment next.

Comment entered 2010-04-13 19:45:21 by alan

BZDATETIME::2010-04-13 19:45:21
BZCOMMENTOR::Alan Meyer
BZCOMMENT::11

What follows is a 962 line 35 KB Python program. Should be longer than any actual comment anyone would enter.

#----------------------------------------------------------------------
#

  1. $Id: FileSweeper.py 9285 2009-10-20 12:44:47Z bkline $
    #

  2. Sweep up obsolete directories and files based on instructions in

  3. a configuration file - deleting, truncating, or archiving files

  4. and directories when required.
    #

  5. $Log: not supported by cvs2svn $

  6. Revision 1.6 2008/04/08 23:17:51 ameyer

  7. Small fixes: 1) Add .TEST to output filenames in test mode. 2) Do not

  8. prevent making large numbers of files with same name in test mode.

  9. 3) Validate OutputFileName element exists if needed. 4) Fix broken

  10. method of writing exception tracebacks to log file.
    #

  11. Revision 1.5 2006/01/24 23:57:34 ameyer

  12. Added some debugging checks to TruncateArchive to assure that file sizes

  13. after processing are as expected.
    #

  14. Revision 1.4 2005/09/20 18:16:02 ameyer

  15. Fixed bug causing files to be truncated even if none exceeded the

  16. size threshold.

  17. Made some other small cleanups the need for which was revealed by the

  18. bug fix.
    #

  19. Revision 1.3 2005/07/15 02:09:07 ameyer

  20. Eliminated requirement to specify and OutputFile name if we are not

  21. actually going to output any files.
    #

  22. Revision 1.2 2005/07/01 03:20:28 ameyer

  23. Bug fix.
    #

  24. Revision 1.1 2005/07/01 02:31:55 ameyer

  25. Cleanup program for old log and output files.
    #
    #
    #----------------------------------------------------------------------
    import sys, os, os.path, getopt, glob, shutil, time
    import traceback, re, xml.dom.minidom, tarfile, cdr

  1. Logfile
    LF = cdr.DEFAULT_LOGDIR + "/FileSweeper.log"

  1. Don't go wild creating output files
    MAX_OUTPUT_FILES_WITH_ONE_NAME = 5

  1. Size for read/write
    BLOCK_SIZE = 4096

  1. Date constants, YEARS_OLD is max time we'll look back, sanity check
    DAY_SECS = 86400
    YEAR_DAYS = 365.25
    YEARS_OLD = 5
    LONG_TIME = DAY_SECS * YEAR_DAYS * YEARS_OLD

#----------------------------------------------------------------------

  1. Class encapsulating actions on one file
    #----------------------------------------------------------------------
    class qualFile:

def init(self, fileName):

  1. Default values
    self.fileName = fileName

  1. Stat file
    fstat = os.stat(self.fileName)

  1. Save info
    self.fsize = fstat.st_size
    self.mtime = fstat.st_mtime

  1. Nothing done to this file yet
    self.archived = False
    self.truncated = False
    self.deleted = False

#----------------------------------------------------------------------

  1. Class encapsulating the elements in a specification
    #----------------------------------------------------------------------
    class SweepSpec:

def init(self, specNode):
"""
Constructor loads SweepSpec from a dom node.

Pass:
DOM node of a SweepSpec element in a configuration file.
"""

  1. Initialize specification invalid values
    self.specName = "Unknown" # Name for report
    self.action = None # What to do with files
    self.root = None # Where to look for files
    self.inFiles = [] # File paths to look for
    self.outFile = None # Output file for archive
    self.oldSpec = None # If at least one file older than this
    self.youngSpec = None # Files must be older than this
    self.maxSizeSpec = None # If file bigger than this
    self.truncSizeSpec = None # Truncate file to this size

  1. Set this flag to true when the archive is successfully saved
    self.okayToDelete = False

  1. These fields track what actually matches the specification

  2. Initialized to invalid values, filled in by self.statFiles()
    self.oldestDate = 0 # Date of oldest file found in root/inFiles
    self.youngestDate = 0 # Date of youngest found
    self.biggestSize = 0 # Size of biggest file found
    self.smallestSize = 0 # Size of smallest
    self.totalList = [] # All names of files found in root/inFiles
    self.qualifiedList = [] # qualFile objects qualified for action
    self.totalBytes = 0 # Total bytes in all files
    self.qualifiedBytes= 0 # Total bytes in qualified files
    self.archivedFiles = 0 # Number successfully archived
    self.archivedBytes = 0 # " " "
    self.truncFiles = 0 # Number successfully truncated
    self.truncBytes = 0 # " " "
    self.msgs = [] # Messages accrued during processing
    self.statted = False # Info has been collected

  1. All times relative to right now, normalized to previous midnight
    now = normTime(time.time())

  1. Load all significant fields
    for node in specNode.childNodes:
    if node.nodeType == xml.dom.minidom.Node.ELEMENT_NODE:
    elem = node.nodeName
    if elem == 'Name':
    self.specName = cdr.getTextContent(node)
    elif elem == 'Action':
    self.action = cdr.getTextContent(node)
    elif elem == 'InputRoot':
    self.root = cdr.getTextContent(node)
    elif elem == 'InputFiles':
    for child in node.childNodes:
    if child.nodeType == xml.dom.minidom.Node.ELEMENT_NODE:
    if child.nodeName == 'File':
    self.inFiles.append(\
    cdr.getTextContent(child))
    elif child.nodeName == 'Comment':
    pass
    else:
    fatalError(\
    'Unrecognized element "%s" in SweepSpec "%s"' \
    % (child.nodeName, self.specName))
    elif elem == 'OutputFile':
    self.outFile = cdr.getTextContent(node)
    elif elem == 'Oldest':

  2. Convert to system time = seconds since epoch
    days = int(cdr.getTextContent(node))
    self.oldSpec = now - (days * DAY_SECS)
    elif elem == 'Youngest':
    days = int(cdr.getTextContent(node))
    self.youngSpec = now - (days * DAY_SECS)
    elif elem == 'Biggest':
    self.maxSizeSpec = int(cdr.getTextContent(node))
    elif elem == 'Smallest':
    self.truncSizeSpec = int(\
    cdr.getTextContent(node))
    elif elem == 'Comment':
    pass
    else:
    fatalError('Unrecognized element "%s" in SweepSpec "%s"' \
    % (elem, self.specName))

  1. Validate
    if self.specName == "Unknown":
    fatalError("No Name subelement in one of the SweepSpec elements");
    if not self.action:
    fatalError('No Action in SweepSpec "%s"' % self.specName);
    if self.action not in ('Archive', 'Delete', 'TruncateArchive',
    'TruncateDelete'):
    fatalError('Invalid Action "%s" in SweepSpec "%s"' % \
    (self.action, self.specName))
    if self.inFiles == []:
    fatalError('No File (or InputFiles?) in SweepSpec "%s"' % \
    self.specName);

  1. Validate combinations of specs
    if not self.outFile and self.action in ('Archive','TruncateArchive'):
    fatalError(
    "No output file specified for SweepSpec %s with Action=%s" % \
    (self.specName, self.action))
    if not (self.oldSpec and self.youngSpec):
    if self.action == 'Archive':
    fatalError(\
    'Must specify Oldest/Youngest for Archive SweepSpec "%s"' \
    % self.specName)
    if not (self.maxSizeSpec and self.truncSizeSpec):
    if self.action.startswith('Truncate'):
    fatalError(\
    'Must specify Biggest/Smallest for Truncate SweepSpec "%s"'\
    % self.specName)

  1. Times must be reasonable e.g., now until 5 years before now
    if self.oldSpec:
    if self.oldSpec >= now or self.youngSpec >= now:
    fatalError('A date >= current date in SweepSpec "%s"' % \
    self.specName)
    longAgo = now - LONG_TIME
    if self.oldSpec < longAgo or self.youngSpec < longAgo:
    fatalError('A date is older than %d years in SweepSpec "%s"' % \
    (YEARS_OLD, self.specName))

if self.oldSpec and self.maxSizeSpec:
fatalError("Can't specify both big/small and old/young")

#----------------------------------------------------------------------

  1. Find files matching a spec
    #----------------------------------------------------------------------
    def statFiles(self):
    """
    Find all files matching a SweepSpec specification.
    Only finds those that qualify "Youngest" or "Smallest" limit.
    Also finds maximum oldest, youngest, biggest, smallest values
    so that caller can decide whether there is anything at all to
    do for the SweepSpec.

Pass:
Void

Return:
True = There is at least one file matching the SweepSpec.
False = There aren't any.
"""

  1. Get a list of all files matching all input specs
    for fileSpec in self.inFiles:
    self.totalList.extend(glob.glob((fileSpec)))

  1. Were there any files found at all?
    if len(self.totalList) == 0:
    return False

  1. Stat each one
    for fileName in self.totalList:

  1. Create a stat'ed object for it

  2. Force string format in case name is all digits
    fileObj = qualFile(normPath(str(fileName)))

  1. Shorthand names for last modified time and file size
    mtime = fileObj.mtime
    fsize = fileObj.fsize

  1. Update number of bytes in directory we've examined
    self.totalBytes += fsize

  1. Does this file qualify?
    if (self.youngSpec and mtime < self.youngSpec) or \
    (self.truncSizeSpec and fsize > self.truncSizeSpec):

  1. Yes, remember it
    self.qualifiedList.append(fileObj)

  1. Update summary dates for SweepSpec

  2. XXX Should I erase these if required criterion not met?
    if self.oldestDate == 0 or mtime < self.oldestDate:
    self.oldestDate = mtime
    if mtime > self.youngestDate:
    self.youngestDate = mtime

  1. Summary of sizes
    if self.smallestSize == 0 or fsize < self.smallestSize:
    self.smallestSize = fsize
    if self.biggestSize == 0 or fsize > self.biggestSize:
    self.biggestSize = fsize

  1. And cumulate number of bytes we'll remove
    self.qualifiedBytes += fsize
    if self.truncSizeSpec:

  2. Already know file is bigger than max
    self.qualifiedBytes -= self.truncSizeSpec

  1. Signify completion of statFiles
    self.statted = True

  1. Was at least one required criterion met?
    if self.oldSpec and self.oldestDate > self.oldSpec:
    return False
    elif self.maxSizeSpec and self.maxSizeSpec > self.biggestSize:
    return False

  1. We should take action on this spec
    return True

#----------------------------------------------------------------------

  1. Stringify entire spec
    #----------------------------------------------------------------------
    def str(self):
    """
    Display spec for debugging purposes.
    """

  2. Convert date times for display
    old = young = oldest = youngest = None
    if self.oldSpec:
    old = time.strftime("%Y-%m-%d",
    time.localtime(self.oldSpec))
    young = time.strftime("%Y-%m-%d",
    time.localtime(self.youngSpec))
    if self.oldestDate > 0:
    oldest = time.strftime("%Y-%m-%d",
    time.localtime(float(self.oldestDate)))
    youngest = time.strftime("%Y-%m-%d",
    time.localtime(self.youngestDate))

  1. Basics from config file
    specStr = """
    SweepSpec: "%s"
    Action: %s
    InputRoot: %s
    Files: %s
    OutputFile: %s
    Oldest: %s
    Youngest: %s
    Biggest: %s
    Smallest: %s
    """ % (self.specName, self.action, self.root, self.inFiles, self.outFile,
    old, young,
    self.maxSizeSpec, self.truncSizeSpec)

  1. If statFiles() called, report statistics
    if self.statted:
    specStr += """
    Statistics:
    oldest: %s
    youngest: %s
    biggest: %d
    smallest: %d
    total files: %d
    qualified files: %d
    total bytes: %d
    qualified bytes: %d
    message count: %d
    """ % (oldest, youngest, self.biggestSize, self.smallestSize,
    len(self.totalList), len(self.qualifiedList),
    self.totalBytes, self.qualifiedBytes, len(self.msgs))

  1. Messages
    if self.msgs:
    specStr += "Messages:\n"
    i = 1
    for msg in self.msgs:
    specStr += "%2d: %s\n" % (i, msg)
    i += 1

return specStr

#----------------------------------------------------------------------

  1. Archive files needing to be archived
    #----------------------------------------------------------------------
    def archive(self, testMode):
    """
    Copy all qualified files into an archive file.

If archive is successful, deletes the files.

Pass:
testMode
True = Don't actually delete file - for debugging
False = Delete after archiving
"""

  1. Don't do anything if there's nothing to do
    if len(self.qualifiedList) == 0:
    return

  1. Create the output compressed tar archive
    try:
    tar = tarfile.open(self.outFile, "w:bz2")
    except Exception, info:
    fatalError('Could not open tarfile "%s" for writing' \
    % self.outFile)

  1. Process each qualifying file
    for fobj in self.qualifiedList:

  2. Archive it
    try:
    tar.add(fobj.fileName);
    except Exception, info:
    self.addMsg("Tar error: %s" % info)
    self.addMsg("Abandoning this SweepSpec")
    return

  1. Stats
    self.archivedFiles += 1

try:
tar.close()
except Exception, info:
fatalError('Could not close tarfile "%s"' % self.outFile)

  1. If here, everything okay, deletion can proceed
    self.delete(testMode)

#----------------------------------------------------------------------

  1. Truncate files needing to be truncated
    #----------------------------------------------------------------------
    def truncate(self, testMode):
    """
    Left truncate any files that exceed the specified maximum
    size, saving the truncated size in place of the original. By
    "left truncate" we mean delete bytes from the beginning of
    the file, saving bytes that were at the end.

For TruncateArchive, the removed data is archived to an output
file, as with the archive() function. For TruncateDelete, the
removed data is lost.

We'll move or copy the file to the output directory
using a unique name generated by our name generator.

Truncation is done by copying the "right" end of the
file back to its original location, overwriting the
longer original.

If TruncateDelete'ing, we can delete the copy in the
output directory.

If TruncateArchive'ing, we retain it there, where it
will eventually get compressed.

Pass:
Output directory
Build output archive in this directory.

testMode
True = Don't actually replace file - for debugging
False = Replace the original file with truncated data
"""

  1. Don't do anything if there's nothing to do
    if len(self.qualifiedList) == 0:
    return

  1. Create the output compressed tar archive
    if self.action == "TruncateArchive":
    try:
    tar = tarfile.open(self.outFile, "w:bz2")
    except Exception, info:
    fatalError('Could not open tarfile "%s" for writing' \
    % self.outFile)

  1. Process each qualifying file
    for fileObj in self.qualifiedList:

  1. Name of the input file
    inFile = fileObj.fileName

  1. Create a temporary file for output of the part to be saved
    tmpFile = inFile + ".Truncation"

  1. If the temp file exists from a previous run, delete it
    if os.path.exists(tmpFile):
    self.addMsg('Warning: overwriting old temporary output "%s"' \
    % tmpFile)
    os.remove(tmpFile)

  1. Truncation point is truncSizeSpec before end of file
    truncPoint = fileObj.fsize - self.truncSizeSpec

  1. Copy the part we want to save into the new file

  2. This creates the truncated file
    try:
    srcp = open(inFile, "rb")
    destp = open(tmpFile, "wb")
    srcp.seek(truncPoint)
    done = False
    while not done:
    bytes = srcp.read(BLOCK_SIZE)
    if bytes:
    destp.write(bytes)
    else:
    done = True
    destp.close()
    srcp.close()
    except Exception, info:
    self.addMsg(\
    """WARNING: Unable to create truncation of "%s::%s":
    %s
    Truncation was aborted""" % (self.specName, inFile, info))
    continue

  1. Sanity debug checks
    if not os.path.exists(tmpFile):
    fatalError('Temporary file "%s" not found - internal error' \
    % tmpFile)
    tmpstat = os.stat(tmpFile)
    if tmpstat.st_size != self.truncSizeSpec:
    self.addMsg(
    'WARNING: Temp file "%s" size=%d, but truncsize=%d\n' \
    (tmpFile, tmpstat.st_size, self.truncSizeSpec) +
    ' Input file may have changed during processing')

  1. If archiving, truncate and save the uncopied part

  2. of the temporary file
    if self.action == "TruncateArchive":

  1. But don't truncate if in test mode
    if not testMode:
    try:
    srcp = open(inFile, "ab+")
    srcp.truncate(truncPoint)
    srcp.close()
    except Exception, info:
    self.addMsg('Unable to truncate "%s::%s": %s' % \
    (self.specName, inFile, info))
    continue

  1. Sanity debug checks
    fstat = os.stat(inFile)
    if fstat.st_size != truncPoint:
    fatalError(\
    'Truncation file "%s" size=%d, truncsize=%d' %\
    (inFile, fstat.st_size, self.truncSizeSpec))
    if (tmpstat.st_size + fstat.st_size) != fileObj.fsize:
    fatalError(\
    'File "%s": Truncated and remaining sizes %d + %d != original size %d' %\
    (inFile, tmpstat.st_size, fstat.st_size, fileObj.fsize))

  1. Archive the truncation
    try:
    tar.add(inFile);
    except Exception, info:
    self.addMsg("Tar error: %s" % info)
    self.addMsg("Abandoning this SweepSpec")

  1. If doing it for real, replace the original with the new

  2. truncated version
    if not testMode:
    try:
    shutil.move(tmpFile, inFile)
    except Exception, info:
    fatalError('Unable to replace original file "%s":\n %s'\
    % (inFile, info))

  1. If we got this far, the truncation has occurred
    fileObj.truncated = True

  1. Close archive
    if self.action == "TruncateArchive":
    try:
    tar.close()
    except Exception, info:
    fatalError('Could not close tarfile "%s"' % self.outFile)

#----------------------------------------------------------------------

  1. Delete files
    #----------------------------------------------------------------------
    def delete(self, testMode):
    """
    Delete all files named in a specification.

Pass:
testMode
True = Just testing, don't delete anything.
False = Delete them.
"""
for fileObj in spec.qualifiedList:
nameIsDir = False
try:
if not testMode:
if os.path.isdir(fileObj.fileName):

  1. Recursivley remove directory if it's empty
    nameIsDir = True
    os.system("rm -r %s" % fileObj.fileName)
    else:

  2. Remove ordinary file
    os.remove(fileObj.fileName)
    fileObj.deleted = True
    else:
    self.addMsg('Test mode, not deleting "%s"' % \
    fileObj.fileName)
    except Exception, info:
    if nameIsDir:
    self.addMsg("Error removing directory %s: %s" % \
    (fileObj.fileName, info))
    else:
    self.addMsg("Unable to remove file %s: %s" % \
    (fileObj.fileName, info))

#----------------------------------------------------------------------

  1. Create a full output file name
    #----------------------------------------------------------------------
    def makeOutFileName(self, outPath, testMode):
    """
    Create an absolute path name for an output file in zip or
    tar format.

Path is stored in self.outFile

Filename includes dates as follows:
For Archived files, use the dates of the oldest and youngest
files in the archive, i.e.:
filename.YYYYMMDD-YYYYMMDD.tar

For Archived truncations of files, use today's date, i.e.:
filename.YYYYMMDD.{tar or zip}

Pass:
Output directory to prepend to filename stored in the Spec
configuration.
"""

  1. Construct output base name
    if not (self.outFile.startswith("/") or self.outFile[1:2] == ":/"):
    outFile = os.path.join(outPath, self.outFile)
    else:
    outFile = self.outFile

  1. Strip off suffixes that get added later
    if outFile.endswith(".bz2") or outFile.endswith(".BZ2"):
    outFile = outFile[:-4]
    if outFile.endswith(".tar") or outFile.endswith(".TAR"):
    outFile = outFile[:-4]

  1. Add appropriate date suffixes
    if self.action == 'Archive':

  2. Add dates for oldest file destined for the archive
    outFile += time.strftime(".%Y%m%d",
    time.localtime(self.oldestDate))
    outFile += time.strftime("-%Y%m%d",
    time.localtime(self.youngestDate))
    else:
    outFile += time.strftime(".%Y%m%d", time.localtime())

  1. If running in test mode, indicate that in the output filename

  2. Allows us to easily find and delete such files
    if testMode:
    outFile += ".TEST"

  1. Add conventional .tar.bz2 suffix
    outFile += ".tar.bz2"

  1. Normalize slashes. Cygwin tar likes forward slashes
    outBase = normPath(outFile)

  1. Make sure we don't overwrite existing archive
    outFile = makeFileNameUnique(outBase, MAX_OUTPUT_FILES_WITH_ONE_NAME)

  1. Sanity check
    if not outFile:
    fatalError(
    "Too many output files with base name '%s', in SweepSpec '%s'" %\
    (outBase, self.specName))

self.outFile = outFile

#----------------------------------------------------------------------

  1. Report results via HTML
    #----------------------------------------------------------------------
    def reportHTML(self):
    """
    Construct HTML to summarize what happened with this SweepSpec

Return:
HTML string
"""
html = """
<h3>%s</h3>
<table width='80%%' align='center' border='1'>
<tr><td>Action</td><td>%s</td></tr>
<tr><td>Num files examined</td><td>%d</td></tr>
<tr><td>Num files processed</td><td>%d</td></tr>
<tr><td>Num bytes processed</td><td>%d</td></tr>
""" % (self.specName, self.action, len(self.totalList),
len(self.qualifiedList), self.qualifiedBytes)

  1. Any errors or warnings?
    if len(self.msgs) > 0:
    for msg in (self.msgs):
    html += "<tr><td colspan='2'>%s</td></tr>\n" % msg

html += "</table>\n"

return html

#----------------------------------------------------------------------

  1. Process a message
    #----------------------------------------------------------------------
    def addMsg(self, msg):
    """
    This version appends messages to a list for this spec.
    Might do something else sometime.

Pass:
message
"""
self.msgs.append(msg)

#----------------------------------------------------------------------

  1. Load configuration file
    #----------------------------------------------------------------------
    def loadConfigFile(fileName):
    """
    Load the configuration file.

Pass:
Path to config file.

Return:
Sequence of SweepSpec objects.
Each object represents one file sweeper specification.

Fatal error if:
Unable to find or load file.
Unable to parse file.
File contents invalid.
"""

  1. Parse file from disk
    try:
    dom = xml.dom.minidom.parse(fileName)
    except Exception, info:
    fatalError("Error loading config file %s: %s" % (fileName, info))

  1. List of loaded specifications
    spec = []

  1. Load specifications
    docElem = dom.documentElement
    if dom.documentElement.nodeName != 'SweepSpecifications':
    fatalError("SweepSpecifications not found at root of config file %s" \
    % fileName)

for node in docElem.childNodes:
if node.nodeType == xml.dom.minidom.Node.ELEMENT_NODE:
if node.nodeName == 'SweepSpec':
spec.append(SweepSpec(node))

return spec

#----------------------------------------------------------------------

  1. Usage message
    #----------------------------------------------------------------------
    def usage(msg=None):
    if msg:
    sys.stderr.write("%s\n" % msg)

sys.stderr.write("""
usage: FileSweeper.py {-t} configFileName {outputDir}
-t = Test mode - create output files but delete nothing.
configFileName = Full or relative path to configuration file.
outputDir = Optional path to prepend to archive file output directory.
""")
sys.exit(1)

#----------------------------------------------------------------------

  1. Normalize a path
    #----------------------------------------------------------------------
    def normPath(path):
    return re.sub(r"
    ", r"/", path)

#----------------------------------------------------------------------

  1. Normalize a time value in seconds to the nearest previous midnight

  2. Converts UTC to local time and makes the change
    #----------------------------------------------------------------------
    def normTime(timeVal):
    return (timeVal - (timeVal % DAY_SECS) + time.altzone - DAY_SECS)

#----------------------------------------------------------------------

  1. Make a filename unique by adding a suffix if needed
    #----------------------------------------------------------------------
    def makeFileNameUnique(inFile, maxSuffix=1):
    """
    Check if an input absolute or relative file name is unique.
    If so:
    Return it.
    Else:
    Try to make it unique by appending a suffix like .01, .02, etc.
    to it. But don't exceed some reasonable number of tries.

Pass:
Input file name.
Max allowable suffix number.

Return:
Filename. Maybe be modified from original.
None if no unique name could be generated within the passed
constraint.
"""

  1. Sanity check
    if maxSuffix < 1:
    fatalError('makeFileNameUnique maxSuffix value error\n' +
    ' inFile="%s", maxSuffix=%d' % (inFile, maxSuffix))

  1. Generate names until we create a unique one
    fileNum = 1
    outFile = inFile
    while os.path.exists(outFile):
    outFile = inFile +".%02d" % fileNum
    fileNum += 1

  1. Did we get to a surprising number of files with the same name
    if fileNum > maxSuffix:

  2. If they're test files, it's okay, otherwise let's complain
    if outFile.find('.TEST') < 0:
    return None

return outFile

#----------------------------------------------------------------------

  1. Fatal error
    #----------------------------------------------------------------------
    def fatalError(msg):
    """
    Log and display error message.
    Then exit.

Pass:
Error message.
"""
msg = "FATAL error: %s\n" %msg
cdr.logwrite(msg, LF)
sys.stderr.write(msg)
sys.exit(1)

def log(msg):
cdr.logwrite(msg, LF)
sys.stderr.write(msg)

#----------------------------------------------------------------------

  1. MAIN
    #----------------------------------------------------------------------

if name == 'main':

  1. Don't allow two filesweepers to run at the same time
    lockFileName = "/cdr/log/FileSweeper.lockfile"
    if not cdr.createLockFile(lockFileName):
    fatalError("""
    It appears that another copy of FileSweeper is currently running.

Only one copy may run at a time.

If you are certain that no other copy is running, then manually
remove the file "%s" to enable FileSweeper to run.
""" % lockFileName)

  1. Command line args
    (opts, args) = getopt.getopt(sys.argv[1:], "t", ["test",])
    if len(args) < 1 or len(args) > 2:
    usage()
    configFile = args[0]
    if len(args) > 1:
    outputDir = args[1]
    else:
    outputDir = ""

  1. Run separator in log file
    cdr.logwrite("""

                      • Beginning File Sweep -------------
                        Args:
                        %s
                        """ % sys.argv, LF)

  1. Test mode?
    testMode = False
    if len(opts):
    if opts[0][0] in ("t", "-test"):
    testMode = True

  1. XXX For testing only

  2. testMode = True

  1. Load the configuration file, fatal if fails
    specList = loadConfigFile(configFile)

  1. Current working directory
    cwd = os.getcwd()

  1. Create absolute output directory path
    if outputDir:

  1. Normalize path name
    outputDir = normPath(outputDir)

  1. If output directory is relative, prepend cwd
    if not (outputDir[0:1] == "/" or outputDir[1:2] == ":/"):
    outputDir = normPath(os.path.join(cwd, outputDir))

else:

  1. Output directory not specifed. Use the current working directory
    outputDir = normPath(cwd)

  1. Output directory must exist
    if not os.path.exists(outputDir):

  2. Try to make it
    try:
    os.makedirs(outputDir)
    except Exception, info:
    fatalError(
    """Directory "%s" does not exist, can't create it: %s""" \
    % (outputDir, info))
    if not os.path.isdir(outputDir):
    fatalError('Command line output name "%s" is not a directory' \
    % outputDir)

  1. Process each archive specification
    try:
    for spec in specList:

  2. Change to input file root directory
    try:
    os.chdir(spec.root)
    except Exception, info:
    spec.addMsg("Unable to cd to root: %s" % info)
    spec.addMsg('"%s" not processed' % spec.specName)
    continue

  1. Find files to process
    if spec.statFiles():

  1. If we're archiving files, process output filename
    if spec.outFile and spec.outFile.find("Delete") == -1:

  1. Combine command line path with stored output path
    spec.makeOutFileName(outputDir, testMode)

  1. Create the directory path if necessary

  2. Already created outputDir, but we may need more
    (fileBase, fileName) = os.path.split(spec.outFile)
    if not os.path.exists(fileBase):
    try:
    os.makedirs(fileBase)
    except Exception, info:
    fatalError('Error creating directory "%s": %s' \
    % (fileBase, info))
    if not os.path.isdir(fileBase):
    fatalError('Config output name "%s" is not a directory')

  1. Perform action
    if spec.action == "Archive":
    spec.archive(testMode)
    elif spec.action.startswith("Truncate"):
    spec.truncate(testMode)
    else:
    spec.delete(testMode)

  1. Back to where we started
    try:
    os.chdir(cwd)
    except Exception, info:
    fatalError(\
    """SweepSpec "%s" could not return to directory "%s" - can't happen""" \
    % (spec.specName, cwd))

  1. Report results to log file
    cdr.logwrite(str(spec), LF)

except Exception, info:
sys.stderr.write('Exception halted processing on: %s' % str(info))
traceback.print_exc(file=sys.stderr)
logf = open(LF, "a", 0)
traceback.print_exc(file=logf)

Comment entered 2010-04-13 19:46:59 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2010-04-13 19:46:59
BZCOMMENTOR::Volker Englisch
BZCOMMENT::12

I just wanted to say something about the CSS not displaying correctly but apparently Stas knew I would say that and fixed it before I could complain. :-)

Comment entered 2010-04-13 19:48:06 by priced

BZDATETIME::2010-04-13 19:48:06
BZCOMMENTOR::Stas Firstov
BZCOMMENT::13

(In reply to comment #9)
> Our skins are lost. Has the new Bugzilla been installed in a new directory?

Alan, since it was a huge upgrade (skipping too many versions) I had to install
it into the new folder. I'm not sure if the skins are inter-compatible between
these versions...

Comment entered 2010-04-13 19:48:40 by alan

BZDATETIME::2010-04-13 19:48:40
BZCOMMENTOR::Alan Meyer
BZCOMMENT::14

I extracted my long comment to the clipboard and compared it to the original that I put it. The only changes were a couple of lines that got wrapped - which is expected.

So the long comment worked fine.

Comment entered 2010-04-13 19:49:54 by alan

BZDATETIME::2010-04-13 19:49:54
BZCOMMENTOR::Alan Meyer
BZCOMMENT::15

(In reply to comment #13)
> (In reply to comment #9)
> > Our skins are lost. Has the new Bugzilla been installed in a new directory?
>
> Alan, since it was a huge upgrade (skipping too many versions) I had to install
> it into the new folder. I'm not sure if the skins are inter-compatible between
> these versions...

No problem.

Volker is very good at skinning Bugs and Cats.

Comment entered 2010-04-13 19:54:43 by alan

BZDATETIME::2010-04-13 19:54:43
BZCOMMENTOR::Alan Meyer
BZCOMMENT::16

Added a quip.

Ran reports.

Sorted reports.

All OK.

Comment entered 2010-04-13 19:57:17 by alan

BZDATETIME::2010-04-13 19:57:17
BZCOMMENTOR::Alan Meyer
BZCOMMENT::17

Changed priority to P7, normal.

Ran report.

Then changed back to P5, enhancement.

Comment entered 2010-04-13 20:03:40 by alan

BZDATETIME::2010-04-13 20:03:40
BZCOMMENTOR::Alan Meyer
BZCOMMENT::18

Volker noticed a couple of places where things we did before (skins and changing the "restrict to this IP" weren't carried over.

However, those don't affect our ability to use it. It looks like the basic functions all work correctly.

I have some real postings I'll want to make tonight. Rather than delay them, I propose that we keep the new version and start using it for real. I'll make the postings before I go home tonight.

If anyone tries anything that looks like a show stopper, please post it tonight and I'll hold off live postings while we figure it out. Otherwise I'll probably make some live postings some time after 11 pm.

Thanks Stas. That was quick work.

Comment entered 2010-04-13 20:05:29 by alan

BZDATETIME::2010-04-13 20:05:29
BZCOMMENTOR::Alan Meyer
BZCOMMENT::19

I just noticed another customization that may be gone.

It looks like we need to re-implement the rule that copies all bugs to Reza and Lakshmi - unless this category of bug isn't supposed to go to them.

Comment entered 2010-04-13 20:06:28 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2010-04-13 20:06:28
BZCOMMENTOR::Volker Englisch
BZCOMMENT::20

(In reply to comment #9)
> Our skins are lost. Has the new Bugzilla been installed in a new directory?

Skins are back! Just had to copy the directories from the
skins/contrib
directory.

Comment entered 2010-04-13 20:08:21 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2010-04-13 20:08:21
BZCOMMENTOR::Volker Englisch
BZCOMMENT::21

(In reply to comment #18)
> Thanks Stas. That was quick work.

I second that. That's what I call a quick turn-around.

Thanks a lot!!!

Comment entered 2010-04-13 20:25:19 by alan

BZDATETIME::2010-04-13 20:25:19
BZCOMMENTOR::Alan Meyer
BZCOMMENT::22

(In reply to comment #19)
> I just noticed another customization that may be gone.
>
> It looks like we need to re-implement the rule that copies all bugs to Reza and
> Lakshmi - unless this category of bug isn't supposed to go to them.

I was mistaken about that.

Reza did get the email and Lakshmi doesn't have a rule requiring that
she get everything. I was misled by the fact that she is a viewer on
so many bugs.

Comment entered 2010-04-13 20:26:50 by alan

BZDATETIME::2010-04-13 20:26:50
BZCOMMENTOR::Alan Meyer
BZCOMMENT::23

(In reply to comment #20)

> Skins are back!

I knew Volker was good at skinning Bugs.

Comment entered 2010-04-13 21:07:19 by priced

BZDATETIME::2010-04-13 21:07:19
BZCOMMENTOR::Stas Firstov
BZCOMMENT::24

thanks everybody for your help with testing and fixes!

(In reply to comment #23)
> (In reply to comment #20)
>
> > Skins are back!
>
> I knew Volker was good at skinning Bugs.

Comment entered 2010-04-29 18:25:22 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2010-04-29 18:25:22
BZCOMMENTOR::Volker Englisch
BZCOMMENT::25

Bugzilla is running smoothly for the past two weeks.

About to close this issue.

Comment entered 2010-04-29 18:25:42 by Englisch, Volker (NIH/NCI) [C]

BZDATETIME::2010-04-29 18:25:42
BZCOMMENTOR::Volker Englisch
BZCOMMENT::26

Closing issue.

Attachments
File Name Posted User
bugzilla_upgrade_log.txt 2010-04-13 19:39:07

Elapsed: 0:00:00.000599