#528 get rid of instance specific scripts
Closed: Fixed None Opened 7 years ago by rmeggins.

Instead of having per-instance scripts in /usr/lib*/dirsrv/slapd-INSTANCE, like db2ldif, ldif2db, etc. we should instead have them in /usr/sbin or /usr/bin. They should accept the specific instance to work on as a command line argument/option, and if there is only one instance, they should just work on that one instance without requiring to have it specified. Bonus points for giving the user a menu of instances and having the user choose which instance to work on.


Created global scripts for /usr/sbin. The old instance specific scripts are still there, but they are now wrappers for the real scripts in /usr/sbin. These scripts have a new option -Z to specify the server instance. If there is only one instance, then there is no need to use this option.

./db2bak -Z localhost ... ...

The scripts also accept the server id as: slapd-localhost, dirsrv-localhost, or localhost.

This bit from remove-ds.pl
{{{
+use lib qw(@perlpath@);
use Getopt::Long;
use DSUtil;
+my $prefix = DSUtil::get_prefix();
+use lib "$prefix/lib/dirsrv/perl"; # this is only needed for "prefix" installations
use Resource;
use DSCreate qw(removeDSInstance);
}}}
If this is a "prefix" installation, how can it find DSUtil? DSUtil.pm, Resource.pm, and DSCreate.pm are all in $prefix/lib/dirsrv/perl

Otherwise, looks good.

Replying to [comment:5 rmeggins]:

This bit from remove-ds.pl
{{{
+use lib qw(@perlpath@);
use Getopt::Long;
use DSUtil;
+my $prefix = DSUtil::get_prefix();
+use lib "$prefix/lib/dirsrv/perl"; # this is only needed for "prefix" installations
use Resource;
use DSCreate qw(removeDSInstance);
}}}
If this is a "prefix" installation, how can it find DSUtil? DSUtil.pm, Resource.pm, and DSCreate.pm are all in $prefix/lib/dirsrv/perl

Because "use lib qw(@perlpath@)" gets converted to: "use lib qw(/data/LDAP/lib/dirsrv/perl)"

So they are redundant, and we don't need to do the prefix processing at all - I'll clean this up.

Otherwise, looks good.

git merge ticket528
Updating 1a1db0f..cca2990
Fast-forward
Makefile.am | 53 ++-
Makefile.in | 53 ++-
ldap/admin/src/scripts/DSCreate.pm.in | 12 +-
ldap/admin/src/scripts/DSMigration.pm.in | 4 +-
ldap/admin/src/scripts/DSUtil.pm.in | 59 ++
ldap/admin/src/scripts/bak2db.in | 118 +++
ldap/admin/src/scripts/bak2db.pl.in | 191 +++++
ldap/admin/src/scripts/cleanallruv.pl.in | 222 +++++
ldap/admin/src/scripts/db2bak.in | 118 +++
ldap/admin/src/scripts/db2bak.pl.in | 176 ++++
ldap/admin/src/scripts/db2index.in | 117 +++
ldap/admin/src/scripts/db2index.pl.in | 252 ++++++
ldap/admin/src/scripts/db2ldif.in | 190 +++++
ldap/admin/src/scripts/db2ldif.pl.in | 319 +++++++
ldap/admin/src/scripts/dbverify.in | 112 +++
ldap/admin/src/scripts/dn2rdn.in | 95 +++
ldap/admin/src/scripts/fixup-linkedattrs.pl.in | 207 +++++
ldap/admin/src/scripts/fixup-memberof.pl.in | 219 +++++
ldap/admin/src/scripts/ldif2db.in | 144 ++++
ldap/admin/src/scripts/ldif2db.pl.in | 277 ++++++
ldap/admin/src/scripts/ldif2ldap.in | 106 +++
ldap/admin/src/scripts/monitor.in | 98 +++
ldap/admin/src/scripts/ns-accountstatus.pl.in | 879 +++++++++++++++++++
ldap/admin/src/scripts/ns-activate.pl.in | 884 ++++++++++++++++++++
ldap/admin/src/scripts/ns-inactivate.pl.in | 884 ++++++++++++++++++++
ldap/admin/src/scripts/ns-newpwpolicy.pl.in | 330 ++++++++
ldap/admin/src/scripts/remove-ds.pl.in | 10 +-
ldap/admin/src/scripts/restart-slapd.in | 92 ++
ldap/admin/src/scripts/restoreconfig.in | 101 +++
ldap/admin/src/scripts/saveconfig.in | 101 +++
ldap/admin/src/scripts/schema-reload.pl.in | 206 +++++
ldap/admin/src/scripts/setup-ds.pl.in | 2 +-
ldap/admin/src/scripts/start-dirsrv.in | 1 +
ldap/admin/src/scripts/start-slapd.in | 82 ++
ldap/admin/src/scripts/stop-dirsrv.in | 5 +
ldap/admin/src/scripts/stop-slapd.in | 83 ++
ldap/admin/src/scripts/suffix2instance.in | 104 +++
ldap/admin/src/scripts/syntax-validate.pl.in | 216 +++++
ldap/admin/src/scripts/template-bak2db.in | 60 +--
ldap/admin/src/scripts/template-bak2db.pl.in | 113 +---
ldap/admin/src/scripts/template-cleanallruv.pl.in | 150 +----
ldap/admin/src/scripts/template-db2bak.in | 54 +--
ldap/admin/src/scripts/template-db2bak.pl.in | 103 +---
ldap/admin/src/scripts/template-db2index.in | 32 +-
ldap/admin/src/scripts/template-db2index.pl.in | 190 +----
ldap/admin/src/scripts/template-db2ldif.in | 85 +--
ldap/admin/src/scripts/template-db2ldif.pl.in | 247 +------
ldap/admin/src/scripts/template-dbverify.in | 48 +-
ldap/admin/src/scripts/template-dn2rdn.in | 22 +-
.../src/scripts/template-fixup-linkedattrs.pl.in | 135 +---
.../src/scripts/template-fixup-memberof.pl.in | 146 +---
ldap/admin/src/scripts/template-ldif2db.in | 55 +--
ldap/admin/src/scripts/template-ldif2db.pl.in | 205 +-----
ldap/admin/src/scripts/template-ldif2ldap.in | 26 +-
ldap/admin/src/scripts/template-monitor.in | 25 +-
.../src/scripts/template-ns-accountstatus.pl.in | 827 +------------------
ldap/admin/src/scripts/template-ns-activate.pl.in | 827 +------------------
.../admin/src/scripts/template-ns-inactivate.pl.in | 827 +------------------
.../src/scripts/template-ns-newpwpolicy.pl.in | 270 +------
ldap/admin/src/scripts/template-restart-slapd.in | 12 +-
ldap/admin/src/scripts/template-restoreconfig.in | 27 +-
ldap/admin/src/scripts/template-saveconfig.in | 28 +-
.../admin/src/scripts/template-schema-reload.pl.in | 135 +---
ldap/admin/src/scripts/template-start-slapd.in | 10 +-
ldap/admin/src/scripts/template-stop-slapd.in | 9 +-
ldap/admin/src/scripts/template-suffix2instance.in | 26 +-
.../src/scripts/template-syntax-validate.pl.in | 146 +---
ldap/admin/src/scripts/template-upgradedb.in | 28 +-
ldap/admin/src/scripts/template-upgradednformat.in | 62 +--
.../scripts/template-usn-tombstone-cleanup.pl.in | 164 +----
ldap/admin/src/scripts/template-verify-db.pl.in | 238 +-----
ldap/admin/src/scripts/template-vlvindex.in | 27 +-
ldap/admin/src/scripts/upgradedb.in | 100 +++
ldap/admin/src/scripts/upgradednformat.in | 121 +++
ldap/admin/src/scripts/usn-tombstone-cleanup.pl.in | 237 ++++++
ldap/admin/src/scripts/verify-db.pl.in | 303 +++++++
ldap/admin/src/scripts/vlvindex.in | 110 +++
ldap/servers/slapd/entry.c | 4 +-
78 files changed, 8154 insertions(+), 5202 deletions(-)
create mode 100755 ldap/admin/src/scripts/bak2db.in
create mode 100644 ldap/admin/src/scripts/bak2db.pl.in
create mode 100644 ldap/admin/src/scripts/cleanallruv.pl.in
create mode 100755 ldap/admin/src/scripts/db2bak.in
create mode 100644 ldap/admin/src/scripts/db2bak.pl.in
create mode 100755 ldap/admin/src/scripts/db2index.in
create mode 100644 ldap/admin/src/scripts/db2index.pl.in
create mode 100755 ldap/admin/src/scripts/db2ldif.in
create mode 100644 ldap/admin/src/scripts/db2ldif.pl.in
create mode 100755 ldap/admin/src/scripts/dbverify.in
create mode 100755 ldap/admin/src/scripts/dn2rdn.in
create mode 100644 ldap/admin/src/scripts/fixup-linkedattrs.pl.in
create mode 100644 ldap/admin/src/scripts/fixup-memberof.pl.in
create mode 100755 ldap/admin/src/scripts/ldif2db.in
create mode 100644 ldap/admin/src/scripts/ldif2db.pl.in
create mode 100755 ldap/admin/src/scripts/ldif2ldap.in
create mode 100755 ldap/admin/src/scripts/monitor.in
create mode 100644 ldap/admin/src/scripts/ns-accountstatus.pl.in
create mode 100644 ldap/admin/src/scripts/ns-activate.pl.in
create mode 100644 ldap/admin/src/scripts/ns-inactivate.pl.in
create mode 100755 ldap/admin/src/scripts/ns-newpwpolicy.pl.in
create mode 100644 ldap/admin/src/scripts/restart-slapd.in
create mode 100755 ldap/admin/src/scripts/restoreconfig.in
create mode 100755 ldap/admin/src/scripts/saveconfig.in
create mode 100644 ldap/admin/src/scripts/schema-reload.pl.in
create mode 100755 ldap/admin/src/scripts/start-slapd.in
create mode 100755 ldap/admin/src/scripts/stop-slapd.in
create mode 100755 ldap/admin/src/scripts/suffix2instance.in
create mode 100644 ldap/admin/src/scripts/syntax-validate.pl.in
create mode 100755 ldap/admin/src/scripts/upgradedb.in
create mode 100755 ldap/admin/src/scripts/upgradednformat.in
create mode 100644 ldap/admin/src/scripts/usn-tombstone-cleanup.pl.in
create mode 100644 ldap/admin/src/scripts/verify-db.pl.in
create mode 100755 ldap/admin/src/scripts/vlvindex.in

git push origin master
Counting objects: 125, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (85/85), done.
Writing objects: 100% (86/86), 48.65 KiB, done.
Total 86 (delta 55), reused 0 (delta 0)
To ssh://git.fedorahosted.org/git/389/ds.git
1a1db0f..cca2990 master -> master

Sorry, I was too hasty in my earlier review.

There are many hardcoded paths in the new code that should use parameterized paths or macros - for example:

/usr/sbin - @sbindir@ - location of ns-slapd, the new scripts
/etc/dirsrv - @instconfigdir@ - location of the slapd-INST instance directories
/etc/sysconfig - @initconfigdir@ - location of the dirsrv-INST files used by init, scripts
/var/lib/dirsrv - the default db dir is @localstatedir@/lib/@PACKAGE_NAME@

Use of prefix - when using with /etc or /var, you use prefix in addition to this directory - when the directory begins with /usr, you use prefix to replace the /usr part e.g.

configure --with-fhs:
/etc
/var
/usr

configure --prefix=/opt/dirsrv
/opt/dirsrv/etc
/opt/dirsrv/var
/opt/dirsrv/lib
/opt/dirsrv/share

so instead of /opt/dirsrv/usr/lib we have /opt/dirsrv/lib - instead of /opt/dirsrv/usr/share we have /opt/dirsrv/share

$prefix replaces /usr except when the top level directory is not /usr (/etc, /var, and so on)

There is a lot of duplicated code in the new scripts - they all have code to 1) find all of the instances 2) check to see that a server id was given - if not set the default - if so, fix it if it begins with slapd- or dirsrv- 3) code to check for the existence of the server config dir. It would be better to move this and other common code to DSUtil or some common shell script that can be sourced. Since you have introduced the use of code from DSUtil, we should make more use of it.

The old code did have a lot of copy/paste duplication - we had no common "library" that we could use.

Replying to [comment:8 rmeggins]:

Sorry, I was too hasty in my earlier review.

There are many hardcoded paths in the new code that should use parameterized paths or macros - for example:

/usr/sbin - @sbindir@ - location of ns-slapd, the new scripts
/etc/dirsrv - @instconfigdir@ - location of the slapd-INST instance directories
/etc/sysconfig - @initconfigdir@ - location of the dirsrv-INST files used by init, scripts
/var/lib/dirsrv - the default db dir is @localstatedir@/lib/@PACKAGE_NAME@

Use of prefix - when using with /etc or /var, you use prefix in addition to this directory - when the directory begins with /usr, you use prefix to replace the /usr part e.g.

configure --with-fhs:
/etc
/var
/usr

configure --prefix=/opt/dirsrv
/opt/dirsrv/etc
/opt/dirsrv/var
/opt/dirsrv/lib
/opt/dirsrv/share

so instead of /opt/dirsrv/usr/lib we have /opt/dirsrv/lib - instead of /opt/dirsrv/usr/share we have /opt/dirsrv/share

$prefix replaces /usr except when the top level directory is not /usr (/etc, /var, and so on)

There is a lot of duplicated code in the new scripts - they all have code to 1) find all of the instances 2) check to see that a server id was given - if not set the default - if so, fix it if it begins with slapd- or dirsrv- 3) code to check for the existence of the server config dir. It would be better to move this and other common code to DSUtil or some common shell script that can be sourced. Since you have introduced the use of code from DSUtil, we should make more use of it.

The old code did have a lot of copy/paste duplication - we had no common "library" that we could use.

Ok I'll work on this.

git merge ticket528
Updating a550a91..7a736ad
Fast-forward
Makefile.am | 3 +-
Makefile.in | 3 +-
ldap/admin/src/scripts/DSSharedLib.in | 68 +++++++++++
ldap/admin/src/scripts/DSUtil.pm.in | 90 +++++++++++++--
ldap/admin/src/scripts/bak2db.in | 88 ++++-----------
ldap/admin/src/scripts/bak2db.pl.in | 66 +----------
ldap/admin/src/scripts/cleanallruv.pl.in | 65 +----------
ldap/admin/src/scripts/db2bak.in | 93 ++++------------
ldap/admin/src/scripts/db2bak.pl.in | 69 +----------
ldap/admin/src/scripts/db2index.in | 96 ++++------------
ldap/admin/src/scripts/db2index.pl.in | 67 +----------
ldap/admin/src/scripts/db2ldif.in | 121 +++++---------------
ldap/admin/src/scripts/db2ldif.pl.in | 68 +----------
ldap/admin/src/scripts/dbverify.in | 104 ++++-------------
ldap/admin/src/scripts/dn2rdn.in | 88 ++++-----------
ldap/admin/src/scripts/fixup-linkedattrs.pl.in | 68 +----------
ldap/admin/src/scripts/fixup-memberof.pl.in | 67 +----------
ldap/admin/src/scripts/ldif2db.in | 82 +++-----------
ldap/admin/src/scripts/ldif2db.pl.in | 67 +----------
ldap/admin/src/scripts/ldif2ldap.in | 87 +++-----------
ldap/admin/src/scripts/monitor.in | 83 +++-----------
ldap/admin/src/scripts/ns-accountstatus.pl.in | 67 +----------
ldap/admin/src/scripts/ns-activate.pl.in | 66 +----------
ldap/admin/src/scripts/ns-inactivate.pl.in | 67 +----------
ldap/admin/src/scripts/ns-newpwpolicy.pl.in | 44 +-------
ldap/admin/src/scripts/restart-slapd.in | 72 +++---------
ldap/admin/src/scripts/restoreconfig.in | 94 ++++------------
ldap/admin/src/scripts/saveconfig.in | 91 ++++------------
ldap/admin/src/scripts/schema-reload.pl.in | 67 +----------
ldap/admin/src/scripts/start-slapd.in | 66 +++--------
ldap/admin/src/scripts/stop-slapd.in | 65 +++--------
ldap/admin/src/scripts/suffix2instance.in | 88 +++-----------
ldap/admin/src/scripts/syntax-validate.pl.in | 67 +----------
ldap/admin/src/scripts/upgradedb.in | 77 ++-----------
ldap/admin/src/scripts/upgradednformat.in | 98 ++++-------------
ldap/admin/src/scripts/usn-tombstone-cleanup.pl.in | 66 +----------
ldap/admin/src/scripts/verify-db.pl.in | 39 +------
ldap/admin/src/scripts/vlvindex.in | 92 ++++------------
38 files changed, 596 insertions(+), 2173 deletions(-)
create mode 100644 ldap/admin/src/scripts/DSSharedLib.in

git push origin master
Counting objects: 86, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (43/43), done.
Writing objects: 100% (44/44), 11.26 KiB, done.
Total 44 (delta 41), reused 0 (delta 0)
To ssh://git.fedorahosted.org/git/389/ds.git
a550a91..7a736ad master -> master

Hi Mark, I'm having some problems to use utilities...
1) db2bak

./db2bak

./db2bak: line 27: [-Z: command not found
{{{

diff -twU4 /tmp/db2bak.orig /usr/sbin/db2bak

--- /tmp/db2bak.orig 2013-02-28 17:55:11.810582189 -0800
+++ /usr/sbin/db2bak 2013-02-28 18:00:11.463386510 -0800
@@ -23,9 +23,9 @@
fi

if [ "$#" -gt 0 ]
then
- if ["$1" != "-*" ]
+ if [ expr "$1" : "-.*" -eq 0 ]
then
bak_dir=$1
shift
fi
}}}

2) upgradednformat
./upgradednformat: invalid option -- Z
{{{

diff -twU4 /tmp/upgradednformat.orig /usr/sbin/upgradednformat

--- /tmp/upgradednformat.orig 2013-02-28 17:37:56.767427827 -0800
+++ /usr/sbin/upgradednformat 2013-02-28 18:07:25.686867022 -0800
@@ -23,9 +23,9 @@
{
echo "Usage: $0 [-N] [-Z serverID] -n backend_instance -a db_instance_directory"
}

-while getopts "vhd:a:n:D:N" flag
+while getopts "vhd:a:n:D:NZ:" flag
do
case $flag in
Z) servid=$OPTARG;;
v) args=$args" -v";;

}}}

3) bak2db also does not issue a reasonable error message unless specifying the right backup dir.

./bak2db

[28/Feb/2013:18:13:05 -0800] - Warning: Unable to read dbversion file in /usr/sbin/-Z
[28/Feb/2013:18:13:05 -0800] - Restore: backup directory /usr/sbin/-Z does not exist.
[28/Feb/2013:18:13:05 -0800] - archive2db: Failed to read backup file set. Either the directory specified doesn't exist, or it exists but doesn't contain a valid backup set, or file permissions prevent the server reading the backup set. error=53 (Invalid request descriptor)

HI Noriko,

Thanks for catching this. I'm going to address this in ticket 332. Issue 1 was already been resolved, and I'll take care of 2 and 3.

Thank you for the ack, Rich!

Pushed to master: commit 4b2ee49
Pushed to 389-ds-base-1.3.1: commit 2566c96

Steps to reproduce on F20 as of today:

  • install F20 + updates
  • make sure the system has a proper hostname
  • install needed packages
  • yum install 389*
  • touch /tmp/ts
  • setup-dl.pl
  • use all defaults
  • the script reports new DS instance created ok
  • find / -xdev -newer /tmp/ts -print
  • finds 33 new instance specific files under /usr

On F20 389-ds-base is 1.3.2.16-1 and 389-ds is 1.2.2-5. setup-dl.pl is part of 389-ds-base.

Replying to [comment:20 myllynen]:

Steps to reproduce on F20 as of today:

  • install F20 + updates
  • make sure the system has a proper hostname
  • install needed packages
  • yum install 389*
  • touch /tmp/ts
  • setup-dl.pl
  • use all defaults
  • the script reports new DS instance created ok
  • find / -xdev -newer /tmp/ts -print
  • finds 33 new instance specific files under /usr

Can you paste the list of files under /usr in this ticket?

On F20 389-ds-base is 1.3.2.16-1 and 389-ds is 1.2.2-5. setup-dl.pl is part of 389-ds-base.

setup-ds.pl

Replying to [comment:22 rmeggins]:

Can you paste the list of files under /usr in this ticket?

Sure, please see below:

/usr/lib64/dirsrv/slapd-instance
/usr/lib64/dirsrv/slapd-instance/bak2db
/usr/lib64/dirsrv/slapd-instance/bak2db.pl
/usr/lib64/dirsrv/slapd-instance/cleanallruv.pl
/usr/lib64/dirsrv/slapd-instance/db2bak
/usr/lib64/dirsrv/slapd-instance/db2bak.pl
/usr/lib64/dirsrv/slapd-instance/db2index
/usr/lib64/dirsrv/slapd-instance/db2index.pl
/usr/lib64/dirsrv/slapd-instance/db2ldif
/usr/lib64/dirsrv/slapd-instance/db2ldif.pl
/usr/lib64/dirsrv/slapd-instance/dbverify
/usr/lib64/dirsrv/slapd-instance/dn2rdn
/usr/lib64/dirsrv/slapd-instance/fixup-linkedattrs.pl
/usr/lib64/dirsrv/slapd-instance/fixup-memberof.pl
/usr/lib64/dirsrv/slapd-instance/ldif2db
/usr/lib64/dirsrv/slapd-instance/ldif2db.pl
/usr/lib64/dirsrv/slapd-instance/ldif2ldap
/usr/lib64/dirsrv/slapd-instance/monitor
/usr/lib64/dirsrv/slapd-instance/ns-accountstatus.pl
/usr/lib64/dirsrv/slapd-instance/ns-activate.pl
/usr/lib64/dirsrv/slapd-instance/ns-inactivate.pl
/usr/lib64/dirsrv/slapd-instance/ns-newpwpolicy.pl
/usr/lib64/dirsrv/slapd-instance/restart-slapd
/usr/lib64/dirsrv/slapd-instance/restoreconfig
/usr/lib64/dirsrv/slapd-instance/saveconfig
/usr/lib64/dirsrv/slapd-instance/schema-reload.pl
/usr/lib64/dirsrv/slapd-instance/start-slapd
/usr/lib64/dirsrv/slapd-instance/stop-slapd
/usr/lib64/dirsrv/slapd-instance/suffix2instance
/usr/lib64/dirsrv/slapd-instance/syntax-validate.pl
/usr/lib64/dirsrv/slapd-instance/upgradednformat
/usr/lib64/dirsrv/slapd-instance/usn-tombstone-cleanup.pl
/usr/lib64/dirsrv/slapd-instance/verify-db.pl
/usr/lib64/dirsrv/slapd-instance/vlvindex

Thanks.

Yes these scripts were left behind for backwards compatibility. These scripts, under /usr/lib64/dirsrv/slapd-instance, are just wrappers to the new "real" scripts under /usr/bin & /usr/sbin

This ticket really should have been named "centralize instance specific scripts in system bin and sbin directories"

Closing this ticket.

The next step will be to add a configure option to disable instance specific scripts

Metadata Update from @rmeggins:
- Issue assigned to mreynolds
- Issue set to the milestone: N/A

2 years ago

Login to comment on this ticket.

Metadata