INFNGRID v1.1 installation procedure

Note No. 1/2000
September 2000

INFNGRID v1.1 installation procedure

Flavia Donno, Andrea Sciaba', Massimo Sgaravatto, Zhen Xie

  1. Introduction
  2. System Prerequisites
  3. INFN Customizations
  4. A very fast guide
  5. Special Environments
  6. Appendix A: included fixes
  7. Appendix B: production of a precompiled distribution

1. Introduction

The aim of this document is to describe the steps to be performed in order to install the INFNGRID v1.1 software at a site. The INFNGRID software is a special distribution of GLOBUS v1.1.3 which contains patch fixes and special additions to configure GLOBUS in the INFN environment, such as registration with the INFN CA (Certification Authority) and the configuration of a local GIIS (Grid Index Information Service), where the local GRISes (GRID Resource Information Service) register, following the INFN GIS (Grid Information Service) architecture. For more information about Globus v1.1.3, please refer to the
Globus ToolKit 1.1.3 System Administration Guide However, in the INFNGRID distribution, the changes proper for an INFN site have been added to the Globus software in a loose way. It is always possible to install INFNGRID following the standard globus configuration and excluding the INFN customizations. Before starting the installation of the INFNGRID software you should take some decisions such as:
  1. Would you like to include INFN customization in your installation ?
  2. Which machine should be the Globus Gatekeeper ?
  3. Which underlying resource management system for GRAM will you use ?
  4. Which machine will be your local GIIS ?
  5. What is the directory where you want to copy the INFNGRID tar files ?
  6. What is the directory where Globus and the related software will be installed ?
  7. Where will Globus be deployed ?

2. System Prerequisites

The operating systems supported by INFNGRID v1.1 are Linux RedHat 6.1 and SUN Solaris 2.6. The INFNGRID distribution contains a pre-compiled version of the Globus software for these two platforms.
The disk space requirements for installation are of about 80MB. Before starting, you should create a globus account on the machine where you intend to install the software. For a successful installation, it is assumed that the environmental variable PATH for the globus account is set in such a way that all standard UNIX utilities can be found. In fact, each command is searched in the following directories:


Some steps need to be performed as root.

3. INFN Customizations

In the following sections we illustrate which are the customizations specific to INFN. In particular, the Globus/Grid Information Service architecture and the mechanism to ask for and install host and user certificates following the specifications of the INFN Certification Authority are described. An INFN specific Globus installation can still use the Globus Certification Authority (CA) certificates and work in the standard Globus environment. Details for doing so are also provided.

3.1 The INFN GIS Architecture

At INFN it has been decided to adopt a preliminary GIS (GRID Information Service) architecture. Following this model, each Globus gatekeeper at a site runs a GRIS (GRID Resource Information Server) daemon. Such daemon, collects the info releated to the resources managed by that particular gatekeeper and contacts the node running the local GIIS (GRID Index Information Server). The local GIISes of each INFN section register with the INFN central top level GIIS.
Figure 1 shows a picture of the INFN GIS architecture.

Figure 1

As shown in the picture, the implementation of GIISes for the experiments is also forseen. Those GIISes will collect info related to the resources geografically distributed, available for a given HEP experiment.
Tools are provided to browse the info kept at the INFN central GIIS (
The infngrid-install procedure performs the step of configuring an INFN GRIS on the node which runs the gatekeeper, everytime the Globus Deploy step is executed and the INFN setup has been chosen.

3.2 The INFN CA and the registration mechanism

INFN has created an internal Certification Authority which can sign INFN certificates. The INFNGRID installation toolkit provides a semi-automatic mechanism for requesting host and user certificates that can be used inside Globus. In the following, we describe the steps that need to be done in order to get and install a user certificate. You will need such a certificate to be able to use Globus, and also to sign the request for a host certificate (the gatekeeper) that has to be sent to:
Subsection 2 describes the steps needed for a host certificate. Such steps are mostly executed by the infngrid-install procedure. We outline what are the steps that the user needs to do by hand and that the procedure infngrid-install does not cover.
If you use an INFN installation of the infngrid software, you can still use the certificates obtained from the Globus CA. After installing globus and before issuing a globusrun request, you should use the command:
	% grid-proxy-init -cert <user-cert> -key <user-key>
where <user-cert> and <user-key> are the globus user certificate and key issued by the considered CA.
However, the INFN customized installation modifies the script grid-cert-request. So, such a script will only be able to generate host certificates requests for the INFN CA. The original Globus script is saved with the name grid-cert-request-old. So this copy can still be used to generate Globus certificate requests.

3.2.1 Requesting and installing an INFN user certificate

  1. Using Netscape, request a personal certificate from:
  2. Follow the instructions you receive via e-mail, using Netscape from the same machine used to request the certificate or from a machine which shares the same home.
  3. On the Netscape browser window, push the button "Security".
  4. Choose Certificates, Yours; select the INFN certificate and push Export.
  5. Enter the password used for your own Netscape certificates database.
  6. Enter a null password.
  7. Save your certificate as cert.p12.
  8. After the execution of infngrid-install (which installs openssl on your machine, if you have chosen the INFN setup), and the execution of globus-root-setup (which defines the right user PATH for using openssl), run the following command on the machine where you executed infngrid-install:
    % <grid-install-dir>/openssl/bin/openssl pkcs12 -in cert.p12 -out cert.pem
  9. When asked, use the null password.
  10. Enter the PEM pass phrase when asked and confirm it.
  11. Delete the file cert.p12
  12. Run the script <grid-install-dir>/infn-scripts/converti in the directory where the file cert.pem is. After, delete the file cert.pem for security reasons.
  13. Move the files usercert.pem and userkey.pem in the directory ${HOME}/.globus.
  14. Execute the following command:
    % chmod 600 $HOME/.globus/userkey.pem

3.2.2 Requesting and installing an INFN host certificate

  1. If you don't have already a certificate for your host, after the installation of the INFNGRID software, you must request a certificate from the INFN CA.
    For this purpose, you must send the file: <grid-deploy-dir>/etc/globus-gatekeeper.request to You must sign this e-mail, using your personal certificate issued by the INFN Certification Authority. Read section 3.2.1 for further details on getting a personal certificate.
  2. To send the mail to the INFN CA, use Netscape messenger.
    Before sending the mail, push the Netscape Security button, and check the box Include my security certificate in this message and use a digital signature to sign this message.
  3. As a result of the previous actions, you will receive the certificate via e-mail, save it as root in the file: ${GLOBUS_DEPLOY_PATH}/etc/globus-gatekeeper.cert and execute the command:

  4. % chmod 600 $GLOBUS_DEPLOY_PATH/etc/globus-gatekeeper.key
  5. Save the files:
    in a safe place.
  6. Test what you have done executing as root the command:

  7. # $GLOBUS_DEPLOY_PATH/sbin/globus-gatekeeper \
          -conf $GLOBUS_DEPLOY_PATH/etc/globus-gatekeeper.conf -test
    verifying that the output is close to the following:

              Testing gatekeeper
              Local user id (uid)      : root
              Home directory           : /opt/globus
              Libexec directory        : /opt/globus/libexec
              Gatekeeper subject name  : "/O=Grid/O=Globus/"
              Gatekeeper test complete : Success!

              Gatekeeper shutting down!

4. A very fast guide

In this section we describe step by step how to run the procedure infngrid-install to install and configure the INFNGRID software on a machine. A globus user should exist on the machine where you execute the procedure. All steps described below should be executed as globus, unless otherwise specified.
  1. Create three directories: <grid-download-dir>, <grid-install-dir>, <globus-deploy-dir>.
    The directory <grid-download-dir> should be able to hold about 20MB of data, the directory <grid-install-dir> about 64MB and the directory <globus-deploy-dir> about 10MB.

  2. Download the script infngrid-install from:
    or via ftp from:

    If AFS (the Andrew File System) is not available at your site, together with infngrid-install, copy all files that you find in the anonymous ftp directory: /pub/GRID/1.1/<Flavor>, where <Flavor> is either Linux or SunOS.

  3. Run the script:
    % infngrid-install
    You will be prompted with four questions:

    Please, enter the download directory path, default is the current directory:

    Enter your <grid-download-dir>

    Please, enter the grid install path:

    Enter your <grid-install-dir>

    Please, enter the deploy directory path:

    Enter your <globus-deploy-dir>

    Would you like the INFN setup (Y/N) [Y]?:

    Enter Y if you want to use the INFN certificates and the INFN GIS Architecture.

  4. A simple menu with 7 steps will appear.
    If AFS is not available, download the whole package via ftp, as described in 2, and skip Step 1.

  5. Each step is completed when you see a message : Step n: done . The Globus Deploy step will configure the machine to use a GIIS (which can be the same machine where you are running or another one) and to run a GRIS.

  6. If you have AFS installed on your machine, and therefore you didn't download the package via ftp, choose option 1 ( Copy INFNGRID tar files from /afs/<Flavor> to download dir) Answer y at the question: Would you like INFNGRID tar files to be copied now ?.

  7. When the message Step 1: done appears, choose option 2 ( Decompress and untar INFNGRID distribution files in install dir).

  8. When the message Step 2: done appears, choose option 3 ( Configure INFNGRID software).

    Note: One must run Step 3 after Step2.

  9. When the message Step 3: done appears, choose option 4 (Globus Setup).
    Answer Y at the question: Do you wish to continue (y/n).
    Choose 1 (Hostname for site's organization server) and type the IP name of the GIIS in your site (this GIIS could already be running on an other machine, or you could configure the machine where you are installing the software as a GIIS).
    Choose 2 (Port number for site's organization server).
    If there is already a running GIIS at your site, type the port number associated to the GIIS server (it must be 2167, if you are using the INFN setup/architecture), or, if you are configuring the machine where you are installing the software now as a GIIS for your site, enter a port number of your choice (the port MUST be 2167 for the INFN setup).
    The DN for the organization should already be defined as:
         [ dc=xx, dc=infn, dc=it, o=Grid ]
         where xx represents the site (i.e. pi for Pisa, mi for Milano, etc ...)
    Choose 4 (Save settings).
    Choose 5 (Quit).
    Answer Y at the question: Do you wish to continue (y/n)
    The base DN for user certificates should already be defined as:
          [, o=Globus, o=Grid ]   
          (where xx represents the site)
    and the base DN for host certificates should already be defined as:
       [ o=Globus, o=Grid ] 
    Choose q (save, configure the GSI and Quit)

  10. When the message Step 4: done appears, choose option 5 (Configure GRAM services).
    You will choose this option ONLY if you want to add job schedulers such as CONDOR or/and LSF to the GRAM services.
    You do not need to perform this step if you choose to use only the fork as the job manager.
    Note: The configuration of the job scheduler PBS will be added in the next release.
    Note: After the installation, make sure that the file :
    ${GLOBUS_INSTALL_PATH}/etc/<arch>/ contains the right path to the commands for the job resource manager of your choice.

  11. When the message Step 5: done appears, choose option 6 (Globus local deploy).
    If you are redeploying the Globus software, and you already have a certificate/key pair, at the question:
    Please, enter the full filename of the INFN gatekeeper certificate if you already have it, otherwise press Enter to generate a request:
    Type the pathname of the certificate, otherwise press Enter.
    If you give the certificate full file name, you will then be prompted to provide the full pathname of the host/gatekeeper key file, as well.
    Answer n at the question:
    Do you want to register this host with the Globus Project?
    You can safely ignore the instructions that describe how to complete the deployment (a script will be used later to automatically perform these operations)

  12. When the message Step 6: done appears, choose option 7 ONLY if you want to configure this machine as a GIIS for your site. You do not need to perform this step if you do not choose the INFN setup.

  13. When the message Step 7: done appears, choose option q (Quit).

  14. When you are done, send <globus-deploy-dir>/etc/globus-gatekeeper.request to or to if you have chosen to use the INFN certification.
    You need to do that only if you do not have already a certificate for your host.
    Section 3.2.2 to know more about how to get and install INFN gatekeeper certificates.

    Save the globus-gatekeeper.cert and the globus-gatekeeper.key to a safe place and put another copy in the directory <globus-deploy-dir>/etc.

  15. Go to the directory <grid-install-dir>/infn-scripts and, as root, run the script globus-root-setup for setting up globus, as follows:
      % su
      (for Bourne shell)
      # GLOBUS_INSTALL_PATH=<grid-install-dir>/globus ; export GLOBUS_INSTALL_PATH
      # GLOBUS_DEPLOY_PATH=<globus-deploy-path> ; export GLOBUS_DEPLOY_PATH
      (for C-sh shell)
      # setenv GLOBUS_INSTALL_PATH <grid-install-dir>/globus
      # setenv GLOBUS_DEPLOY_PATH <globus-deploy-path>
      # ${grid-install-path}/infn-script/globus-root-setup
    Note: GLOBUS_INSTALL_PATH is <grid-install-dir>/globus

    The globus-root-setup will present you a menu as the following:
    (1) modify system files and reactivate the inetd daemon
    (2) change owner to root of certain files for tighter security
    (3) modify system wide login files
    (4) start/restart Globus now
    (5) configure gsi-wuftpd and restart the inetd daemon
    You should execute steps 1 to 4 in order.
    Since this procedure modifies system files, it is recommended that you look at the source code or review the files that the procedure reports as modified.
    The procedure can, optionally, configure the gsi-wuftpd daemon. If you want to do so, just select step 5.
    During step 5, if the directory /etc/grid-security exists already or its a symbolic link to another directory, the procedure will not attempt to modify its content.
    It is your responsability to make sure you have the correct files in place. Check the gsi-wuftpd installation manual.

    Note: If you do not use HEPIX at your site, globus-root-setup modifies the system wide /etc/csh.cshrc, /etc/.profile, and /etc/.login to include the setup of the environemnt for globus. Please, review those files after this step is executed.
    If you use HEPIX, the files /etc/hepix/sys.conf.csh and /etc/hepix/ are modified. Please, review those files as well.

  16. Now, for each user who will use globus, get a user certificate.
    If you are using the INFN certification authority, just follow the steps described in Section 3.2.1.
    If instead you are using the Globus certification, as the user who wants to get the certificate, execute the following:
    % grid-cert-request
    Follow all instructions given by the procedure and, when prompted, enter a passphrase of at least 8 characters, with at least one numeral. The procedure will generate a key file (userkey.pem) and a request file (usercert-request.pem) in the directory ~<user>/.globus. Send the request to the globus CA: A certificate will be sent to you.
    No matter what CA you use, either Globus or INFN, the user key and certificate should reside in the directory ~<user>/.globus and should be called respectively userkey.pem and usercert.pem. The file userkey.pem should only be readeable by the owner.

    Note: If the user home directory is on AFS space, link the directory ~<user>/.globus/.gass_cache to a local filesystem (recommended) or make the directory writeable by all users.

  17. For each user of the GRID who wants to use resources on the machine where you just installed the software, enter in the file:
    $GLOBUS_DEPLOY_PATH/etc/grid-mapfile (editable only by root) a line which defines the mapping between the subject of the user certificate, obtainable using the command:
    grid-cert-info -subject
    and a local user login name, as in the following examples:

    "/C=US/O=Globus/O=Istituto Nazionale di Fisica Nucleare/OU=Sezione di Pisa/CN=Flavia Donno" flavia
    "/C=US/O=Globus/O=Istituto Nazionale di Fisica Nucleare/OU=Sezione di Pisa/CN=Zhen Xie" cmsprod

    If you hold an INFN certificate the lines above become:

    "/C=IT/O=INFN/L=Pisa/CN=Flavia Donno/" flavia
    "/C=IT/O=INFN/L=Pisa/CN=Zhen Xie/" cmsprod

  18. Test your installation and setup.
    As root execute the command:

    # $GLOBUS_DEPLOY_PATH/sbin/globus-gatekeeper \
          -conf $GLOBUS_DEPLOY_PATH/etc/globus-gatekeeper.conf -test
    verifying that the output is close to the following:

              Testing gatekeeper
              Local user id (uid)      : root
              Home directory&nb sp;          : /opt/globus
              Libexec directory        : /opt/globus/libexec
              Gatekeeper subject name  : "/C=IT/O=INFN/OU=gatekeeper/L=PI/"
              Gatekeeper test complete : Success!

              Gatekeeper shutting down!

    Login as a normal user (neither globus or root). Make sure that you have an entry in the globus grid-mapfile of the machine you are testing globus on. Make sure that in your .globus directory the files usercert.pem, userkey.pem are present and with the right protections.

    As a normal user, run command grid-proxy-init to get a proxy. Then run globus-setup-test. If this test is successful, it means your gatekeeper is correctly installed and configured.

    If you have chosen an INFN set-up, go to and check if your host is present in the list of the central INFN MDS server.

Special Environments

In this section we would like to give some hints on special installation cases, such as those where you want to configure a machine to only enable users to issue globus commands or you want to share a single software distribution tree among multiple machines all configured as GRISes with a local job manager.

5.1 Installation of Globus client machines

A machine can be configured only as client machine. This means that this machine won't be used as a Globus resource (i.e. it is not possible to submit jobs on this machine), but from it a user can submit jobs to other machines in the GRID, can query the Grid Information Service, etc.

To configure such a machine, you just need to create a grid-install-dir. Then, as globus user, run the script infngrid-install and execute ONLY the first 4 steps:
(1) Copy INFNGRID tar files from /afs/ to download dir
(2) Decompress and untar INFNGRID distribution files in install dir
(3) Configure INFNGRID software
(4) Globus Setup

You do not need to perform the Globus Local Deployment step, nor to get a host/gatekeeper certificate.

As root, go to the directory <grid-install-dir>/infn-scripts and run the script globus-root-setup, as described in step 14 of section 4, but execute ONLY step 3 (modify system wide login files) if you need/want to setup the user environemnt/path for globus. Otherwise you do not need to execute the globus-root-setup script at all.

5.2 Sharing the Grid install directory among gatekeepers

While each machine must have its own <globus-deploy-directory> (and it is reccomended to have this directory on a local file system), multiple Globus gatekeepers of the same architecture can share the same grid install directory.

Let's assume that you want to install the INFNGRID software on 2 hosts (host1 and host2), that share the same <grid-install-dir>. First, you need to install the software on one host1, as described in Section 4.
Then, to configure the software on host2, run the script infngrid-install skipping the first three steps.

During the deployment, you will have to answer to some new questions:

<host2> is not listed in the
Should I add it (y/n) [y] 

<host2> is not listed in the
Should I add it using the defaults (y/n) [y] : 

<host2> is not listed in the
Should I add a default jobmanager (y/n) [y] :

Note: The directory tree <grid-install-dir> must be writable when the script infngrid-install is executed. On the other side, to use the INFNGRID software the directory tree can be just readable.
Note: It is not possible at the moment to define more than one job scheduler when sharing the installation directory among gatekeepers. ONLY THE DEFAULT JOB MANAGER FORK IS SUPPORTED.

Appendix A: included fixes

  1. globus-user-setup fix.
    The file ${GLOBUS_INSTALL_PATH}/etc/globus-user-setup.csh has been modified substituting the line:
    setenv GLOBUS_PATH "`${GLOBUS_INSTALL_PATH}/bin/globus-tools-path -bindir`"
    the line: setenv GLOBUS_PATH "`${GLOBUS_INSTALL_PATH}/bin/globus-services-path -bindir`"
    The same for the file ${GLOBUS_INSTALL_PATH}/etc/ where the line: GLOBUS_PATH="`${GLOBUS_INSTALL_PATH}/bin/globus-tools-path -bindir`"
    has been substituted with: GLOBUS_PATH="`${GLOBUS_INSTALL_PATH}/bin/globus-services-path -bindir`"
    Problem found and patch provided by the INFNGRID Release Team (F. Donno, A. Sciaba', Z. Xie).

  2. globus-local-deploy fix.
    In the routine _check of
    the following lines have been removed:
     if [ ! -w "$1" ] ;  then
          echo "Unable to write to \"$1\"!"
    and the variable
    has been changed to
    Patch provided by Steven Fitzgerald (GLOBUS Team)

  3. DN fix.
    Before building globus, the file
    in the source tree has to be modified with the correct version reported in
    Appendix B. In particular:
    Around line 124:
    case "\1"
    is replaced with:
    TRY_DN=\`echo "\$1" | ${GLOBUS_SH_TR-tr} "[a-z]" "[A-Z]"\`
    case "\$TRY_DN"
    and around line 157:
    DN_PATTERN=`echo "${SET_DN}" \
    | sed -e 's/ /\\\\ /g' \
          -e 's/)/\\\\)/g'`
    is replaced with:
    DN_PATTERN=`echo "${SET_DN}" \
    | ${GLOBUS_SH_TR-tr} "[a-z]" "[A-Z]" \
    | sed -e 's/ /\\\\ /g' \
          -e 's/)/\\\\)/g'`
    Problem found by G. Lo Biondo (INFNGRID Team). Patch provided by Karl Czajkowski (GLOBUS Team).

  4. BASH2 fix
    Before building globus, the following file is changed in the following way:
    746: if test_rdn ${search_attr} $1  (take out the = sign)
    760: search_rdns_expanded $1 $2     (take out the comma)
    Patch provied by

  5. $GLOBUS_INSTALL_PATH/services/<platform>/libexec/globus-script-condor-submit fix.
    The block:
        if [ "${condor_universe}" = "vanilla" ] ; then
           echo "Initialdir = /var/tmp"
           echo "Input = /dev/null"
           echo "Output = /dev/null"
           echo "Error = /dev/null"
    has been replaced with:
         if [ "${condor_universe}" = "vanilla" ] ; then
           eval echo "Initialdir = ${grami_directory}"
           eval echo "Input = ${grami_stdin}"
           eval echo "Output = ${grami_stdout}"
           eval echo "Error = ${grami_stderr}"
    otherwise standard Input/Output/Error for condor vanilla jobs is redirected to /dev/null.
    Problem found by M. Sgaravatto (INFNGRID Team). Patch provided by James Frey (CONDOR Team).

  6. $GLOBUS_INSTALL_PATH/services/<platform>/libexec/globus-script-condor-poll fix.
    The line:
    if [ 0 -lt `echo " R I " | grep -c " $val "` ]; then
    has been replaced with:
    if [ 0 -lt `echo " R " | grep -c " $val "` ]; then
    and the line:
    if [ 0 -lt `echo " U " | grep -c " $val "` ]; then
    has been replaced with:
    if [ 0 -lt `echo " U I " | grep -c " $val "` ]; then

    Problem found by M. Sgaravatto (INFNGRID Team). Patch provided by Stuart Martin (GLOBUS Team).

  7. $(SRC)/Miscellaneous/globusrun/globusrun.c fix.
    Before building globus the line:
    err == GLOBUS_SUCCESS;
    has been replaced with:

    This is to avoid the problem that in some cases globus-job-clean is unable to do its job. This happens when the status of the job as given by globusrun -status happens to be DONE but the return code of globusrun is different from 0 (see cancelJob() in globus-job-clean). This is due to a mistyping in globusrun.c, line 1983.

    Problem found and patch provided by F. Giacomini (INFNGRID Team).

Appendix B: Production of a precompiled distribution

  1. Untar and build, on a given machine (es., and in the directory /usr/local/grid as user globus, the following packages: OpenLDAP-1.2.7-globus.tar.gz SSLeay-0.9.0b.tar.gz
  2. Archive the directories ldap e ssl produced by the installation as ldap.tar.gz e ssl.tar.gz
  3. Untar in /usr/local/grid the file globus-1.1.3.tar.gz. Substitute the file /usr/local/grid/globus-1.1.3/InformationServices/mds/services/ with the correct version (obtained following the instructions and
  4. Execute globus-install, as explained in
  5. Copy the following files in $GLOBUS_INSTALL_PATH/share/certificates:
    and archive $GLOBUS_INSTALL_PATH as globus.tar.gz
Last update 27-Sep-2000