MCNP5PCs

LA-UR- Approved for public release; distribution is unlimited. Title: Author(s): Submitted to: Form 836 (8/00) Los Alamos National Laboratory, an affirmative action/equal opportunity employer, is operated by the University of California for the U.S. Department of Energy under contract W-7405-ENG-36. By acceptance of this article, the publisher recognizes that the U.S. Government retains a nonexclusive, royalty-free license to publish or reproduce the published form of this contribution, or to allow others to do so, for U.S. Government purposes. Los Alamos National Laboratory requests that the publisher identify this article as work performed under the auspices of the U.S. Department of Energy. Los Alamos National Laboratory strongly supports academic freedom and a researcher’s right to publish; as an institution, however, the Laboratory does not endorse the viewpoint of a publication or guarantee its technical correctness. Nuclear Mathematical and Computational Sciences: A Century in Review, A Century Anew Gatlinburg, Tennessee, April 6-11, 2003, on CD-ROM, American Nuclear Society, LaGrange Park, IL (2003) MCNP 5TM IMPROVEMENTS FOR WINDOWS PCS Tim Goorley, Forrest Brown, and Lawrence J. Cox Los Alamos National Laboratory X-Division, X-5, MS F663 Los Alamos, NM 87545 jgoorley@lanl.gov, fbrown@lanl.gov, ljcox@lanl.gov ABSTRACT With the release of MCNP 5, much more emphasis has been placed on improving its functionality on PCs running Microsoft Windowsâ (9X/NT/2000/XP). Enhanced installation and build systems, support for more Fortran compilers, integration with X11 graphics build files, and MPI and PVM parallel capabilities have been implemented in MCNP 5 for Windows PCs. MCNP 5 can be installed with an InstallShield setup programs, similar to other Windows programs, for users who only need to install executables and data libraries. For those users who need to compile the source, the GNU make utility can used in conjunction with three supported Fortran compilers: Compaq Visual Fortran (CVF), Lahey Fortran 95 (LF95) or Absoft Fortran 95 (AF95). Alternatively, Compaq Developer Studio can be used to compile MCNP 5. The X-Windows plotting capabilities have been improved, and all the appropriate open source X11R6 files for compiling MCNP 5 are bundled with the MCNP 5 source code. X client software is still needed, however, to display geometry, cross-section or tally plots. Parallel capabilities which exist on other platforms have been extended to Windows PCs, allowing users to utilize dual CPU PCs, clusters of homogeneous Windows PCs (preferably with MPI), or heterogeneous clusters (preferably with PVM). Wall-clock runtimes show that MCNP 5 compiled with CVF runs 1.4 times faster than when it is compiled with LF95 or AF95. Wall-clock runtimes also show MCNP 5 with MPI and the balance keyword more effectively utilizes a dual-processor and a small heterogeneous cluster Windows 2000 PC than MCNP 5 with PVM. Key Words: MCNP 5, Microsoft Windows, Parallel, MPI, PVM 1. INTRODUCTION MCNP Version 5[1,2,3]1, a general purpose radiation transport computer code, has been released to the Radiation Safety Information Computational Center[4]. MCNP is developed and supported by the Eolus team at Los Alamos National Laboratory (LANL) to support Accelerated Strategic Computing Initiative (ASCI) projects. With this release, MCNP has been rewritten into Fortran 90 and several new features have been added. These new features inc lude a new build and installation process, color plotting enhancements, Doppler energy broadening for photons, a neutral particle radiography imaging capability, improved source options, time importance variance reduction, support for parallel communications with MPI, new random number generators, dynamic memory on all platforms, and a new mesh tally capability[2]. 1 MCNP is a trademark of the Regents of the University of California, Los Alamos National Laboratory. T. Goorley, F. Brown, L.J. Cox American Nuclear Society Topical Meeting in Mathematics & Computations, Gatlinburg, TN, 2003 2/16 Enhancements for Microsoft Windowsâ PCs have also been included with this release, since desktop PCs increase their power and popularity for scientific computing applications. In addition to the new code features which are available on all platforms, improvements for Windows PCs include two new methods for installing and compiling the source, support for three commercial FORTRAN compilers, and easier building of plotting versions with included X11 graphics files. Support for parallel capability using Message Passing Interface MPI[5,6] or Parallel Virtual Machine PVM[7,8] protocols, originally on other platforms[9], has been extended to Windows PCs. These new parallel capabilities allow MCNP to be run on homogeneous and heterogeneous clusters and/or dual processor Microsoft Windows PCs. These PC enhancements are the subject of this paper. The following two sections describe how to install and build MCNP 5 on a PC. The subsequent two sections discuss a comparison of the wall clock runtimes for MCNP 5 on a dual processor desktop and a network of different CPU speed laptops. 2. INSTALLING MCNP 5 ON WINDOWS PCS There are two different ways to install MCNP 5 on a PC running a Windows operating system (95/98/NT/2000/XP/ME). The simplest method is to use the InstallShieldâ setup programs, similar to that of other Windows programs. The first setup program copies the MCNP executables, source code, and test problems to a user-selected directory and then sets two environmental variables. The MCNP Visual Editor and MCNP documentation are also installed. The second setup program installs the data libraries, MAKXSF (a cross-section library compression program), the files XSDIR and SPECS, and sets an environmental variable. The user is then asked to logout and login and then run the test problems to verify that MCNP has been installed correctly. The main advantage of using this method is that no compilers are needed and no source code needs to be built. This option meets the needs of most users. Alternatively the user can copy the MCNP directory tree to the desired location and use the install script to build MCNP 5, MAKXSF, and/or run the test problem suite. The install script can be used to build MCNP only if the appropriate compilers are already installed. The advantage to this method is that the executables can be rebuilt to apply patches or modifications to the source. After installing the MCNP executable, additional software may need to be installed and appropriate environmental variables may need to be set or changed to take advantage of X11 graphics or parallel communications capabilities in MCNP. X-windows client software, not provided with MCNP 5, needs to be running to display geometry, tally, or cross-section plots. X-Windows software is discussed in section 2.4 of this paper. To use the parallel versions of the MCNP executables, parallel communications software will need to be installed prior to running with this capability, and prior to building a parallel version of MCNP, if the InstallShield executables are not used. Specific instructions on how to install MPI and PVM are given in section 2.5. The environmental variables PATH and DATAPATH can be modified to make file management easier. DISPLAY may need to be set to use the plotting capabilities of MCNP. MCNP environmental variables are discussed in section 2.6. MCNP 5 IMPROVEMENTS FOR WINDOWS PCS American Nuclear Society Topical Meeting in Mathematics & Computations, Gatlinburg, TN, 2003 3/16 2.1. The InstallShield Setup Program The InstallShield programs for MCNP 5 are similar to that of other windows applications. Double clicking on the setup file will start the InstallShield program. After starting the MCNP 5 Executables installer, the initial setup window is displayed, then the next two windows present the Copyright notice and Software License Agreement, and request user information. The following window asks where MCNP should be installed. It does not need to be in the default directory of /Program Files/LANL/MCNP5/. The installation package will then copy the plotting-sequential and parallel executables, MCNP source code, documentation (including the MCNP Manual), Visual Editor, and problem test suite into the chosen directory. The final screen queries the user if it can change the appropriate environmental variables: PATH and DISPLAY. Since PATH is already present, the MCNP5 directory path is appended to this variable. If these environmental variables are changed, the user must log out and then log back in (or reboot for some operating systems) before they will take effect. The user should be aware that if another executable with the name MCNP5 is already present in the path, the first executable in the path, i.e. the previously existing MCNP5, may be unintentionally used. The InstallShield setup program cannot be used to build MCNP executables. The second InstallShield setup program installs the data libraries, the MAKXSF program and the XSDIR and SPECS files. The installer queries the user for a directory to place these files, which may or may not be a subdirectory where MCNP5 was installed. After this directory is specified, the environmental variable DATAPATH is set. Administrative privileges are also required, as well as write permission to at least 2.5 Gigabytes of hard disk space. This large amount of space is mostly used by the ASCII format (type 1) updated ENDF/B-VI cross-section libraries, which can be compressed with MAKXSF and the SPECS file to ~800 Megabytes. For more information about using MAKXSF, see section XIV of Appendix C in the MCNP 5 Manual. After the InstallShield programs are completed, the user should run the test suite to verify that the executable has been installed and operates correctly on the user’s specific operating system and hardware. This testing procedure can be started by double clicking on the runprob.bat icon located in the directory Installation, where MCNP5 was installed. After the test problems are run, files that list the differences between the tally or output files generated and the expected results are displayed. These difference files should be reviewed by the user to determine if the differences are simple round-off errors or something more substantial, indicating incompatible hardware or software and that MCNP may give incorrect results. To uninstall either the MCNP 5 Executables or MCNP Data install packages, the user should remove them via the Windows Control Panel, with the Add/Remove Programs function. This will delete any files that were installed (files created while running the test problems will not be deleted) and will modify the registry appropriately. The environmental variables will not be removed, however, but these can be removed manually. 2.2. The Install Script A second method to install MCNP 5 uses an install script, which interactively queries the user for various build options and then executes the make utility to build MCNP, MAKSXF and/or run T. Goorley, F. Brown, L.J. Cox American Nuclear Society Topical Meeting in Mathematics & Computations, Gatlinburg, TN, 2003 4/16 the test problem suite. The various build options include which Fortran and C compilers should be used, the location of appropriate X11 files, and the path to the xsdir file. The script also gives the user the opportunity to only generate custom makefiles. These custom makefiles contain the build options selected, and will be automatically used whenever the make utility is used. An answer file, which lists the options chosen in the install script, is also created. The answer file can only be used to set options in the install script, and only if it is specified on the install command line. If MCNP is to be built, or the test suite is to be run, then install script will then execute gmake, (the GNU version of make), and write most of the output to the file install.log. This script, which uses the gmake utility to build the code, can only be used if a Unix based shell is installed. 2.3. Installing a Unix shell - Cygwin If the install script or gmake utility is used, a Unix command shell must be installed. Unix command shells are not standard on Windows operating systems and must be installed. Cygwin, a freeware port of a Unix command shell for Windows PCs, can be obtained at http://www.cygwin.com or http://www.redhat.com/apps/download/. The setup program can be downloaded or run from the Redhat website. This setup program will step users through the Cygwin installation process, allowing them to select web installation or download installation files to a local drive. The location where the cygwin software should be installed (a path without spaces is recommended) and a temporary directory where files can be downloaded are specified in the next two windows. After selecting a website to download or install from, the user selects Cygwin packages to install. In addition to the Cygwin packages that are selected by default, the gcc and make packages (located under the Devel directory) will also need to be selected to build MCNP with the make utility. The perl package (located under the Interpreters directory) is also recommended, and is required if the Absoft or Lahey Fortran compilers are going to be used to compile MCNP. 2.4. Installing Plotting Software – X Windows Client In order to display MCNP plots, an X11 windows client software package is needed. Several commercial X-windows clients are available: Reflection X (http://www.wrq.com/products/), Hummingbird’s Exceed_NT (http://www.hummingbird.com/products/nc/exceed/index.html), and Starnet’s X-win32 (http://www.starnet.com/). No single commercial product is recommended. A freeware X client is also available with Cygwin, X-Free86. It has also been tested with MCNP5. These client software packages do not need to be the developer or professional versions, since the X11 header and library files are included with the MCNP 5 distribution. 2.5. Installing Parallel Software – MPICH.NT or PVM In order to use or build MCNP 5 with parallel capabilities, appropriate parallel communications software must be installed. Either MPI or PVM communications protocols are supported. To build a parallel version of MCNP 5 for Windows, see the section 3.6. MCNP 5 IMPROVEMENTS FOR WINDOWS PCS American Nuclear Society Topical Meeting in Mathematics & Computations, Gatlinburg, TN, 2003 5/16 The MPI port for Windows is MPICH.NT, developed at Argonne National Laboratory, and can be downloaded from http://www-unix.mcs.anl.gov/~ashton/mpich.nt/. This website also offers the helpful references MPICH.NT FAQ and MPICH Users Manual. If there is no need to rebuild parallel MCNP 5, only the runtime dlls and mpirun package will be needed (mpich.nt.1.2.4.exe). This package uses an InstallShield setup program which requires installation from an administrative account on all PCs in the cluster. If MCNP 5 needs to be built, then the source code (package mpich.nt.1.2.4.src.exe) should be downloaded and unzipped as well. The program MPIConfig must be run on each computer after MPICH.NT installation. The local host name must be added and the settings applied. Once MPICH.NT is installed, a few additional steps are required. MPI enabled MCNP must be copied to the same directory on all hosts. MCNP can be executed through either the Windows MPIRun GUI or command line MPIRun. For the MPIRun GUI, hosts must be added by selecting or typing their names in the hosts section. The DATAPATH may need to be set under the advanced options. For the command line MPIRun, the hosts must be specified by the –hosts option, which can be used to specify the number of processes started on each host. Typically the number of processes is equal to the number of CPUs utilized plus one. The first process listed is the master process, and does not run any histories. When “balance” is specified at the end of the command line, MCNP will load balance its MPI processes. For example, the command to start three MCNP MPI processes on CompA (a dual CPU machine) and one process on CompB is: mpirun –hosts 2 CompA 3 CompB 1 mcnp5mpi inp=test balance Alternatively, MCNP 5 can be built and run with PVM, developed at Oak Ridge National Laboratory. The PVM port for Windows can be downloaded from http://www.csm.ornl.gov/~sscott/PVM/Software/. The file ParallelVirtualMachine3.4.3.zip contains the source and binaries needed to install and run PVM on a single computer. This InstallShield program must be run on all Windows PCs in the cluster. PVM requires additional communications software, a remote shell (RSH) client/server package, before it will run across a cluster of Windows PCs. Two commercial RSH packages can be obtained from http://www.winrshd.com/ and http://www.ataman.com/. The RSH package must be installed on each computer in the cluster, and the permissions must be set to allow RSH or REXEC connections for the desired user accounts. The Ataman RSH package was successfully tested. Similar post- installation steps are required to run the PVM version of MCNP 5 on Windows PCs. The MCNP executable must be copied into the %PVM_ROOT%/pvm3/bin/WIN32 directory on all hosts. PVM must be started on a single computer before a PVM enabled MCNP can be executed. After PVM is started, additional hosts can be added to the PVM cluster, with the “add host” command from the PVM console prompt. PVM operability can be tested and verified with the PVM example programs, such as hello. MCNP can then be started from a separate command shell with the following command: mcnp5pvm inp=test tasks n where n is the number of slave processes. The number of tasks is usually the number of CPUs in the cluster. A negative number entered for n causes MCNP to skip the initial load-balancing T. Goorley, F. Brown, L.J. Cox American Nuclear Society Topical Meeting in Mathematics & Computations, Gatlinburg, TN, 2003 6/16 feature, and is recommended for a homogeneous cluster. Additional information on how to install MPI or PVM can be obtained in their respective user’s manuals. Additional information on running MCNP in parallel can be found in Appendix C, section VIII of the MCNP 5 Manual. 2.6. Setting Environmental Variables After installation, it may be necessary to change or add three Windows environmental variables. They are set differently for Windows 95/NT/2000/XP/ME, but for each of these operating systems the variables can be viewed the same way. The value of an environmental variable PATH, for example, can be printed in a command shell window (i.e. a “DOS prompt”) with the command echo %PATH%. The first environmental variable that may need to be changed is PATH, a semicolon separated list of directories used to find executables and dynamic link libraries. The directory where MCNP is installed should be included, so that MCNP can be executed from any directory, making file management more convenient. If several programs with the same name exist within PATH, then the executable in the first occurring directory will be executed. Appending the directory of the newly installed version of MCNP to the PATH may not change which MCNP is executed if an older version with the same name is given earlier in the directory listing. Failure to change PATH will mean that the MCNP executable must be located in the directory where the input file is located. To use the plott