Using the following workstation computers:
Acer Aspire E700 Intel Core 2 Q6600 Quad 64 Architecture 2.4GHz
Desktop Computer (ASE700-EQ662A) PSU 300W Lite-On
DDR2, PC2-5300 (333 MHz), 2048 MB memory
ATI X1650SE 256MB video card and a 500GB HDD
and having 3 hours, u, x, y, and z.
I booted the existing Vista
Moved data from D: partition to a directory on C:
Selected graphical install by waiting or pressing enter
Selected English OK, us OK, HTTP OK
Enable IPv4 Manual:
Gateway: yyy.yyy.yyy.yyy
Name Server: zzz.zzz.zzz.zzz
OK, Web site name:
CentOS directory for x86_64: /centos/5.5/os/x86_64
OK, be patient... select Next
Selected "Create Custom Layout"
Deleted /dev/sda3
clicked "Back", selected "Review and modify partitioning layout" and "Use free space on selected drives and create default layout".
Doubled /boot providing:
sda1 9993 Mb ntfs (Vista recovery?)
sda2 233609 Mb ntfs (Vista half of drive)
sda3 196 Mb ext3 (boot partition)
sda4 233138 Mb LVM
LVM 4000 Mb swap (double the physical memory)
LVM 229120 Mb / (remaining space)
Next, Next Set hostname, add Secondary DNS uuu.uuu.uuu.uuu
Next Toronto, Next root password
Next Select all but Gnome
Would like to add epel repository but selecting Extras at this point
causes a crash and reboot, thus add epel later
Next, Next, wait an hour... Reboot, eject disk, Forward, select all but
FTP and telnet, Forward, permissive, Forward, Forward, Forward, create your account,
Forward, Forward, Finish, login, system->admin->users->add...
It is possible to use to make sure all centos software is installed during OS install but it also does the partitioning automatically so thus start a terminal as root to do the following:
yum groupinstall -y \
"Administration Tools" \
"Authoring and Publishing" \
Beagle \
"Cluster Storage" \
Clustering \
"DNS Name Server" \
"Development Libraries" \
"Development Tools" \
"Dialup Networking Support" \
Editors \
"Educational Software" \
Emacs \
"Engineering and Scientific" \
"Fedora Packager" \
"FTP Server \
"FreeNX and NX \
"GNOME Desktop Environment" \
"GNOME Software Development" \
"Games and Entertainment" \
"Graphical Internet" \
Graphics \
"Hardware Support" \
Horde \
Java \
"Java Development" \
"KDE (K Desktop Environment)" \
"KDE Software Development" \
"Legacy Network Server" \
"Legacy Software Development" \
"Legacy Software Support" \
"Mail Server" \
Mono \
"MySQL Database" \
"Network Servers" \
"News Server" \
"Office/Productivity" \
"OpenFabrics Enterprise Distribution" \
"PostgreSQL Database" \
"Printing Support" \
Ruby \
"Server Configuration Tools" \
"Sound and Video" \
"System Tools" \
"Text-based Internet" \
Tomboy \
Virtualization \
"Web Development" \
"Web Server" \
"Window Managers" \
"Windows File Server" \
"X Software Development" \
"X Window System" \
"XFCE-4.4" \
"Yum Utilities"
cd /usr/src/redhat/RPMS/x86_64/
wget \
rpm -Uvh epel-release-5-4.noarch.rpm
yum install -y hdf5-devel.x86_64
yum install -y gsl-devel.x86_64
yum install -y atlas-devel.x86_64
yum install -y blas-devel.x86_64
yum install -y guile-devel.x86_64 lapack-devel.x86_64
yum install -y fftw3-devel.x86_64
yum install -y openmpi-devel.x86_64
yum install -y libXp-devel.x86_64
cd /usr/src/redhat/SPECS
wget \
cd /usr/src/redhat/SOURCES
wget \
cd /usr/src/redhat
rpmbuild -v -bb --clean SPECS/harminv.spec
cd /usr/src/redhat/RPMS/x86_64
yum install -y --nogpgcheck harminv-1.3.1-16.x86_64.rpm
yum install -y --nogpgcheck harminv-devel-1.3.1-16.x86_64.rpm
cd /usr/local/src
wget \
gunzip libctl-3.1.tar.gz
tar xf libctl-3.1.tar
cd /usr/local/src/libctl-3.1
make install
cd /usr/local/src
wget \
gunzip meep-1.1.1.tar.gz
tar xf meep-1.1.1.tar
cd /usr/local/src/meep-1.1.1
export \
echo 'export \
PATH="/usr/lib64/openmpi/1.4-gcc/bin:$PATH" ' \
>> /etc/profile
./configure --with-mpi
make install
cd /usr/local/src
wget \
gunzip h5utils-1.12.1.tar.gz
tar xf h5utils-1.12.1.tar
cd /usr/local/src/h5utils-1.12.1
make install
cd /opt
scp -r working_machine:/opt/matlab .
ln -s /opt/matlab/bin/matlab /usr/local/bin/matlab
/sbin/restorecon -r *
export \
echo 'export \
LD_LIBRARY_PATH="/usr/lib64/openmpi/1.4-gcc/lib/:$LD_LIBRARY_PATH" ' \
>> /etc/profile
nano /etc/gconf/gconf.xml.defaults/%gconf-tree.xml

and use ^w to search: user_switch to find and change the following entry from false to true:
Set this to TRUE to offer an option in the unlock dialog to switch to a different user account.
rpm -ivh
yum install -y flash-plugin
yum install -y alpine.x86_64
DISPLAYMANAGER="KDE"' >> /etc/sysconfig/desktop
mkdir /home/user/test
Given several workstations choose the fileserver and add:
echo \
>> /etc/exports
chkconfig --levels 345 nfs on
might still need: 'service nfs start' ???
On the remainder add:
echo \
' /home/user/test nfs rw,hard,intr 0 0' \
>> /etc/fstab
sleep 180
On one of the machines:
echo \
' slots=2 max-slots=4 slots=2 max-slots=4 slots=2 max-slots=4' >> /home/user/test/.my_hostfile
cd /home/user/test
mpicxx -O3 -fstrict-aliasing -march=core2 holey-wvg-cavity.cpp -o \
holey-wvg-cavity.dac -lmeep_mpi -lhdf5 -lz -L/usr/local/lib \
-L/usr/lib/gcc/x86_64-redhat-linux/3.4.6 \
-L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../../../lib64 \
-L/usr/lib/gcc/x86_64-redhat-linux/3.4.6/../../.. -L/lib/../lib64 \
-L/usr/lib/../lib64 -lharminv -llapack -lblas -lfrtbegin -lg2c -lm \
-lgcc_s -lfftw3
nice -n 19 mpirun \
-mca btl ^openib -np 2 -machinefile /home/user/test/.my_hostfile \

works only from a prompt on
hangs if -np >2

Output when it works:
Process 1 on out of 2
Using MPI version 2.1, 2 processes
Process 0 on out of 2
holey-wvg-cavity.cpp starting.
diearray[95][0]: 1.444400
diearray[96][0]: 13.000000
Halving computational cell along direction y
time for set_epsilon = 0.0416429 s
Halving computational cell along direction y
time for set_epsilon = 0.0522659 s
creating output file "./eps-000000.00.h5"...
creating output file "./eps-000000.00.h5"...
volume sources added...
simulating structure without holes...
simulating structure...

Field time usage:
connnecting chunks: 0.0177128 s
time stepping: 1.03118 s
communicating: 0.0948257 s
outputting fields: 0.104485 s
Fourier transforming: 0.451195 s
everything else: 1.49551 s

Field time usage:
connnecting chunks: 0.0268929 s
time stepping: 0.752941 s
communicating: 0.0690033 s
outputting fields: 0.015548 s
Fourier transforming: 0.416427 s
everything else: 0.318057 s

holey-wvg-cavity.cpp finished.

Elapsed run time = 3.32725 s

Most usefull output when it does not work:
Process 2 on out of 3
Using MPI version 2.1, 3 processes
Process 0 on out of 3
holey-wvg-cavity.cpp starting.
diearray[95][0]: 1.444400
diearray[96][0]: 13.000000
Process 1 on out of 3
Halving computational cell along direction y
time for set_epsilon = 0.0256131 s
Halving computational cell along direction y
time for set_epsilon = 0.0328779 s
creating output file "./eps-000000.00.h5"...
[][[37707,1],2][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] connect() to failed: Connection
refused (111)
mpirun: killing job...

mpirun noticed that process rank 2 with PID 8431 on node exited on signal 0 (Unknown signal 0).
3 total processes killed (some possibly by mpirun during cleanup)
mpirun: clean termination accomplished

Perhaps related to

Using the command 'iptables -F' as root to turn off the firewall for risky temporary testing
does allow the prompts on to function however -np >3 still fails.


Look into how TCP can be properly opened between trusted nodes.

'chkconfig --levels 235 httpd on' starts Apache at boot time
nano /etc/httpd/conf/httpd.conf ^w userdir.c
chcon -R -t httpd_sys_content_t /home/user/public_html/

Meep: creating a functioning libctl.spec has been difficult on CentOS,
thus meep-1.0.3.tar.gz was installed without it. Creating a functioning
meep.spec for meep-1.1.1.tar.gz has been difficult on CentOS without
libctl. Thus rpms on hold.

Matlab: we are short on licenses.

Octave: CentOS is compatible with octave-2.9.9-2.fc7.x86_64.rpm but
octave-3.1 or better compatibility is needed to create publication
ready plots.

Ubuntu: More up to date and thus much easier to use the above
software however Cadence is compiled for CentOS.

meep-1.1.1.tar.gz with OpenMPI (need to test inter-node
If you had MPICH2
yum remove mpich2 (removes mpich2, meep, h5utils - if not consider make uninstall and make clean but likely they will not help so reinstalling OS may be necessary - then use above to replace with openmpi, meep, h5utils)

cd /usr/local/src/meep-1.1.1
make check (works properly and all 16 tests are passed but may need
the below)
export \

Path of least resistance: meep-1.1.1 with Open MPI, Matlab on KDE,
LyX with BibTEX or Office 2007 with RefWorks