XSAN File System

March 29, 2010 admin

The XSAN filesystem is Apples version of a file system that can be used in a SAN environment. It is apparently based off ADIC StorNext product before. While the idea of a SAN filesystem is typically used in an enterprise environment for storage consolidation, the fact that one has a single storage back end that is accessible in the same way from any node make the same idea good for clusters. However, there are many ways that this can be achieved. Sometimes, via NFS, SAN or Parallel File systems.

XSan Filesystem – The original document form TWiki

Performance Tuning on SNFS

There is quite a fair bit of information in the StorNext file system tuning guide which can be useful. It is attached below for reference.

SNFS Tuning Guide

Apple, Opensource

Ganglia

March 29, 2010 admin

Taken from http://ganglia.sourceforge.net/.

“Ganglia is a scalable distributed monitoring system for high-performance computing systems such as clusters and Grids. It is based on a hierarchical design targeted at federations of clusters. It leverages widely used technologies such as XML for data representation, XDR for compact, portable data transport, and RRDtool for data storage and visualization. It uses carefully engineered data structures and algorithms to achieve very low per-node overheads and high concurrency. The implementation is robust, has been ported to an extensive set of operating systems and processor architectures, and is currently in use on thousands of clusters around the world. It has been used to link clusters across university campuses and around the world and can scale to handle clusters with 2000 nodes.”

Tools required to build Ganglia

The main tools that are required is as follows:-

freetype2
libart-2.0
libpng
rrdtool
ganglia

For web based monitoring, PHP is required.

Building Ganglia on OSX

Here is my build configuration for the various tools stated above. I have previously faced difficulty bulding some of these tools in the past using older versions and on thePowerPC^? platform. However, when i tried again recently, things seems to be able to build without much problems. The versions and configuration options that i used is as follows:-

freetype2 (2.3.4)
- ./configure –prefix=/usr/local
libart-2.0 (2.3.17)
- ./configure –prefix=/usr/local
libpng (1.2.16)
- ./configure –prefix=/usr/local
rrdtool (1.2.19)
- export LDFLAGS=”-L/usr/local/lib”
- export CPPFLAGS=”-I/usr/local/include/freetype2 -I/usr/local/include/libart-2.0″
- ./configure –prefix=/usr/local –disable-ruby –disable-perl –disable-python –disable-tcl
ganglia (3.0.4)
- ./configure –prefix=/usr/local –with-gmetad

Note that the accompanying configuration file for gmond can be obtained by running the following:-

gmond -t > gmond.conf

A default gmetad.conf file is found within the gmetad directory in the ganglia source.

On boot starting on OSX

Here are the example files for launchd starting of Ganglia.

Ganglia launchd startup files

Showing the number of SGE jobs on Ganglia

It is useful to try to push information to be charted into Ganglia so that you can have one simple interface to look at all the various information related to your cluster. Here is a simple script that allows one to be able to see the number of SGE jobs that are in the queue and the number of SGE jobs that are running.

Save the below as a script file and add it into your CRON job to get these information into Ganglia. Simple

#!/bin/sh

GMETRIC="/usr/bin/gmetric"
source /etc/bashrc

# Might have to change path to reflect your SGE install.. 
QSTAT=`qstat | grep -v "$(qstat | head -n2)" | wc -l`
$GMETRIC --name SGEJOBS --type uint16 --units jobs --value $QSTAT

QSTAT=`qstat | grep -v "$(qstat | head -n2)" | grep @ | wc -l`
$GMETRIC --name SGEJOBS_RUNNING --type uint16 --units jobs --value $QSTAT

Apple

Setting up DNS on Mac OSX Server

March 29, 2010 admin

Setting up the DNS service on Mac OSX is really simple. In short, it is using the Server Admin tool and clicking on the DNS selection. A summary of the steos required is as follows.

Login to the system with the admin username and password
Select the “Serveradmin” application in your “Applications” folder.
Make sure that you have your server marked and available in the left plane. If not, select “Add Server” and include the relevant parameters
Select the “DNS” entry and select “Settings” somewhere near the bottom
Create a new zone and enter the following
- zone name : The domain name you wish the DNS to be a part of
- Server name : DNS hostname
- Server IP : Local IP address of the DNS machine
Select “Machines” and add the machine IP addresses and names into the table provided.
- NOTE: if you have a large number of machines that you need to input into the DNS tables, there is a small “file” icon on the bottom right which you can drag and place in your desktop. This will be a plist file which you can then open by using “Texedit”.
- Once you have “exported” the .plist file, you can edit the file by hand or via a script such that you can enter many compute nodes at a time. Doing so will be faster for a large cluster of nodes.
- You can re-import this by dragging the plist file back into the DNS settings window
- NOTE2: if you want to do a zone transfer or zone forwarding, you need to add entries into the “Secondary Zone” for the queries to be forwarded.
Start the DNS service by selecting start on the top bar
Start the terminal application and ensure that your network interface is activated and up. (use ifconfig or something)
Use “host” or “nslookup” to check for both forward and reverse name resolutions

One thing to note.. unlike linux where you can set up DNS without having a network connection, in MAC, you seem to actually NEED to have a wire connected, so that the interface is brought up, BEFORE you can test the DNS. So ensure that you actually have a physical ethernet connection before you test the DNS.

Apple

Open Directory (OD)

March 29, 2010 admin

Open Directory is essentially Apples way of handling large scale authentication, system and user management. It is based off LDAPv3 and integrates with a variety of other authentication protocols such as:-

flat files
active directory
NIS
other LDAP servers

It also provides a way to easily setup Kerberosv5. Unlike normal uses of LDAP systems which comprises mainly of Address Books, as well as normal Authentication, Apple uses the very same system to also store user configurations, system setup preferences, mount point and configuration information and many more. While this makes LDAP a little slow due to the large number of records, it provides a very standard way of being able to access the data off any system. It is therefore possible to map LDAP entries from the Mac into any other system.

The LDAP based configuration also allows administrators to be able to login from any point in the network to perform management if required to do so. Replication and synchronization is also possible.

Open Directory Setup and Replication notes

Setting OD is relatively straight forward and doesn’t require too much configuration if the defaults are to be used. Unless you want to setup Kerberos, simply selecting OD setup within Server Admin tool is enough to get the OD up and running. Setting up replication of OD servers is also a simple case of selecting the option in Server Admin.

Changing passwords for users in the terminal can be done with either

passwd, or
kpasswd

I have also come across web based password changing applications such as

OSXPass (http://sourceforge.net/project/showfiles.php?group_id=102051)
MOSXSWebPassword (http://ps-enable.com/software/MOSXSWebPassword.dmg/view)

Of the above, i prefer the former to the latter as it does not use Web Objects.

In either case, the password will be synced going through the password service. If replication is setup, it can take as long as 8 minutes or even longer before the passwords can be synced to all the replicas. In some of the testings that is done by customers, they have found that the optimal number of replicas for a tolerable update time (perhaps 5-8 minutes) is 3 replicas. Anything more than that seems to take a much longer time to replicate. The log files are located in:-

/Library/Logs/PasswordService
/Library/Logs/slapdconfig.log

These two files will provide alot of clues on the activities of OD as well as issues if they should arise.

Some other K-base articles includes

Oversights during a OD tear down

As everything is directory dependent, I have made some mistakes during OD tear down resulting in weird configurations which needs cleaning up. Mainly, the job would be to review the environment and to remove all Directory configurations of share points before the teardown begins. Not doing so will result in many rouge entries which you cannot clean up after.

However, if it so happens that there are rouge entries, after reverting the OD master to standalone configuration, you can use Workgroup Manager tool, and set to “show all”. Following that, check the “Share Points” directory data and remove unwanted entries.

Also remember to check up the actual IP addresses of the various replicas and remove them first. Take note of the exact interface that the replication is going through in the log files. I have encountered that depending solely on the Server Admin tool’s report of IP can be wrong.

Re-Importing Users

Within the Server Admin tool, there is a archive and restore ability which can be used to backup the OD settings. While i have successfully tried that out in a classroom environment, i have yet to try it in a live environment.

Within the Workgroup Manager tool, there is another ability to export users. However, the exporting of these users will cause everyone to lose their passwords. Whiel i have so far been able to re-import the users list successfully, there has always been problems after. These problems includes:-

the directory admin (diradmin) losing its ability to login to the domain from workgroup manager
users not being able to login after re-import

Directory Admin being unable to login

In this case, there seems to be some error during the import that completely knocks out the original setup diradmin password. The way to prevent this is by:-

During the export of the user list from the original OD, ensure that the diradmin entry is NOT exported
If you don’t have the luxury to re-export, edit the exported file and examin the entries. Remove the block of entires that corresponds to the diradmin user.

However, you will find that even after that, it is still not possible to login from the Workgroup Manager tool. Don’t worry, the diradmin password is still correct, but somehow, it can’t login. To get around this, use the root user name and its password to login to the Workgroup Manager tool. Once authenticated, login to the directory by selecting the “lock” icon near the top right, and use the diradmin username and password. You will find that you will be authenticated and logged in. From there, you will be able to make all the required changes to the directory as per normal.

I have no idea why you have to use the root user to login to Workgroup Manager initially, but the above method works for me.

Note: Remember to reboot the machine when changes are made to the directory as the config settings are picked up only during boot time.

Users unable to login after re-import

This happens as the user list is exported without passwords. One will find that under “Advanced” in Workgroup Manager, all the user passwords will be “Crypt Passwords”. All that needs to be done is to reset them to “Open Directory” Passwords and everything will be ok.

I have heard of a tool called Passenger from http://macinmind.com/. But i have yet to be able to use it successfully.

Once done, the user should be able to login to CLI and change passwords as per normal. However, like mentioned before, if many replicas are used, it will take some time for all the passwords to sync up.