All posts by admin

Ganglia

Taken from http://ganglia.sourceforge.net/.

“Ganglia is a scalable distributed monitoring system for high-performance computing systems such as clusters and Grids. It is based on a hierarchical design targeted at federations of clusters. It leverages widely used technologies such as XML for data representation, XDR for compact, portable data transport, and RRDtool for data storage and visualization. It uses carefully engineered data structures and algorithms to achieve very low per-node overheads and high concurrency. The implementation is robust, has been ported to an extensive set of operating systems and processor architectures, and is currently in use on thousands of clusters around the world. It has been used to link clusters across university campuses and around the world and can scale to handle clusters with 2000 nodes.”

Tools required to build Ganglia

The main tools that are required is as follows:-

  • freetype2
  • libart-2.0
  • libpng
  • rrdtool
  • ganglia

For web based monitoring, PHP is required.

Building Ganglia on OSX

Here is my build configuration for the various tools stated above. I have previously faced difficulty bulding some of these tools in the past using older versions and on thePowerPC? platform. However, when i tried again recently, things seems to be able to build without much problems. The versions and configuration options that i used is as follows:-

  • freetype2 (2.3.4)
    • ./configure –prefix=/usr/local
  • libart-2.0 (2.3.17)
    • ./configure –prefix=/usr/local
  • libpng (1.2.16)
    • ./configure –prefix=/usr/local
  • rrdtool (1.2.19)
    • export LDFLAGS=”-L/usr/local/lib”
    • export CPPFLAGS=”-I/usr/local/include/freetype2 -I/usr/local/include/libart-2.0″
    • ./configure –prefix=/usr/local –disable-ruby –disable-perl –disable-python –disable-tcl
  • ganglia (3.0.4)
    • ./configure –prefix=/usr/local –with-gmetad

Note that the accompanying configuration file for gmond can be obtained by running the following:-

gmond -t > gmond.conf

A default gmetad.conf file is found within the gmetad directory in the ganglia source.

On boot starting on OSX

Here are the example files for launchd starting of Ganglia.

Ganglia launchd startup files

Showing the number of SGE jobs on Ganglia

It is useful to try to push information to be charted into Ganglia so that you can have one simple interface to look at all the various information related to your cluster. Here is a simple script that allows one to be able to see the number of SGE jobs that are in the queue and the number of SGE jobs that are running.

Save the below as a script file and add it into your CRON job to get these information into Ganglia. Simple

#!/bin/sh

GMETRIC="/usr/bin/gmetric"
source /etc/bashrc

# Might have to change path to reflect your SGE install.. 
QSTAT=`qstat | grep -v "$(qstat | head -n2)" | wc -l`
$GMETRIC --name SGEJOBS --type uint16 --units jobs --value $QSTAT

QSTAT=`qstat | grep -v "$(qstat | head -n2)" | grep @ | wc -l`
$GMETRIC --name SGEJOBS_RUNNING --type uint16 --units jobs --value $QSTAT

Setting up DNS on Mac OSX Server

Setting up the DNS service on Mac OSX is really simple. In short, it is using the Server Admin tool and clicking on the DNS selection. A summary of the steos required is as follows.

  1. Login to the system with the admin username and password
  2. Select the “Serveradmin” application in your “Applications” folder.
  3. Make sure that you have your server marked and available in the left plane. If not, select “Add Server” and include the relevant parameters
  4. Select the “DNS” entry and select “Settings” somewhere near the bottom
  5. Create a new zone and enter the following
    • zone name : The domain name you wish the DNS to be a part of
    • Server name : DNS hostname
    • Server IP : Local IP address of the DNS machine
  6. Select “Machines” and add the machine IP addresses and names into the table provided.
    • NOTE: if you have a large number of machines that you need to input into the DNS tables, there is a small “file” icon on the bottom right which you can drag and place in your desktop. This will be a plist file which you can then open by using “Texedit”.
    • Once you have “exported” the .plist file, you can edit the file by hand or via a script such that you can enter many compute nodes at a time. Doing so will be faster for a large cluster of nodes.
    • You can re-import this by dragging the plist file back into the DNS settings window
    • NOTE2: if you want to do a zone transfer or zone forwarding, you need to add entries into the “Secondary Zone” for the queries to be forwarded.
  7. Start the DNS service by selecting start on the top bar
  8. Start the terminal application and ensure that your network interface is activated and up. (use ifconfig or something)
  9. Use “host” or “nslookup” to check for both forward and reverse name resolutions

One thing to note.. unlike linux where you can set up DNS without having a network connection, in MAC, you seem to actually NEED to have a wire connected, so that the interface is brought up, BEFORE you can test the DNS. So ensure that you actually have a physical ethernet connection before you test the DNS.

Open Directory (OD)

Open Directory is essentially Apples way of handling large scale authentication, system and user management. It is based off LDAPv3 and integrates with a variety of other authentication protocols such as:-

  • flat files
  • active directory
  • NIS
  • other LDAP servers

It also provides a way to easily setup Kerberosv5. Unlike normal uses of LDAP systems which comprises mainly of Address Books, as well as normal Authentication, Apple uses the very same system to also store user configurations, system setup preferences, mount point and configuration information and many more. While this makes LDAP a little slow due to the large number of records, it provides a very standard way of being able to access the data off any system. It is therefore possible to map LDAP entries from the Mac into any other system.

The LDAP based configuration also allows administrators to be able to login from any point in the network to perform management if required to do so. Replication and synchronization is also possible.

Open Directory Setup and Replication notes

Setting OD is relatively straight forward and doesn’t require too much configuration if the defaults are to be used. Unless you want to setup Kerberos, simply selecting OD setup within Server Admin tool is enough to get the OD up and running. Setting up replication of OD servers is also a simple case of selecting the option in Server Admin.

Changing passwords for users in the terminal can be done with either

  • passwd, or
  • kpasswd

I have also come across web based password changing applications such as

Of the above, i prefer the former to the latter as it does not use Web Objects.

In either case, the password will be synced going through the password service. If replication is setup, it can take as long as 8 minutes or even longer before the passwords can be synced to all the replicas. In some of the testings that is done by customers, they have found that the optimal number of replicas for a tolerable update time (perhaps 5-8 minutes) is 3 replicas. Anything more than that seems to take a much longer time to replicate. The log files are located in:-

  • /Library/Logs/PasswordService
  • /Library/Logs/slapdconfig.log

These two files will provide alot of clues on the activities of OD as well as issues if they should arise.

Some other K-base articles includes

Oversights during a OD tear down

As everything is directory dependent, I have made some mistakes during OD tear down resulting in weird configurations which needs cleaning up. Mainly, the job would be to review the environment and to remove all Directory configurations of share points before the teardown begins. Not doing so will result in many rouge entries which you cannot clean up after.

However, if it so happens that there are rouge entries, after reverting the OD master to standalone configuration, you can use Workgroup Manager tool, and set to “show all”. Following that, check the “Share Points” directory data and remove unwanted entries.

Also remember to check up the actual IP addresses of the various replicas and remove them first. Take note of the exact interface that the replication is going through in the log files. I have encountered that depending solely on the Server Admin tool’s report of IP can be wrong.

Re-Importing Users

Within the Server Admin tool, there is a archive and restore ability which can be used to backup the OD settings. While i have successfully tried that out in a classroom environment, i have yet to try it in a live environment.

Within the Workgroup Manager tool, there is another ability to export users. However, the exporting of these users will cause everyone to lose their passwords. Whiel i have so far been able to re-import the users list successfully, there has always been problems after. These problems includes:-

  • the directory admin (diradmin) losing its ability to login to the domain from workgroup manager
  • users not being able to login after re-import

Directory Admin being unable to login

In this case, there seems to be some error during the import that completely knocks out the original setup diradmin password. The way to prevent this is by:-

  1. During the export of the user list from the original OD, ensure that the diradmin entry is NOT exported
  2. If you don’t have the luxury to re-export, edit the exported file and examin the entries. Remove the block of entires that corresponds to the diradmin user.

However, you will find that even after that, it is still not possible to login from the Workgroup Manager tool. Don’t worry, the diradmin password is still correct, but somehow, it can’t login. To get around this, use the root user name and its password to login to the Workgroup Manager tool. Once authenticated, login to the directory by selecting the “lock” icon near the top right, and use the diradmin username and password. You will find that you will be authenticated and logged in. From there, you will be able to make all the required changes to the directory as per normal.

I have no idea why you have to use the root user to login to Workgroup Manager initially, but the above method works for me.

Note: Remember to reboot the machine when changes are made to the directory as the config settings are picked up only during boot time.

Users unable to login after re-import

This happens as the user list is exported without passwords. One will find that under “Advanced” in Workgroup Manager, all the user passwords will be “Crypt Passwords”. All that needs to be done is to reset them to “Open Directory” Passwords and everything will be ok.

I have heard of a tool called Passenger from http://macinmind.com/. But i have yet to be able to use it successfully.

Once done, the user should be able to login to CLI and change passwords as per normal. However, like mentioned before, if many replicas are used, it will take some time for all the passwords to sync up.

Proper OD migration for any upgrade activities

Apparently, the best way to migrate the OD to perform any form of upgrade activity is to do the following:-

  1. Switch off the OD master (server A) and bring it off the network
  2. Promote one of the replicas ( server B) to OD master
  3. Set server A to stand alone and perform the upgrade of server A
  4. Re-attach server A to the network and set it up as a replica to server B
  5. Switch off server B and bring it off the network
  6. Re-promote server A to OD Master
  7. Set server B as stand alone and perform the upgrade on server B
  8. Re-attach server B to the network and set it up as a replica to server A
  9. and so on and so forth….

Apple Lights Out Management

Apple introduced what was termed as Lights Out Management when they released their new Intel Based Servers. This is essentially nothing new when Intel based Server boards are concerned. I suppose this is just a different term to what is known as IPMI.

Functionality

LOM allows one to be able to both monitor hardware status of the server, but it also allows one to be able to power on and power off the hardware remotely. This means if the server is down, one would be able to cold power on the hardware. This functionality is very similar to the Remote Service Adapter in IBM.

Configurations

The configuration of LOM sits within Server Monitor provided by the Server Admin tools that is installed with either the XServe or the Admin tools package within the Server DVD. All one has to do is to launch the Server Monitor application, and select the Server menu item, followed by the Configure Local Machine item. Once this is done, a dialog with IP address settings is presented. Configure these with the desired IP addresses and the user name and password to access the LOM. Once that is done, Appleand exit.

Do note that there is no extra Ethernet interface that is attached to the server. This means that perhaps a virtual Ethernet port is created that uses the existing 2 Ethernet port connected on-board the server.

Quirks

There are several irregularities that i have observed when using LOM.

  1. A local machine cannot monitor itself using the defined LOM IP address. But 127.0.0.1 will work. But perhaps this is related to the point below.
  2. The IP address defined within the LOM setting would have to be setup such as a monitoring machine will be able to access its network parameters
    • This means that if you have a machine that is meant to do management, a separate network is required on top of the typical IP network.
    • This is in exception to the fact IF you setup all the IP addresses reachable within the access network to be in the same network range as the actual Ethernet ports. For example, if your actual Ethernet configuration is setup as 10.0.0.1/255.255.255.0 and your LOM network ip is set to be 10.0.0.101/255.255.255.0, This is fine (but not a suggested configuration due to security).