January 10, 2010

Part 4 of the 5 part series on the step by step installation post focuses on setting up the Sybase cluster. Part 1 has a list and links to the download pages for this software.

The 5 parts of this series are:

The instructions designed that non-Sybase DBA can perform these tasks. Sybase DBA will find that the steps are very similar to your standard ASE installation.

Please click here to read about how to install the Sybase software.

Sybase Setup

Run uafstartup.sh

Before continuing with the cluster install, each node needs to have a running agent waiting for the connection. Without this agent Sybase Cluster Edition will not install nor run.
On asece1 and asece2 as user sybase run this command.

$SYBASE_UA/bin/uafstartup.sh &

This is the out from running the command. Make sure no errors are displayed.

sybase@asece1:~> $SYBASE_UA/bin/uafstartup.sh &
[1] 12142
sybase@asece1:~>
Node: asece1
Operating System: Linux
OS Architecture: x86_64
OS Version: 2.6.16.60-0.21-default
JAVA Runtime: /sybase/shared/jre150_012
java version "1.5.0_12"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_12-b04)
Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_12-b04, mixed mode)
Message level: WARN
Creating a UAF nodes subdirectory representing the node asece1
Agent Home: /sybase/UAF-2_5/nodes/asece1
2008-11-25 15:44:53,500 INFO [main] Starting Unified Agent...
2008-11-25 15:44:53,508 INFO [main] Version: 2.5.0(983)
2008-11-25 15:44:53,705 INFO [main] Initializing BootstrapService...
2008-11-25 15:44:53,705 INFO [main] Starting BootstrapService...
2008-11-25 15:44:54,196 INFO [main] Host: asece1.cluster.local
2008-11-25 15:44:54,199 INFO [main] Agent home: /sybase/UAF-2_5/nodes/asece1
2008-11-25 15:44:54,409 INFO [main] Loading 7 primordial services...
2008-11-25 15:44:54,616 INFO [main] Registered Agent Service
2008-11-25 15:44:55,253 INFO [main] Initializing Agent Service...
2008-11-25 15:44:55,257 INFO [main] Starting Agent Service...
2008-11-25 15:44:55,802 INFO [main] Registered Environment Service
2008-11-25 15:44:56,516 INFO [main] Registered Configuration Service
2008-11-25 15:44:57,229 INFO [main] Registered Session Service
2008-11-25 15:44:57,543 INFO [main] Registered Security Service
2008-11-25 15:44:58,180 INFO [main] Registered Service Registration Service
2008-11-25 15:44:58,218 INFO [main] Initializing Service Registration Service...
2008-11-25 15:44:58,219 INFO [main] Starting Service Registration Service...
2008-11-25 15:44:58,656 INFO [main] Validated 9 extended services.
2008-11-25 15:44:58,656 INFO [main] Check circular dependencies...
2008-11-25 15:44:58,657 INFO [main] OK. No circular dependencies are found.
2008-11-25 15:44:58,837 INFO [main] Initializing RMI Service...
2008-11-25 15:44:58,928 INFO [main] Starting RMI Service...
2008-11-25 15:44:59,588 INFO [main] Agent URL in long format: service:jmx:rmi:///jndi/rmi://asece1.cluster.local:9999/agent
2008-11-25 15:44:59,588 INFO [main] Agent URL in short format: rmi://asece1.cluster.local:9999
2008-11-25 15:44:59,589 INFO [main] TDS Service will not be registered because registerOnStartup is false.
2008-11-25 15:44:59,592 INFO [main] SNMP Service will not be registered because registerOnStartup is false.
2008-11-25 15:44:59,593 INFO [main] DeploymentService: Found 1 dependencies.
2008-11-25 15:44:59,593 INFO [main] Dependent service RemoteShellService needs to be registered...
2008-11-25 15:44:59,809 INFO [main] DeploymentService: successfuly resolved dependencies.
2008-11-25 15:44:59,931 WARN [main] The template directories has been preset to: [/sybase/UAF-2_5/nodes/asece1/templates]
2008-11-25 15:45:00,275 INFO [main] SelfDiscoveryService: Found 1 dependencies.2008-11-25 15:45:00,279 INFO [main] Dependent service RMIService already registered.
2008-11-25 15:45:00,280 INFO [main] SelfDiscoveryService: successfuly resolved dependencies.
2008-11-25 15:45:00,801 INFO [main] Initializing Plugin Registration Service...2008-11-25 15:45:00,801 INFO [main] Starting Plugin Registration Service...
2008-11-25 15:45:00,875 INFO [main] Validated 2 plugin directories.
2008-11-25 15:45:01,238 INFO [main] Plugin registered. Updating lookup info...
2008-11-25 15:45:01,392 WARN [main] The Sybase home directory should be specified in the com.sybase.sysam agent-plugin.xml file.
2008-11-25 15:45:01,786 WARN [main] The Sysam home directory should be specified in the com.sybase.sysam agent-plugin.xml file.
2008-11-25 15:45:01,789 WARN [main] Log file could not be located. This may be that the license manager has not been started
2008-11-25 15:45:02,240 INFO [main] Plugin com.sybase.ase.snmp_2.0.0: Found 2 dependencies.
2008-11-25 15:45:02,419 INFO [main] Validated 9 extended services.
2008-11-25 15:45:02,420 INFO [main] Check circular dependencies...
2008-11-25 15:45:02,420 INFO [main] OK. No circular dependencies are found.
2008-11-25 15:45:02,425 INFO [main] TdsService: Found 1 dependencies.
2008-11-25 15:45:02,425 INFO [main] Dependent service RMIService already registered.
2008-11-25 15:45:02,426 INFO [main] TdsService: successfuly resolved dependencies.
2008-11-25 15:45:02,849 INFO [main] Validated 9 extended services.
2008-11-25 15:45:02,850 INFO [main] Check circular dependencies...
2008-11-25 15:45:02,850 INFO [main] OK. No circular dependencies are found.
2008-11-25 15:45:03,770 INFO [main] Plugin com.sybase.ase.snmp_2.0.0: successfuly resolved dependencies.
2008-11-25 15:45:04,059 INFO [main] Plugin registered. Updating lookup info...
2008-11-25 15:45:04,076 INFO [main] Finished loading primordial services.
2008-11-25 15:45:04,076 INFO [main] Bootstrap completed successfully.
sybase@asece1:~>

Run sybcluster

On the node asece1 execute the cluster creation command. This interactive session will be used to create the configuration file that the can be used to create the cluster with a batch file. This way unforeseen problems can be corrected in the batch file and the cluster creation can restart.

sybase@asece1:~> sybcluster -U uafadmin -P -C mycluster -F "asece1,asece2"

> create cluster

Cluster mycluster – Enter the maximum number of instances: [ 4 ]
How many nodes (hardware hosts) will participate in this cluster: [ 4 ] 2
Verifying the supplied agent specifications…
1) asece1.cluster.local 9999 2.5.0 Linux
2) asece2.cluster.local 9999 2.5.0 Linux
Enter the number representing the cluster node 1: [ 2 ] 1
2) asece2.cluster.local 9999 2.5.0 Linux
Enter the number representing the cluster node 2: [ 2 ] 2
—————— Quorum Device ———————
The quorum device is used to manage a cluster. It contains information shared between instances and nodes.
Enter the full path to the quorum disk: /dev/raw/raw2
Enter any traceflags:
————— Master Database Device —————-
The master database device controls the operation of the Adaptive Server and stores information about all user databases and their associated database devices.
Enter the full path to the master device: /dev/raw/raw1
Enter the size the Master Device (MB): [ 80 ] 180
Enter the size the Master Database (MB): [ 60 ]
Enter the page size in kilobytes: [ 2 ]
———— Sybase System Procedure Device ——–
Sybase system procedures (sybsystemprocs) are stored on a device.
Enter the System Procedure Device path: /dev/raw/raw4
Enter System Procedure Device size (MB): [ 150 ] 180
Enter the System Procedure Database size (MB): [ 135 ]
————– System Database Device ——————
The system database (sybsystemdb) stores information about distributed transactions.
Enter the System Database Device path: /dev/raw/raw5
Enter the System Database Device size (MB): [ 12 ] 80
Enter the System Database size (MB): [ 12 ]
——————————————————–
Does this cluster have a secondary network: [ Y ]
——————————————————–
Enter the SYBASE home directory: [ /sybase ]
Enter the environment shell script path: [ /sybase/SYBASE.sh ]
Enter the ASE home directory: [ /sybase/ASE-15_0 ]
Enter the interfaces directory: [ /sybase ]
Enter path to the dataserver config file: [ /sybase/mycluster.cfg ]
——————————————————–
You will now be asked for the instance information on a node by node basis.

— Cluster: mycluster – Node: asece1.cluster.local – Agent: asece1.cluster.local:9999 —
Enter the name of the cluster instance: ase1
Enter the interface file query port number for instance ase1: 19786
Enter the primary protocol address for ase1: [ asece1.cluster.local ] asece1-ppriv
Enter the secondary protocol address for ase1: [ asece1.cluster.local ] asece1-spriv
————— Local System Temporary Database ———
The Local System Temporary Database Device contains a database for each instance in the cluster.
Enter the LST device name: mycluster1_tempdb
Enter the LST device path: /dev/raw/raw7
Enter LST device size (MB): 1000
Enter the LST database name: [ mycluster_tdb_1 ]
Enter the LST database size (MB): [ 900 ] 900
——————————————————–
Do you want to add another instance to this node? (Y or N): [ N ]
— Cluster: mycluster – Node: asece2.cluster.local – Agent: asece2.cluster.local:9999 —
Enter the name of the cluster instance: ase2
Enter the interface file query port number for instance ase2: 19786
Enter the primary protocol address for ase2: [ asece2.cluster.local ] asece2-ppriv
Enter the secondary protocol address for ase2: [ asece2.cluster.local ] asece2-spriv
————— Local System Temporary Database ———
The Local System Temporary Database Device contains a database for each instance in the cluster.
Enter the LST device name: mycluster2_tempdb
Enter the LST device path: /dev/raw/raw8
Enter LST device size (MB): 1000
Enter the LST database name: [ mycluster_tdb_2 ]
Enter the LST database size (MB): [ 900 ] 900
——————————————————–
Do you want to add another instance to this node? (Y or N): [ N ]
Would you like to save this configuration information in a file? [ Y ] Y
Enter the name of the file to save the cluster creation information: [ /home/sybase/mycluster.xml ]
——————————————————–
Create the cluster now? [ Y ] N

There are certain circumstances when the install will fail. Most of the time it is a size mismatch of the raw devices. If this happens just scale down the raw device size down a little bit. It seems that the size check for the master device causes problems if set to the max size of the raw device.

Once the problem is correct in the .xml file, restart the sybcluster tool and enter the following command.

sybase@asece1:~> sybcluster -U uafadmin -P -C mycluster -F "asece1,asece2"
> create cluster mycluster file /home/sybase/mycluster.xml
During this run answer the questions about reuse the configuration with Y to complete this step.
Here's the output from the batch run:
INFO - Creating the Cluster Agent plugin on host address asece1.cluster.local using agent: asece1.cluster.local:9999
A cluster agent plugin at asece1.cluster.local:9999 is already managing a cluster by the name mycluster.
Should this agent plugin be reused for the cluster: [ N ] y
INFO - The Cluster Agent Plugin on agent asece1.cluster.local:9999 will be reused.
INFO - Creating the Cluster Agent plugin on host address asece2.cluster.local using agent: asece2.cluster.local:9999
A cluster agent plugin at asece2.cluster.local:9999 is already managing a cluster by the name mycluster.
Should this agent plugin be reused for the cluster: [ N ] y
INFO - The Cluster Agent Plugin on agent asece2.cluster.local:9999 will be reused.
Would you like to check whether this device supports IO fencing capability (Y/N)? [ Y ] n
INFO - Cluster "mycluster" creation in progress.
INFO - Choosing the first instance to be created using the connected agent...
INFO - The Sybase home directory is /sybase.
INFO - The ASE home directory is /sybase/ASE-15_0.
INFO - Retrieving environment variables from /sybase/SYBASE.sh.
INFO - The first instance created will be ase1.
INFO - Warning: Unable to verify /dev/raw/raw1 device size. Please verify that this
INFO - device is not already in use and that it has sufficient space available.
INFO - Warning: Unable to verify /dev/raw/raw4 device size. Please verify that this
INFO - device is not already in use and that it has sufficient space available.
INFO - Building Adaptive Server 'ase1':
INFO - Writing entry into directory services...
INFO - Directory services entry complete.
INFO - Building master device...
INFO - Master device complete.
INFO - Starting server...
INFO - Server started.
INFO - Building sysprocs device and sybsystemprocs database...
INFO - sybprocs device and sybsystemprocs database created.
INFO - Running installmaster script to install system stored procedures...
INFO - installmaster: 10% complete.
INFO - installmaster: 20% complete.
INFO - installmaster: 30% complete.
INFO - installmaster: 40% complete.
INFO - installmaster: 50% complete.
INFO - installmaster: 60% complete.
INFO - installmaster: 70% complete.
INFO - installmaster: 80% complete.
INFO - installmaster: 90% complete.
INFO - installmaster: 100% complete.
INFO - installmaster script complete.
INFO - Creating two-phase commit database...
INFO - Two phase commit database complete.
INFO - Installing common character sets (Code Page 437, Code Page 850, ISO Latin-1,
INFO - Macintosh and HP Roman-8)...
INFO - Character sets installed.
INFO - Server 'ase1' was successfully created.
INFO - Adding cluster and instance entries to the interfaces file at /sybase.
INFO - Connecting to the dataserver using the host and query port asece1.cluster.local:19786.
INFO - Creating the Local System Temporary device mycluster1_tempdb at /dev/raw/raw6 of size 1000M.
INFO - Creating the Local System Temporary device mycluster2_tempdb at /dev/raw/raw7 of size 1000M.
INFO - ase1: Creating the Local System Temporary database mycluster_tdb_1 on mycluster1_tempdb of size 1000M.
INFO - ase2: Creating the Local System Temporary database mycluster_tdb_2 on mycluster2_tempdb of size 1000M.
INFO - The cluster is now configured. Shutting down this first instance.
The cluster mycluster was successfully created.
INFO -
>

By default the cluster will be down. To start the cluster enter the following commands.

> connect
mycluster> start cluster
.
.
Startup output messages
.
.

Install Auxiliary Servers

Install a Backup Server

sybase@asece1:~> sybcluster -U uafadmin -P -C mycluster -F "asece1,asece2"

> connect

mycluster> create backupserver
Enter the Backup Server name: [ mycluster_BS ]
Enter Backup Server log file path: [ /sybase/ASE-15_0/install/mycluster_BS.log ]
Do you want to create Backup Server for node "asece1.cluster.local"? [ Y ]
Enter the Backup Server port number for node "asece1.cluster.local": 19799
Do you want to create Backup Server for node "asece2.cluster.local"? [ Y ]
Enter the Backup Server port number for node "asece2.cluster.local": 19799
The Backup Server "mycluster_BS" was successfully defined.
mycluster>

It is not necessary to install the backup server on both nodes. The second node is failover node. Initiating a backup on the cluster edition is the same as with the standard ASE backup. Simply execute the commands dump database or dump tran. To restore the database the same commands as with the standard ASE version is used. Load database will reload a database dump to the system.

Install Job Scheduler

Before installing the job scheduler a device named sybmgmtdev must be created. This device must be shared with all nodes.

sybase@asece1:/sybase/ASE-15_0/scripts> isql -Usa -P -Smycluster -i ./installjsdb

(return status = 0)
Starting the setup of sybmgmtdb.
Using device sybmgmtdev for the sybmgmtdb database.
CREATE DATABASE: allocating 86016 logical pages (168.0 megabytes) on disk
'sybmgmtdev' (86016 logical pages requested).
CREATE DATABASE: allocating 6144 logical pages (12.0 megabytes) on disk
'sybmgmtdev' (6144 logical pages requested).
Caution: You have set up this database to include space on disk 5 for both data
and the transaction log. This can make recovery impossible if that disk fails.
Database option 'trunc log on chkpt' turned ON for database 'sybmgmtdb'.
Running CHECKPOINT on database 'sybmgmtdb' for option 'trunc log on chkpt' to
take effect.
(return status = 0)
Database option 'select into/bulkcopy/pllsort' turned ON for database
'sybmgmtdb'.
Running CHECKPOINT on database 'sybmgmtdb' for option 'select
into/bulkcopy/pllsort' to take effect.
(return status = 0)
Executing checkpoint in sybmgmtdb
Database log is 6144 pages, 30% log free is at 1843 pages, the last chance
threshold is at 448 pages.
installing sp_js_logthreshold at 1843 pages.
Adding threshold for segment 'logsegment' at '1843' pages.
DBCC execution completed. If DBCC printed error messages, contact a user with
System Administrator (SA) role.
(return status = 0)
Created js_jobs.
Created js_commands.
Created js_schedules.
Created js_scheduledjobs.
Created js_seqsql.
Created js_callouts.
Created js_history.
Created js_output.
Created js_keys.
Created js_templates.
Created js_xml.
Created js_daynames.
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 1)
(return status = 0)
Setup of sybmgmtdb is done.
Password correctly set.
Account unlocked.
New login created.
(return status = 0)
Account locked.
(return status = 0)
Authorization updated.
(return status = 0)
Authorization updated.
(return status = 0)
Option changed.
(return status = 0)
Option changed.
(return status = 0)
New user added.
(return status = 0)
Installing Job Scheduler Task js_seqsql sequencer code
: js_seqsql installed token 0 - BOOTSTRAP FIRMWARE
: js_seqsql installed token 1 - PROBE JS_CALLOUTS
: js_seqsql installed token 2 - PROBE JS_HISTORY
: js_seqsql installed token 3 - CLEAN JS_CALLOUTS
: js_seqsql installed token 4 - POPULATE JS_CALLOUTS
: js_seqsql installed token 5 - GET CALLOUT SIZE
: js_seqsql installed token 6 - LOAD CALLOUT
: js_seqsql installed token 7 - LOAD JS_SCHEDULEDJOBS
: js_seqsql installed token 8 - GET JS_SCHEDULES
: js_seqsql installed token 9 - SET JSC_START
: js_seqsql installed token 10 - GET MAX JSH_EXID
: js_seqsql installed token 11 - GET SYB_JSAGENT
: js_seqsql installed token 12 - SET RUNNABLE
: js_seqsql installed token 13 - SET COMPLETED
: js_seqsql installed token 14 - SET TERMINATED
: js_seqsql installed token 15 - SET EXPIRED
: js_seqsql installed token 16 - EXEC JSAGENT
: js_seqsql installed token 17 - EXEC DIAGJSAG
: js_seqsql installed token 18 - INSERT CALLOUT
: js_seqsql installed token 19 - DELETE CALLOUT
: js_seqsql installed token 20 - ADD JSTASK
: js_seqsql installed token 21 - UPDATE CALLOUT
: js_seqsql installed token 22 - SET JSC_STATE
: js_seqsql installed token 23 - CLEAN JSH_STATE
: js_seqsql installed token 24 - EXPIRE SCHEDULE
: js_seqsql installed token 25 - UNLOCK JSTASK
: js_seqsql installed token 26 - CHECK WAKEUP
: js_seqsql installed token 27 - VERIFY CALLOUT
Sequencer code installed in js_seqsql ...
Created sp_js_check_id
Created sp_js_dates2bits
Created sp_js_bits2dates
Created sp_js_days2bits
Created sp_js_bits2days
Created sp_js_getkey
Created sp_js_putkey
Created sp_js_checksched
Created sp_js_properties
Created sp_createjob
Created sp_createschedule
Created sp_createscheduledjob
Created sp_bindscheduledjob
Created sp_unbindscheduledjob
Created sp_addjobtext
Created sp_dropjobtext
Created sp_sjobmodify
Created sp_sjobcreate
Created sp_sjobcmd
Created sp_sjobcopy
Created sp_sjobcontrol
Created sp_sjobsetstatus
Created sp_sjobdrop
Created sp_sjobhelp
Created sp_sjobhistory
Created sp_modifyscheduledjob
Created sp_modifyschedule
Created sp_dropscheduledjob
Created sp_dropjob
Created sp_dropschedule
Created sp_copyjob
Created sp_copyschedule
Created sp_listscheduledjob
Created sp_listjob
Created sp_listschedule
Created sp_listjobtext
Created sp_controljob
Created sp_js_agent
Created sp_listjobhistory
Created sp_dropjobhistory
Created sp_dropjoboutput
Created sp_readjoboutput
Created sp_writejoboutput
Created sp_createjobtemplate
Created sp_modifyjobtemplate
Created sp_readjobtemplate
Created sp_writejobtemplate
Created sp_dropjobtemplate
Created sp_listjobtemplate
Created sp_sjobclean
Created sp_js_history_thresh
Database data is 86016 pages, installing free data threshold 8601 pages,
sp_js_history_thresh
Adding threshold for segment 'default' at '8601' pages.
DBCC execution completed. If DBCC printed error messages, contact a user with
System Administrator (SA) role.
(return status = 0)
Loading of sybmgmtdb database is complete.
(return status = 0)
sybase@asece1:/sybase/ASE-15_0/scripts>

Install a XP Server

sybase@asece1:~> sybcluster -U uafadmin -P -C mycluster -F "asece1,asece2"

> connect
mycluster> create xpserver
Enter the XP server port number for instance “ase1”: 19798
Enter the XP server port number for instance “ase2”: 19798
The XP Server was successfully defined for each instance.
mycluster>

Install a Monitor Server

sybase@asece1:~> sybcluster -U uafadmin -P -C mycluster -F "asece1,asece2"

> connect
mycluster> create monitorserver
Enter the Monitor server log file location for instance “ase1”: [ /sybase/ASE-15_0/install/ase1_MS.log ]
Enter the Monitor server port number for instance “ase1”: 19797
Enter the Monitor server log file location for instance “ase2”: [ /sybase/ASE-15_0/install/ase2_MS.log ]
Enter the Monitor server port number for instance “ase2”: 19797
Enter a username: sybmon
Enter a password:
mycluster>

Check The Interfaces File

After installing all the auxiliary servers, the interfaces file will look like this:

sybase@asece1:~> cat /sybase/interfaces

ase1
master tcp ether asece1.cluster.local 19786
query tcp ether asece1.cluster.local 19786

ase2
master tcp ether asece2.cluster.local 19786
query tcp ether asece2.cluster.local 19786

mycluster
query tcp ether asece1.cluster.local 19786
query tcp ether asece2.cluster.local 19786

mycluster_BS
master tcp ether asece1.cluster.local 19799
query tcp ether asece1.cluster.local 19799
master tcp ether asece2.cluster.local 19799
query tcp ether asece2.cluster.local 19799

ASE1_XP
master tcp ether asece1.cluster.local 19798
query tcp ether asece1.cluster.local 19798

ASE2_XP
master tcp ether asece2.cluster.local 19798
query tcp ether asece2.cluster.local 19798
sybase@asece1:~>

Test the cluster

Login to the cluster to execute a view commands.

sybase@asece1:~> sybcluster -U uafadmin -P -C mycluster -F "asece1,asece2"

> connect
mycluster> start cluster
.
.
Startup output messages
.
.
mycluster> show cluster status
INFO – Listening for the cluster heartbeat. This may take a minute. Please wait… (mycluster::AseProbe:434)

Id Name Node State Heartbeat
— —- —————– —– ———
1 ase1 asece1.cluster.local Up Yes
2 ase2 asece2.cluster.local Up Yes
— —- —————– —– ———

This is the easiest way to check on the status of the cluster.

mycluster> show cluster config

** Cluster configuration for “mycluster” **
Interface Path “/sybase”
Trace Flags:
There were are no trace flags.
Maximum Instances “4”
Quorum “/dev/raw/raw2”
Master Device “/dev/raw/raw1”
logfile ase1 /sybase/ASE-15_0/install/ase1.log
run_parameters ase1
logfile ase2 /sybase/ASE-15_0/install/ase2.log
run_parameters ase2
Primary Interconnect “udp”
Server[1] ase1 asece1-ppriv 15100 15119
Server[2] ase2 asece2-ppriv 15120 15139
Secondary Interconnect “udp”
Server[1] ase1 asece1.spriv 15181 15200
Server[2] ase2 asece2.spriv 15201 15220
mycluster>

This completes the Sybase ASE 15 Cluster Edition installation examples. There is one more step to walk you through. This will explain how to add an additional node to an existing cluster.