Posts tagged ·



TIBCO Rendezvous and MS NLB Cluster

Comments Off on TIBCO Rendezvous and MS NLB Cluster

TIBCO Rendezvous is multicast-based messaging. Network Load Balancing (NLB) is a way to configure a pool of machines so they take turns responding to requests. It’s commonly implemented in server farms: identically configured machines that spread out the load for a web site or work as terminal services cluster.

Task was to cross both of these things – Rendezvous based application on servers in MS NLB terminal services cluster. I’ve done some tests using different settings, but the result was an inappropriate. I received RV messages only on one server or one message on the first server, next message on second, and so on, it depend on “Filtering mode”. NLB for multicast packets works even better than I would like! But users of an application work on every server and need all messages delivered to all users on all servers.

What happens with every frame that the Network Load Balancing driver (wlbs.sys) receives is:

  1. on every node wlbs.sys checks if the received packet is send to a virtual IP
  2. on every node wlbs.sys checks the source IP and port
  3. one node decides to accept the packet and passes it up to the TCP/IP driver
  4. all other nodes drop the packet

The issue is that there is no special treatment for multicast IPs. NLB driver treats them like every other IP that is not the dedicated IP of that machine.

What are the possible solutions?

  • Receive the IP multicast traffic over a NIC where no NLB is bound to. Additional NIC in every server.
  • Use TCP connection to remote Rendezvous daemon (rvd). Daemon parameter in RV transport: -daemon "tcp:remotemachine:7500"
  • Use local Rendezvous routing daemon (rvrd) instead of rvd. It requires rvrd on every terminal server and additional rvrd somewhere in the network.

If you would like read more, here is the list of clustering and high availability cluster resources from MS.

Comments Off on TIBCO Rendezvous and MS NLB Cluster

Using TIBCO EMS with Storage Foundation Cluster File System

1 comment

Here is the article on Symantec site. I can agree that installing TIBCO Enterprise Message Service (EMS) on Veritas Storage Foundation Cluster File System (CFS) is the best high availability and high performance solution. If your company really need 99.999% uptime, 50000 messages per second and ready to pay much for that.

Whenever we need to choose the most suitable solution, we must consider the various options. For example, if the SLA is required to ensure very high uptime, but not expected tens of thousands of messages per second, you can consider EMS options using NFS or HA Database for messages store. In the opposite case can be encouraged to use the failover cluster with shared cluster disk.

Installing TIBCO on Windows Failover Cluster – part 4

no comments

Continued from part 1, part 2, part 3. Now I ready to add my virtual cluster host TST00-TIB01 to TIBCO Domain. I will start DomainUtility (D:\tibco\tra\5.6\bin\domainutility.exe) on the first (Master) node and select Machine Management => Add Machine. Then change machine name to cluster resource name TST00-TIB01, specify TIBCO EMS tcp://TST00-TIB01:7223 as a transport, enter admin credentials for domain and for EMS, and press “Discover..”. My domain TIB01 will be discovered. Then mark “Machine is Logical” checkbox and enter its virtual IP address as it is in cluster. After confirmation, virtual host TST00-TIB01 will be in the domain and new Hawk Agent service will be created. Now I can start this service.

In the TIBCO Administrator new machine will appear, but first node shown as disconnected. It is happened as Hawk Agent which is responsible for this node was uninstalled and replaced by new TST00-TIB01 Agent. I will fix this later. Now I will initialize TIBCO EMS plugin to manage both EMS in Administrator.

To do that I will start DomainUtility again and select TIBCO EMS Plugin => Add TIBCO EMS Server. Then I will select domain and enter admin credentials, specify machine, where is my EMS is running, provide information about EMS itself. On the confirmation screen I can test connection to the EMS to be sure that all provided info is OK. As everything was fine and server was successfully added to domain, I will do the same for the second EMS. After that, both EMS shown in the Administrator as running service instances and now is possible to manage them using TIBCO Administrator GUI.

As my process is going well, I can start moving TST00-TIB01 Hawk Agent from services on the master node to cluster resources. First of all I will stop this service and copy whole content of D:\tibco\tra\domain\TIB01 folder to T:\tibco\tra\domain\TIB01 folder. Then it is necessary to correct all paths related to TST00-TIB01 in hawkagent_TIB01.tra file. Also check and replace related paths in hawkagent.cfg file.

This Hawk Agent application should be added as a cluster resource. To do that, I will open Cluster Management Console and select Add a resource => Generic Application, and enter path to the Agent binary. After confirmation, new resource will be created. In its properties, I will mark “Use network name for computer name” checkbox and define dependences: cluster volume where Agent’s files are, IP address and binded name, EMS 7223 as it provides service transport for this agent also. To continue, bring this Agent online. TST00-TIB01 status is OK now in TIBCO Administrator.

On the next step is special wrapped application has to be installed in cluster for both EMS. It will be used by Hawk Agent to query EMS status and display in TIBCO Administrator. Path to binary file and parameters you can see on the screenshot. Next, in properties I will mark “Use network name for computer name” checkbox and define dependences: hawkagent_TIB01 and EMS Server 7222. Then I will install second wrapper for second EMS: accordingly propFile will be TIBCOServers-EMS_7223.tra and in dependences will be same hawkagent_TIB01 and EMS Server 7223.

After all, everything has to be green, this is final picture for cluster.

Last step is restore Hawk Agent on the first (Master) node. I will start DomainUtility (D:\tibco\tra\5.6\bin\domainutility.exe) again on the first node and select Machine Management => Add Machine. Machine name should be S-TST01-TIB01 now. If transport and credentials are valid, domain TIB01 will be discovered. After confirmation, Hawk Agent on this server will be replaced back to original for the first node. I will start this service, then both nodes and cluster name should be green in TIBCO Administrator.

My newly created TIBCO Domain is ready to serve TIBCO BusinessWorks applications that can be installed as fault tolerant pairs on both nodes. Everything should work fine in this configuration in case any one node fails.

Installing TIBCO on Windows Failover Cluster – part 3

no comments

Continued from part 1, part 2. TIBCO Domain Creation. In my configuration I’d like to use TIBCO EMS as domain transport and store domain information in MS SQL Instance. Let’s start from installation of two EMS instances in cluster.

As I said, in my TIBCO Domain will be two EMS instances: one for production data, listen on default TCP port 7222, and second for service domain transport on TCP port 7223. On the node where my cluster service group TST00-TIB01 exists, I clone default EMS configuration from D:\tibco\tibco\cfgmgmt\ems\data to cluster disk into two folders: T:\tibco\ems\7222\data and T:\tibco\ems\7223\data. Actually, only one file is necessary: tibemsd.conf. All other configuration files will be created in paths specified in tibemsd.conf when EMS will be started for the first time. I need to correct paths in tibemsd.conf to correspond with current location on cluster disk. To specify path here use slash (/) instead of backslash (\). Important thing is setup log file and trace items for future EMS monitoring. Good idea to specify max size of logfile to enable log rotation. Trace items I prefer are: DEFAULT, +CONNECT, +DEST.

For second EMS instance files will be located in T:\tibco\ems\7223\data, I will set the paths accordingly. But one more things here: it is necessary to change listener TCP port to 7223 as it is will be second EMS instance on the same host.

Next step is remove existing TIBCO EMS Server and TIBCO EMS Multicast Daemon from services on the both nodes:
emsntsrg.exe /r tibemsd
emsntsrg.exe /r tibemsmcd
And then register two new instances also on the both nodes:
emsntsrg.exe /i tibemsd D:\tibco\ems\5.1\bin D:\tibco\ems\5.1\bin "-config T:\tibco\ems\7222\data\tibemsd.conf" 7222
emsntsrg.exe /i tibemsd D:\tibco\ems\5.1\bin D:\tibco\ems\5.1\bin "-config T:\tibco\ems\7223\data\tibemsd.conf" 7223

System services were created, so I can add these services as cluster resources. Put them under the cluster control, which will be responsible for their execution on one of two nodes.

In the cluster manager I will select my group TST00-TIB01 and click Add a resource => Generic Service. Then wizard will be started, where I can select one of my EMS service. Just confirm it then and it will be successfully created and configured as cluster resource.

But I need to tune this, I will change properties. Necessary to use network name for computer name for this service. And to add dependencies from network resources and disk.

Now it’s time to do the same for EMS 7223, bring both online and check log files to be sure that everything is up and running good. Default admin user on EMS is ‘admin’ with empty password. Next I will connect to each EMS using EMS Administration Tool and set new password for admin:
connect TST00-TIB01
set password admin [password]

Now I will continue to create new TIBCO domain on primary (Master) server. Domain Utility window still open from TIBCO Administrator installation. I will select master and press “Next” to continue. I have to enter new domain name, let it be TIB01. Then I will select UTF-8 encoding. Next, I mark “Show Advanced” checkbox and select TIBCO EMS as my transport. URL to connect is tcp://TST00-TIB01:7223 as I will use this EMS instance for domain transport. Also I will specify my EMS credentials.

TIBCO Administrator is web application running under Tomcat web server. On the next screen we can change default HTTP port 8080 to something else. For me default port is OK as it’s free on my nodes. Next screen is place to define administrator credentials for new domain: username and password twice. Remember well! It is possible to change it later but not very easy. Also it is possible to configure password policy here. Next screen to provide database configuration information. This DB will be used to store whole domain information. Next step is the final confirmation before domain creation. After that, TIBCO Administrator will be configured and new TIBCO Domain will be created. Two system services will be also created: one is TIBCO Administrator and second one is TIBCO Hawk Agent. I will go to Services and start both.

On the other host I will install secondary TIBCO Administrator server. To do that just point to “Secondary” in TIBCO Domain Utility. Then I need mark “Show Advanced” checkbox, and then specify TIBCO EMS as a transport and enter its URL. Then press “Discover”. Just created on the first node domain should be discovered. Next is necessary to provide the administrator credentials in order to add the machine to domain. After confirmation second server will join domain and two services will be created: Administrator and Hawk Agent. It’s time to start both.

Now I can connect to the web interface to see the result. After login, two my hosts will be shown as domain members. In the part 4 I will describe how to add virtual cluster host TST00-TIB01 and both EMS into this domain.

Installing TIBCO on Windows Failover Cluster – part 2

no comments

1-2-1Continued from part 1. The following installation process must be done on the both nodes. The first step when you installing TIBCO SOA products is TIBCO Runtime Agent (TRA) installation. As TIBCO doesn’t have x64 TRA yet, I have to use 32 bit installer on my 64 bit system: TIB_tra-suite_5.6.0_win_x86_32.exe. At this time, 5.6 is the latest minor release of TRA.

After welcome screen, I will scroll down and accept license agreement, next read notes and select custom install. List of products included in the TRA package will appear on the screen. You may suggest that on production server is no need to install designer, but it is not so, I’m faced with the dependence of certain components, such as TIBCO SmartMapper, from its designer libraries. My suggestion is install full suite. Next step is specify default installation home that will be used for all TIBCO products. I prefer to use “D:\tibco” folder. Some next steps allow me to install documentations for components and the last screen before files copying is installation preview and confirmation.

After all files are copied, will be offered the opportunity to install an additional database JDBC driver. You can use this opportunity to install an additional driver or do it later manually like me. After that installation summary screen will appear and installation will be finished.

Now it is time to install TRA patch release. The latest is 5.6.1, file is TIB_tra-suite_5.6.1_win_x86.exe. Patch release is not includes all TRA files, so it must be installed after latest minor release. Installation looks very similar to TRA installation, but is is update only: next, next, next, finish.

Installing TIBCO ActiveMatrix BusinessWorks is not more difficult than installing Runtime Agent. At this time minor version is 5.7, file name is TIB_bw-simple_5.7.0_win_x86.exe. After welcome screen, I will scroll down and accept license agreement, read notes and look at detected an established installation home – “D:\tibco” in my case. Custom setup type helps me make sure I install all the components. Installation preview screen do the same thing. After short files copying process TIBCO BusinessWorks installation will be finished.

Installing patch release 5.7.1 is even easier. File is TIB_bw-simple_5.7.1_win_x86.exe. Next, accept, next, next, next, finish.

As x64 EMS is available, I will use latest 64 bit version 5.1.3 in archive. To install TIBCO EMS from version 5 new TIBCOUniversalInstaller will be used. By the way, this new installer is also used to install TIBCO Rendezvous from version 8 if you need install it as complete standalone product not as limited TRA part. Looks like TIBCO decided to switch all the products in this new installer. Package distributed as zip file and size increased twice compare to old binary installer package. For me old installer was lighter than new. Fundamental change in the new installer is that now we can specify different installation environments and it helps to install and manage many instances of same product on one host.

Then I will select custom installation as usual to install full set. If you install TIBCO EMS on MS Windows platform, installer will install EMS as a system service. Next question will be concerning service startup type: manual or auto. Let it be manual, in fact it doesn’t matter in my case as I will remove this default service and install it later a little differently. On the next screen pre-install summary will be displayed and after press to “install” button TIBCO EMS installation process will be completed.

The last main product to install is TIBCO Administrator. The latest version at this time is 5.6.0 and I will install from TIB_tibcoadmin-epe-simple_5.6.0_win_x86_32 as it only 32 bit version available. Installation is very simple: accept license agreements, read notes, look at gray read-only installation folder name as it is old installer again, select custom type and leave administrator binaries and its documentation checked. After confirmation, installation process will be completed.

TIBCO Domain Utility will be started automatically. Do not close this window, I will continue with domain creation in the part 3.

Installing TIBCO on Windows Failover Cluster – part 1


1-1-0This is divided into four parts description of installation process of TIBCO domain on Windows 2008 R2 Failover Cluster. The aim of this is to get running on two hosts fault-tolerant TIBCO domain which contain two TIBCO EMS servers. Each TIBCO EMS server is running as cluster application and use cluster disk for data store. This solution helps to get fault-tolerant TIBCO EMS design and there is no need to have shared volumes, which is expensive solution and not available on the Windows platform yet (CSV is recently appeared exclusion made for Hyper-V). The final configuration shown in the TIBCO Administrator screenshot on the right, where is S-TST01-TIB01 is the first node and S-TST02-TIB01 is the second node. TIBCO Administrator is running on each node as master/secondary pair. Test application is running on the same hosts as fault-tolerant group in active/standby mode. But the most interesting part of domain is TST00-TIB01 – cluster service which acting as virtual host where both EMS servers live in TIBCO domain. I will explain from the beginning.

1-1-1First of all I’m going to prepare two nodes cluster. Base software is MS Windows Server 2008 R2 Enterprise Edition. Here are two disk connected to both cluster nodes. In my case storage system connected to servers via iSCSI interface. Two volumes were prepared: one small 5G will be act as cluster quorum disk, second large 50G will be used for clustered application data store. Both are raw volumes at this time.

1-1-2I’m going to put both volumes online and initialize it. If you will do it like me, do it only on one node! After this step I can create NTFS file system on quorum disk. Start “New Simple Volume Wizard”, let it to use whole disk, assign drive letter, I prefer “Q” for quorum, then perform a quick format with default settings.

1-1-3I suppose that both my nodes are ready for failover cluster. But the best practice to check it before. So, I will start cluster validation process. Here a lot of steps for testing cluster core resources availability such as network and storages. During testing, access to shared volumes will be checked from both nodes. After completion of testing, validation report will be generated. In my case warning message was received as my network adapter has no redundancy: between my servers only one network path and it will be a single point of failure. In production cluster it is better to avoid things like this, I will correct this before going to production.

1-1-4As all other things were good, I’m ready to create failover cluster. I just start “Create Cluster Wizard” and add the names of all the servers that I want to have in the cluster. Then type the cluster name. After confirmation, cluster creation and configuration process will be running. When cluster will be formed, summary message and final report will be generated. Everything looks very good in my case, so I’m ready to prepare storage for cluster data disk. In future, it will be used as storage for TIBCO EMS and TIBCO Hawk. The steps are the same as for quorum disk: “New Simple Volume Wizard”, drive letter assign, let it be “T” for TIBCO, NTFS quick format.

1-1-5After that my storage is available as cluster resource and I’m prepared to create cluster resources that will be used for TIBCO services. I will start with “Create Empty Service or Application” action, then rename this new service to TST00-TIB01 and add client access point with the same name: TST00-TIB01. IP address will be assigned automatically by DHCP. Then it’s time to add storage to this service. I will select my available Cluster Disk 1.

1-1-6 Last step is bring this service group online. At this point of time both nodes are ready for TIBCO software installation. I will continue from this point in the part 2.