Posts tagged ·



Problem with libeay32.dll and ssleay32.dll


You may experience this problem, when can’t run TIBCO Administrator, Hawk, BW process or any other TIBCO component. In log files you can find messages related to OpenSSL libraries libeay32.dll and ssleay32.dll. The reason is that TIBCO messaging components use OpenSSL, but other applications may also use OpenSSL and have already installed these dlls in C:\WINDOWS\System32 or C:\WINDOWS\SysWOW64 depending on the platform.

When you start the application, it unsuccessful attempts to find dll files near the binary, then tries to find in the System32 folder. But dlls from other version of OpenSSL are there. Your application will use it, instead correct version from TIBCO Rendezvous bin folder for example, even you have this folder in the PATH. TIBCO product can’t call the necessary functions, gives an error and stops working.

Solution is take libeay32.dll and ssleay32.dll from TIBCO Rendezvous bin folder to the bin folders of every installed TIBCO component. Or just copy and replace to C:\WINDOWS\System32 or C:\WINDOWS\SysWOW64.

Update: these libraries are needed for SSL communication using secured daemons (RVSD or RVSRD) only, if you are not using RVSD or RVSRD then just remove the tibrvjsd.jar file from the classpath.

TIBCO Rendezvous and MS NLB Cluster

Comments Off on TIBCO Rendezvous and MS NLB Cluster

TIBCO Rendezvous is multicast-based messaging. Network Load Balancing (NLB) is a way to configure a pool of machines so they take turns responding to requests. It’s commonly implemented in server farms: identically configured machines that spread out the load for a web site or work as terminal services cluster.

Task was to cross both of these things – Rendezvous based application on servers in MS NLB terminal services cluster. I’ve done some tests using different settings, but the result was an inappropriate. I received RV messages only on one server or one message on the first server, next message on second, and so on, it depend on “Filtering mode”. NLB for multicast packets works even better than I would like! But users of an application work on every server and need all messages delivered to all users on all servers.

What happens with every frame that the Network Load Balancing driver (wlbs.sys) receives is:

  1. on every node wlbs.sys checks if the received packet is send to a virtual IP
  2. on every node wlbs.sys checks the source IP and port
  3. one node decides to accept the packet and passes it up to the TCP/IP driver
  4. all other nodes drop the packet

The issue is that there is no special treatment for multicast IPs. NLB driver treats them like every other IP that is not the dedicated IP of that machine.

What are the possible solutions?

  • Receive the IP multicast traffic over a NIC where no NLB is bound to. Additional NIC in every server.
  • Use TCP connection to remote Rendezvous daemon (rvd). Daemon parameter in RV transport: -daemon "tcp:remotemachine:7500"
  • Use local Rendezvous routing daemon (rvrd) instead of rvd. It requires rvrd on every terminal server and additional rvrd somewhere in the network.

If you would like read more, here is the list of clustering and high availability cluster resources from MS.

Comments Off on TIBCO Rendezvous and MS NLB Cluster

Security advisories for TIBCO products

no comments

Yesterday TIBCO announced vulnerability in TIBCO Runtime Agent (TRA). To be more specific, in TIBCO Domain Utility (domainutility and domainutilitycmd). To say even more specifically, vulnerability is that the local users (whether they are on your server?) have read access to the properties files where administration domain credentials are stored. Not in clear text, by the way. Here is the advisory.

Here is the list of all security advisories for TIBCO products.

I like TIBCO.

TIBCO Hawk NoClassDefFoundError issue

1 comment

On the Windows platform you can experience java.lang.NoClassDefFound Error when starting Hawk Agent or Hawk Display. In my example I use TIBCO EMS as a transport for Hawk messaging and issue arose after upgrade EMS from version 4 to version 5 on the server. The reason is that in EMS version 5.1 paths to java class libraries are: <Tibco_Root>\ems\5.1\lib\jms.jar; <Tibco_Root>\ems\5.1\lib\tibjms.jar; <Tibco_Root>\ems\5.1\lib\tibrvjms.jar; <Tibco_Root>\ems\5.1\lib\tibcrypt.jar;
In EMS 4.x. paths were: <Tibco_Root>\ems\clients\java\jms.jar; <Tibco_Root>\ems\clients\java\tibjms.jar; <Tibco_Root>\ems\clients\java\tibrvjms.jar; <Tibco_Root>\ems\clients\java\tibcrypt.jar; <Tibco_Root>\ems\clients\java\jaxp.jar; and they remained in the Hawk configuration.

On windows this configuration is stored in the registry. Just open regedit and modify three classpath strings under HKLM\SOFTWARE\Tibco Software\TIB/Hawk\<version>.

For TIBCO Rendezvous transport situation can be the same, if path to used jar <Tibco_Root>\tibrv\lib\tibrvj.jar; was changed.

About classpath and Hawk. In general, NoClassDefFoundError is a Java (JVM) error that occurs when a class needed to run a Java program cannot be found. Here are Hawk is Java program and classes (a set of dynamically loadable libraries that Java applications can call at runtime) in the jar files mentioned above. Classpath is an argument that tells the JVM where to look for user-defined classes and packages in Java programs.

On Unix in the startagent startup script for TIBCO Hawk Agent for example, you can find something like this:
# If EMS_ROOT is set, add EMS jars files.
if [ ! -z "$EMS_ROOT" ]; then
# Add jar files for RV.
# Add existing CLASSPATH environment variable to class path.

Here are the same classes and classpath variable. So, please keep in mind this.

What is TIBCO EMS?


Remark: this is a brief overview from the admin perspective. If you ask a developer or architect, then their views may differ greatly from what you find below :)

TIBCO Enterprise Message Service (EMS) is fully compliant Java Message Service (JMS) implementation from TIBCO with some enterprise-class enhancements. What is it? In general, from JMS FAQ:

The Java Message Service makes it easy to write business applications that asynchronously send and receive critical business data and events.

The Java Message Service defines a common enterprise messaging API that is designed to be easily and efficiently supported by a wide range of enterprise messaging products.

The Java Message Service supports both message queuing and publish-subscribe styles of messaging (topics).

It is main part of Enterprise Backbone, Enterprise Middleware and Enterprise SOA. Unlike TIBCO Rendezvous, where publishers and subscribers communicate directly without server, EMS represent dedicated server, hub which connects all clients and passes through itself all messages.

Better to see once than hear a hundred times. Installation process for EMS server is very simple, I slightly described installation on Windows platform in this post. On Solaris or other *nix we can run installer with [-console] option if X11 isn’t configured:
bash-3.2# ./TIBCOUniversalInstaller-sol-sparc.bin -console

New TIBCO Universal Installer will store configuration files and message storages separately from binaries and allows to have multiple environments on the same host, you must specify both paths. EMS is not required any additional components like TIBCO Runtime Agent (TRA), everything is included in the archive.

To start EMS manually with output to console, just execute tibemsd or tibemsd64 (depend on platform) with option [-config] and path to tibemsd.conf file. Later in production it will run as a system service on Windows (install or remove service using emsntsrg utility) or as a daemon in Unix. Start process from console is also good for debugging purposes, if an error somewhere in the configuration files and service isn’t running.

All EMS configuration stored in the configuration files and these files are read when the EMS process going up. Main file is tibemsd.conf: it’s contain service name, listening TCP port, links to other configuration files, logging options and etc. If start EMS without specifying tibemsd.conf file, it will try to find it near binary, if unsuccessful then conf files will be created near binary with default values.

The most of EMS configuration, like new user, new queue or bridge, performed using administration tools on the live system and become active immediately, no restart required. Then changes saved in the corresponding conf files to be restored when you restart the service. But it is also possible to modify conf files manually. Moreover, some parameters, like message storages location or log file name must be predefined in conf files and EMS restart is necessary. Before each change make sure to have fresh backup of the configuration files!

For EMS administration tasks some tools are available: tibemsadmin – command line administration tool provided with EMS, EMS plugin for TIBCO Administrator, Gems (Graphical Administration Tool for EMS) by Richard Lawrence, HermesJMS. Using these tools admin can manage topics, queues, bridges, users and so on. For monitoring EMS offers many options for logging and trace. Also, admin can subscribe to system topics beginning with “$sys.monitor.” for live evens monitoring (easiest using tibemsmonitor utility).

Files used to store messages will be created on the first start using parameters in the stores.conf file (before version 5 in tibemsd.conf). In the normal operational mode, messages may accumulate in the topics and queues if no recipients – files will grow when needed, and therefore it is necessary to continuously monitor, otherwise the service may become unusable. It is possible to predefine minimal size of those files, it will take some time to build files for the first time if predefined size is large, but help to avoid fragmentation. Shrink or truncate large files to predefined minimum is also possible. When EMS restarts, all persistent messages will be recovered, but it will take some time to recover if files are large. In general, when you upgrade EMS from 4.x to 5.x, all stores will be upgraded automatically. Downgrade or rollback is also possible using tibemsdb5revert.

To provide high availability, two EMS servers can run as active-standby fault-tolerant pair. The main requirement of this configuration is the simultaneous access to store files – Cluster File System is required. Veritas Storage Foundation Cluster File System as expensive enterprise solution example. Some variants with network shares or NFS are also possible but guaranteed uptime and messages rate can be much lower. Alternative approach – failover cluster with shared volume.

Starting with EMS version 5 it became possible to use a database to store the messages. This simplifies the creation of fault-tolerant pair – no need to create a shared file system, enough to configure two servers to the same database. So far I haven’t collected a sufficient pro and cons, if you have such please share in the comments.

And lastly few words about connecting clients. Usually, when your application must be integrated into TIBCO middleware, means that you need communicate to EMS only. The most of SOA oriented applications are ready to communicate with JMS, Java clients can use JMS classes. TIBCO ActiveMatrix BusinessWorks has JMS palletes. Applications must be able to work with the fault-tolerant pair of two servers, provide reconnection in case of connection failure, support authentication.

Useful links:

How to change admin password in TIBCO Administrator


If you need to change password for admin user in TIBCO Administrator please keep in mind that it is not enough just to change the admin password from Administrator GUI. But whole procedure is not so complex.

  1. Always have full backup of Administrator configuration files and repository files or database!
  2. Change admin password in the Administrator GUI.
  3. Start DomainUtility on each machine in the TIBCO Domain, select Server Settings => Update Domain Credentials and change password there or use domainutilitycmd and ChangeDomainCredentials.xml as a template.
  4. Make sure that password has been changed in tibcoadmin_<domain>.tra file. Or you can put new password manually there like this repo.securePassword=#!tibco and use obfuscate utility to encrypt it.
  5. Restart Administrator daemon and Hawk Agents everywhere in TIBCO domain.

That’s all. But, if you change password in the Administrator GUI only, and Administrator service (on the Windows) or daemon (on the Unix) has been restarted as nothing is working as before, then you can find picture like this on your screen when you try to login into Administrator. If your daemon has been started using nohup utility as mine, then you have a chance to find little more in the nohup.out file:
com.tibco.infra.repository.RepoSecurityException: Can not read policy domain for repository server HM : Failed in authentication.

If you start DomainUtility at this time and try to change password there, following error will appear.

To solve this problem you need to disable security option in the tibcoadmin_<domain>.tra file

then restart Administrator.

Now it is possible to continue the procedure from step 3: DomainUtility will work. When password will be changed, you can enable security back in tra file. Do not forget to restart Administrator daemon and Hawk Agents everywhere in the domain!

After all, redeployment of all applications may be necessary.

TIBCO Designer issue on Ubuntu 9.10 (Karmic Koala)

1 comment

After successful upgrade Ubuntu Linux from 9.04 (Jaunty Jackalope) to 9.10 (Karmic Koala) on my laptop, I’ve found that my TIBCO Designer is no longer works here:

./designer: error while loading shared libraries: cannot open shared object file: No such file or directory

I’ve found that isn’t available in Karmic repos anymore but still in Jaunty here: You can download and install it:

sudo dpkg -i libstdc++5_3.3.6-17ubuntu1_i386.deb

After that Designer runs great on Karmic.