Archive for

January, 2010

...

How to add Solaris 10 server into MS Active Directory domain

15 comments

Here are my notes applicable for Solaris 10. First of all install latest patches – a lot of related things fixed (but new bugs may appear :))

  1. Synchronize the system clock with AD server
    domain ntp server(s) must be in /etc/inet/ntp.conf
    then restart ntp daemon svcadm restart /network/ntp
  2. Solaris server must have a record in the DNS
  3. Domain name and name servers (DNS servers) must be in /etc/resolv.conf
  4. In the /etc/nsswitch.conf file dns and files must be specified for hosts and ipnodes
    ...
    hosts: dns files
    ipnodes: dns files
    ...
  5. In the /etc/nodename and /etc/hostname.<nic> files host name must be specified only, not a fully qualified domain name
  6. Run adjoin script. You can find it here. It will:
    • auto-detects the Active Directory domain controllers
    • creates a machine account (also called a Computer object) for the Solaris host in Active Directory and generates a random password for this account
    • configures the Solaris host as a Kerberos client of the Active Directory domain controller by using the /etc/krb5/krb5.conf file
    • configures the /etc/krb5/krb5.keytab file on the Solaris host by using the keys for the machine account (also called host credentials)

    Execute adjoin script with following options:
    ./adjoin -d <domain_name> -p <administrator_principal> -f -x
    where -f to delete any pre-existing computer account for this host and -x to debug output.

    If your domain if geographically distributed with a lot of domain controllers (DC), script can detect inappropriate controllers. Just before entering admin password, check prepared adjoin-krb5.conf.XXXXXX file in the /tmp folder and remove unnecessary controllers from it.

    Adjoin script can stop work with pkcs11_kernel.so syntax error on some SUN servers:
    + ./adjoin[859]: /usr/lib/security/$ISA/pkcs11_kernel.so:: syntax error
    Then all you need is just to temporary rename this file and execute adjoin again
    mv /usr/lib/security/$ISA/pkcs11_kernel.so /usr/lib/security/$ISA/pkcs11_kernel.so.orig
    when adjoin finished successfully, rename it back

  7. Run ldapsearch and klist to check Kerberos
    ldapsearch -R -T -h dc1.xxxxxx.com -o authzid= -o mech=gssapi -b CN=Computers,DC=xxxxxx,DC=com -s sub cn=<computer_name>
    klist
    klist -e -k /etc/krb5/krb5.keytab
  8. Enable dns client and cache daemons
    svcadm enable /network/dns/client
    svcadm enable /system/name-service-cache
  9. In the /etc/nsswitch.ldap file dns and files must be specified for hosts and ipnodes
    ...
    hosts: dns files
    ipnodes: dns files
    ...
  10. Set up a server as a client of an LDAP. Execute ldapclient
    ldapclient -v manual \
    -a credentialLevel=self \
    -a authenticationMethod=sasl/gssapi \
    -a defaultSearchBase=dc=xxxxxx,dc=com \
    -a defaultSearchScope=sub \
    -a domainName=xxxxxx.com \
    -a defaultServerList="dc1.xxxxxx.com dc2.xxxxxx.com dc3.xxxxxx.com" \
    -a attributeMap=passwd:gecos=cn \
    -a attributeMap=passwd:homedirectory=unixHomeDirectory \
    -a objectClassMap=group:posixGroup=group \
    -a objectClassMap=passwd:posixAccount=user \
    -a objectClassMap=shadow:shadowAccount=user \
    -a serviceSearchDescriptor="passwd:ou=Accounts,ou=European office,dc=xxxxxx,dc=com?sub;ou=Accounts,ou=American Office,dc=xxxxxx,dc=com?sub" \
    -a serviceSearchDescriptor=group:ou=Groups,dc=xxxxxx,dc=com?sub

    ldapclient should finish without errors. To check use ldapclient list
  11. Edit the /etc/nsswitch.conf file: files and ldap must be specified for passwd and group only
    ...
    passwd: files ldap
    group: files ldap
    hosts: dns files
    ipnodes: dns files
    networks: files
    protocols: files
    ...

    remove ldap from everywhere else
  12. Restart LDAP client
    svcadm restart /network/ldap/client
  13. Add pam_krb5.so.1 in the /etc/pam.conf file
    ...
    login auth sufficient pam_krb5.so.1
    krlogin auth required pam_krb5.so.1
    krsh auth required pam_krb5.so.1
    ktelnet auth required pam_krb5.so.1
    other auth sufficient pam_krb5.so.1
    other account required pam_krb5.so.1
    other password sufficient pam_krb5.so.1
    ...

To ensure that users could login on the host under their AD accounts, accounts in AD must have following additional attributes:
uid the same as sAMAccountName
uidNumber unique number
gidNumber number
unixHomeDirectory for example /tmp
loginShell for example /usr/bin/bash or /bin/false

To check it use getent or ldapsearch
getent passwd <uid>
ldapsearch -R -T -h dc1.xxxxxx.com -b "ou=Accounts,ou=American Office,dc=xxxxxx,dc=com" -o mech=gssapi -o authzid='' "uid=<uid>"

If you would like read more: link to SUN’s article “Using Kerberos to Authenticate a Solaris 10 OS LDAP Client With Microsoft Active Directory”.

TIBCO Rendezvous and MS NLB Cluster

Comments Off

TIBCO Rendezvous is multicast-based messaging. Network Load Balancing (NLB) is a way to configure a pool of machines so they take turns responding to requests. It’s commonly implemented in server farms: identically configured machines that spread out the load for a web site or work as terminal services cluster.

Task was to cross both of these things – Rendezvous based application on servers in MS NLB terminal services cluster. I’ve done some tests using different settings, but the result was an inappropriate. I received RV messages only on one server or one message on the first server, next message on second, and so on, it depend on “Filtering mode”. NLB for multicast packets works even better than I would like! But users of an application work on every server and need all messages delivered to all users on all servers.

What happens with every frame that the Network Load Balancing driver (wlbs.sys) receives is:

  1. on every node wlbs.sys checks if the received packet is send to a virtual IP
  2. on every node wlbs.sys checks the source IP and port
  3. one node decides to accept the packet and passes it up to the TCP/IP driver
  4. all other nodes drop the packet

The issue is that there is no special treatment for multicast IPs. NLB driver treats them like every other IP that is not the dedicated IP of that machine.

What are the possible solutions?

  • Receive the IP multicast traffic over a NIC where no NLB is bound to. Additional NIC in every server.
  • Use TCP connection to remote Rendezvous daemon (rvd). Daemon parameter in RV transport: -daemon "tcp:remotemachine:7500"
  • Use local Rendezvous routing daemon (rvrd) instead of rvd. It requires rvrd on every terminal server and additional rvrd somewhere in the network.

If you would like read more, here is the list of clustering and high availability cluster resources from MS.

Comments Off

Winter in Russia

1 comment

Security advisories for TIBCO products

no comments

Yesterday TIBCO announced vulnerability in TIBCO Runtime Agent (TRA). To be more specific, in TIBCO Domain Utility (domainutility and domainutilitycmd). To say even more specifically, vulnerability is that the local users (whether they are on your server?) have read access to the properties files where administration domain credentials are stored. Not in clear text, by the way. Here is the advisory.

Here is the list of all security advisories for TIBCO products.

I like TIBCO.

TIBCO Hawk NoClassDefFoundError issue

no comments

On the Windows platform you can experience java.lang.NoClassDefFound Error when starting Hawk Agent or Hawk Display. In my example I use TIBCO EMS as a transport for Hawk messaging and issue arose after upgrade EMS from version 4 to version 5 on the server. The reason is that in EMS version 5.1 paths to java class libraries are: <Tibco_Root>\ems\5.1\lib\jms.jar; <Tibco_Root>\ems\5.1\lib\tibjms.jar; <Tibco_Root>\ems\5.1\lib\tibrvjms.jar; <Tibco_Root>\ems\5.1\lib\tibcrypt.jar;
In EMS 4.x. paths were: <Tibco_Root>\ems\clients\java\jms.jar; <Tibco_Root>\ems\clients\java\tibjms.jar; <Tibco_Root>\ems\clients\java\tibrvjms.jar; <Tibco_Root>\ems\clients\java\tibcrypt.jar; <Tibco_Root>\ems\clients\java\jaxp.jar; and they remained in the Hawk configuration.

On windows this configuration is stored in the registry. Just open regedit and modify three classpath strings under HKLM\SOFTWARE\Tibco Software\TIB/Hawk\<version>.

For TIBCO Rendezvous transport situation can be the same, if path to used jar <Tibco_Root>\tibrv\lib\tibrvj.jar; was changed.

About classpath and Hawk. In general, NoClassDefFoundError is a Java (JVM) error that occurs when a class needed to run a Java program cannot be found. Here are Hawk is Java program and classes (a set of dynamically loadable libraries that Java applications can call at runtime) in the jar files mentioned above. Classpath is an argument that tells the JVM where to look for user-defined classes and packages in Java programs.

On Unix in the startagent startup script for TIBCO Hawk Agent for example, you can find something like this:
# If EMS_ROOT is set, add EMS jars files.
if [ ! -z "$EMS_ROOT" ]; then
JARFILE="$JARFILE:$EMS_ROOT/clients/java/jms.jar"
JARFILE="$JARFILE:$EMS_ROOT/clients/java/tibjms.jar"
JARFILE="$JARFILE:$EMS_ROOT/clients/java/tibrvjms.jar"
JARFILE="$JARFILE:$EMS_ROOT/clients/java/tibcrypt.jar"
fi
# Add jar files for RV.
JARFILE="$JARFILE:$RV_ROOT/lib/tibrvj.jar"
# Add existing CLASSPATH environment variable to class path.
JARFILE="$JARFILE:$CLASSPATH"

Here are the same classes and classpath variable. So, please keep in mind this.

What is TIBCO EMS?

4 comments

Remark: this is a brief overview from the admin perspective. If you ask a developer or architect, then their views may differ greatly from what you find below :)

TIBCO Enterprise Message Service (EMS) is fully compliant Java Message Service (JMS) implementation from TIBCO with some enterprise-class enhancements. What is it? In general, from JMS FAQ:

The Java Message Service makes it easy to write business applications that asynchronously send and receive critical business data and events.

The Java Message Service defines a common enterprise messaging API that is designed to be easily and efficiently supported by a wide range of enterprise messaging products.

The Java Message Service supports both message queuing and publish-subscribe styles of messaging (topics).

It is main part of Enterprise Backbone, Enterprise Middleware and Enterprise SOA. Unlike TIBCO Rendezvous, where publishers and subscribers communicate directly without server, EMS represent dedicated server, hub which connects all clients and passes through itself all messages.

Better to see once than hear a hundred times. Installation process for EMS server is very simple, I slightly described installation on Windows platform in this post. On Solaris or other *nix we can run installer with [-console] option if X11 isn’t configured:
bash-3.2# ./TIBCOUniversalInstaller-sol-sparc.bin -console

New TIBCO Universal Installer will store configuration files and message storages separately from binaries and allows to have multiple environments on the same host, you must specify both paths. EMS is not required any additional components like TIBCO Runtime Agent (TRA), everything is included in the archive.

To start EMS manually with output to console, just execute tibemsd or tibemsd64 (depend on platform) with option [-config] and path to tibemsd.conf file. Later in production it will run as a system service on Windows (install or remove service using emsntsrg utility) or as a daemon in Unix. Start process from console is also good for debugging purposes, if an error somewhere in the configuration files and service isn’t running.

All EMS configuration stored in the configuration files and these files are read when the EMS process going up. Main file is tibemsd.conf: it’s contain service name, listening TCP port, links to other configuration files, logging options and etc. If start EMS without specifying tibemsd.conf file, it will try to find it near binary, if unsuccessful then conf files will be created near binary with default values.

The most of EMS configuration, like new user, new queue or bridge, performed using administration tools on the live system and become active immediately, no restart required. Then changes saved in the corresponding conf files to be restored when you restart the service. But it is also possible to modify conf files manually. Moreover, some parameters, like message storages location or log file name must be predefined in conf files and EMS restart is necessary. Before each change make sure to have fresh backup of the configuration files!

For EMS administration tasks some tools are available: tibemsadmin – command line administration tool provided with EMS, EMS plugin for TIBCO Administrator, Gems (Graphical Administration Tool for EMS) by Richard Lawrence, HermesJMS. Using these tools admin can manage topics, queues, bridges, users and so on. For monitoring EMS offers many options for logging and trace. Also, admin can subscribe to system topics beginning with “$sys.monitor.” for live evens monitoring (easiest using tibemsmonitor utility).

Files used to store messages will be created on the first start using parameters in the stores.conf file (before version 5 in tibemsd.conf). In the normal operational mode, messages may accumulate in the topics and queues if no recipients – files will grow when needed, and therefore it is necessary to continuously monitor, otherwise the service may become unusable. It is possible to predefine minimal size of those files, it will take some time to build files for the first time if predefined size is large, but help to avoid fragmentation. Shrink or truncate large files to predefined minimum is also possible. When EMS restarts, all persistent messages will be recovered, but it will take some time to recover if files are large. In general, when you upgrade EMS from 4.x to 5.x, all stores will be upgraded automatically. Downgrade or rollback is also possible using tibemsdb5revert.

To provide high availability, two EMS servers can run as active-standby fault-tolerant pair. The main requirement of this configuration is the simultaneous access to store files – Cluster File System is required. Veritas Storage Foundation Cluster File System as expensive enterprise solution example. Some variants with network shares or NFS are also possible but guaranteed uptime and messages rate can be much lower. Alternative approach – failover cluster with shared volume.

Starting with EMS version 5 it became possible to use a database to store the messages. This simplifies the creation of fault-tolerant pair – no need to create a shared file system, enough to configure two servers to the same database. So far I haven’t collected a sufficient pro and cons, if you have such please share in the comments.

And lastly few words about connecting clients. Usually, when your application must be integrated into TIBCO middleware, means that you need communicate to EMS only. The most of SOA oriented applications are ready to communicate with JMS, Java clients can use JMS classes. TIBCO ActiveMatrix BusinessWorks has JMS palletes. Applications must be able to work with the fault-tolerant pair of two servers, provide reconnection in case of connection failure, support authentication.

Useful links:

How to change admin password in TIBCO Administrator

5 comments

If you need to change password for admin user in TIBCO Administrator please keep in mind that it is not enough just to change the admin password from Administrator GUI. But whole procedure is not so complex.

  1. Always have full backup of Administrator configuration files and repository files or database!
  2. Change admin password in the Administrator GUI.
  3. Start DomainUtility on each machine in the TIBCO Domain, select Server Settings => Update Domain Credentials and change password there or use domainutilitycmd and ChangeDomainCredentials.xml as a template.
  4. Make sure that password has been changed in tibcoadmin_<domain>.tra file. Or you can put new password manually there like this repo.securePassword=#!tibco and use obfuscate utility to encrypt it.
  5. Restart Administrator daemon and Hawk Agents everywhere in TIBCO domain.

That’s all. But, if you change password in the Administrator GUI only, and Administrator service (on the Windows) or daemon (on the Unix) has been restarted as nothing is working as before, then you can find picture like this on your screen when you try to login into Administrator. If your daemon has been started using nohup utility as mine, then you have a chance to find little more in the nohup.out file:
com.tibco.infra.repository.RepoSecurityException: Can not read policy domain for repository server HM : Failed in authentication.

If you start DomainUtility at this time and try to change password there, following error will appear.

To solve this problem you need to disable security option in the tibcoadmin_<domain>.tra file
#repo.isSecurityEnabled=true
repo.isSecurityEnabled=false

then restart Administrator.

Now it is possible to continue the procedure from step 3: DomainUtility will work. When password will be changed, you can enable security back in tra file. Do not forget to restart Administrator daemon and Hawk Agents everywhere in the domain!

After all, redeployment of all applications may be necessary.