Remark: this is a brief overview from the admin perspective. If you ask a developer or architect, then their views may differ greatly from what you find below :)
TIBCO Enterprise Message Service (EMS) is fully compliant Java Message Service (JMS) implementation from TIBCO with some enterprise-class enhancements. 
What is it? In general, from JMS FAQ:
The Java Message Service makes it easy to write business applications that asynchronously send and receive critical business data and events.
The Java Message Service defines a common enterprise messaging API that is designed to be easily and efficiently supported by a wide range of enterprise messaging products.
The Java Message Service supports both message queuing and publish-subscribe styles of messaging (topics).
It is main part of Enterprise Backbone, Enterprise Middleware and Enterprise SOA. Unlike TIBCO Rendezvous, where publishers and subscribers communicate directly without server, EMS represent dedicated server, hub which connects all clients and passes through itself all messages.
Better to see once than hear a hundred times. Installation process for EMS server is very simple, I slightly described installation on Windows platform in this post. On Solaris or other *nix we can run installer with [-console] option if X11 isn’t configured:
bash-3.2# ./TIBCOUniversalInstaller-sol-sparc.bin -console
New TIBCO Universal Installer will store configuration files and message storages separately from binaries and allows to have multiple environments on the same host, you must specify both paths. EMS is not required any additional components like TIBCO Runtime Agent (TRA), everything is included in the archive.
To start EMS manually with output to console, just execute tibemsd or tibemsd64 (depend on platform) with option [-config] and path to tibemsd.conf file. 
Later in production it will run as a system service on Windows (install or remove service using emsntsrg utility) or as a daemon in Unix. Start process from console is also good for debugging purposes, if an error somewhere in the configuration files and service isn’t running.
All EMS configuration stored in the configuration files and these files are read when the EMS process going up. Main file is tibemsd.conf: it’s contain service name, listening TCP port, links to other configuration files, logging options and etc. 
If start EMS without specifying tibemsd.conf file, it will try to find it near binary, if unsuccessful then conf files will be created near binary with default values.
The most of EMS configuration, like new user, new queue or bridge, performed using administration tools on the live system and become active immediately, no restart required. Then changes saved in the corresponding conf files to be restored when you restart the service. But it is also possible to modify conf files manually. Moreover, some parameters, like message storages location or log file name must be predefined in conf files and EMS restart is necessary. Before each change make sure to have fresh backup of the configuration files!
For EMS administration tasks some tools are available: 
tibemsadmin – command line administration tool provided with EMS, EMS plugin for TIBCO Administrator, Gems (Graphical Administration Tool for EMS) by Richard Lawrence, HermesJMS. 
Using these tools admin can manage topics, queues, bridges, users and so on. For monitoring EMS offers many options for logging and trace. Also, admin can subscribe to system topics beginning with “$sys.monitor.” for live evens monitoring (easiest using tibemsmonitor utility).
Files used to store messages will be created on the first start using parameters in the stores.conf file (before version 5 in tibemsd.conf). In the normal operational mode, messages may accumulate in the topics and queues if no recipients – files will grow when needed, and therefore it is necessary to continuously monitor, otherwise the service may become unusable. It is possible to predefine minimal size of those files, it will take some time to build files for the first time if predefined size is large, but help to avoid fragmentation. Shrink or truncate large files to predefined minimum is also possible. When EMS restarts, all persistent messages will be recovered, but it will take some time to recover if files are large. In general, when you upgrade EMS from 4.x to 5.x, all stores will be upgraded automatically. Downgrade or rollback is also possible using tibemsdb5revert.
To provide high availability, two EMS servers can run as active-standby fault-tolerant pair. The main requirement of this configuration is the simultaneous access to store files – Cluster File System is required. Veritas Storage Foundation Cluster File System as expensive enterprise solution example. Some variants with network shares or NFS are also possible but guaranteed uptime and messages rate can be much lower. Alternative approach – failover cluster with shared volume.
Starting with EMS version 5 it became possible to use a database to store the messages. This simplifies the creation of fault-tolerant pair – no need to create a shared file system, enough to configure two servers to the same database. So far I haven’t collected a sufficient pro and cons, if you have such please share in the comments.
And lastly few words about connecting clients. Usually, when your application must be integrated into TIBCO middleware, means that you need communicate to EMS only. The most of SOA oriented applications are ready to communicate with JMS, Java clients can use JMS classes. TIBCO ActiveMatrix BusinessWorks has JMS palletes. Applications must be able to work with the fault-tolerant pair of two servers, provide reconnection in case of connection failure, support authentication.
Useful links:
