Introduction
In TINE as in any control system, user application programs (clients) make use of the services provided by front-end or middle-layer servers. Essential to smooth operation is then the locating of the services offered, i.e. server address resolution. This in general amounts to matching a human-readable device server name to its network address. A trivial method of matching names to addresses is to provide a file containing this information, in analogy with a 'hosts' file for host address resolution. New servers would then need to be entered into a file by a control system administrator.
Another method of address resolution would be to 'broadcast' the name of the server in question. The server with the matching name would then presumably respond and thereby deliver the address information.
The first method noted above requires a client to have a file mount onto a file system containing the address database file. This is of course manageable but does involve synchronization problems where multiple platforms (and multiple file systems) are supported. It also necessitates a control system administrator entering in the information for new servers.
The second method noted above requires a well-known control subnet (or subnets) on which to broadcast, or perhaps a well-known multicast group. If a client is outside the 'control' net, it won't be able to find its partner by these means.
Primarily, TINE makes use of yet another mechanism, namely an equipment name server (ENS), which is a running process which manages address resolution for the control points. This is a specific server process (analogous to a domain name server) which returns address information via the TINE protocol. It also allows servers, upon startup, to 'plug' themselves into the ENS database, thus obviating the need for a control system administrator to add new server addresses by hand. Similarly, address changes are verified by the ENS when a server seeks to make use of a new network address. First a query is sent to determine if a server by the same name is still in operation under the old address. If so, a 'address in use' message is returned to the startup server and the address change is refused.
Plug and Play
This scenerio realizes a 'plug-and-play' functionality for the participating control system elements. That is, new servers can plug themselves into the database just as address changes can plug themselves into the database. Clients seeking the server addresses can find them by querying the ENS. A running client can in fact discover an address change without requiring a restart. This is possible since an address change will of necessity require a server restart which will automatically update the address information stored at the ENS. The client program will in the mean time be receiving connection time-outs, which is itself a signal to re-acquire the server address.
Note that this by no means implies that the ENS is a 'single point of failure'. Indeed, you can configure multiple equipment name servers (at DESY for instance there are two). Even if an address request finds no ENS running, an attempt to resolve the server address is made by consulting the last cached address on the local file system (where there is a local file system), and finally by consulting a local database if one exists. In fact, you can run TINE without an ENS in this manner, namely by guaranteeing that a local copy of the equipment database is available on the local file system. You can also run clients and servers on the same machine in "STAND_ALONE" mode if necessary.
Clients must be able to find either the ENS(es) or the local database files. The addresses of all running ENSes must be supplied in a startup file called 'cshosts.csv'. This file as well as any local database files are to be found in the location specified by the environment variable 'TINE_HOME'. If this environment variable is not set, then the current working directory is scanned.
An example of a 'cshosts.csv' file containing two running ENSes is shown below.
Naming Scheme
We should briefly review the TINE naming hierarchy for clarification. The TINE name space distinguishes between devices and their properties. Identifying a device can contain up to three levels of hierarchy. The root level of the device name hierarchy is the device 'Context'. This is frequently used to designate the 'facility' or 'machine' name, but can be as simple as 'TEST'. This root element is actually optional as far as name resolution is concerned. It can be omitted. If omitted, the so-called 'default' context is used, meaning that the first match of the device server name is returned. As long as this is unambiguous, there is no problem. The second level in the device name hierarchy is the device server name. This is actually the most important level and is not optional. This name must be found by the ENS in order to return the associated network address. Indeed, the name resolution from the ENS ends here at the second level, and the address is returned to the caller. The device server name can also be a 'logical' server, that is a device 'group' (see the discussion below).
The third level in the device name hierarchy is the specific device name of the desired module. This is resolved at the device server and not at ENS. To this end, the specific device name is also optional, in that the device server can ignore it or use a default value if omitted. Likewise the device Property is resolved at the device server and not at the ENS. The device Property is not however optional and must match a property exported from the device server.
Queries for device contexts and device servers are addressed then specifically to the ENS. Queries for device names and properties are addressed to the corresponding device servers. Information regarding the device 'Sub-system' is also recorded at the ENS. Although this is not a part of the name space, one can still query the ENS for all device servers belonging to the Sub-system 'Magnets' for instance.
Group Server
As noted above, one class of device 'servers' refers in reality to a 'logical' server or rather a device group. A phyisical server might belong to a device group and can just as easily be reached via its group name as its physical device server name. The great advantage is that several individual device servers can belong to the same group, for instance when they are basically equivalent but control different devices in different parts of the system. An initializing server can 'join' a group or an administrator can update the group database by hand. Either way, you will have to run the TINE Group Equipment Name Server on the same machine(s) as your ENS for this to work. This is another server process, which assumes that it is running on the same machine as an ENS and uses the TINE Port Offset 101 by default (the ENS uses 0).
If you run more than one instance of the TINE ENS (you should!) so that there is no single point of failure, be sure than you also run an instance of the TINE GENS on each machine where the ENS is running. You should also have a mechanism for synchronizing the database repository from the primary to the secondary machine(s).
One caveat concerning a physical server joining a 'group' is that in order for this to make sense to someone using or browsing the 'group' server the devices registered for each member of the group must be unique! The GENS will do nothing more than maintain a device list for each group along with redirection information pertaining to each device. If for instance the second member re-registers a device with the same name as the one from the first member, the second registration 'wins'. Where the registered device names refer to different instances at different locations, they are liable to be different at each instance of a device server joining a group. If on the other hand, the registered device names refer to a 'category' of devices or are otherwise an identical set at each instance of a device server joining a group, then an optional device prefix and/or postfix should be applied in order to have a means of distinguishing among the different elements. This can be accomplished in a plug-and-play manner at the server side either by making use of the GROUP_DEVICE_PREFIX and/or GROUP_DEVICE_POSTFIX columns (.csv file) or tags (.xml file) or via the extended registration API call JoinEquipmentGroupEx().
Administration
Due to the plug-and-play features described above, the TINE ENS can operate without intervention by a control system administrator. Nonetheless, if obsolete servers are not removed from time to time, the equipment database can contain a lot of unnecessary information, although there is automatic removal of inactive servers (past an 'allowed' down time - more later).
It might also become desireable to add 'alias' names to the equipment database or to otherwise make changes by hand.
An administrator is always free to locate the database files read by the ENS and edit them by hand. The ENS checks regularly for external changes to its database and can re-read its database on the fly. This amounts to editing with any text editor the files 'eqpdbase.csv', which matches device server names to equipment modules names and front end computer (FEC) names, and 'fecaddr.csv' which provides the network addresses for the FECs.
When the administrative activity primarily involves removing obsolete entries, the preferred method is to make use of the ENS administrator program shown below.
It is also adviseable to use the above tool to establish a FEC's 'importance'. This is one characteristic which must be applied by a control system administrator, and can be one of "NONE" (default), "IMPORTANT", "ESSENTIAL", or "CRITICAL". A server's 'importance' can be used as a filtering criterion when using for example the Remote Fec Control Panel. Also, servers such as the Central Alarm Server (CAS) or Fec statistics server will query the ENS for those servers with importance >= IMPORTANT if they are started without a database. Generally, "NONE" is the default for all new servers; "IMPORTANT" should signify something that should be running when there are operations; "ESSENTIAL" should signify a server that is necessary for operations (for instance the Central Archive Server is "IMPORTANT" but not "ESSENTIAL"); and finally "CRITICAL" should signify a server without which operations are impossible.
Users who are responsible for the FEC entry are allowed to remove their own creations. Otherwise, control system administrators are allowed to remove any and all elements. The list of control system administrators must be supplied in ENS startup file 'admins.csv'. The server developers must include their user names in the 'Responsible' column of the 'fecid.csv' at the server side, or provide this information via the RegisterFecName() API call if that method is chosen.
A site can have more than one running equipment name server. This is in fact adviseable as it offers both redundancy and load balancing. In such cases it is also advisable to provide a 'forwards.csv' list which gives the locations of the other name servers which should receive update information, such as the addition or removal of a device server or front end controller.
The ENS will regularly ping all server entries at hourly intervals and remove those entries that have not responded within the so-called 'allowed down time'. This is by default 90 days. The default value can be adjusted at any time during run time by using the ENS administration panel and selecting "Options" -> "Allowed Down Time" and then entering the desired interval. This new value will revert to the default upon a restart of the ENS. In order to change the default, the environment variable
ENS_ALLOWED_DEADTIME
should be set to the desired time in seconds (!). An administrator can also force the ENS to ping all servers in a given context and remove the non-responders immediately. To do this, select "Options" -> "Flush Inactive Servers".
ENS Configuration
At a new site, you can start the ENS server without a database. It will make its own, where it will include itself as the initial entry. A secondary ENS running on another machine is always a good idea. However this is one address entry you will have to add by hand as an ENS adminitrator. This is because the ENS itself does not plug itself into the ENS. A common tactic is to make an alias for the primary ENS within the eqpdbase.csv file by copy and paste of the entry whose 'Name' and 'FecName' entries are both 'ENS'.
Simply change the pasted names from 'ENS' to 'ENS1'. Then make another paste and change the names from 'ENS' to 'ENS2'. Now we will need an entry in the fecaddr.csv for 'ENS1' and 'ENS2'. So edit the fecaddr.csv file and copy and paste the entry whose 'FecName' entry is 'ENS' and change 'ENS' to 'ENS1'. Make another paste and change the 'ENS' to 'ENS2'. For this last entry you will have to change the 'IPaddr' by hand to that of the server where the secondary ENS is running. You will then have to copy all of the pasted lines by hand to the database files on the secondary, so that both refer the same machines calling themselves 'ENS1' and 'ENS2'.
Note: You do NOT need to do this at all, but it will aid in explicitly browsing to the known primary and secondary ENSes.
The ENS will check its own self-entry for consistency, but you can prevent this if necessary by makeing use of the environment variable
NO_ENS_CHECK
set to TRUE. This is sometimes necessary if, due to network configuration, the IP address seen by the process itself is not the same as that used by remote hosts to access it. For example, if you use the loopback address in the ENS database, or if there are multiple interfaces on the machine running the ENS.
ENS Administrators
Every other server in the system will plug itself in as they are brought to life. If there is no administrators file, then every user in the site will be able to remove device servers, etc. by using the ENS adminitration tool. This is generally not desireable, so you will probably want to supply an administrators file. This is a simple .csv file located in the working directory of the ENS. For example:
If you maintain both a primary and secondary ENS, you should attempt to keep them synchronized. Common practice is the supply a users.csv file in the working directory of the secondary ENS containing only the entries 'ENS' and 'GENS' as users.
This will block any attempt of an initializing server to plug itself into the secondary ENS. Note: do NOT use a users.csv file on the primary ENS, or else no server will be able to plug itself in!
ENS forwarding
Then you can either invent a copy process that periodically schedules itself and copies the primary database files to the secondary, or make use of a forwards.csv file, which will automatically forward new entries or removals to all ENSes included in the forwards file. Caution: this will only handle those new entries arriving via plug and play or changes made by the ENS administration tool. Changes made to the database by hand will not be forwarded. The forwards.csv file can have an optional column called "CONTEXT" which will then only forward changes to servers in the context given. This is a good way to selectively forward information from ENSes accross a site boundary. If the CONTEXT column is absent or its context empty, then all changes are forwarded to the ens given. For example:
The same admins.csv and forwards.csv should also be used for the GENS process.
ENS allowed subsystems and contexts
As a final note, other optional features of the ENS include the usage of a subsystems.csv and/or a root_contexts.csv file. If the ENS sees a subsystems.csv upon startup, it will restrict the allowed subsystems to the entries contrained within. Futhermore it will establish system known subsystem name tags and attempt to match incoming registration information against its own list. If this file is absent, then all registered subsystems are accepted. This is sometimes an issue depending on how well enforced a site's naming convention is, and whether names such as "FB","Feedback", "FEEDB" all refer to the same thing or not.
An example subystems.csv file is shown below.
In the ENS sees a root_contexts.csv file upon startup, it will likewise restrict any new incoming server registrations to use one of the given root contexts maintained in the file. That is to say, the 'root' context is that given by stripping off any context decorations (anything following a '.') and making a case insensitive comparison against the allowed root contexts found in the list. If an attempt is made to register a server in an invalid context then the registration call will receive the error 'invalid context' and the server will not be visible to the control system.
An example root_contexts.csv file is shown below.
