Real Time Solutions of America, Inc.
1101
Seventeenth Street NW, Suite 1100
Washington
D.C. 20036-4798
United
States of America
Real Time Solutions of America Inc. designs and develops computer
software for real-time systems and provides training and consulting services
for customers. Our expertise ranges from supervisory control and data
acquisition (SCADA) systems to management information and data mining
solutions. Today Real Time Solutions of America is highly specialized in
integrating security printing presses in enterprise IT systems.
For more
information please contact us at: office@rtsol.com
RTA - Real Time Architecture®[1]
and its use in press consoles
February 2005
This document shows the usage of
RTA - Real Time Architecture® and its specific implementation in
press consoles. The shown architecture is an example of one solution that Real
Time Solutions of America Inc. has implemented in the past.
Table of Contents:
The base
system: RTA – Real Time Architecture ®
System Overview of
an application using RTA
ORACLE and the
RtDB Interconnection
Stopping,
Debugging And Starting Programs.
Server - Console
Redundancy Concept
Important Notice:
Due to the highly exclusive
nature of our business we do not, neither during nor after a project, forward
received information to a third party (i.e. person, institution etc.) which
is/was not directly involved in the project. Even participating project members
may receive information from us only to that extend that is required for
completing their task. For this reason we do not advertise, do not compete on
the common computer software market and provide company descriptions only in a
personal form and only on demand for people that require our company profile.
For this reason this document
does not provide any information of the press console by itself. It explains
only how RTA® supports such a console concepts.
General features of RTA
Based on RTA® numerous computer software
systems are installed on newspaper and security printing presses worldwide.
Alone in the US RTA® is installed a couple of dozen times on
security printing presses, with the first installation starting back in 1989
for the BEP in Ft. Worth, Texas. For the newspaper printing presses RTA®
is used in the US by every single site of Dow Jones to print the Wall Street
Journal. Also others like the New
York Daily have ABB MPS 730 installed, which is based on RTA®.
With RTA®
we meet the demand of clients that ask for state-of-the-art software technology
in security printing environments. However this requires that the presses
provide certain functionality in order to allow RTA® to maintain the
following features, which are essential for our customers:
·
Operating systems
WindowsNT (NT 4.0 / 2000 / XP)
·
ORACLE Database with RTA
specific real-time interface
·
Enterprise network
(TCP/IP) and press real-time network (ARCNET) interfaces
·
Data logging subsystem
(historical time driven and real-time event driven)
·
Open architecture for
interfacing to Management Information Systems (all information in SQL tables,
KPI processing with PL/SQL)
·
Redundancy management -
RTA based clients (e.g. press consoles) can keep running without interruption
and data loss even if the connection to the back-end is lost.
Base Architecture
The base
system: RTA – Real Time Architecture ®
RTA® is a software development platform that is solely
based on international standards and especially designed for the implementation
of real-time applications in all
different areas of process and workflow automation. It features:
RtIpC - Real Time Interprocess Communication
a network-wide inter-process communication that allows data
transmission rates between applications from 5,000 messages per second via
standard networks up to 25,000 messages per second local on a 40 MIPS hardware
(DECstation 5000/240).
RtDb - Real Time Database
a network-wide, distributed, active, real-time database that allows up
to 32.000 read operations per second on a 40 MIPS hardware (DECstation
5000/240).
RtRM - Real Time Redundancy Manager
a supervisor function that has network-wide control over nodes and
applications in redundant systems to allow real-time MASTER - STANDBY switching
in case of system malfunctions.
The above figures have been
measured based on 10Mbit networks. Using 100Mbit or even 1Gbit networks in a
distributed environment would result in much higher transmission rates. The CPU
tests were based on DEC MIPS hardware. Modern Intel based (or compatible)
processors can achieve much higher processing rates.
RtIpC - Real Time
Interprocess Communication
RtIpC is a task-to-task
communication, which is primarily designed for high-speed data transmission between
applications. Data transmission can be done between applications within single
nodes as well as between applications on different nodes in a network.
Communication is even possible between nodes using different operating systems
and network protocols.
RtDB - Real Time
Database
In contrast to conventional real-time capable
database systems, RtDB represents a combination of an active, real-time and a relational database. Thus RtDB provides
on the one hand an interface to the active real-time database for high-speed
data processing and a SQL interface to the relational database for standardized
data access. The integrity of the data between the real-time database and the
relational database is assured by a synchronization mechanism, which automatically
keeps track of changes in either of the databases. It provides for distribution
of changed data between the two databases as well as providing for distribution
of data changes between different nodes in a distributed or redundant system.
RtDB uses RtIpC to distribute data to other nodes. Thus real-time database
tables can be distributed in a network.
RtRM - Real Time
Redundancy Manager
The real time redundancy manager
is required for the implementation of redundant systems. It has network-wide
control over all nodes and even over all important applications on each single
node. Its job is to watch over all system components for malfunction and, in
case a component fails, to activate specific routines like system restart or
node switching (MASTER-STANDBY switching).
RTA® Structure
RtVars are Real Time Variables that can
be accessed using PL (Process Language) or Visual Studio program languages
(like Visual Basic, Visual-C) via ActiveX.
RtVars provide access to all necessary
components in the system like RtDB tables, Oracle tables, operating system
information (environment, timers etc.). RtVars can be active or passive. Active
RtVars allow external triggers to be activated whenever the value of the RtVar
changes. Due to that a system based on RTA® is very much event-driven.
System Overview of an application using RTA
The system consists of 5 major
components:
Press and its high speed network
– which connects the press PLC’s together
Console - the machine control station (a data
acquisition station)
Ethernet network – which
connects more presses to the server
Server – with the database for a
whole production line
MIS - (Management Information System).
These applications usually run on PCs that are connected to the Ethernet
network.
The components are integrated
into a complete system as follows:
The SERVER serves as central
database server. Therefore all CONSOLES and MIS stations connected to the
Ethernet network have access to the data. The SERVER provides no direct
coupling to the process, rather the data is received solely via the CONSOLE or
MIS stations. All data is stored in the database of SERVER. If a separate
SERVER is not available (e.g. stand-alone-press) the CONSOLE takes over the
database functions as well having a local DB installed.
RTA® Inside
ORACLE and
the RtDB Interconnection
By incorporating RtDB the system provides the
following features:
·
standardized access
to all system data by using the Oracle-SQL interface, allowing standard
applications like EXCEL, ACCESS etc. to retrieve system data easily
·
high-speed data
collection using RtDB, the database for the process drivers, providing a
procedural API for C or VB programs
·
active database
system that provides external triggers
·
maximum possible
speed in accessing the Oracle by making full use of the Oracle array-interface
The use of RTA-RtDB allows avoiding the typical
problems that can occur in a process automation environment. The example below
shows one of these possible problems:
Several
drivers are connected to a machine getting data in real-time and trying to
deliver that data into Oracle. Oracle might be too busy to insert or update a
record immediately or a low priority process might lock a record. In this case
no real-time can be guaranteed. A backlog could be the result, back to the
process that is not able to deliver data to the locked component. This could in
the worst case, depending on the implementation, cause a stop of the process
itself. In case of a printing press this would happen after blocking the
press-internal-network for more than 50 to 100 milliseconds. After that time
the redundancy system of the press would, for security reasons, shut down the
press by turning the main power off (emergency stop).
RtDB does NOT implement any kind of rollback
mechanism. The reason is that in a process automation environment a rollback is
in most cases not necessary or even technically not possible => a printing
press cannot move sheets backwards due to a rollback, the printing process just
keeps going. Other solutions are provided to ensure consistency.
Table
definitions in RtDB
RTDB table definitions look very much
like normal SQL 'create table' statements except that there are many functions that
are specific to RTDB and for that reason the ORGANIZATION clause has been
extended:
CREATE
TABLE WORKCODE (
CODE INT, /* PrimKey */
MESSAGE_TYPE INT,
PRESS_TYPE STRING(15),
RELEASE STRING(3),
TEXT STRING(80)
)
ORGANIZATION
(
SIZE
WORKCODE_SIZE,
KEEP
ALL,
LOCATION
MEMORY,
STORAGE
IS VARIABLE,
DATABASE
IS PRESS_DATABASE,
MASTER
IS SQL DELAY IS CONFIG_DELAY,
NODE_SYNCHRONIZATION
IS OFF, /* i.e. not used */
READACCESS
FOR ( SERVER, CLIENT ),
WRITEACCESS
FOR ( SERVER )
);
RtDB can handle much faster update,
insert and delete operations than Oracle. Therefore in order to synchronize
date RtDB is delaying the update to the Oracle until a certain time has elapsed
(DELAY - this time is configurable per each RtDB table). All changes are
processed immediately in the RtDB table but the transfer into Oracle is
deferred. Due to that RtDB is able to handle peek loads for a certain
time (depending on the SIZE of the RtDB table and the DELAY).
The Update from RtDB to the Oracle is
done by a single RTA-Task (RtDbSqlSync) thus minimizing the problem of Oracle
record locks and speeding up the update process by making full use of the
Oracle array-interface. The related ORGANIZATION clause is shown below:
ORGANIZATION
(
SIZE
max_number_of_records_in_table,
KEEP
ALL,
LOCATION
MEMORY,
STORAGE
IS VARIABLE,
DATABASE
IS BOTH,
MASTER
IS RTDB DELAY IS seconds_update_to_SQL_is_delayed
);
Since RtDB tables are memory resident
the size of each RtDB table is limited to a certain number of records.
Currently the limit is 65.536 (2^16) per each RtDB tables.
In order to provide an RtDB interface
also to larger tables like archives RtDB implements tables of type KEEP HIGHEST
and KEEP LOWEST. For these tables only a certain number of records are kept in
RtDB. Depending on the organization of the table these are the highest
or lowest records, requiring that there is a strictly monotonous
increasing or decreasing primary key. Below is the example of the CON_ARC table
(event archive for the console):
CREATE
TABLE CON_ARC (
ARC_TIME SECONDS NOT NULL,
ARC_MSEC SMALLINT NOT NULL,
ARC_STATION UNSIGNED SMALLINT NOT NULL,
...
ARC_KEY STRING(16) ,
ARC_TEXT STRING(120) )
ORGANIZATION
(
DATABASE
BOTH,
LOCATION
IS MEMORY,
STORAGE
IS VARIABLE,
MASTER
IS RTDB,
NODE_SYNCHRONIZATION
IS ON,
READACCESS
(SERVER,CLIENT),
WRITEACCESS
(SERVER,CLIENT),
SIZE
IS CON_ARC_Size,
KEEP HIGHEST
);
CREATE
UNIQUE INDEX PRIMKEY ON CON_ARC (ARC_TIME,
ARC_MSEC, ARC_STATION)
INDEX_ORGANIZATION
(
PRIMARY
KEY,
TYPE
IS SORTED,
LOCATION
IS MEMORY,
STORAGE IS VIRTUAL,
DATABASE
BOTH,
SEQUENTIAL
BOTH);
Only records with a primary key HIGHER/LOWER than
the highest/lowest record in the table are inserted into KEEP HIGHEST / KEEP
LOWEST tables.
When RTA-RtDB (or a similar tool) is
not being used in an automation project the following has usually to be
considered when adequate functionality should be provided:
classical approach
The
driver, pre- or postprocessor task has to provide for a kind of cache for the
process data between the fast process and the (naturally) much slower SQL
database. Usually this is implemented as an internal memory cache in form of a
linked or hashed list. Unless this cache is kept in shared memory ALL client
components, like archive program or visualization etc, that require process
information have to QUERY the driver or postprocessor (the owner of this cache)
to transmit the requested information. This results usually in a higher CPU
consumption that increases with the number of clients (due to the required
inter-process communication and the context switching that comes with it). This
happens even in systems that have only clients requiring READ access. A typical
client requiring mostly read access would be a visualization (VIS) task. This
means that a system implementing the above architecture slows down
proportionally with the number of (e.g. VIS) clients connected.
more sophisticated approach
A more
sophisticated approach is to keep the process information in shared memory
(what RTDB does). This approach avoids unnecessary data transfer between the
clients and is therefore much faster. Due to that this approach is much more
appropriate for real-time environments. But it is harder to implement and it is
also a bit tricky. One has to know the nuts and bolts of real-time. Because the
data is in shared memory, resource-locking mechanisms (e.g. table, record or
column locks) have to be provided in order to control access to the shared
information. This and the fact that there might be processes with high priority
(e.g. a driver that has to deliver data in milliseconds in to the database) and
processes with low-priority (e.g. an archive that writes information every
minute into a time raster archive SQL table) are the ingredients that can
finally lead into a malicious situation know as priority-inversion. This is
when a low priority task has locked a resource, e.g. some records in the RTDB,
and is (slowly) processing them and not willing to unlock the records until it
has finished its processing. At the same time a real-time task (e.g. the driver)
needs instance access to that records. RTDB has of course implemented solutions
for that kind of problem as well. But anyone who is not using RTDB or a similar
product has to consider those and other real-time related challenges and
provide appropriate solutions in order to make sure that real-time remains not
just a phrase.
Data types
used
A press console distinguishes between the following
data types:
·
process data
·
configuration data
·
product data
process data
This is
data with the following characteristics:
·
frequency is high,
data source is the press
·
MASTER IS RTDB
(changes are made in RTDB and sync. to SQL)
·
event occurs at
machine (events from the process)
·
driver receives data
from press real-time network
·
date is pre-processed
and RTDB tables are updated
·
post processing
occurs
configuration data
This is
data with the following characteristics:
·
frequency is low,
data source is the user (via DB backend)
·
MASTER IS SQL
(changes are made in SQL and sync. to RTDB)
·
Mr. Jones (a user)
changes configuration information in SQL (Oracle)
·
RTA®
transfers the changes automatically from Oracle into RTDB
·
hot reconfiguration
takes place firing triggers in RTDB
·
conditions coupled
with those triggers are evaluated and events executed
product data
This is
data with the following characteristics:
·
frequency is low,
data source is the press, operator and user (backend)
·
data is in Oracle
(SQL) only
·
product related data
comes from the press, the operators and users
·
as product data
arrives Oracle triggers fire and KPIs are calculated
Programs
on the console
The following programs are running on the console:
RtNodeCtrl - Node and task
handling
This task is responsible for node and task
supervision and re-start control. Restart is done in two phases. At first all
the subsystems must be started and initialized (e.g. RtDb synchronization).
Later the application programs will start. Restart is finished when all
important application programs have reached the state "ready".
RtNetCom - Network communication
The communication task that transports task-to-task
messages over the network (TCP/IP).
RtDbNodeSync - RtDb
Node-Synchronization
Each node in a network contains one task named
RtDbNodeSync. This task is responsible for synchronizing RtDb-tables on
different nodes. Insert and update operations are processed immediately.
Updates operations are buffered and transmitted at configurable intervals.
Buffering of updates improves performance due to decreased network traffic and
increased block sizes.
RtDbSqlSync – RTDB SQL Synchronization
Each node configured with a SQL database can run RtDbSqlSync. This
task is responsible for synchronizing data of RtDb and SQL.
RtSqlSqlSync - SQL to SQL
Synchronization
This task synchronizes tables defined in RtDb and
tables defined in Oracle. With the advent of advanced replication features in
Oracle 9 the value of this tasks functionality is diminished for systems that
use Oracle user database subsystem.
DRIVER - Driver for connection to
press
Sends and receives telegrams from the press
controllers. The driver is connected to the press via a network or bus. It
stores incoming telegrams immediately in RTDB tables and it receives outgoing
telegrams also via RTDB tables. For each telegram type a different RTDB table
is created (automatically generated from the telegram definition). The columns
of the tables represent the individual data items of the telegram.
Barcode - Barcode Driver for
connection a manual scanner
This driver allows integration of various barcode
readers for effortless input of all kind of production and maintenance related
information.
Purge - Purges Files and SQL
Tables
Depending on the actual configuration (customer
dependent) this task purges unnecessary files from the hard drive and also
obsolete data from the Oracle database. This keeps the press related part of
the Oracle database basically maintenance free.
OraCheck - Checks the connection
to Oracle
This task checks permanently the connection to
Oracle. If the network connection goes down, the server is shut down or the
Oracle connection gets lost for any reason, this task reports an error message
to RTA.
DCS – Preprocessing of data
The DCS is the AL/PL program that does all specific
data processing. It is like a preprocessor in SCADA systems. The program is
solely event driven. The program logic is described in form of ECA
(Event-Condition-Action) rules. The ECA rules process mainly (but not
exclusively) data in the RTDB. As soon as new telegrams arrive triggers a fired
and the telegram specific processing is activated. The resulting data is also
written into RtDB tables (specific tables that describe the business model).
This allows customer specific post-processing being attached in a fairly simple
way.
VIS – Console visualization
The user interface of the press console. This program is also event
driven and implements triggers (mostly VB ActiveX components) that are attached
to various RTDB items and automatically updated when changes occur in the
database.
Stopping,
Debugging And Starting Programs
In order to stop or debug a program you have to
move the mouse above the program and then press the RIGHT mouse button. A menu
appears where you can choose what action you want to do:
In order to start a program you have to press
CTRL-ESC to launch the Windows Start Menu and then you have to select the path
shown below.
Server -
Console Redundancy Concept
The RTA® console delivers, during normal
operation, all data immediately to the Server's Oracle database via SQL*NET. In
case that either the Server or the network is down, not allowing the RTA®
console to deliver data to the Server, RTA automatically goes into 'client'
mode and starts collecting information locally.
This information is not accessible in the Server
database until the Server or the network is up and running again.
On the RTA® console a red indicator
shows when the Server is not available. Some functions on the console, those
that require data to be read FROM the server, are disabled until the server is
available again. The system design makes sure that all data required for the
redundant functionality is available in RTDB tables and as such still available
even Oracle is offline.
As soon as the Server goes on-line again, the red
indicator light at the console turns green, indicating that the server is up
and running again and all data that was collected in the meantime is
automatically (without operator interaction) transferred to the Server.
This method allows for a loose coupling of the RTA®
consol clients and the Oracle server thus improving the overall system
efficiency and production up time in 24/7 environments.