387 lines
15 KiB
Groff
387 lines
15 KiB
Groff
$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$
|
||
%$% %$%
|
||
$%$ Electronic Switching System Faults $%$
|
||
%$% %$%
|
||
$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$%$
|
||
|
||
|
||
|
||
|
||
"Notes from No 2 ESS Administration and Maintenance Plan,"
|
||
"BSTJ Vol 48, 1969"
|
||
|
||
"Data Maintenance"
|
||
|
||
|
||
Memory mutilation results from hardware faults and program bugs.
|
||
During nonsynchronous operation mismatch detection not available so
|
||
there may be a long period of time during which mutilation occurs.
|
||
Mismatch detection useless in finding data mutilation caused by program bugs.
|
||
|
||
Data maintenance aided by
|
||
ease of communication among programs,
|
||
absence of linked lists, and
|
||
per call memory allocation (Call processing program addressing is relative to the allocated memory, reducing scope of data accesses).
|
||
|
||
Defensive programming techniques:
|
||
|
||
Range check table indexes,
|
||
Zero check derived transfer-to addresses, and
|
||
Distinct program and data errors prevent programs being read as data.
|
||
Audit programs detect bad data.
|
||
Audits run periodically or as requested from tty.
|
||
Separate audits for different memory blocks
|
||
Audits correct by idling memory blocks containing bad data.
|
||
System recovery initiated by control unit switch during simplex operation, control
|
||
unit switch can be caused by bad data or bugs that cause sanity time out.
|
||
|
||
System recovery Funtions:
|
||
Make call store consistent with state of periphery.
|
||
Clear memory associated with program in control at time of recovery,
|
||
Run audits,
|
||
Repeat the above with widening scope of memory initialization until sanity obtained
|
||
|
||
|
||
|
||
|
||
"Notes from Design of Recovery Strategies for A Fault Tolerant No. 4 ESS"
|
||
"by R. J Willet - BSTJ vol 61, no 10, 4-13-82"
|
||
|
||
"Objectives"
|
||
|
||
616,000 call attempts/hour
|
||
100,000 acive terminations
|
||
Downtime less than 2 hours in 40 years
|
||
Not cost-effective (or possible) to remove all software errors - minimize
|
||
number of service effecting errors and analyze data for cause.
|
||
|
||
|
||
"Software Recovery"
|
||
Reconstruct data from associated information - slow, disturbs few calls.
|
||
Reinitialize memory structure - fast, disturbs many calls.
|
||
|
||
|
||
"Audit Programs"
|
||
Provide for integrity of system memory
|
||
Structured into mutilation detection and correction modules
|
||
Detection modules run continiously in background
|
||
Detection modules augmented by defensive checks in operational programs
|
||
Call correction modules to correct errors found by background audits or
|
||
defensive checks.
|
||
|
||
|
||
"System Integrity Programs"
|
||
Provide for integrity of programs
|
||
Monitor job scheduling and sequencing for frequency and execution times
|
||
Use sanity timers
|
||
Call audits or reinitialize system to correct errors.
|
||
|
||
|
||
"Recovery from software problems"
|
||
Software problems caused by program errors or bad data
|
||
Out-of-range accesses trigger hardware interrupt, recovery
|
||
requires correction of data, or killing of call and return of control
|
||
to a safe point.
|
||
Inhibit (pest) interrupts while audits are correcting problem,
|
||
risky, but assumes single software fault.
|
||
In cases where the out-of-range error can be isolated to a single unit can use frame level pesting, otherwise use system level pesting.
|
||
Software recovery does not consider the possibility of a hardware fault.
|
||
Recovery cannot fix a program bug. Running pested may allows the system to
|
||
operate in a degraded fashion while maintenance personnel analyze data and
|
||
correct program.
|
||
The buffer overflow problem - may be caused by program error.
|
||
Buffers protected by hardware overflow interrupts.
|
||
Recovery runs the buffer unloader program to unload the buffer and audits the task dispenser program to ensure the unloader is scheduled properly.
|
||
The overflow interrupt is pested.
|
||
If problem continues, hardware is suspect.
|
||
|
||
|
||
|
||
"No. 4 ESS: Maintenance Software"
|
||
"by M. N. Meyers, W. A. Routt and K. W. Yoder,"
|
||
"BSTJ Vol. 56, No. 7, September 1977"
|
||
|
||
|
||
"Software Error Recovery"
|
||
Since system operation is dependent on data in memories, and memories can be written, there is a possibility the memory will be in a state that precludes operation.
|
||
System must be as error-free as possibile.
|
||
Since system cannot be completely error-free, it must be error tolerant.
|
||
|
||
|
||
"Classification of software errors"
|
||
Errors in interfaces between software modules.
|
||
Non-conformity to systems rules.
|
||
KsO$ H@! fw
|
||
Logic errors.
|
||
Coding errors.
|
||
Complex man-machine interfaces that lead to procedural errors.
|
||
|
||
|
||
"Effects of software errors"
|
||
Loss of viability
|
||
Loss of call processing
|
||
Loss of a facility
|
||
Loss of a functions
|
||
Loss of capacity
|
||
Loss of single call
|
||
No effect
|
||
|
||
|
||
"Error Prevention"
|
||
Standardization,
|
||
Simplification,
|
||
Improved Documentation,
|
||
I wonder why improved testing isn't listed
|
||
|
||
|
||
"Error Tolerance"
|
||
Error tolerance acheived through defensive programming and defensive memory.
|
||
Programs attempt to remain operational in presence of errors
|
||
Restrict access to data to noncritical regions where errors have minor impact. (Using special instrutions to write to critical memory).
|
||
Checking state codes
|
||
Range Checks
|
||
Positive Decisions
|
||
Symbolic Addressing
|
||
Interpreting all possible stimuli
|
||
Link by index rather than absolute address
|
||
Special purpose memory allocators
|
||
|
||
|
||
"Error Handling"
|
||
"Software Integrity Control"
|
||
Receives reports of all software errors, decides and activates corrective action.
|
||
Corrective action: request an audit, or if history indicates repition, request a phase of system initialization.
|
||
Receives reports of overload - must decide if these are true, or are the results of software errors and take appropriate action.
|
||
|
||
|
||
"Audits"
|
||
Detect and correct data errors.
|
||
Control structure and audit routines specific to particular data structures.
|
||
Control structure schedules routine audits.
|
||
Demand audits run when errors found or suspect.
|
||
Error detection through: comparison of duplicate structures, association (correct structures linked together), and semantics (data is reasonably correct).
|
||
|
||
|
||
U"qGu
|
||
"Integrity monitor system"
|
||
Detects shceduling and cycling irregularities and los of major system functions.
|
||
Time monitors detect basic sanity
|
||
Base level monitor verifies validity of base cycle
|
||
Test call program observes progress of test calls
|
||
|
||
|
||
"Recovery"
|
||
Remedial actions - audits - correct specific errors.
|
||
If remedial actions fail undertake system recovery.
|
||
System recovery reconfigures hardware and initializes memory.
|
||
System recovery initiated upon detection of:
|
||
Mutilation of program memory
|
||
Loss of vital system function,
|
||
Loss of major facility,
|
||
Escalation of remedial actions,
|
||
Software integrity monitor problems,
|
||
Sanity Timeout
|
||
Mutilation of software clock,
|
||
Duplex failure
|
||
|
||
System recovery phases:
|
||
Phase 1 - initialize specific areas of transient memory - does not kill calls,
|
||
Phase 2 - reconfigure peripheral hardware and initialize all transient memory - kills
|
||
only non-stable calls.
|
||
Phase 3 - reconfigure processors, initialize all memory - kills only non-stable calls.
|
||
Phase 4 - totally initialize system - kills all calls - manual activation only.
|
||
|
||
More detailed and specific manual recovery procedures can be initiated.
|
||
|
||
|
||
|
||
"Notes from DS5C 3.00.00.00 - Issue 2 No. 5 ESS Maintenance Plan, 40288-500"
|
||
|
||
|
||
Up to 127 interface module processors - each is duplicated.
|
||
No simplex failure causes service loss, duplex failures cause service loss for only a few terminations.
|
||
Central Processor (CP) is duplicated.
|
||
The time multiplexed switch contains a duplicated processor.
|
||
The Control Distribution Units are not duplicated, since a single one is sufficient for inter-processor communications.
|
||
The Input/Output Processor is duplicated.
|
||
CP maintenance is under control of CP programs.
|
||
CP programs have minimal involvement in peripheral unit maintenance.
|
||
IM resposible for own maintenance, with few exceptions (manual requests, etc.).
|
||
IM uses self-checking hardware and operational checks.
|
||
Inter-processor messages use a reliable communications protocol.
|
||
Test for errors during use, scheduled tests for equipment not frequently used or in standby mode.
|
||
Test error detecting capabilities by inducing errors.
|
||
Operational software can detect some hardware errors through in-line checks,
|
||
and keep history to isolate error.
|
||
Audits and defensive checks to reduce software errors.
|
||
Software reliability is obtained at the expense of real time performance.
|
||
|
||
Defensive checks:
|
||
Range checks on pointers and indexes.
|
||
Duplicate copies of data at two different locations to detect mutilation.
|
||
Two-way linked lists.
|
||
Maintain and check redundant status information.
|
||
Positive decisions
|
||
Limited use of GOTOs
|
||
One entry point per module
|
||
Check incoming buffered data for mutilation while in buffer.
|
||
Ack timeouts.
|
||
|
||
Audits check for resources in an invalid state and restore the resource to a valid state.
|
||
Audits try to detect erroneous states before system performance is affected.
|
||
Loss of all of a given class of resource may halt call processing.
|
||
Long loops in programs cause them to consume all of the real time and put the system in overload.
|
||
Auditing of data resident in more than one processor leads to a race condition.
|
||
If audits find one error in a data structure then all associated data is assume to be bad.
|
||
Processes audited by timing, timeout in transient states indicated errors.
|
||
Scheduled audits have low priority.
|
||
Errors detected by audits are output on TTY.
|
||
Audit history is maintained in CP and can be queried.
|
||
Audits are stitched together during initialization.
|
||
Craft can request audits.
|
||
CP has audits for its data, IM has audits for its data, CP controls audits for data shared by
|
||
IM and CP.
|
||
Test calls are generated and monitored to detect timing irregularities.
|
||
Operating system monitors scheduling to detect overload or fault.
|
||
Hardware sanity timers to detect basic processor sanity.
|
||
Overload is caused by a shortage of resources.
|
||
In overload give preference to terminating traffic.
|
||
During growth intervals, audits must be gracefully updated to test new resources.
|
||
|
||
|
||
|
||
"Notes from DS5D 5.00.02.00 - Issue 1 Feature Control Development Specification Case 40288-100"
|
||
|
||
|
||
|
||
Feature control programs must explicitly record state during real time break, even if the
|
||
state can be derived implicitly by the program.
|
||
Restricted control of the process stack improves reliability.
|
||
Program design represented by finite state machine.
|
||
|
||
|
||
|
||
"Notes from 5ESS AUDITS Description and Procedures Manual, Issue 2, 1983"
|
||
|
||
|
||
|
||
Audits are 10% of 5ESS software.
|
||
Audits are powerful enough to disconnect stable calls and initialize the system.
|
||
|
||
5ESS software is divided into 5 areas:
|
||
operating system,
|
||
call processing,
|
||
Administration,
|
||
database, and
|
||
maintenance (The largest).
|
||
|
||
Audits are responsible for data in each area.
|
||
The maintenance area uses five system integrity features:
|
||
Asserts (defensive checks),
|
||
integrity monitors (sanity timers and other checks),
|
||
initializations, and audits.
|
||
|
||
Application audits run under OSDS which provides:
|
||
Process management,
|
||
Resource allocation,
|
||
Process synchronization and communication,
|
||
Inter-processor communication,
|
||
IM interrupt and fault handling,
|
||
Timing services.
|
||
|
||
DMERT has audits that run in the System Integrity Monitor environment at the CP, not descibed here.
|
||
Audit control in charge of audits in all environments, different audits
|
||
are run in different environments.
|
||
Routine audits are segmented - 20 millisecond segments for error detection,
|
||
40 milliseconds for error correction.
|
||
|
||
Audits may run in 5 modes:
|
||
routine segmented,
|
||
(run at lowest priority, take real-time breaks, output results to file,
|
||
escalate if critical error found)
|
||
elevated segmented,
|
||
(run at second lowest priority, take real-time breaks, caused by; input messages,
|
||
asserts, single process purges, other audits, if cannot correct errors in 40 msec,
|
||
either escalate or return to safe point)
|
||
nonsegmented,
|
||
(no real-time breaks)
|
||
initialization,
|
||
(run sets of audits back-to-back with no real-time break)
|
||
postinitialization.
|
||
(used to restore non-critical data after initialization)
|
||
|
||
A single event number is assigned to all audit actions resulting from the
|
||
same error.
|
||
Asserts placed in operational code (includeing audit code) to detect:
|
||
Out-of-range indexes and pointers
|
||
Redundancy on duplicate data,
|
||
Point-to/point-back linkage,
|
||
consistency between related data,
|
||
invalid function return codes.
|
||
|
||
Failing asserts usually trigger audits.
|
||
Asserts are a convention for defensive programming that standardize history keeping, and output messages.
|
||
|
||
Audits classify errors as: process errors, non-process errors, or audit failures.
|
||
Process errors cause single process purge.
|
||
Non-process errors are accumulated until end of segment and then an elevated
|
||
audit is scheduled to correct them.
|
||
Audit failures (the error is such that the audit can't continue) result in
|
||
scheduling of audits to fix the error.
|
||
Interprocessor audits check duplicate data in different processors.
|
||
Interprocessor audits run in routine and elevated segmented modes.
|
||
Interprocessor audits triggered manually, by asserts, and routinely.
|
||
Interprocessor audits correct data by making CP agree with IM.
|
||
Interprocessor audits divided into partitions that run on a single processor.
|
||
Interprocessor audit execution controled by CP, which receives completetion
|
||
reports from each partition.
|
||
About 70 application audits.
|
||
|
||
Current software development methodologies incapable of producing fault-free
|
||
programs.
|
||
Operator mistakes introduce incorrect data into system.
|
||
Hardware faults cause data mutilation.
|
||
Early detection and correction of fault effect retains system availability,
|
||
at a cost (usually mishandling of single job).
|
||
|
||
|
||
"DEFINITIONS"
|
||
|
||
Fault - improper action
|
||
Error - the result of some faults - an erroneous states
|
||
Failure - incorrect system output (wrong or delayed)
|
||
Outage - Failure effecting significant number of jobs
|
||
Fault tolerant systems prevent errors from becoming failures and failures
|
||
from becoming outages.
|
||
Faults that do not produce an erroneous state are not detectable
|
||
|
||
|
||
"CLASSIFICATION"
|
||
"ESS Software Reliability"
|
||
"Objectives"
|
||
|
||
Same as achieved by electro-mechanical systems:
|
||
No more than 2 hours total outage time in 40 years (720 seconds allocated to outages called by software faults).
|
||
No more than 2 calls out of 10,000 mishandled.
|
||
About 100-200 call attempts/sec
|
||
Tolerable response times range from milli-seconds to seconds.
|
||
"System Characteristics - 1ESS, 1AESS, 4ESS"
|
||
|
||
Base level cycle partitioned into A,B,C,D,E and interject priorities.
|
||
Interrupts for timing and hardware exceptions.
|
||
I/O at interrupt level
|
||
Time shared CPU with segmented (interleaved) processing
|
||
System memory paritioned into:
|
||
Text
|
||
Parameters (describe office size)
|
||
Translations (describe connections)
|
||
Shared Memory
|
||
|
||
|
||
Note: This file was in EMACS , I hope I edited it good enough to read, so that it provides use. -DT
|
||
|
||
|
||
----------------------------------------------------------------------------
|
||
|
||
|