Introduction to mTCA

Introduction to MicroTCA
Ray Larsen
SLAC MicroTCA Review
June 4-5, 2012
June 4-5, 2012
SLAC MicroTCA Standards Review
1
Outline
I. MicroTCA for Physics Basics
II. Physics System Requirements
III. COTS & Lab Developments
IV. xTCA Market Growth Projections
V. Summary Conclusions
June 4-5, 2012
SLAC MicroTCA Standards Review
2
I. MicroTCA Basics
June 4-5, 2012
SLAC MicroTCA Standards Review
3
PICMG-ese
Term
Definition
PICMG
PCI Industrial Computer Manufacturers Group, 250 corporations
ATCA
Advanced Telecommunications Computing Architecture large board
Carrier
ATCA or µTCA board that supports smaller standard board
Shelf
Crate, ATCA (large) or µTCA (small)
RTM/µRTM
Rear/Micro Rear Transition Module
AMC
Advanced Mezzanine Card mounting on ATCA Carrier, µTCA shelf
Micro/µTCA
Crate designed to support AMCs directly
MCH
MicroTCA Carrier Hub switch module for µTCA shelf
PU, CU
Power Unit (Module), Cooling Unit (fan or fan tray)
IPMI
Intelligent Platform Management Interface
Shelf Mgr
Shelf board hosting IPMI controller (BMC, MMC controllers)
Wide, High
High (vertical module height), Wide (front panel width)
xTCA
ACTA and /or MicroTCA standard platforms
June 4-5, 2012
SLAC MicroTCA Standards Review
4
MicroTCA Basics
• First a word about ATCA
– ATCA board, shelf is first modular computer architecture with
completely serial multi-Gbps backplane, dual star or mesh
– Board designed with standard Mezzanine card (AMC) for
ease in upgrading as technology evolves (Moore’s Law)
– ATCA managed system (IPMI), standard diagnostics detect
problems at ATCA, AMC levels; take remedial action to evade
machine/system interruption
– Target Availability for ATCA shelf (crate) is 0.99999 or greater
– In Telecom automated load switchover is common in systems
of many identical processor blades
• Less practical in accelerator, but less required for subsystems with high
redundancy
June 4-5, 2012
SLAC MicroTCA Standards Review
5
ATCA Shelf Carrier
ATCA Dual Star 14-Slot Shelf
June 4-5, 2012
ATCA Carrier w/ 4 AMCs
SLAC MicroTCA Standards Review
6
ATCA Fabric: Dual Star & Mesh
June 4-5, 2012
SLAC MicroTCA Standards Review
7
ATCA System Features, Extensions
• ATCA Board and RTM
–
–
–
–
–
Designed for ATCA board hot swap from front
Designed so all or most I/O from rear via RTM
IPM system isolates problem board, calls maintenance
Main board or AMC swapped while RTM remains in place
IPM returns to service with no shelf-wide interruption
• Extensions for Physics - RTM
– PICMG 3.0 RTM had no management features, standard
connectors, keying
– Physics Committee designed PICMG3.8 with IPM power
management, high density fabric I/O connectors, JTAG
June 4-5, 2012
SLAC MicroTCA Standards Review
8
PICMG3.8 RTM
Interface Standard
Rear View
120 - IO Channels (3x40)
IPMI, Power Connector (blue)
2 Mechanical Keys
Courtesy M. Huffer et al, SLAC
June 4-5, 2012
SLAC MicroTCA Standards Review
9
IPMI System Basics
June 4-5, 2012
SLAC MicroTCA Standards Review
10
• IPMI ~1990
pre-dates
ATCA for
use in
Mainframe
computers,
PCs etc.
– Architecture,
protocols,
chip sets
developed by
Intel, HP,
NEC, DELL
June 4-5, 2012
SLAC MicroTCA Standards Review
11
MicroTCA Shelf
• ATCA Carrier Board Transitions to AMC Shelf
– Segment of ATCA committee developed AMC shelf to lower
infrastructure costs
– Key was 1-wide multi-tongue MCH combining star switching,
shelf manager and IPMI communications
– MicroTCA shelves developed: 1U, 2U, 3U, horizontal and
vertical card orientation; no single shelf format
– Double wide AMC had space for RTM but undeveloped
June 4-5, 2012
SLAC MicroTCA Standards Review
12
MicroTCA Infrastructure for Physics
• Physics TC declared 2-wide w/ RTM essential
– 3X larger clean analog design space than 1-wide AMC alone
– Rear I/O, hot-swap options
– Technical Committee design included:
• 2-wide AMC, mirror image RTM
• AMC extended IPMI management to RTM for power, temperature, user
diagnostics, hot swap
• 3x30-pair fabric balanced line data connectors (same style as ATCA)
• Separate new connector for all RTM power and IPMI control
• Mechanical & E-keying, JTAG
• Hot swap handles, indicators, for both AMC, RTM
– Major Goal: Higher level of interoperability of lab-industry products
by defining I/Os of generic AMCs to support multiple applications
June 4-5, 2012
SLAC MicroTCA Standards Review
13
AMC-RTM Mechanics-Front/Rear Panels
Module sizes
June 4-5, 2012
SLAC MicroTCA Standards Review
14
MTCA.4 Mechanical Details
June 4-5, 2012
SLAC MicroTCA Standards Review
15
June 4-5, 2012
SLAC MicroTCA Standards Review
16
MTCA.4 Full Payload Shelf
• RFQ developed for Industry
– Infrastructure costs least when all payload slots filled
– Hot swap dictates all components be plug-modular including
power and cooling
– Features:
• 12 payload slots mid-size front panel 2-wide modules
• Push-pull cooling bottom to top, front to back with redundant controllable
hot swappable fans
• Redundant intelligent power modules of 1 KW nominal (smaller if not
needed)
• Full function 4-tongue MCHs with Shelf Manager
• Dual star backplane with special layer for timing, triggering, point-to-point
• Designation of spare lines (extended options region) for switched point-topoint low jitter timing (~1 psec)
June 4-5, 2012
SLAC MicroTCA Standards Review
17
MTCA.4 AMC-RTM-Shelf Concept
AIR OUT
I/
O
I/
O
INPUT/OUTPUT SIGNAL
CONDITIONING
FRONT END MODULE
text
ADVANCED
MEZZANINE CARD (AMC)
B
B
A
P
C
K
C
PL
O
A
N
N
N
E
AIR IN
REAR TRANSITION MODULE
(RTM)
USER DEFINED I/O CONNECTORS
FAN TRAY-HOTSWAP
FAN TRAY-HOT SWAP
June 4-5, 2012
SLAC MicroTCA Standards Review
18
12 Payload Slots
2-Wide w/ RTM
Dual Star
Dual MCHs
Dual PM, CU
June 4-5, 2012
SLAC MicroTCA Standards Review
19
MTCA.4 Backplane Extensions
12 Slot MTCA.4
Backplane
Point-toPoint Links
8 x M-LVDS:
Trigger
June 4-5, 2012
20
MTCA.4 Clock, Trigger & Interlocks
June 4-5, 2012
21
IPMI Extension to RTM
• When adding RTM, AMC supplies all power,
management
– 80W max was spit to 50W AMC, 30W RTM
– New connector pins sized accordingly
• Extension IPMI to RTM required protocol
compatibility w/ ATCA
– ATCA committee worked on parallel effort for managed
RTM, but no standard connectors as done for MTCA.4
– Physics committee assured protocol compliance
– Following slides show implementation
June 4-5, 2012
SLAC MicroTCA Standards Review
22
IPMI Extensions
Additional RTM control signals for MTCA.4
MP (+3.3V)
PP (+12V)
Power On
Enable
Presence
MCH 1..2
PP (+12V)
PP (+12V)
MP (+3.3V)
MP (+3.3V)
MMC
Enable
Presence
Presence
AMC
AMC 1..12
1..12
MP (+3.3V)
PP (+12V)
PM 1..4
µRTM 1..12
EMMC
Enable
Presence
June 4-5, 2012
Backplane
CU 1..2CU Front
1..2 fans
SLAC MicroTCA Standards Review
RearRear
cooling
fans
23
IPMI Extensions 2
Management extensions in MTCA.4
IPMB-L
 Connects the MCMC on
the MCH to the MMC on
the AMC Modules
 Radial architecture
IPMB-0
 Connects the MCMC on
the MCH to the EMMC on
the PM and CU
 Bused architecture
I2C-bus
 Connects the AMC to the
µRTM
 The µRTM is treated as
managed FRU of the AMC
June 4-5, 2012
SLAC MicroTCA Standards Review
24
II. Physics System Requirements
June 4-5, 2012
SLAC MicroTCA Standards Review
25
Physics Requirements 1
• Control Systems
– MTCA.4 optimized for controls requirements
– Basic Infrastructure:
• Powered shelf star, dual star backplane
• MCH, Processor, IPMI
• Timing Generator/Receiver
– Applications AMCs – Generic- 3 Types:
• Fast multichannel high resolution low noise ADC-DAC
• FPGA programmable Processor
• Industry Pack Carrier
– Each leverages multiple applications with single design
– Strategy: Develop min. 2 suppliers each generic AMC
June 4-5, 2012
SLAC MicroTCA Standards Review
26
Physics Requirements 2
• Applications Specific RTMs
– Generics can serve multiple requirements via RTMs
– Application specific RTMs vary in type of design skill
• GHz Low Level RF, high performance wide dynamic range
• Low speed <60 MHz 12 bit analog interlock fault detection
• industrial monitoring and control -- vacuum, temperature,
pressure, voltage, current
– RTMs also can be adaptable, programmable
– Strategy: Build RTMs to standard generic AMC
interfaces; develop COTS sources when quantities
justify
June 4-5, 2012
SLAC MicroTCA Standards Review
27
Physics Requirements 3
• AMC-RTM Examples
Generic AMC
Typical RTMs
Fast 16 bit ADC-DAC
LLRF Feedback
Beam Position Monitor
Cavity BPM
Toroid Precision Charge
FPGA Medium Performance Fast-Slow12 bit ADC
Interlocks , FPGA
RF Interlocks
Beam Loss Monitor
Beam Length Monitor
3- Industry Pack Carrier
June 4-5, 2012
Motor Controllers
Wire scanners
Vacuum monitor/control
Temperature monitor/control
SLAC MicroTCA Standards Review
28
III. COTS & Lab Developments
June 4-5, 2012
SLAC MicroTCA Standards Review
29
Shelves
• 3 Models
– 6 Payload slot development unit half-rack 6U
•
•
•
•
Produced early while main production unit being designed
Power and cooling non-standard, incompatible
New units do have “Physics” backplane timing layer
Schroff, ELMA
– 12 Payload slot production rack-mount 8U
• Fully redundant dual star, MCH, power & cooling units
• Schroff, ELMA, PT (Performance Tech)
– 6 Payload slot production rack-mount 2U
• Fully redundant dual star production unit, 3 slots w/RTMs
• PowerBridge to Schroff specification (First prototype delivered)
June 4-5, 2012
SLAC MicroTCA Standards Review
30
Shelves
12 Slot Fully Redundant 40GbE
PT Performance Tech
6-Slot Development
Non-Redundant
Schroff, Elma
12-Slot Fully Redundant
Schroff, Elma
June 4-5, 2012
6 Slot Non redundant
3 slots w/ RTMs PowerBridge
SLAC MicroTCA Standards Review
31
MCH Carrier Manager and Data Hub
• Standard MCHs
– Support all hub switch, IPMI operations in Physics
backplane except radial extended options clock lines
which SLAC currently does not need
– Currently evaluated by SLAC are NAT, Vadatech
– Some firmware differences were resolved to meet
MTCA.4 standards
– NAT currently developing 1 psec jitter switch card for
radial lines for DESY
– Vadatech will also produce this option in future
June 4-5, 2012
SLAC MicroTCA Standards Review
32
MCH Units
N.A.T MCH
Manages 12 AMCs, 2 cooling units,
1-4 power modules
Supports PCIe, Serial Rapid IO
June 4-5, 2012
Vadatech UTC002
Currently using in LLRF
System tests
Fully featured, compliant,
preferred front panel
network I/O
SLAC MicroTCA Standards Review
33
Processors
• All are single wide totally compatible with
ATCA carriers
• Processors evaluated to date are:
– AdLink AMC 1000 Core 2 Duo
– Kontron AM4020M i7
• Other processors available
– Concurrent, Radisys, GE, etc.
• On order
– Vadatech AMC720 i7 (delayed due to silicon late
delivery)
June 4-5, 2012
SLAC MicroTCA Standards Review
34
Timing Interim Solution: PMC Adapter
• Existing solution for LLRF test system and lab prototyping
uses MRF (Micro Research Finland) EVR PMC adapter.
– Not a permanent solution as all cabling is via external coaxes and
SMA or Lemo connectors; timing backplane bypassed; however
works fine for test purposes
– New highly efficient solutions discussed in following slides
MRF EVR on
Vadatech PMC
Adapter
June 4-5, 2012
SLAC MicroTCA Standards Review
35
Timing Modules
• DESY-Stockholm Generator-Receiver
– Single wide unit operational for over a year at DESY
– SLAC has a unit but not implemented
– Not compatible with SLAC’s MRF (Micro-Research
Finland) EVR (Event Receiver) timing network protocol
• Gen 2 Prototype with RTM
–
–
–
–
2nd Generation 2-wide with RTM nearing completion
Generator will be optional plug-in daughter-card
Additional I/O Tx-Rx options via RTM
To be produced commercially
June 4-5, 2012
SLAC MicroTCA Standards Review
36
Timing AMC-µRTM 1&2-Wide MTCA.4
•
•
•
•
Both single & double-wide MTCA.4 compliant
Double-wide RTM allows rear expansion to multiple receivers
Accesses all parallel, point-to-point serial switched lines
SW Committee looking into standardizing protocols
MTCA.4 Backplane
June 4-5, 2012
New Developments in I&C Standards for
Physics
37
Timing Modules 2
• International Technologies (I-Tech) Receiver
EVRX
– EVR & MTCA.4 Compatible single wide in prototype stage
– Will also be configurable as generator
– To be shown at xTCA Workshop June 9-10 2012 Berkeley
• 3
EVRX PC Board
Top View
June 4-5, 2012
SLAC MicroTCA Standards Review
38
I-Tech Timing EVRX
• EVRX 3D Models
– Courtesy International
Technologies
June 4-5, 2012
SLAC MicroTCA Standards Review
39
Example Timing Solution - BPMs
• Current solution: 4 network cables per BPM
– BPMs are rackmount Pizza boxes with 4 long timing & data network
cables per box, expensive, consume large rack space
– Cables are primary trigger, calibration triggers, clock, data
• MicroTCA Solution: 1 EVR cable to BPM shelf
– BPM RTM contains filters and calibrate circuits 4 input Ch.
– Single parallel bus trigger ahead of beam from timing module starts all
units sampling ringing filtered input signals, processing data in FPGA,
delivering to processor on demand
– Each BPM digitizer FPGA when ready delivers 2 consecutive timed
calibrate signals via RTM
– After 3 cycles unit is ready for next pulse
– Large network cable plant eliminated; rack space minimized to 1 shelf
June 4-5, 2012
SLAC MicroTCA Standards Review
40
Lab-Industry 1: Fast ADC RTM & ADC-DAC
Includes IPMI Extension to RF Chassis,
Rear clock, trigger in & DAC out
Courtesy A. Young, SLAC & Struck Co.
June 4-5, 2012
New Developments in I&C Standards for
Physics
41
Example 2: Generic FPGA & Fast-Slow 12
bit ADCs RTM for interlocks
Courtesy D. Brown, SLAC & TEWS Co.
June 4-5, 2012
New Developments in I&C Standards for
Physics
42
V. xTCA Market Growth
Projections
June 4-5, 2012
SLAC MicroTCA Standards Review
43
Survey Data
• Limitations
– Survey 1 for total ATCA market commissioned by
industry leader, Emerson
– Survey 2 looked at only blade sales which appear to be
about half the total market
– Market segmentation shows 80%+ total market going to
Telecom
– MicroTCA is represented partly in remaining 20%
– Total of $800M/yr projected for 2012
– Close to entire VME market which is 80% military
June 4-5, 2012
SLAC MicroTCA Standards Review
44
Survey 1: ATCA Total Market Forecast
Note:
Stated target of
PICMG for
ATCA was
eventual
penetration of
global market of
10% of $100B
Note - Current
ATCA market is
approx same as
total VME market
June 4-5, 2012
SLAC MicroTCA Standards Review
45
Survey 1: ATCA Adoption Potential in
Adjacent Markets
• Markinetics also believes the broad ATCA software
ecosystem and developer community must also evolve in
parallel in order for ATCA to achieve greater acceptance
across multiple industries, including a number of missioncritical enterprise sectors.
• For instance, one specific opportunity on which some ATCA
suppliers have been attempting to capitalize is the
widespread reallocation of military and defense-related
spending toward network-centric cyber-security initiatives,
with ATCA as a common, open-standard platform option.
June 4-5, 2012
SLAC MicroTCA Standards Review
46
Survey 1: Forecast Analysis
• Looking ahead, the growth trajectory for ATCA adoption certainly will be
affected over the next five years, Markinetics believes, by a number of
important factors, including:
– 1. Improving macro- and micro-economic conditions and the gradual return
to an investment (rather than a cost-cutting) posture among the world’s
leading telecommunications services providers
– 2. Broader acceptance across various information-critical industries
adjacent to the traditional telecom sector
– 3. A far more engaged software developer community focused on enabling
robust ATCA-based solutions
– 4. Incremental advances in ATCA-compliant technology (e.g., engineering,
design, reduced energy consumption, thermal management, etc.)
– 5. A unique capability and capacity to adequately provide a range of support
services that complement the hardware product
June 4-5, 2012
SLAC MicroTCA Standards Review
47
Survey 2: Market Segmentation
Note:
Segmentation
in rough
agreement
with Survey 1
Note:
Current
telecom
market over
80% of total
June 4-5, 2012
SLAC MicroTCA Standards Review
48
VI. Summary & Conclusions
June 4-5, 2012
SLAC MicroTCA Standards Review
49
Summary
• Summaries of COTS and Lab initiatives,
latter principally in RTM sections, shown in
following sllides.
June 4-5, 2012
SLAC MicroTCA Standards Review
50
MTCA.4 Shelves 6/7 & 12 Slot
Item- Shelves
Mfgr
Type
Available
6-slot
Schroff
1-Star backplane.
√ Non redundant
7 slot
Schroff
1-Star backplane.
√ Standard PS
Removable fan
6-slot
ELMA
1-Star backplane.
√ Non redundant
12-slot
Schroff
2-Star redundant
√ Managed PU, CU
12-slot
ELMA
2-Star redundant
√ Managed PU, CU
12-slot
Perf Tech
2-Star redundant
6/12 for β test
6-slot
Power
Bridge
1 Star non
redundant
2U, 3 RTM slots
12 Slot VT811
Vadatech
2-Star redundant
6/12 release
June 4-5, 2012
SLAC MicroTCA Standards Review
51
MTCA.4 Power Modules
Item- Power
Modules
Mfgr
Type
Available
Puma
µBlade
Puma 900W
Unreliable, RF Noise
6-pack mod.
Schroff
300W Devmt
√ Incompatible w/12 slot
6-pack mod.
Elma
300W Devmt
√ Incompatible w/12 slot
12-pack mod. Schroff
900W
Due 8/12
6-Pack mod
PowerBridge
600W
Prototype delivered
12 Pack mod
Performan
ce Tech
1000W, 1200W
modules
Beta unit 6.12
12-pack mod. Vadatech
1000W W/ AC in
Due 7/12
12-pak mod.
Vadatech
1200W, -DC in
Due 7/12
12-pak mod
Wiener
900W AC in
Due 7/12
June 4-5, 2012
SLAC MicroTCA Standards Review
52
MCH Modules
ItemProcessors
Mfgr
Type
UT002
Vadatech
Full featured
√
PCIe Gen3
NAT
Full featured
√
?
Performance Tech PT
Full featured
√ Beta test
unit 6/12
?
Kontron
Full featured?
√
June 4-5, 2012
SLAC MicroTCA Standards Review
Available
53
MTCA.4 Processors
ItemProcessors
Mfgr
Type
Available
AMC121
Performance Tech PT
Intel core 2 duo
√ Beta test
unit 6/12
ASLP11
GE
Intel core 2 duo
√
AMC1000
Adlink
Intel core 2 duo
Unreliable
AM31x
Concurrent Tech
Intel i7
AM930
Concurrent Tech
2-wide i7
PCIeGen3
AMC 720
Vadatech
Intel i7
On Order
Radisys
Inteli7
√
June 4-5, 2012
SLAC MicroTCA Standards Review
√
9/12
54
MTCA.4 Generic AMC Modules
Item- AMCs
Mfgr
Type
Available
Digitizer 8300
Struck
125MSPS 16 bit 10 CH
√
Digitizer 720
Vadatech
125MSPS 16 bit 10 CH
Q3 12
Digitizers
SP Devices
1.6-7 GSPS 8-14 bits
No RTM
Q4 12
IP Adapter
Hytek
3-IP adapter w/RTM
√
TAMC220
TEWS
3-IP adapter w/RTM Intfce
√
TAMC651
TEWS
FPGA Processor Spartan 6
w/ RTM Intfce
√
June 4-5, 2012
SLAC MicroTCA Standards Review
55
Application RTM Adapters 2
ItemRTMs
Lab/
Co.
Type
Mates
With
Available
SIS 8300
√
LLRF
SLAC
Passive DC, IPMI Extend
Interlock
SLAC
Fast/Slow 12 bit ADCs
TEWS651
BPM
SLAC
Filter, Calibration Interface
SIS8300
1st proto in
test
LLRF
DESY
Downconverter 1.3-3.9 GHz
SIS8300
√
Sensors
Interface
XFEL
Avalanche photo diode
pulse stretcher
SIS 8300
√
TAMC002
TEWS
3-Industry Pack RTM
TAMC651
√
?
Hytek
3-Industry Pack RTM
?
√
ADC/DAC
DESY
8ch 16bit ADC, 8ch 16bit DAC DAMC2
1st proto in
test
ADC
DESY
Coupler interlock
1st proto
June 4-5, 2012
DAMC2
SLAC MicroTCA Standards Review
56
Application RTM Adapters 2
ItemRTMs
Lab/
Co.
Type
Digital IO
DESY
Machine protection system
DAMC2
1st proto in
production
ADC
DESY
Coupler interlock
DAMC2
1st proto test
ADC
DESY
Beam loss monitors
DAMC2
Development
ADC
DESY
Toroid protection / readout
DAMC2
Development
UCL
Clock & trigger control for exp.
DAMC2
1st proto test
Interface
DESY
Wire scanner
DAMC2
Development
BPM
DESY
Low charge button/stripline
SIS8300
1st proto test
June 4-5, 2012
Mates
With
SLAC MicroTCA Standards Review
Available
57
Why MicroTCA?
• New standards essential as technology develops, legacy
systems no longer cost-effective or supportable
– NIM (1966), CAMAC (1976), FASTBUS (1986), VMEp (1998)
• Last decade revolution in analog, digital, communication
technologies accelerates need for change
–
–
–
–
–
Programmable FPGA’s obsolete discrete logic components,
Multilayer board design enables Gigabit backplanes >10 GHz BW
Integrated SERDES communications obsolete parallel bus backplanes
LVDS balanced multi Gbps backplanes minimize discrete switch blades
Intelligent platform management diagnoses enterprise wide problems to
board level, initiates corrective action
– Redundant architectures => shelf Availability of 5 nines or better (0,99999)
June 4-5, 2012
SLAC MicroTCA Standards Review
58
Summary of Performance Advantages
• Excellent analog performance
• Modular partition AMC-RTM allows planned upgrades FPGA,
processor parts (Moore's Law) separate from analog and
interface module
• Partition interface enables different design skills to work on
complete system
• System based on high performance scalable state-of-art
communication technology only (PCIe and Ethernet)
• Common interfaces and operating systems for control systems
• High availability architecture and remote management/
maintenance, upgrades of firmware
•
June 4-5, 2012
SLAC MicroTCA Standards Review
59
V. Summary & Conclusions
• MicroTCA Advantages being recognized increasingly in
Europe and Asia and more labs are signing on (ESS,
ESSB, GSI, Spring8)
• PICMG industry support is solid and growing, much
stronger in US than lab support
• Standards collaborations under PICMG will continue to
refine timing protocols, software-firmware interfaces as
more users raise practical problems.
• MicroTCA has potential to be physics standard of choice for
new projects, upgrades for next 1-2 decades
June 4-5, 2012
SLAC MicroTCA Standards Review
60
END OF SLIDES
June 4-5, 2012
SLAC MicroTCA Standards Review
61