COSMIC Data Analysis and Archive Center (CDAAC) Overview and

COSMIC Data Analysis and Archive Center (CDAAC)
Overview and Status
B. Schreiner, B. Kuo, S. Sokolovskiy, D. Hunt, X. Yue, J. Zeng,
K. Hudnut, M. Sleziak, T.-K. Wee, T. Vanhove, J. Lin
UCAR COSMIC Program
6th FORMOSAT-3 / COSMIC Data Users Workshop 2012
Outline
• 
• 
• 
• 
• 
• 
COSMIC System Overview and Status
CDAAC Overview
Precise Orbit Determination and Excess Phase
Neutral Atmospheric Retrievals
Ionosphere and Space Weather
Status and Outlook
Getting
COSMIC
Results to Weather
Centers
COSMIC
Operational
Processing
JCSDA
TACC
RTSs:
Alaska
Norway
McMurdo
Hanscom
AFB
Input Data
-  COSMIC data
-  Attitude data
-  GPS ground data
-  GPS NDM Bits
-  GFS Forecast
-  IGS/ORB/CLK
-  Bernese Config
C
D
A
A
C
UCAR/Unidata’s
LDM
Research
WGET Community
1000-2000 WMO
BUFR Files
per day with
Latency ~ 75-90min
SFTP
Science & Archive
AFWA
N
E
S
D
I
S
NCEP
ECMWF
GTS
CWB
UKMO
JMA
Meteo
France
Canada Met.
Providing data to > 1,937 registered users from 66 countries > 3.4 Million Profiles in Real Time
4/21/06 – 10/21/2012
Change in Satellite Ops 4
COSMIC Firmware Status
•  New firmware uploaded to all
spacecraft in Jan 2012
-  L2C tracking for setting occultations
are high quality with few L2 failures
-  Improved tracking for rising
occultations after reboot
•  Possible future enhancements to
firmware
-  Implement L2C tracking for rising
occultations
-  Implement L2C tracking for reference
satellites
-  Track RO signals to lower HSL
routinely
-  Test 100 Hz tracking capability
Acknowledgement to JPL for
developing COSMIC receiver
firmware updates!
And to NSPO for uploading them
to COSMIC spacecraft!
Examples of ionospheric scintillation
L1CA and L2C signals
F-region scintillation
Sporadic E-clouds
L2C Tracking on COSMIC
Since Jan 2012, COSMIC receivers are configured to track L2C (setting occultations)
L2C Less noisy than L2P
L2P ~ 25% fail QC
L2C ~ 1-2% fail QC
Statistical comparison of COSMIC to
ECMWF in the stratosphere
Higher stability to fluctuations
L2C can be used down to lower
heights than L2P (L2P often fails
at sharp tropopause)
Benefits of L2C Tracking
•  Improves L2 SNR
•  Reduces BA noise in the stratosphere
•  Use of L2 down to lower heights for ionospheric correction
•  L2C Phase-Lock Loop tracking allows significant reduction of
errors due to diffractional effects for occultations affected by
the ionospheric scintillation
•  Combined use of L1CA and L2C Open-Loop signals may
reduce impact of noise in lower troposphere inversions
Data Latency
FM3 Out of Contact ~90% available within 3 hrs Average 75-­‐90 min for single orbit dump 9
Outline
• 
• 
• 
• 
• 
• 
COSMIC System Overview and Status
CDAAC Overview
Precise Orbit Determination and Excess Phase
Neutral Atmospheric Retrievals
Ionosphere and Space Weather
Status and Outlook
CDAAC Objectives
•  Provide high quality COSMIC RO data to the community in NRT,
post-processing, re-processing (re-analysis)
•  Improve data processing algorithms to maximize impact of RO
data on research and operations
•  Support data users, TACC
•  Support other RO missions of opportunity (C/NOFS, SAC-C, ..)
•  Perform validation and science investigations
CDAAC Design
tipLv1
(Radiances)
TIP
scnLv1 (S4/σΦ)
CDAAC File types in RED
Scint.
podObs
Atmospheric processing
atmPhs
RINEX
LEO Science
and SOH data
GNSS Clock
Estimation
Excess
Phase
Abel
Inversion
atmPrf
(α,N,T,P)
1-D Var
Moisture
Correction
Level 0--level 1
wetPrf
bfrPrf
LEO POD
Real time Task Scheduling Software
Profiles
ionPrf
(EDPs)
Ionospheric processing
GNSS ground data
GNSS NDM Bits
IGS/IGU ORB/CLK
Bernese Config files
GFS Forecast
Absolute
TEC
Excess
Phase
Ionospheric
Inversion
ionPhs
podTec (TEC)
12
COSMIC Data Products
•  LEO POD and excess phase (<15 cm, <0.15mm/s 3D rms) •  > 3.4M neutral atmospheric profiles: -  Bending angle noise ~1.5-­‐2.0 μ-­‐rad between 60-­‐80km •  > 3M Absolute TEC data arcs: -  Absolute accuracy ~ 1-­‐3 TECU •  > 3.3M Electron Density Profiles: -  NmF2/hmF2 Accuracy ~20%/20km (compared to Ionosondes) •  ScinWllaWon Indices (S4): -  ~3M available from occultaQon profile events (alQtudes < 120 km) -  1 Hz data available from > 3M line of sight tracks to all GPS in view •  TIP Night-­‐side Radiances: -  TIP set new standard for sensiQvity of UV instruments, ~500 counts/s/Rayleigh -  > 17,800 hours of quality controlled data  
New Level 3 gridded climate data product: mmcGrd -  Includes monthly means of BA, Ref, T, P, GeopHt
How do we know our CDAAC RO data
processing is optimal?
•  Review of algorithms and software internally and via publication
•  Identify/Analyze QC failures
•  Compare results (POD, BA, N, T, WV, TEC, EDP) from mission
to mission
•  Comparison of results with other data processing software and
investigate and understand differences
–  more detailed RO-Trends studies
–  ROPP comparisons
•  Comparison with correlative data and models
•  Get feedback from science application studies
Outline
• 
• 
• 
• 
• 
• 
COSMIC System Overview and Status
CDAAC Overview
Precise Orbit Determination and Excess Phase
Neutral Atmospheric Retrievals
Ionosphere and Space Weather
Status and Outlook
LEO POD at CDAAC
with Bernese v5.0
• 
• 
• 
• 
Zero-Difference ionosphere-free carrier phase
observables with reduced-dynamic processing
(Svehla and Rothacher, 2003)
Dynamic Model: Gravity - EIGEN1S, Tides (3rd body, solid Earth, ocean)
State Parameters:
–  6 initial conditions (Keplerian elements)
–  9 solar radiation pressure parameters (bias
and 1 cycle per orbital revolution
accelerations in radial, transverse, and
normal directions)
–  pseudo-stochastic velocity pulses in R-T-N
directions every 12 minutes
–  real valued ambiguities
–  LEO clock offset
Quality Control
–  Post-fit residuals
–  Internal overlaps
Errors in satellite velocity
contribute directly to errors in
RO temperature retrievals
METOP/GRAS Post-Processed
External Overlaps
UCAR – EUMETSAT (2007.274-305)
Radial
Along-Track Cross-Track
3-D RSS
POS [cm]
(VEL: [mm/s])
POS [cm]
(VEL: [mm/s])
POS [cm]
(VEL: [mm/s])
POS [cm]
(VEL: [mm/s])
Mean
0.7
(-0.01)
0.9
(-0.01)
2.9
(0.00)
-
STD
4.2
(0.05)
6.0
(0.08)
4.1
(0.04)
8.6
(0.08)
•  POD quality
-  COSMIC ~ 15-20 cm (0.15-0.2 mm/s) 3D RMS
-  METOP/GRAS < 10 cm (0.1 mm/s) 3D RMS
Velocity error of 0.1 mm/s results in bending angle error
of ~2e-8 rad
POD and Atmospheric Excess Phase
Processing Status
•  Current Problems/Issues
–  Stig Syndergaard reported variations in
excess phase that were correlated with
variations in GPS/LEO positions – software
fix being tested
–  UCAR detected inconsistency between GPS/
LEO velocities and differentiation of positions
– under investigation
–  Barbara Scherllin-Pirscher noted systematic
differences between Metop/GRAS and
COSMIC data – under investigation
•  Planned Improvements
–  Software restructured to be more modular
and maintainable
–  Compute/Apply PCV maps
–  Compute/Apply 5-sec GPS clocks
Excess Phase Bug Reported by Stig S.
BA noise between 60-80 km
COSMIC - SD
Metop/GRAS - ZD
2012.090
2007.274
OLD
Software
2007.274
2012.090
Agrees well
With GRAS
data from
EUMETSAT
NEW
Software
Under
test
Excess Phase Bug Reported by Stig S.
UCAR Metop/GRAS - ECMWF
OLD Software
2007.274
NEW Software under test
Software fix has insignificant impact on BA’s as compared to ECMWF
Outline
• 
• 
• 
• 
• 
• 
COSMIC System Overview and Status
CDAAC Overview
Precise Orbit Determination and Excess Phase
Neutral Atmospheric Retrievals
Ionosphere and Space Weather
Status and Outlook
Raw Phase/Amplitude
Neutral Atmospheric
Data Processing
Precise Orbit Determination / excess phase process
No error propagation
Excess Phase (Doppler)
local transform
no error propagation
Bending angle
non-local transform (Abel inversion)
error propagation downward
Refractivity (density)
non-local transform (hydrostatic integration)
error propagation downward
Pressure, temperature
22
Gorbunov, M.E., A.V. Shmakov, S.S. Leroy, and K.B. Lauritsen, COSMIC Radio Occultation
Processing: Cross-Center Comparison and Validation. DOI:10.1175/2011JTECHA1489.1
Quantification of Structural Uncertainty of RO retrievals data among centers using
profile-to-profile results among 5 centers
The time series of fractional refractivity anomalies among five centers for the 8-12 km layer for
(a)  the 20ºN to 60ºN zone, (c) the 20ºN to 20ºS zone, and (e) the 60ºS to 90ºS zone, and in the
(b)  20-30 km layer for (b) the 20ºN to 60ºN zone, (d) the 20ºN to 20ºS zone, and (f) the 60ºS to
(c)  90ºS zone.
Ho, et al. 2012, The Reproducibility of GPS Radio Occultation for Climate Monitoring:
Profile to Profile Inter-comparison of CHAMP Climate Bending Angle, Refractivity,
Temperature, Geo-potential Height, and Pressure Records 2002-2008 from Different Data Centers.
JGR, in press.
24
Statistical comparison of COSMIC retrieved refractivities to ECMWF
over the tropical ocean
Red lines: mean deviation
Green lines: +/- standard deviation
Blue lines: penetration (bottom heights of retrieved profiles)
GPS azimuth: 0-10 deg
SNR ~ 700-900 V/V
~ 80% profiles penetrate to 1 km
GPS azimuth: 50-60 deg
SNR ~ 400-600 V/V
~ 20% profiles penetrate to 1 km;
mean and standard deviation
(inversion errors)
are substantially larger
Neutral Atmospheric Retrievals
QC
•  COSMIC data from Jan-Feb 2009
-  Green failures due to poor quality of L2 signal
-  Tan failures due to large departures from climatology
-  Red failures due to problem with rising occultation tracking
70-90 N
FM1 vs Latitude
sudden stratospheric
warming event
NEW Neutral Atmospheric Retrievals SW
Undergoing test/evaluation
• 
• 
• 
• 
• 
• 
Software restructured to be more modular and maintainable
Wave Optics (Phase Matching) Algorithm
Fixed height (20 km) for GO/WO transition
New statistical optimization algorithm
MSISE added to library of BA climatologies
Filtering of Doppler for GO processing: window consistent
with Fresnel scale (different in time for different occultations)
•  Calculation of lat. & lon. of TP: use of BA from climatology
(CIRA+Q) instead of raw BA obs.
•  QC
•  Additional scalar output parameters
Optimization of the bending angles for Abel inversion
Current approach: Lohmann, Radio Sci., 2005 (dynamic error estimation; log-fitting
of background to obs. BA; dynamic estimation of the height interval for the fitting)
Revised approach (eliminates dynamic estimates in favor of climate applications)
1) Direct fitting of background to obs. BA (eliminates bias)
2) Fixed height interval for the fitting: 35 - 60 km
3) Mixing obs. BA with fitted background in fixed interval 35 - 60 km
4) Mixing of mixed (3) BA with background in fixed interval 55 - 65 km
direct fitting
log-fitting
mixing
Neutral Atmospheric Profiles
Scalar Output Parameters
Area of Interest
atmPrf Scalar Variable
Description
Bending angle uncertainty for an
occultation
smean
Mean deviation of ionosphere-free BA from
climatology
Bending angle uncertainty for an
occultation
stdv
Standard deviation of ionosphere-free BA
from climatology
Signal-to-Noise
Ratio for an occultation
snr1avg, snr1del
snr2avg, snr2del
Mean SNR, linear trend of SNR,
for L1 and L2, between 60 and 80 km
Tropopause Height and
Temperature
trtwmo, trhwmo
trtwmo2, trhwmo2
trtcp, trhcp
Tropopause height and temperature from
WMO definition (incl. 2nd tropopause), and
cold point definition
Atmospheric Boundary Layer
characterization
balmax
zbalmax
max. BA lapse in variable sliding window
and median height
Atmospheric Boundary Layer
characterization
bpnmax
zbpnmax
N break point (lapse of N-gradient by sliding
linear regression in 0.5 km window) and
median height
Ionospheric Scintillation
S4
New Scalar Parameters
Sporadic E-clouds
Detection of multiple Es effects: heights of
TP and intervals of amplitude fade
Atmospheric Boundary Layer
characterization
max. N lapse in sliding window 0.3 km and
median height
Convection in the moist
troposphere
Characterization of moist convection, depth
of convective layer (under revision)
New Retrieval Results
Jan.1-15, 2009
decrease of the
bias and stand.
dev. at 40 km
(polar winter)
increase of the
bias and stand.
dev. at 40 km
(polar summer)
lat > 60N
lat < 60S
OLD
OLD
NEW
NEW
New Retrieval Results
30S < lat < 30N
Jan.1-15, 2009
decrease of the
bias in lower
Troposphere
Reduction of
penetration depth
OLD
OLD
NEW
NEW
Neutral Atmospheric Retrieval
Status
•  Currently under consideration
–  20 km fixed height for GO/WO transition
–  Implement new BA file
–  Improve error profiles
–  Tune QC algorithm
•  Longer Term Plans
–  Combination of L1CA and L2C OL signals
–  Minimize small-scale ionospheric residuals due to
diffraction effects via Back Propagation
–  Minimize large-scale ionospheric residuals via ray-tracing
through ionospheric re-analysis
Outline
• 
• 
• 
• 
• 
• 
COSMIC System Overview and Status
CDAAC Overview
Precise Orbit Determination and Excess Phase
Neutral Atmospheric Retrievals
Ionosphere and Space Weather
Status and Outlook
TEC Precision from Collocated Tracks
COSMIC (2006.230-245)
Data Assimilation Retrieval of Electron Density
Profiles from Radio Occultation Measurements
Comparison of standard COSMIC Abel
retrieval and data assimilation retrieval with
Ionosonde data
Simulation of retrieval errors for standard
COSMIC Abel retrievals and data assimilation
retrievals
Truth
Abel
DA
Abel
Error
Large
Errors
DA
Error
Yue, X., W. S. Schreiner, Y.-C. Lin, C. Rocken, Y.-H. Kuo,
and B. Zhao, 2010: Data Assimilation Retrieval of Electron
Density Profiles from Radio Occultation Measurements. J.
Geophys. Res., 116, A03317, doi:10.1029/2010JA015980"
Geomagnetic latitude and altitude variations of
electron density during noon time (LT=13)
Data sources for Re-Analysis
•  Reanalysis
evaluation 1:
Compare with
RO retrieved
results
•  Reanalysis
evaluation 3:
Compare with
Poker Flat ISR
（65N，147W）
Ionosphere and Space Weather
•  Current Space Weather Data Products
– 
– 
– 
– 
– 
> 3 Million Absolute TEC data arcs (1-3 TECU accuracy)
> 3 Million EDPs
Large amount of L-band scintillation data
~90% available within 3 hrs
TEC/EDP data available for other missions
•  Available in short term
–  Ionospheric Re-Analysis will be published on website soon
–  Improved COSMIC Scintillation data (10-sec average S4)
–  Improved Re-Analysis version based on theoretical model (TIEGCM)
•  Long Term
–  Accurate Nowcast/Short-Term Forecast system of Ionosphere/
Thermosphere based on huge amount of COSMIC-2 data
CDAAC Near Term Plans
(in next 3-4 months)
•  CDAAC System
-  Add new data products (daily netCDF files, climatologies, ..)
•  POD and Excess Phase processing
-  Finalize bug fixes and validation of software
•  Neutral Atmospheric Products
-  Finalize development/validation of NEW retrieval software
•  Continue Re-processing efforts with NEW software
•  Provide test dataset of NEW products to weather
centers
•  Ionospheric Products
-  Publish Re-Analysis products
-  Scintillation improvements (provide 10-sec average S4)
CDAAC Data Access
•  New Batch Data Delivery System
-  Online since Oct. 2, 2012
-  Available file types: atmPrf, atmPhs,
wetPrf, avnPrf, ncpPrf, ecmPrf, sonPrf,
ionPhs, ionPrf
-  Average processing time is under 2 min
-  5 GB/user/day with unlimited file counts
-  When the job is done, user will get
notification email with download links
-  Finished data archive files will be kept on
our server for 48 hours
-  ~1M files downloaded in two weeks
-  Soon to add daily tar files for selected file
types to speed up downloading
•  Other download methods
-  GTS, LDM, GoogleMaps
-  FTP, HTTP, Loaded HD
http://cdaac-www.cosmic.ucar.edu/cdaac/DBif/cdaac_highlevel.cgi
CDAAC Support to the Community
•  Supporting 1937 data users from 66
countries
•  Providing NRT data to NWPs, AFWA
•  Providing NRT, Post-Processed, and
climate Re-analysis products from 9
RO missions
•  Development of improved RO data
processing algorithms
•  Providing data, user support, and
science data processing software to
community
Data Downloaded ~250TB
(1/3 from university community)
Acknowledgments
• 
• 
• 
• 
• 
NSF
Taiwan’s NSPO
NASA/JPL, NOAA, USAF, ONR, NRL
Broad Reach Engineering
Other RO Missions, CHAMP, SAC-C, GRACE,
TerraSAR-X, C/NOFS, Metop-A/GRAS
Questions?
Backup Slides
Single-Difference details
Subtract ionosphere-free reference link phase, and eliminate other terms to get:
Smooth reference link L1-L2 to reduce L2 noise,
but increases the uncorrected small-scale
ionospheric effects (Rocken et al., 1997,
Schreiner et al., 1998, Beyerle et al., 2005)
denotes
smoothing
with a 2 sec
window
Upper stratosphere and lower troposphere are regions
of maximum uncertainty for GPS RO inversions
at what height to
start using signal
for inversion
?
In the lower troposphere:
the signal reduces below noise level
in terms of the amplitude
In the upper stratosphere:
the signal reduces below noise level
in terms of the phase (Doppler)
Fixed Height for GO/WO Transition
Largest difference between old and new
processings
is in the moist lower troposphere
(region of large structural uncertainty)
RH filtering tested: slightly reduces
the structural uncertainty, but degrades
resolution of the sharp top of ABL
Is Wave Optics needed up to 20 km
to resolve the tropical tropopause?
Is Geometric Optics OK there?
- currently for modeling observable (excess phase or full phase) to calculate BA,
positions and velocities from Bernese SW are used
- generally, inversion of the excess phase or the full phase yields same results - most occs.
- some occultations: significant difference (bias) in BA; can be explained by inconsistency
of velocities and position derivatives in Bernese SW
- inversion of the full phase is sensitive to velocity errors, less sensitive to position errors
- inversion of the excess phase is sensitive to position errors, less sensitive to velocity errors
First level 3 CDAAC data type
New gridded climate data product: mmcGrd
  Useful for study of climate trends in RO data
  Defined by an international group of climate scientists for the
rotrends radio occultation processing inter-comparison project
  A netCDF format which contains monthly means of various RO
observables, gridded into latitude bands
  Includes:
 
- 
- 
- 
- 
 
 
Bending angle
Refractivity
Dry temperature and pressure
Geopotential height
Available for the most recent reprocessing only: metopa2011
Will be available for other missions as they are reprocessed
Other Missions
•  SAC-C power anomaly. Return to service possible by Dec 2012
•  C/NOFS nominal. Possible to increase # of soundings
•  Metop-A/GRAS nominal. Metop-B/GRAS launched
•  Tandem-X to provide rising occultations
•  KOMPSAT-5 (IGOR+), Launch ~ late 2012
•  Spanish PAZ mission (IGOR+), Launch ~ late 2013
•  EQUARS, Launch 2014: Having discussions with INPE on data
processing