Value for Money Audit of the Labour Market Re-entry Program

Value for Money Audit of the Labour Market Re-entry Program
WSIB
Executive Summary
of Final Report
1
Table of Contents
•
Background……………………………………………………………………………………..3
•
Project Scope & Objectives…………………………………………………………..…….…4
•
Summary Audit Findings………………….…………………………………………………...7
•
Findings & Conclusions
•
– Alignment Review…………………………………….…………………………….….....8
– Risk Management & Control……………………………………………………..….....10
– LMR Program Performance Assessment……………………………………………..14
Prioritized Recommendations….…………………………………..……...….……………..28
•
Summary Opinion and Auditors’ Report…..………………………………………….….....43
2
Background
• To meet legislative requirements, the Workplace Safety and Insurance Board (WSIB) is
required to undertake an annual Value for Money Audit (VFMA) of at least one program
delivered under the Act.
• For 2003, the Labour Market Re-entry (LMR) Program was identified as the focus for this
year’s VFMA. Judy Geary, General Manager of Specialized Claim Services, is the
executive sponsor for this audit.
• Following a formal request for proposal and vendor selection process, Deloitte & Touche
LLP was engaged to conduct the VFMA.
3
Project Scope & Objectives
Project Scope
The audit begins from the point in the RTW process where a referral for LMR services is made, and focuses
on WSIB’s involvement and responsibilities in the LMR program. More specifically, these responsibilities
include:
• Establishing criteria for selecting, approving, and maintaining a roster of LMR Service Providers.
• Identifying those workers who may benefit from an LMR assessment
• Determining whether an LMR plan is required to enable the worker to re-enter the labour market and
that the plan is the most appropriate for the worker, and cost effective.
• Reviewing the recommended Suitable Employment or Business (SEB) to ensure that it will enable
the worker to reduce or eliminate his/her loss of earnings resulting from the injury.
• Monitoring and evaluating the services provided by the LMR Providers.
Project Objectives
The purpose and main objective of this engagement is to conduct an external VFMA in order to review the
outcomes generated by WSIB’s LMR program. Specifically, the audit will:
• Provide an audit opinion as to whether there is value for money in the outcomes generated by the
LMR program provided by the WSIB and recommend improvements that could be made.
• Assess whether WSIB is getting value for money from external providers of LMR services.
• Comment on the validity and/or recommend valid performance measurement indicators that relate to
the WSIB stated outcomes for the LMR program, and provide opportunities to enhance operations.
• Provide practical recommendations on how to improve the Value-for-Money of services that the
WSIB provides, and possible implementation plans and options.
• As requested by the WSIB, assist WSIB management in the formulation of the WSIB management
response.
4
Project Approach: Specific Audit Criteria
• From a “statutory” perspective, the purpose of a Value for Money Audit is to
examine and report on the following:
• The adequacy of management systems (including performance
standards and measurement), controls and practices, including those
intended to control and safeguard assets, to ensure due regard to cost,
efficiency and effectiveness. (SYSTEMS)
• The extent to which resources have been managed in conducting
relevant activities with due regard to cost and efficiency. (RESOURCES)
• The extent to which programs, operations or activities of an entity have
been effective. (OUTCOMES)
• Our audit approach involves three components, the combined results of
which yield our audit conclusions:
• Alignment Review
• Risk Management & Control
• Program Performance Assessment
5
The Journey to LMR Optimal Performance
Developmental
Stage
Optimal
Performance
X
Emerging
January 1998
Legislative
Change
October 1998
Change from
VR to Provider
Model
January
2004
WSIB has made significant progress towards its
goals as related to LMR since the introduction of
new legislation. This progress is evidenced by
initiatives such as the following:
• Development and roll out of PBAS system,
designed to assist with the day-to-day management
and monitoring of LMR Program, including Primary
Provider relationships and costs
• Development of LMR training initiatives
• Formation of LMR Evaluation unit within QI
• Initial development of new RTW strategy
incorporating LMR
• Preliminary conceptual development of RTW unit to
include LMR, as well as the associated
accountability model
• Formation of LMR Protocols Committee
• Undertaking the LMR VFM Audit as a checkpoint to
confirm that future direction is appropriate
WSIB has made steady progress to date. Continued development and greater focus in particular
areas is now required to enable WSIB to attain optimal LMR performance.
6
Summary Audit Findings
The graphic below portrays our summary audit findings of each component as it relates to the LMR program
only. These findings are outlined in more detail in the following report:
Strategic Alignment
Risk Management &
Control (RMC)
LMR Program Performance
Measurement & Management
(LMR Program PMM)
Policy and
Legislation
Systems
Systems
Resources
Resources
Outcomes
Outcomes
Corporate
Outcomes &
Legislation
LMR Activities to
Corporate
Outcomes
LMR
Activities &
Legislation
WPP
Expectations
& Legislation
Key:
Acceptable
Some Improvements
Required
Significant Improvements
Required
Critical
7
Project Approach: Alignment Review
Corporate Outcomes
•
In the absence of a standalone LMR
Strategy (combined RTW and LMR
strategy document currently under
development) Deloitte conducted the
alignment review with reference to
Legislation
•
We first performed a higher level
alignment review of legislation with
corporate outcomes, policy, and WPP
expectations
•
We then conducted the risk self
assessment workshop to identify any
further areas of misalignment between
these components
•
We then performed a lower level
alignment review to assess the extent to
which the current LMR Program supports
the legislation
Policy
Legislation
Activities
WPP Expectations
8
Alignment Review: Conclusions
Policy and
Legislation
– Having completed an analysis exercise to assess the degree of alignment
between the WSIB LMR policy and relevant legislation, we assess the policies to
be acceptable in keeping with legislation
Corporate Outcomes
& Legislation
– Having reviewed the alignment between the legislation and corporate outcomes,
we are satisfied that the level of alignment is acceptable
LMR Activities to
Corporate Outcomes
– Having reviewed the alignment between LMR activities and Corporate Outcomes,
we are satisfied that LMR activities assist with the promotion of broader WSIB
objectives, which is an acceptable result.
LMR Activities &
Legislation
WPP
Expectations
& Legislation
– We have also reviewed the alignment between the LMR activities and relevant
legislation, which we consider acceptable.
– We believe that some improvements are required with respect to the alignment
between workplace party expectations and legislation. While WSIB’s activities
are clearly in alignment with its legislated mandate, or in some instances exceed
the mandate (as indicated by the preceding alignments) this does not meet all
workplace party expectations.
– There is a disconnect between the workplace parties’ desire for employment
and the “employability” mandate of legislation, which is a function of both lack
of knowledge and / or fundamental disagreement with the legislation
– As such, WSIB may need to develop improved communications to workplace
parties to enable better alignment of expectations and/or modify delivery efforts.
However, improved communication will not assist those parties who understand
the legislation yet fundamentally disagree.
9
Project Approach: Risk Management & Control
• Using the input from the documentation review, alignment review and the interview
process, we assisted WSIB to perform a self-assessment on the likelihood and
potential impact of the risks associated with LMR
– respondent scores were averaged and plotted on the matrix below, based on the perceived
impact and likelihood of the risk
• Following the identification of compensating mitigating controls, we then tested for
the adequacy and existence of these controls
Average Response to all Polls
Audit Opinion Dimensions
Impact
Systems – “Are controls in place to
manage system risks?”
Resources – “Are controls in place to
manage resource risks?”
Outcomes – “Are controls in place to
manage outcome risks?”
Likelihood
10
Risk Management & Control Self-Assessment: Findings
Ris k
Re f.
L
A
N
M
P
C
J
R
E
B
F
Q
K
I
O
H
G
D
Potential Risk
Risk of inadequate performance measurement and monitoring of LMR
(system risk)
Risk of inappropriate referral to LMR Program (late, early or
unnecessary) (system & outcome risk)
Risk of WSIB’s reluctance to remove poorly performing Primary
Service Providers (outcome risk)
Risk that LMR Evaluation Unit audit process is not sufficient to
address the risks (system risk)
Risk that Primary Service Provider costs are not effectively monitored
and contained by WSIB (resource risk)
Risk of an inability to obtain buy-in/support from WSIB staff
regarding current service model (resource risk)
Risk of inadequate Secondary Service Providers service delivery
resulting in poor quality outcomes (outcome risk)
Risk of adverse public opinion (government, worker, worker
representatives, employer, employer representatives) (system risk)
Risk of inconsistent application, or absence of, WSIB policy (system
risk)
Risk of unclear LMR ownership and accountability within WSIB
(system risk)
Risk of ineffective LMR internal processes, from referral to closure
(outcome risk)
Risk that WSIB does not know total LMR costs (resource risk)
Risk of a failure to ensure that LMR plans are suitable, appropriate and
tailored to meet the needs of the worker (system & outcome risk)
Risk of inadequate WSIB contract and controls to monitor Primary
Service Provider quality and outcomes (system & resource risk)
Risk of non-arms length transactions between Primary and Secondary
Service Providers (resource risk)
Risk of a lack of collaboration between WSIB / Primary Service
Providers and the Employer (system risk)
Risk of a lack of collaboration between WSIB / Primary Service
Providers and the Worker (system risk)
Risk of inadequate security and privacy measures (system risk)
Score (Impact
x Likelihood)
14.1
12.2
11.5
10.9
10.9
10.6
10.6
10.2
9.9
9.9
9.4
9.3
9.0
9.0
8.1
7.6
7.0
5.7
11
Risk Management & Control: Conclusions
Systems:
•
In our opinion, WSIB has demonstrated that controls are in place to manage certain
systems risks to an acceptable level of exposure with potentially some improvement
opportunities. For other systems risks, the degree of risk exposure was selfassessed as either Significant Improvements Required or Critical.
•
We believe that although much data has been collected, it has not been
appropriately tailored to track meaningful performance measures for the LMR
program. Meaningful performance measures would facilitate the management of a
number of the top operational risks identified. In addition, the PBAS tool has not
been used to its full potential, both due to system issues and a lack of user
knowledge and accessibility.
•
Further, while some activity level roles and accountabilities are clear, there are
areas in which it is less clear which may be adversely impacting outcomes. In
particular, there is currently no overall program accountability.
•
Further, there is a need for both PSP (Primary Service Provider) and SSP
(Secondary Service Provider) performance to be linked to outcomes. This has not
occurred to date.
•
Overall we deem the risk and controls related to system risks as requiring
significant improvement.
Systems
12
Risk Management & Control: Conclusions
n
Resources
Resources:
•
•
n
Outcomes
Most resource risks were self assessed as requiring some
improvement. Our testing of controls on these risks revealed an
acceptable level of exposure. The key improvement area is the
need to ensure the visibility and integrity of all program costs.
Overall, we assess risk and control related to resource risks as
requiring some improvements.
Outcomes:
•
•
Outcome related risks were self-assessed as requiring some or
significant improvement. One of the primary driving factors behind
this is the fact that PSP and SSP performances are currently not
linked to outcomes. For those outcome risks self assessed as
“acceptable with some improvements”, we confirmed the
existence and adequacy of the controls in place.
Overall, we assess risk and control related to outcome risks as
requiring some improvements.
13
Project Approach – LMR Program Performance Assessment
Using the input from our higher level alignment review, the risk self assessment, the
documentation review and the interview process, we determined the extent to which
the current LMR program is achieving Value for Money for the WSIB.
Audit Opinion
Dimensions
Systems
Selecting/
Managing
PSP
Performance
Management
LMR Needs
Assessment
Performance Management
• Accountability
• Organization Structure
• Activity Alignment With Legislation
Resources
Performance Measurement
• Activity Economy
• Activity Efficiency
Outcomes
Performance Measurement
• Activity Effectiveness
Case
Management
14
LMR Program Performance Assessment - Key LMR Activities
Selecting and
Managing LMR
Service Providers
Assessing the
Need for the
LMR Program
LMR Case
Management
LMR Performance
Management
Select and Extend
Contracts
with Primary
Service Providers
LMR Threshold
Decision
LMR Plan
Implementation
Provide Information
to Providers
Evaluate and
Monitor Quality of
Services Provided
Communicate with
Worker and Accident
Employer (AE)
LMR Addendum
Process
Manage Quality
and Continuous
Improvement
Manage Relationships
with Service Providers
Prepare LMR
Referral
Monitor, Evaluate
and Review
LMR Plan
Manage Costs and
Develop Performance
Measures
LMR Assessment
LMR Billings and
Invoicing
Manage Stakeholders
LMR Plan Approval
LMR Closure
Manage Internal
Resources
For analysis purposes, we divided the LMR process into four phases and corresponding key
activities. These phases and/or activities do not necessarily follow a linear sequence.
15
LMR Program Performance Assessment - Systems Findings
(Performance Measures Existence, Role Clarity, Accountability related to LMR)
M anag e
Relat io ns hip s wit h
Service Pro vid ers
Pro vid e Info rmat io n
t o Pro vid ers
X
S
S
S
S
S
LM R Ad d end um
Pro ces s
S
X
M anag e Qualit y and
Co ntinuo us
Imp ro vement
S
S
S
S
Prep are LM R
referral
S
S
M o nito r, Evaluat e
and Review LM R
Plan
S
X
M anag e Co s t s and
Develo p
Perfo rmance
M eas ures
X
X
LMR As s es s ment
S
S
LM R Billing s and
Invo icing
S
S
M anag e
St akeho ld ers
S
S
S
X
LMR Plan Ap p ro val
S
S
LM R Clo s ure
S
S
M anag e Int ernal
Res o urces
S
S
S
X
X
Co mmunicate with
wo rker and
accid ent emp lo yer
X
S
S
Key: Measure/Role Accountability
Comprehensive/Defined
S: Some/Somewhat Defined
Efficiency
Measures
Effectiveness
Measures
Efficiency
Measures
Effectiveness
Measures
Efficiency
Measures
Effectiveness
Measures
S
Accountability
Defined
S
Role Defined
S
Accountability
Defined
LM R Plan
Imp lement atio n
S
Role Defined
S
LMR Thres ho ld
Decis io n
S
X
LMR Performance
Management
S
Accountability
Defined
S
LMR Case
Management
Accountability
Defined
Evaluate and
M o nito r Qualit y o f
Services Pro vid ed
Assessing the
Need for the
LMR Program
Role Defined
Select and Ext end
Co ntract s wit h
Primary Service
Pro vid ers
Role Defined
Efficiency
Measures
Effectiveness
Measures
Selecting and
Managing LMR
Service Providers
N/A N/A
X: None/Overlap, Gap or No Definition
Note:
•All activities have been assessed only within the scope of LMR.
•Refer to Appendix 3 & 4 for detail on measures and roles
At the current time, limited performance measures exist. While roles and accountabilities are clear
from the point of referral to closure, overall process role and accountability is less clear, particularly
with respect to performance management.
16
LMR Program Performance Assessment – Reduction in Loss of Earnings
Benefits
LMR Closure
Phase 3
Benefit Rates at Referral Date to Benefit Rate 30th
Date After Closure: October 1998 - October 2002
Benefit Rate on 30th Date After LMR Closure
$800.00
Interpreting the 45 Degree Line:
• The 45 degree line indicates the range where there has
been little to no loss of earnings 30 days after the
closure of LMR.
$700.00
• Points below the line indicate a reduction in LOE
benefits after LMR completion.
$600.00
$500.00
• Those points only slightly above the 45 degree line
likely represent those workers granted full benefit,
adjusted for cost of living allowances. Potential reasons
for full benefits include processing lags, 100% LOE
awards and cases closed for medical reasons.
$400.00
$300.00
• Those points significantly above the line show cases
where there has been an increase in LOE. One
reasons for this may be the result of non-cooperative
cases where the worker has re-entered the program.
$200.00
$100.00
$0.00
$0.00 $100.0 $200.0 $300.0 $400.0 $500.0 $600.0 $700.0 $800.0
0
0
0
0
0
0
0
0
Benefit Rate on LMR Referral Date
• WSIB should investigate other factors that contribute to
variations in LOE% on closure.
Source : WSIB Legacy System Data
(includes all cases with a closure code)
The graph suggests that LMR has been fairly effective in reducing LOE 30 days after closure at an average of
approximately $126.60 weekly across all closed cases. However, given various data complexities, it is difficult to draw firm
conclusions. Beyond the 30th date after closure, the trend is for further declines in LOE benefits. WSIB should use this
concept in its development of performance measures for the LMR program.
17
LMR Program Performance Assessment - Total LMR Cost 1998-2003
Manage Costs and Develop Performance Measures
Phase 4
At the current time, all meaningful cost analysis should be done on an aggregate basis for 1998-2003 .
• Annual costs have increased year-over-year as more workers have participated in the program. On program
maturity, costs will stabilize,making comparison easier year-over-year.
• Historically, annual Provider costs have been recognized at time of payment. Given the payment backlog
early in the program, the year of payment may not correspond to the year the cost was incurred, again
making annual comparisons difficult to perform.
1998-2003
%
External Costs
Primary
Secondary
$ 136,588,331 33.4%
185,087,449 45.2%
Worker
59,747,316 14.6%
$ 381,423,096 93.2%
Total External
RTW Advisors
1.2%
Worker
14.6%
Internal Costs
RTW Advisors
Adjudicators
Nurse Case Managers
LMR Program
Adjudicators
4.7%
$
Total Internal
5,092,852
19,226,363
279,341
3,368,444
$ 27,967,001
1.2%
4.7%
0.1%
0.8%
6.8%
Total LMR Cost
$ 409,390,097
100%
Nurse Case
Managers
0.1%
LMR Program
0.8%
Primary
33.4%
Secondary
45.2%
Over the life of the LMR Program, Primary and Secondary Provider Costs have represented close to 80%
of total program costs, which exemplifies the need to have strict monitoring procedures over these areas.
Of note though, administration costs represent only 7%, which is low for program administration (10% is a
typical benchmark).
18
LMR Program Performance Assessment – Employment vs. Employability
Outcomes
Manage Costs and Develop Performance Measures
Phase 4
Outcomes for Completed LMR Programs Jan 99-Oct 02
100%
90%
80%
33.0%
31.0%
22.7%
23.0%
36.0%
37.0%
23.0%
22.0%
17.0%
16.0%
17.0%
16.0%
32.0%
70%
60%
22.0%
50%
40%
13.0%
14.5%
18.0%
30%
20%
10%
0%
17.0%
17.2%
7.0%
5.8%
6.8%
9.0%
Overall
Duration 0-12
months
3.0%
4.0%
Duration 13-24
months
21.0%
5.0%
4.0%
Duration 25-36
months
2.0%
5.0%
Duration 37-48
months
Employed Accident Employer - No Benefits
Employed Accident Employer - Partial Benefits
Employed Other Employer - No Benefits
Employed Other Employer - Partial Benefits
Employable - No Benefits
Employable - Partial Benefits
Source: Board of Directors
Presentation
Tracking of outcome data for LMR is still evolving for those completing the program. Overall 44% achieve
employment with partial or no benefits. 47% of Workers with no benefits are either employed or
considered employable (the main LMR objective).
19
LMR Program Performance Assessment – Complete vs. Incomplete
Outcomes
Phase 4
Manage Costs and Develop Performance Measures
Status LMR Referrals 1998-2003
Declined Employment
0%
No LMR Activity
3%
Breakdown of Drop Out Rate for LMR Cases
1200
Open Cases
29%
1000
800
600
Completed LMR
47%
Open Cases with Plan End
Dates Before January 2004
9%
Drop Out Rate
12%
400
200
0
W o rke r Unwilling
M e d ic a l
Imp a irme nt C o mp e ns a b le
M e d ic a l
Imp a irme nt - No n
C o mp e ns a b le
W it hd ra w n
De c e a s e d
Source :PBAS
• Since the inception of the LMR program, 47% of all cases referred have been completed (employable or
employed) while the drop out rate was 12% of cases.
• Drilling down on the drop out rate, the large number in the “Worker Unwilling” category may indicate a
need for improvement around worker involvement in plan preparation.
• At the time of analysis, 29% of cases were still open.
• Open cases with plan end dates before January 2004 represented 9%. These plans have likely been
completed but have not yet been assigned a closure code. Ensuring that closure codes are accurate / up
to date and consistently input by Adjudicators is important to enable accurate analysis and interpretation
of PBAS data moving forward.
20
LMR Program Performance Assessment – Cost-Benefit Analysis
In order to assist with the formation of the Value for Money Audit opinion we developed a high-level cost-benefit
model, which should serve as a baseline from which WSIB should develop more sophisticated and accurate
analyses.
$
Cost - Benefit Theory
Years of Benefit
Time
Difference in LOE benefit
pre and post LMR
• For each LMR case, WSIB incurs program costs
• The financial benefit accruing to WSIB is the difference
in cost between LOE benefit paid pre and post LMR – it
is assumed that LOE pre LMR would otherwise remain
constant
• The sum of the net present value of the cash flow
streams for each case represents the cost-benefit to
WSIB
LMR Program Cost
$
Cost - Benefit Calculation Assumptions
21 years(1) (3)
Time
Annual Difference in LOE benefits $117m
($6,583(1) for 13,283(3) completed cases)
Cumulative LMR Program Cost Sources: (1) WSIB legacy system
(2)
LMR Cost Benefit: Net Present
Value $363m
$409m(2)
Deloitte Analysis
• Cumulative program costs (1998-2003) have been used
in totality in the absence of accurate annual case level
costs
• Timeframe is based on average age of worker for all
LMR closed cases between 1998-2003
• A discount rate of 10% has been applied, which is not
material to the analysis.
(3)
PBAS
21
LMR Program Performance Assessment – Cost-Benefit Analysis
Interpretation
• This high-level analysis shows that from purely a financial perspective, the WSIB obtains
value from the funds invested in the LMR program.
• It is a matter of interpretation as to whether this equates to value for money from an
outcomes perspective.
• WSIB needs to develop and standardize a more robust cost-benefit calculation. Potential
enhancements would include:
– Segregation of program costs, at a minimum on an annual basis (requires correct accrual
accounting) or ideally on a case level.
– Completion of full LMR cycle (72 months post injury date) so that LOE is finalized
– Individual case level calculation of avoided LOE benefits
– Inflation adjustment for historic costs
22
LMR Program Performance Assessment – Resources and Outcomes
Findings
Phase 4
Manage Stakeholders
As part of the VFMA, we conducted a focus group with worker representatives in order to solicit worker
input on the LMR Program. Key findings are summarized below:
Worker
• Skeptical of quality of LMR assessment and expectation of a more holistic assessment of worker
impairment.
• Desire for WSIB to undertake a more thorough assessment in determining whether a worker is capable
of any type of work, in order to make the decision on appropriateness of LMR (vs. benefit payout).
• Skeptical of quality of secondary service providers.
• Skeptical on profit maximization motives of primary service providers and potential conflict of interest
between Primary and Secondary Service Providers.
• Expectation that WSIB better manage service provider quality and outcomes.
• Desire to shift LMR mandate from “employability” to an assessment of “available and sustainable
employment”.
• Desire for greater worker involvement in the LMR decision and Program.
• Desire for a more direct link between worker and WSIB (versus through Service Provider).
• Expectation that where there are worker representatives, there will be an opportunity for their
involvement in the LMR Program and plan.
• Expectation of Physician involvement in the LMR Plan decision.
23
LMR Program Performance Assessment – Resources and Outcomes
Findings
Phase 4
Manage Stakeholders
As part of the VFMA, we conducted a focus group with employer representatives in order to solicit
worker input on the LMR Program. Key findings are summarized below:
Employer
• Skeptical on level of scrutiny applied to determining appropriateness of LMR decision for worker.
• Expectation that LMR should be focused only on those workers that require it.
• Desire for improved case and medical management of the worker throughout the LMR Program.
• Skeptical of quality/capability and accountability of service providers (Primary and Secondary) to WSIB.
• Expectation that WSIB demand greater accountability from service providers including the linking of
referrals to service provider outcomes.
• Expectation of greater accountability of WSIB to employers including more detailed reports of costs and
outcomes of LMR Program to employers.
• Desire for information as to how the LMR program fits within the context of other WSIB programs.
• Desire for greater involvement of employer in LMR decision and Program.
• Expectation of maximum return on investment for employers, based on a thorough cost/benefit analysis.
There appear to be some areas of commonality in expectations between workers and employers.
24
LMR Program Performance Assessment: Conclusions
•
Systems:
–
Systems
–
–
–
–
Having assessed the performance management and accountability systems in
place, we find that significant improvements to existing systems are required.
It is noted however that WSIB has initiated some work in this direction and is
clearly committed to implementing further improvements, as demonstrated by
the recent development of a new RTW strategy that has started to integrate
LMR, and the planned proposal to develop a corporate level accountability
model for LMR.
In addition, the on-going development and roll-out of the PBAS system should
significantly enhance performance management and accountability. However,
it will be critical for PBAS to deliver reliable reporting information and for WSIB
to share that information throughout all levels of the organization.
Another key area requiring significant improvement is the need to develop
performance measures for the LMR Program. This needs to start at the
strategic level after the development of clearly defined strategic objectives for
LMR, and cascade down through the organization and across service
providers at the activity level.
While the current service provider audit process is a positive step, WSIB does
need to assess the effectiveness of this process and other controls with a
view to addressing current limitations of the primary service provider process
(e.g. audit sample selection, relationship management versus audit
independence), as well as considering options to establish controls over
secondary service providers.
Given all of the above factors and the importance of this area to the success
of the LMR program, we assess this area as requiring significant improvement.
25
LMR Program Performance Assessment: Conclusions
•
Resources:
–
Resources
–
–
–
–
–
Given the ‘proxy’ nature of existing performance measures and the limited
conclusive data available, we can at this time only provide a limited
assessment on the efficiency of the LMR Program.
While PBAS does contain information on provider costs, there is currently no
capability to track overall program costs (internal and external) and match
them to outcomes. As more data becomes available from PBAS and users
become more comfortable, it will be important to develop the level of
sophistication of cost efficiency analysis, including an assessment of case
management cost drivers.
Our preliminary review indicates that PBAS also contains the data to monitor
other efficiency measures such as the elapsed time between key events in a
case. Currently, such measures have not been defined (refer to Systems
conclusion) so are not being reported from PBAS.
While our cost analysis shows an increase in annual costs, it is difficult to
draw firm conclusions since the program has yet to complete a full cycle,
resulting in the only meaningful cost analysis being performed at an
aggregate level.
The preliminary cost-benefit analysis demonstrated that WSIB is obtaining
financial value from the LMR program.
It will be necessary to reassess ‘efficiency’ (including economy or cost) once
appropriate measures are in place and more data available. This also
includes the need to review all costs and efficiency measures once WSIB has
experienced a complete LMR cycle (referral to 72 months post accident) and
has the capability to track this through PBAS.
26
LMR Program Performance Assessment: Conclusions
• Outcomes:
Outcomes
– Given data limitations and the immaturity of the program there is currently limited
conclusive data available. We can at this time therefore only provide a limited
assessment on the effectiveness of the LMR Program.
– Having assessed the current data related to outcomes from the LMR program,
we find that performance is satisfactory with some improvements required. The
PBAS system has significantly enhanced the ability to track outcome data by
case for those workers accepted into LMR.
– Although the goal of the program is employability, approximately 44% of completed
programs result in actual employment, either with zero or partial wage loss
– 47% of workers with no Loss of Earnings Benefits either find employment or become
employable.
– Current data indicates that, since inception of the LMR Program, 47% of cases
have been completed with a 12% drop out rate. Although the balance of cases
are still “open”, the drop out rate appears reasonable given that, in some
instances, there are legitimate reasons. WSIB must closely monitor the drop out
rate and its contributing factors.
– The challenge facing WSIB is to establish targets for improving these outcomes
over time and link them to service provider performance, both primary and
secondary. In addition, WSIB needs to pay close attention to the completion rate
for LMR programs and begin to match outcomes to avoided LOE benefits
resulting from successful LMR. This would allow comparison to the total cost of
providing the LMR program and provide WSIB with a quantified value of the
program.
– It will be necessary to reassess ‘effectiveness’ once appropriate data is available.
This includes the need to review outcomes once WSIB has experienced a
complete LMR cycle (referral to 72 months post accident) and has the capability
to track this through PBAS.
27
Overall Prioritized Recommendations
High Priority (refer to appendix for assessment criteria):
• Single Consolidated LMR Strategy
• Multi-functional integrated LMR Program performance measurement and management
system and processes
• Defined LMR accountability mechanism
• LMR operating principles and procedures for Service Delivery Teams & other process
participants
• LMR Service Provider performance and accountability
- Development of service provider performance management model
- Improvement of PSP Audit Process
Medium Priority:
• Role clarification
• Periodic reassessment of LMR risk management activities
• Action plan to introduce controls for risks rated as requiring significant improvement
• Internal and external communication strategy on mandate/goals and outcomes of LMR
Program
• Monitoring and containment of Service Provider costs by WSIB
28
High Priority Detailed Recommendations & Management Responses
Single Consolidated LMR Strategy:
§
The WSIB has already commenced the process of developing a strategy for RTW that
incorporates the LMR Program. While we completely support the recognition of the need
to view RTW and LMR as integrated processes, we recommend that the current version of
the strategy be updated to reflect some of the findings from this audit, including the need
for clearly defined LMR strategic objectives, in the context of an integrated RTW and LMR
process.
Management Response:
Management agrees that it will be beneficial to provide more comprehensive strategic
direction on the LMR program. This will be developed by end of Q1 2005, and integrated
with the RTW Strategy.
29
High Priority Detailed Recommendations & Management Responses
Multi-Functional Integrated LMR Program Performance Measurement and Management System &
Process
Performance Management System
§ Develop an integrated multi-functional performance management framework that aligns with LMR
strategic principles (refer to Alignment conclusions for the need to develop clearly defined LMR
objectives) and facilitates effective monitoring of performance towards defined LMR goals and
outcomes. This will require significant effort on the part of WSIB.
– Define LMR outcome measures at a corporate, functional and provider level. This is an iterative process that should
be revisited with business changes.
– Critically select/develop those process measures that are essential to achieving LMR program objectives.
– Ensure measures are defined and tracked in a consistent manner throughout WSIB.
– Link outcomes to service provider performance.
– Monitor key measures on an ongoing basis (may be more frequent than annually).
– Develop formalized cost/ benefit calculation for LMR program, ensuring that all costs can be tracked on an annual
basis.
Performance Management Process
§ Develop appropriate performance management mechanisms/processes/systems to ensure
performance results are used to alter practices to facilitate continuous improvement.
– Build upon, integrate and ensure coordination among existing functional and other performance management
initiatives (e.g. QI, Global and External Audits, functional area specific projects, corporate outcomes initiative).
– Leverage PBAS for reporting and maximize availability of information as required throughout the organization.
– Provide users with sufficient and appropriate training of PBAS. Establish a database of identified user problems and
provide training to address these gaps, whenever necessary.
– Provide PBAS user one-stop user technical support.
30
High Priority Detailed Recommendations & Management Responses
Performance Management and Monitoring of LMR (continued):
Management Response:
§
While performance measures for LMR do exist, management agrees that a systematic review will be
appropriate in conjunction with setting strategic direction for LMR.
§
The WSIB will develop an integrated, multi-functional performance management framework and process
for LMR, including both process and outcome measures to supplement those we currently have. This
will be done by the end of Q2, 2005. The new RTW/LMR unit will be responsible for tracking and
monitoring these measures, and for using the results to continuously improve the LMR programs.
Leveraging PBAS will be necessary to this endeavor.
§
The WSIB now has a significant amount of data on LMR in the new PBAS system. A data "cube" is
currently being created in the Enterprise Information Warehouse (EIW) to store the data. A data
extraction tool will be in place by the second quarter of 2004 and will help us translate that data into
information. Management will also identify methods to make information on LMR program performance
readily available throughout the organization and to stakeholders.
§
Several activities are already underway to improve user knowledge of PBAS. Also, significant strides
have been made to improve the functioning of PBAS, and these efforts will continue throughout 2004.
Once these are complete, in 2004, it is anticipated that users will experience improvements as the
system continues to be enhanced. User problems are and will continue to be logged.
31
High Priority Detailed Recommendations & Management Responses
Defined LMR Accountability Mechanism:
•
Develop an accountability mechanism/structure with more defined accountability at the strategic level
for the LMR program. Integration and coordination of all activities and functional and service provider
areas related to LMR is critical.
- Building upon the on-going work in this area, ensure integration with RTW is achieved to
recognize the continuum from RTW to LMR.
- Ensure definition of overall program strategic ownership to include responsibility for all phases in
the program.
- Clarify and potentially redesign and communicate the accountability of the LMR Evaluation unit to
ensure fair, appropriate and consistent organizational expectations of this group.
•
Defined ownership at the strategic level for the LMR program level will also facilitate a focus on
initiatives such as investigating the impact of various variables on LMR outcomes and developing best
practice protocols to impact these outcomes.
Management Response:
In 2003, an accountability mechanism/structure for both RTW and LMR was developed and approved
for implementation in 2004. Program level ownership of LMR has been identified and approved in the
proposal to create a RTW/LMR unit to provide integrated program leadership, development and
oversight to these two key business activities. It will be responsible for researching factors and
practices contributing to better outcomes, and building these findings into program delivery. During
the implementation of the new unit in 2004, the accountability of the LMR evaluation unit will be
articulated and communicated within the context of the mandate for the new RTW/LMR unit.
Implementation of the unit began in Q1, 2004, and will be substantially completed by year-end.
32
High Priority Detailed Recommendations & Management Responses
LMR Operating Principles and Procedures for Service Delivery Teams & Other
Program Participants:
§
§
§
The LMR legislation is high level and as such does not define a specific operating
model, and detail on how to operationalize its intent. Improved clarity would be
achieved by the development of operating principles and procedures around:
the desired balance between WSIB’s social / public policy mandate and cost
efficiency, recognizing that these are not necessarily mutually exclusive. For
example, when reviewing a case, is the first assessment filter applied by an
adjudicator to consider how to maximize the outcome for the worker (employability
at the highest possible wage), or is it to successfully complete the LMR program at
the lowest cost to WSIB?
the manner in which it wishes to operationalize this mandate such that staff can link
individual actions and decisions to LMR objectives
the roles, responsibilities and communications with service providers
Operational principles and procedures should also define guidelines related to regular
and timely worker involvement.
In addition, it will be beneficial to consider developing case management guidelines
along the most significant complexity factors (e.g. injury type/severity, worker age .
Education etc.) which would promote internal consistency and assist with service
provider communications and monitoring. This leverages the concept of performing
research analytics to help develop case management standards.
33
High Priority Detailed Recommendations & Management Responses
LMR Operating Principles and Procedures for Service Delivery Teams & Other
Program Participants (continued):
Management Response:
In 2003, work was conducted through the LMR Roles Protocols Committee to improve
role clarity for those involved in administering the LMR process, and to more clearly
explain to staff how LMR decisions are to be case managed. In 2003 an LMR policy and
process refresher-training program was given to staff who required it. We do agree,
though, that further work needs to be done on an operating model. The model will be
developed by the end of Q2, 2005, and will contain all of the components identified by
the auditors.
In addition, Operational principles and key concepts/definitions will be included in the
strategic direction document for LMR. More detailed procedures and case management
guidelines will be developed by the RTW/LMR unit during the course of their normal work.
34
High Priority Detailed Recommendations & Management Responses
Development of a Service Provider Performance Management Model
•
Develop and enforce a formal performance management model to link accountability to
provider outcomes and assist with the removal of poorly performing service providers
(Primary and Secondary). This requires a shift from WSIB’s current “behavioural
modification” model to a more compliance based “business outcomes” model.
•
Formally monitor Primary Service Provider (PSP) outcomes to drive PSP performance
and accountability (of PSP directly and Secondary Service Provider (SSP) indirectly).
•
Establish a formal schedule to reissue RFP for service providers.
•
Continually review existing accountability contracts with Primary Service Providers to
ensure accurate reflection of changing business needs. Link accountability to provider
outcomes.
Management Response:
We now have six years experience managing the primary service providers, and more
data with which to assess their performance. We agree that the process needs to be
better defined to tie evaluation to outcomes and to ensure that standards are met. This
will be included in a revised provider management model. This model will be developed
and implemented in 2005.
35
High Priority Detailed Recommendations & Management Responses
Improvement of LMR Primary Service Provider Audit Process:
•
There is a need to improve the LMR Primary Service Provider Audit process:
– The current system involves annual audits for each provider. The annual timing of
these audits may enable the primary providers to anticipate these. It would therefore
be useful to supplement these audits with other control mechanisms.
– There is an element of self reporting at the moment in that primary providers are
responsible for entering case information into PBAS. Since the audit sample is
selected from PBAS, there is no control over the completeness of cases within PBAS,
from which to select the sample.
– At the current time, limited financial analysis is performed to identify high risk audit
targets. Such analysis would lead to the development of a more risk-based audit
approach which would allow for a more efficient deployment of audit resources and
potentially a higher return.
– Examine other audit opportunities for the Unit, similar to some of those undertaken in
2003, e.g. look at specific audit subjects such as claims receiving a specific type of
service vs. the more global audit generally performed.
– PBAS has great potential for data mining to tie service providers, both primary and
secondary, to specific outcomes and cost.
•
Since secondary service provider costs represent the largest component of overall LMR
program costs, it is imperative that WSIB examine various audit and control possibilities
for these providers.
36
High Priority Detailed Recommendations & Management Responses
Improvement of LMR Primary Service Provider Audit Process (continued):
Management Response:
Management agrees with this recommendation. The audit process will be revised to
reflect a revised provider management model. The LMR evaluation unit will strengthen
the audit program and will continue to develop a range of audit opportunities, such as
random audits, and subjects, such as audits on specific types of program or issue. WSIB
does currently independently select the audit sample.
In addition, the development of the proposed service provider performance management
model will address audit sample selection, methods, and targeting. It will also identify
other mechanisms to be used in managing provider performance. The model will also
describe the method by which the WSIB will exert greater control over secondary service
providers. PBAS data will be valuable in managing providers once it is more available
and accessible.
37
Medium Priority Detailed Recommendations & Management Responses
Role Clarification:
• Closely linked to the definition of accountabilities, define and clarify roles and case
management responsibilities in the LMR Program.
– While the broad role definition related to the case management component of the LMR
Program is relatively clear, particular focus is required for Phase 1 (Selecting &
Managing LMR Service Providers) and Phase 4 (LMR Performance Management).
– Define all required activities for all phases.
– In defining the role of the adjudicator in LMR, it will be important for the WSIB to take into
account workload and job complexity of current duties.
Management Response:
In 2003, work was conducted through the LMR Roles Protocols Committee to improve role
clarity for those involved in administering the LMR process, and to more clearly explain to
staff how LMR decisions are to be case managed. We agree that further work is required
to clarify roles and accountabilities, particularly in Phases 1 and 4 of the program. This will
be included in the accountability and operating models and the Performance Measurement
framework recommended earlier.
38
Medium Priority Detailed Recommendations & Management Responses
Periodic Reassessment of LMR Risk Management Activities:
• Develop / put in place a mechanism to proactively, systematically and periodically assess
risks and risk management activities of LMR to be consistent with overall WSIB Risk
Management practices:
• Continue to monitor the effectiveness of current controls so that the degree of exposure is
not increased.
• Continue to refine and evolve risk management processes, always aligning such efforts
with Corporate and Divisional LMR outcomes in the context of efficiency, effectiveness
and cost benefit (economy).
• Incorporate gaps / exposure areas and corresponding risk management efforts into the
annual business planning process.
Management Response:
The WSIB has developed and implemented a corporate risk management program. In
addition, risk assessment and management are built into the WSIB project management
methodology. The new RTW/LMR unit will incorporate these risk management methods
into the annual business planning process.
39
.
Medium Priority
Detailed Recommendations & Management Responses
Action Plan to Introduce Controls for Risks rated as requiring Significant Improvement :
• Identify and implement action plans to introduce controls to effectively manage those risks
assessed as requiring significant improvements and critical during the risk selfassessment. These action plans then need to be monitored to ensure the effectiveness of
these controls on a go-forward basis.
Management Response:
An action plan to mitigate the self-assessed risks will be developed in 2004. The
RTW/LMR unit will then be responsible for implementing them and monitoring their
effectiveness. The development and tracking of performance measures and clarifying
LMR ownership and accountability will substantially address all of the self-assessed risks.
40
Medium Priority Detailed Recommendations & Management Responses
Internal & External Communication Strategy:
§ Develop a mechanism to ensure the organization-wide communication of the
integrated RTW and LMR Strategy and its links to policies & performance outcomes
to facilitate “buy-in”.
§ Develop a mechanism to enable better alignment between WPP expectations and
WSIB’s legislated mandate and the operationalization of this mandate, and to
communicate performance outcomes and cost related information.
Management Response:
Management agrees that there is a need to develop communication plans to inform
staff and stakeholders of program performance and new developments as they
occur.
The Communications Division is currently developing a plan to communicate LMR
outcomes and associated costs to stakeholders. Internal and external
communication plans will be developed as other major milestones are met. The
internal and external communication of an integrated LMR strategy will be a
mechanism to better align WPP expectations and the WSIB’s operationalization of its
mandate. In 2004, the WSIB will be increasing communications about return to work,
providing an additional opportunity to communicate LMR information.
41
Medium Priority Detailed Recommendations & Management Responses
Monitoring and Containment of Service Provider Costs by WSIB:
•
Formally compare Primary Service Provider (PSP) fees to market, determining
whether the fees are appropriate and in line with current market conditions.
•
Identify whether there are any programs for which a fixed pricing structure
could be implemented e.g. ESL, Creative Job Search Training.
•
Ensure worker involvement and agreement with plan prior to implementation.
This may or may not include the requirement to sign their LMR plan.
Management Response:
Management is currently conducting a market value assessment to determine
the appropriateness of PSP and Secondary Service Provider (SSP) fees. The
results of this survey will be reflected in the next RFP for LMR services.
The RTW curriculum, LMR refresher training and LMR protocols all speak to
the importance of worker involvement and agreement with the LMR plan. The
RTW training is also providing staff with improved techniques for managing the
transition between the esRTW and LMR phases of return to work.
The issue will continue to be monitored, and additional steps taken to ensure
worker involvement and agreement if required.
42
Summary Opinion
• A key audit objective was to provide an opinion as to whether there is value for money in
the outcomes generated by the LMR program. In this regard, we present the following
summary conclusions:
– WSIB is already achieving financial value from the program because the benefits of
avoided future loss of earnings benefits exceed the costs of the program.
– Notwithstanding the fact that the program has yet to reach maturity, the current data
reveals outcomes that are consistent with WSIB’s legislated mandate for LMR:
– Recognizing that the legislated mandate of the program promotes “employability” for
workers in contrast to “employment”, 44% of completed plans actually achieved
employment, either with zero or partial loss of earnings.
– The percentage of workers that are considered able to achieve zero loss of earnings,
both employed and considered employable, and have completed their plans is 47%.
– The percentage of workers failing to complete their plans is 12%.
– However, given that the program is still relatively new, at this time there is not enough
consistent data available for us to conclude definitively whether the outcomes are
sufficient.
• In our view, the implementation of the recommendations contained throughout this report
will assist WSIB to expedite the improvement of its LMR outcomes.
43
Auditors’ Report
To the Directors, Workplace Safety & Insurance Board:
•
We have conducted a value for money audit of the LMR program at the Workplace
Safety & Insurance Board (WSIB) of Ontario in accordance with WSIB’s request for
proposals No. 2003-016. The objective of the audit, as stated in the request
document, was “to perform a review on the cost, efficiency and effectiveness of the
LMR program” (Please see page 4 of this report for a full description of the project’s
scope and objectives).
•
We conducted our audit in accordance with the value for money audit standards
recommended by the Canadian Comprehensive Auditing Foundation and accordingly
included such tests and other procedures as we considered necessary in the
circumstances.
•
Specific criteria encompassed by this audit are identified on page 5 of this report.
Our findings, conclusions & recommendations are detailed throughout the report,
starting on page 7.
44
Appendix - Prioritizing Recommendations
Recommendations were prioritized on the following criteria:
• Critical success factors in meeting organization goals (greatest impact on outcomes)
• Degree of deficiency/need for improvement in current Program
• Consistency of recommendations across Alignment, Risk Management & Control and
LMR Program Performance Measurement & Management assessments
• Ability to leverage appropriate on-going work
• Time lag for realization of impact
• Time, cost, difficulty to implement
45