Validating the System
Transcription
Validating the System
Validating the System • What is the academic basis to the Insights Discovery system? • Has the Insights Discovery system been validated by official bodies of the psychology profession? • Is the Insights Discovery preference evaluator continually tested and updated? Insights and Jung External Assessment The Insights Discovery System is solidly based on the psychological types theory of Dr Carl G Jung and one of his leading students, Dr Jolande Jacobi. Jung proposed that our personalities are created in the interaction of four functions and two attitudes. When combined, these elements generate eight common personality types. Jung felt that what makes an individual unique are the different balances of these functions and attitudes interacting within them. It is these balances that make up the Insights 4 Colour System. The four colours represent observable behavioural patterns: In 2005 the University of Westminster’s Business Psychology Centre performed an extensive study of the Insights Discovery Evaluator English Version 3.0. Key statistical analysis was used to study the IDE’s “item analysis”, “norm data”, “reliability” and “validity”. Figure 2: Pyramid of key psychometric statistics Validity Construct Validity Factor Analysis Criterion Validity - Concurrent Criterion Validity - Predictive Reliability Temporal Stability For a more comprehensive review of the IDE’s properties, please read “An Overview of the Development, Validity and Reliability of the English Version 3.0 of the Insights Discovery Evaluator” produced at the University of Westminster’s Business Psychology Centre (bpc). Test – re–test correlations Internal Consistency Split-half coefficients Item-total correlations Inter-item correlations Cronbach Alpha coefficients Data on Norms Historic Development - Item Analysis FIE RY H RT Showing no bias Objective Detached Positive Affirmative Bold Assertive Still Tranquil Calming Soothing Cheerful Uplifting Spirited Buoyant RE EN An Overview of the Development, Validity and Reliability of the English Version 3.0 of the Insights Discovery Evaluator Develop ment of the Figures 5 through evaluator usin g item ana Althoug 8 h the grap graphically sho lysis has w there is signific hs antly imp a need for presented here the benefit of evolvin analysis. are both a more g the item roved its quality statistic . al approac visually impactf pairs ove ul r time. h to qua Conseq ntify the and intuitively uen appeali discrim The University of Westminster, 309 Regent Street, London, W1B 2UW, UK ng, inative strength tly, t-tests wer quality e conduc of of the item ted distance the relationship Dr. Corine van Erkom Schurink between s plotted on the item ana lysi the in Below is an exampl two colour sco the graph. A t-te s data in order to con res st e for the gauge the tinuous Director of The Analytical Research Bureau (Pty) Ltd, ‘cool blue within one fram allows you to statistic dete taken betw ly improving vers e, are stat al ’ items from 4 ions of een 31/1 istically rmine if the samples average the evaluat signific 16 Central Avenue, Pinelands 7405, South Africa com across all 1/2003 and 31/7 ant ors. Thi /2004. 25 fram s English paring data acro or not. This exa es, have ss 4 Version For exa mple is scored 4 3.0 sam Stewart Desson mple, the based on or more ple has sha resp evaluat (out of been or, the ‘coo ding of the firs 6) for ‘coo ondents who , t row of on than the l l blue’. Head of Learning, Insights Learning & Development Ltd, the ‘earth gree blue’ statement (exact and table below sho highligh ts that alth n’ statement (cal ws good), they m and eve precise) is success that in the 3 rd Jack Martin Way, Dundee, DD4 9FF, Scotland, UK ough the fram differen n-tempe are nev ce betw highligh red). How fully selected mor e of the ertheles een them ted as a s close e often to problem is statistic ever, the t-test empiric or potentia each other (wh ally driv also This paper presents comprehensive data on the psychometric properties of the Insights Discovery Evaluator (IDE). ich is not ally significant en item l problem analysis. (which ideal). is a can The IDE’s capacity to affect both reliable and valid measures of the primary personality preferences coded by the is Any stat didate for Table 1 ement - T-tes improv model as four colours is assessed. It draws upon an extensive research and development programme undertaken t on item ement thro analysis ugh furt of resp between Insights Learning & Development Ltd. and the University of Westminster, aimed at the development of a ondents UK - Eng her who lish Dr. Stephen Benton It is an or (IDE). ch Evaluat es in whi covery 25 fram ghts Dis sisting of lish Insi scores the s in a igh of the Eng e) evaluator con st’ and then 4 item The Ins ion 3.0 rang st’, a ‘lea 5. Each of the tains vers normative (a l blue’. s a ‘mo ix A con e of 1 to n’ and ‘coo word pair ice) and r scal Append gree a cho fou th of t on ‘ear (forced a choice 6 t and mos shine yellow’, ipsative between oses from in between leas re ‘sun cho , r sco a all ry red’ the use options h giving n across called ‘fie ng two res, eac tic mea t5 erences remaini erence sco ple arithme mple of the firs sure pref e colour pref colours. A sim frame mea exampl ws an exa have 25 the four or will ure 2 sho frames. In this each of urs. Fig ed evaluat all 25 least), for of the four colo from ted. A complet ed (for 0 h t) and rt produc n highligh ed for eac bar cha (for mos ur has bee colo is calculat e of the colour ’s es the item 25 fram exampl or works, and an luat eva frames the ing how explain very ts Disco YE LL OW CO EA G UE BL D RE OL Figure 1: The Insights Discovery 4-Type Wheel r (IDE) Evaluato Director of The Business Psychology Centre (bpc), E IN SH N SU R25 psychometrically robust evaluator. The full methodological and statistical account of this programme may be found in technical papers produced at the University of Westminster’s Business Psychology Centre (bpc). Pair Psychometric science endeavours to impose objectivity upon the measurement of both human performance andtested Pair 7 aspects of personality. In order to do so, any questionnaire based on measurement of human behaviour, especially that based on self-report, must be able to meet certain demonstrable criteria in order to be considered as an Pair 19 Pair 22 PairIt25 objective measure. This paper sets out to explain these psychometric criteria in easily understandable terms. is Pair 26 the authors’ intention to make the statistics and arguments presented understandable to two different professional B9 & Y9 B10 & G10 B13 & G13 B17 & professional assessment of the evidence presented. Secondly, the wider community of Human ResourcePair 51 B17 G17 & R17 Pair 52 professionals require simple non-technical explanations of these same key statistics, in order for them to assess the B18 & G18 Pair 70 B24 & appropriateness of the IDE’s to their business. Four categories of information are presented covering ‘item G24 These colours are measured by the Discovery evaluator; a 25-frame questionnaire of 100 word pairs, which when completed produces the Insights Discovery Personal Profile. Designed by Andi Lothian in the early 1990’s, Andi, and son Andy, founded Insights Learning & Development Ltd in 1993. output es and exa colour luator fram minant’ al’s ‘do ple of eva 2 – Sam an individu Figure cribed as rt is des bar cha ur in this est colo d colour. The high erre t pref their mos analysis’, ‘norms data’, ‘reliability’ and ‘validity’ as these form the bases for demonstrable objectivity.POT Key ENTIAL 6.29 10.10 11.85 -8.71 13.20 10.27 -1.09 8.84 0.000 0.000 0.000 0.000 0.000 0.000 0.276 0.000 0.000 3.59 0.000 0.000 scored lish S2 0.000 7.12 0.000 9.33 22.98 0.000 0.000 12.60 15.67 -1.20 0.000 0.000 4 or more UK - Eng t value on Sig. of t (2paired ttailed) test CI=95% 2.73 0.006 6.00 0.000 10.53 0.002 (out of 6) for cool lish S2.2 32.73 26.45 33.10 38.39 t value on paire d ttest 9.53 40.73 26.51 11.56 0.000 11.78 0.000 36.82 0.000 27.99 0.000 28.01 0.000 0.000 0.000 39.75 10.47 28.90 n a high quality uen item ana quoting lysis doe furt s not nec n selected her from Paul essarily Kline by ry, as has been argu item analysis and in his Psychomet ensure an evaluat ed, to sho or rics Prim the resu lts replicat w the test er (1997) is valid. , ed with is valid a new sam “After the and reli able”. ple, it is ersity Conseq tly, Version 4.0 1 of 70 itemPage s have bee Issued by the University of Westminster’s Business Psychology Centre 22nd December 2005 www.wmin.ac.uk/sshl/page-339 necessa of West 4.0 Version the University Issued by minster’s Business Psycholog y Centre 22nd Dece mber 2005 of 70 Page 6 9 hl/page-33 n.ac.uk/ss www.wmi Version 4.0 Issued by the Univ Insights Learning & Development Ltd JackMartin MartinWay, Way, Claverhouse Business Park, Dundee DD4 9FF, Scotland, UK Jack Claverhouse Business Park, Dundee DD4 9FF, Scotland, UK T: (0)1382 908050 (0)1382 908051 E:Insights@insights.com www.insights.com T:+44 +44(0)1382 908050F: +44 F: +44(0)1382 908051 E:insights@insights.com www.insights.com of West minster’s Business Psycholog y Centre 22nd Dece mber 2005 www.wmi blue UK - Eng t value on Sig. of t (2paired ttailed) test CI=95% 11.85 0.000 23.99 0.000 25.80 0.000 9.23 0.000 18.40 0.232 PROBLE 13.91 M: The -5.89 statistics have been computed for each of these four areas and they have been benchmarked against international 0.000 two item i.e. it is s (t-tes 33.26 but the standards. The paper concludes that the measurement of the four colours is both valid and reliable. PRO distance t) are very clos BLEM: e (low between The two positive the two items (t-tes t-value), items is t-value), t) are very still sign and the ificant close (low distance PROBLE positive/ between M: The nega the two tive two item items is s (t-tes NOT sign and the t) are in distance ificant inverted between order (neg the two items is/is ative t-value), not sign Howeve ificant r, eve GLOBAL HEADQUARTERS Insights Learning & Development Ltd 7.44 14.40 B8 & G8 B9 & G9 psychometricians require key statistics and supporting technical information, in order for them to make a Pair 49 profile mple of 3.11 Pair 28 Pair 37 groups, both of which may have a need to work with the Insights Discovery Evaluator (IDE). Firstly, UK - Eng Blue score>4 VS t value other on Sig. of t (2paired colors ttailed) test CI=95% B3 & G3 B7 & G7 Page 13 n.ac.uk/ss of hl/page-3370 9 lish S3.0 Sig. of t (2tailed) CI=95% 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 Insights Discovery©: Validating the System Item Analysis There are 100 colour items spread over the 25 frames in the IDE. Item analysis involves empirically testing the quality of these 100 items and replacing weaker items with better ones. This is measured by assessing the responses of participants with clear colour preferences, i.e. those whose average across all 25 frames is greater than 5 in one out of the four colours (“Sunshine Yellow, “Fiery Red”, “Earth Green” or “Cool Blue”). Figure 3 shows one of the improvements made in the items from IDE S1.0 (UK) to IDE S2.0 (UK). Figure 3: Example of results of item analysis - graphs show ‘before’ and ‘after’ item changes Note: Green scoring higher than yellow on a yellow item before the alteration. Insights Discovery Preference Evaluator Expressive and hopeful L 1 2 3 4 5 M Decisive and immediate L 1 2 3 4 5 M Powerful and assertive L 1 2 3 4 5 M Analytical and discreet L 1 2 3 4 5 M Thinking and self-contained L 1 2 3 4 5 M Tolerant and laid-back L 1 2 3 4 5 M 7. Demonstrative and persuasive L 1 2 3 4 5 M Questioning and reflective L 1 2 3 4 5 M Logical and controlled L 1 2 3 4 5 M Immediate and initiating L 1 2 3 4 5 M Task-focused and confronting L 1 2 3 4 5 M Stable and concerned L 1 2 3 4 5 M Discussing and spontaneous L 1 2 3 4 5 M L 1 2 3 4 5 M Discreet and valuing L 1 2 3 4 5 M Faithful and helping L 1 2 3 4 5 M Considered and L 1 2 3 4 5 M Consistent and correct L 1 2 3 4 5 M Challenging and determined L 1 2 3 4 5 M Figure 4: Dominant Colour Spread of 75 CFOs As expected, those in charge of financial data tend to have a preference for objective thinking, using facts and performing detailed analysis. It should be noted whilst this data indicates that people in certain roles tend to have a preference for certain colours, it does not correlate or necessarily relate to how well they are doing their job or how capable they are in fulfilling that role. Reliability Cronback-Alpha coefficients, α, measure the error variance on the average inter-item correlations. When the error variance is low, which is desirable, the alpha coefficient approaches 1.00. A value of 0.70 is the commonly accepted inferior limit. Analysing 24,224 completed evaluators shows the four colours to have very high CronbachAlpha coefficients, providing evidence of reliability as shown in Table 1. L 1 2 3 4 5 M L 1 2 3 4 5 M 19. Systematic and L 1 2 3 4 5 M L 1 2 3 4 5 M Encouraging and valuing L 1 2 3 4 5 M Question Responsive and extraverted L 1 Numbers 2 3 4 5 M 1-25 Results-oriented and fast L 1 2 3 4 5 M Counselling and caring Courageous and independent Count of ‘Most Lresponses for people with 1 2 3 4 5 M Total Sunshine Yellow score > 5 L 1 2 3 4 5 M 16. Animated and persuasive L 1 2 3 4 5 M Decisive and immediate Powerful and assertive L 1 2 3 4 5 M Analytical and discreet Consistent and correct 14. Strong-willed and purposeful 9. Reasoned Sensitive and anddiplomatic particular Precise and deliberate Eager and engaging Encouraging andsensitive valuing Concerned and Results-oriented and fast 15. Systematic and principled L L LL 1 1 1 1 2 2 2 2 3 3 3 3 LL 1 1 2 2 3 3 LL 1 1 2 2 3 3 L 1 2 3 L 1 2 3 LL 1 2 1 2 3 3 LL 1 1 2 2 3 3 LL 1 1 2 2 3 3 L 1 2 3 L 1 2 3 LL 1 2 3 1 2 3 LL 1 1 2 2 3 3 LL 1 1 2 2 3 3 L 1 2 3 L 1 2 3 4 4 4 4 5 5 5 5 M M M M 4 4 5 5 M M 4 4 5 5 M M 4 5 M 4 5 M 4 5 M 4 5 M 4 4 5 5 M M 4 4 5 5 M M 4 5 M 4 5 M 4 5 M 4 5 M 4 4 5 5 M M 4 4 5 5 M M 4 5 M 4 5 M Fun-loving and popular L 1 2 3 4 5 M Steadying and moderating L 1 2 3 4 5 M Table 1: Cronbach-Alpha Coefficients for S3.0 (UK) FastIDE and reinforcing L 1 2 3 4 5 M Before L 1 2Changes 3 4 5 M Name:__________________________________ L 1 2 3 4 5 M Upbeat and hopeful Social andand cheerful Analytical evaluating Faithful helping Bold andand objective 0.92 18. Influential and informal Precise and deliberate Immediate and initiating Unassuming and responsive Stable and concerned 13. Constant and attentive 8. Influencing Resolute and confident and expressive α L 1 2 3 4 5 M Social and cheerful Thinking and distant 12. Analysing and painstaking 7. Friendly Demonstrative and persuasive and entertaining Questioning and robust reflective Competitive and Cool Blue 17. Patient and empathetic L 1 2 3 4 5 M Insights Discovery Preference Evaluator N= 24,224 16. Animated and persuasive 8. Resolute and confident 9. Sensitive and diplomatic Data on Norms Count of ‘Most responses for people with L 1 2 3 4 5 M Total Sunshine Yellow score > 5 L 1 2 3 4 5 M 6. Accommodating and helping The norm data for the IDE is of good quality, being segmented by the language of the evaluator completed; the country a respondent is based in; age (in ten-year bands) and occupation. The occupational norm data provides predictive validity. The dominant colour spread of Chief Financial Officers is predominantly ‘Cool Blue’ and ‘Fiery Red’. Name:__________________________________ 6. Relating and amenable L 1 2 3 Table 4 5 M Tolerant and laid-back 22. Objective and daring 17. Relaxed Patient and andempathetic peaceful Logical and Factual and controlled conventional Task-focused and confronting Lively and congenial Discussing and spontaneous 23. Animated and enthusiastic 18. Driving Influential and informal and realistic L 1 2 3 4 5 M L L LL 1 1 1 1 2 2 2 2 3 3 3 3 LL 1 1 2 2 3 3 LL 1 1 2 2 3 3 L 1 2 3 L 1 2 3 LL 1 2 1 2 3 3 Discreet and valuing Compassionate and considerate LL 1 1 2 2 Considered LL 1 Detailed andand attentive 1 2 2 Challenging and determined L 1 2 24. Supporting and steady L 1 2 19. Independent Systematic and L 1 and bold L 1 2 2 4 4 4 4 5 5 5 5 M M M M 4 5 M 4 5 M 4 4 5 5 M M 4 5 M 4 5 M 4 5 4 5 M M 3 4 5 3 4 5 M M 3 3 4 4 5 5 M M 3 4 5 M 3 4 5 M 3 4 5 3 4 5 M M Courageous and independent LL 1 Reflective and thorough 1 2 2 3 3 4 4 5 5 M M Responsive LL 1 2 Good-mixer and and extraverted lively Question 1 Numbers 2 3 3 4 4 5 5 M M 1-25 Counselling and caring 25. Cautious and accurate L 1 2 3 4 5 M After Changes L 1 2 3 4 5 M Forthright and straightforward L 1 2 3 4 5 M Optimistic and upbeat L 1 2 3 4 5 M 2: Average AcceptingFactor and loyalLoadings. L Completed 1 2 3 4 5 M in UK 31/11/03 YOU HAVE COMPLETED THE EVALUATOR, PLEASE ENSURE EVERY BEEN ALLOCATED to 2/7/04. IDEFRAME S3.0HAS(UK) Earth WHENSunONE ‘M’, Fiery ONE ‘L’AND TWO DIFFERENT VALUES SELECTED FROM 1, 2, 3, 4 OR 5. Discovery Evaluator 2 Page (B&W) English UK V3.0 250603Factor 1 Green shine Red N=20,948 © Copyright 1992-2003 Andrew Lothian, Insights. All rights reserved. Insights, Jack Martin Way, Claverhouse Business Park, Dundee DD4 9FF. Tel: 01382-908050 Fax: 01382-908051 Yellow Earth Green 0.56 0.92 12. Analysing and painstaking 0.93 0.92 Competitive and robust Friendly and entertaining Unassuming and responsive L 1 2 3 4 5 M L 1 2 3 4 5 M 22. Objective and daring Relaxed and peaceful L 1 2 3 4 5 M L 1 2 3 4 5 M Sunshine Factual and conventional-0.05 L 1 2 3 4 5 M L 1 2 3 4 5 M Yellow Lively and congenial L 1 2 3 4 5 M Factor 2 0.06 0.32 L 1 2 3 4 5 M 13. Constant and attentive L 1 2 3 4 5 M Test-retest reliability is determined through theL 1 2 3 4 5 M 23. Animated and enthusiastic Influencing and expressive L 1 2 3 4 5 M Driving and realistic L 1 2 3 4 5 M Cool Blue -0.09 -0.57 Analytical and evaluating administration of the same evaluator across L 1 2 3 4 5 M Compassionate and considerate L 1 2 3 4 5 M Bold and objective L 1 2 3 4 5 M Detailed and attentive L 1 2 3 4 5 M Fiery Red -0.59 0.19 time. It helps gauge how robust the items are.L 1 2 3 4 5 M 24. Supporting and steady 14. Strong-willed and purposeful L 1 2 3 4 5 M Reasoned and particular L 1 2 3 4 5 M Independent and bold L 1 2 3 4 5 M Such tests are generally expected to yield Eager and engaging L 1 2 3 4 5 M Reflective and thorough L 1 2 3 4 5 M and sensitive Good-mixer and lively L 1 2 3 4 5 M reliability coefficients ranging Concerned between 0.70 L 1 2 3 4 5 M Assessments/Validations 15. Systematic and principled L 1 2 3 Ongoing 4 5 M 25. Cautious and accurate L 1 2 3 4 5 M and 0.90. Fun-loving and popular L 1 2 3 4 5 M Forthright and straightforward L 1 2 3 4 5 M Steadying and moderating L 1 2 3 4 5 M Optimistic and upbeat L 1 2 3 4 5 M Fast and reinforcing L 1 2 3 The 4 5 M constant Accepting evolution and loyal 1 2 3 4 5 M ofL language means The results of the test-retest analysis WHEN YOU HAVE COMPLETED THE EVALUATOR, PLEASE FRAME HAS BEEN ALLOCATED thatENSURE theEVERY Discovery Evaluator will continually performed on the four colour scores ONE show ‘M’, ONE ‘L’AND TWO DIFFERENT VALUES SELECTED FROM 1, 2, 3, 4 OR 5. evolve. Insights Discovery Evaluator 2 Page (B&W) The English UK V3.0 250603 Research Team monitors very high reliability, translating into coefficients © Copyright 1992-2003 Andrew Lothian, Insights. All rights reserved. Insights, Jack Martin Way, Claverhouse Business Park,Insights Dundee DD4 9FF. Tel: 01382-908050 Evaluator Fax: 01382-908051 in its 30+ the Discovery ranging from 0.81 to 0.86 for the Pearson languages on a daily basis. If you require correlation coefficients and 0.89 to 0.92 for the any information on this process, please Cronbach-Alpha reliability coefficients for the contact ibrady@insights.com. Examples of same data. the Statistical Reliability and Validity tests can be obtained from Insights Learning & Validity Development Ltd on request. Confirmatory Factor Analysis was used to test the hypothesised factor structure of the Insights Discovery model. Specifically it is hypothesised that the four sets of 25 colour based items in the IDE should load onto the factors such that the polar opposite nature of the ‘Fiery Red’ vs. ‘Earth Green’ items is apparent and the polar opposite nature of the ‘Sunshine Yellow’ vs. ‘Cool Blue’ items is apparent. The four colours should load onto their appropriate factor only. The results in Table 2 successfully confirm this hypothesis. Conclusion Over the past decade, the work of Carl Jung has attracted increasing interest as people seek to improve interpersonal dynamics on both personal and professional levels. The scientific research of the Discovery System demonstrates that both Jung’s original typology theory, and Insights’ ongoing research to enhance it, has both strong psychological foundations, and modern scientific application. Depth psychology and empirical science unites within the Insights Discovery System.