Skip to content

Measuring Competency

Measures tested against the outcomes they should predict

Training programs certify readiness with practicum hours, coursework, and local evaluation forms. Many of those forms have unknown psychometric properties and documented measurement and rater biases. The CARE Lab builds competency measures with item response theory and many-facet Rasch modeling, which converts ordinal ratings to interval-level measurement and estimates each rater’s severity explicitly. The lab then tests whether the scores track the training inputs and patient outcomes they should reflect.

The Structure of Competency

Competency is one construct with two parts. Foundational competency is the knowledge, attitudes, and skills a trainee brings. Functional competency is the application of those in professional activities such as assessment and intervention. Research groups in two countries, working with different measures and samples, found this same structure (Gonsalvez et al., 2020; Callahan et al., 2017).

General CompetencyOne dominant factor behind ratings of readiness89.4% of common varianceFoundationalThe knowledge, attitudes, and skillsa trainee bringsAssessed thoroughlyby exams and courseworkFunctionalApplying them in professionalactivities such as assessmentand interventionAssessed less oftenThe foundational and functional factors are reliable but together hold the remaining 10.4% of common variance.

The practical meaning is that a total score is the sound basis for decisions about readiness, because one general factor dominates the ratings. The foundational and functional scores are best kept for feedback that tells a trainee where to grow (Callahan, in press).

What Gets Measured Is Not What Predicts Outcomes

Medical education describes clinical competence as a ladder from knowing, to knowing how, to showing how, to doing (Miller, 1990). Doctoral programs assess the lower rungs thoroughly with examinations and course grades, and the upper rungs, which are performance with real patients, less often. The upper rungs are where measured competency predicts patient outcomes.

IsProfessional values and identity,enacted in everyday conductTrustedJudged ready to workwithout direct supervisionDoesDirect observation,supervisor ratings,patient outcomesShows howSimulated andstandardizedpatient encountersKnows howCase conceptualization,oral and written examsKnowsKnowledge examsand course gradesAdapted from Miller (1990). Is added by Cruess et al. (2016). Trusted added by ten Cate et al. (2021).

The ladder has since been extended upward. One addition is an Is level for the internalization of professional values and identity (Cruess et al., 2016). Another is a Trusted level for the judgment that a learner can work without supervision (ten Cate et al., 2021). The two additions come from separate proposals, so the order shown here is one reading of how they stack. Both name what this framework calls foundational competency, the professional stance a patient meets before any technique is applied.

Competency Grows Where Training Is Designed

In a prospective sample of 94 trainees and 270 supervisor evaluations from three accredited doctoral programs, competency rose at each stage of training, and training level explained 37.8% of the variance (Price et al., 2017).

0123d = 0.0Pre-practicumd = 1.91Internal practicumd = 2.89External practicumEffect size (d) compared with pre-practicumTraining level explains 37.8% of the variance in competency

Researchers express the size of a difference as an effect size, d, which counts how many standard deviations separate two groups. By convention 0.2 is small, 0.5 is medium, and 0.8 is large. A more intuitive reading asks how often a typical person in the later group outscores a typical person in the earlier group, assuming roughly bell-shaped scores.

d = 1.91The typical trainee in internal practicum scores higher than 97% of trainees at the start of training.
d = 2.89The typical trainee in external practicum scores higher than 99% of trainees at the start of training.

Competency Predicts What Happens to Patients

In a training clinic with 192 patients treated by 92 trainees, measured competency predicted two patient outcomes (Dimmick et al., 2023).

0%5%10%15%20%Foundational 7.4%Functional 4.5%11.9% combinedTrainee competencyabout 1%Therapy typePercentage of total variance in patient symptom change

Therapy type means which treatment approach a clinician uses, such as cognitive behavioral or psychodynamic therapy. Across well-conducted comparisons, the specific approach explains about 1% of the differences in how much patients improve (Wampold & Imel, 2015). Who delivers the treatment explains far more.

Who returns

Each one-point gain in the professional values, attitudes, and behaviors domain reduced the relative risk that a patient would not return after the first session by 42%.

Who improves

Foundational competency explained 7.4% and functional competency 4.5% of the variance in symptom change, together more than 10%. Differences between established treatment approaches explain about 1% (Wampold & Imel, 2015).

Who the trainee is matters as well. In a small training-clinic sample of 32 trainees and 133 patients, trainee emotional intelligence and personality characteristics together explained 46% of the variance in how much their caseloads improved (Rieck & Callahan, 2013).

A Problem That Stays Put

Across community settings, about 19.7% of patients end treatment early (Swift & Greenberg, 2012). Across six studies in three training clinics, published between 2005 and 2019, premature termination ranged from 68% to 78%. The pattern has not moved, which points to something structural about how clinical skill is built and checked.

0%20%40%60%80%100%77.9%N=68Callahan &Hynan200577%N=135Swift et al.200977.5%N=40Callahan et al.200967.9%N=42Aubuchon-Endsley& Callahan200977.4%N=31Swift &Callahan201169.9%N=524Al-Jabariet al.2019Dashed line: community norm of 19.7% (Swift and Greenberg, 2012)Dark green, teal, and orange bars are three different training clinics.

Improvement tells the same story. In the community, about 74% of patients improve by 26 sessions (Howard et al., 1986). In a training clinic sample, 31% of patients had improved by 26 sessions (Callahan & Hynan, 2005). A later training clinic sample also responded more slowly than the community model predicts (Schuler et al., 2021). Measuring competency is a way to find out why, and to change it.

74%Community (Howard et al., 1986)31%Training clinic (Callahan & Hynan, 2005)Patients improved by 26 sessions

Supervisors Matter Too

Competency is not only a property of trainees. Supervisors differ in how much they change what happens to patients.

0%5%10%15%20%16.4%Callahan, Almstrom, et al., 2009

Some supervisors move outcomes

In a training clinic, the supervisor accounted for 16.4% of the variance in patient outcomes beyond initial severity and therapist (Callahan, Almstrom, et al., 2009). A later study found that adding the supervisor improved classification of patient outcomes by 12.5 percentage points over chance (Wrape et al., 2015).

What the best supervisors do

Across 13 supervisors, 32 trainees, and 256 patients, lower supervisor agreeableness predicted better patient outcomes (r = -.71). The authors suggest that low agreeableness, used in the service of trainee and patient development, was not experienced as disruptive to the supervision alliance. Neither the supervision alliance nor supervisor emotional intelligence predicted patient outcomes (Rieck et al., 2015).

Generating the Science That Builds Training Clinics of the Future

Competency rises fastest early in training, so the steepest part of the learning curve falls on the first patients a trainee sees. Those patients did not choose to be training cases. The CARE Lab does not run a training clinic. It generates the evidence that training clinics, including those of the UT Dallas doctoral concentration, can build on to protect patients while trainees learn.

Measures that show what works

Valid competency measures tied to patient outcomes tell a clinic which training changes improve care and which only add hours.

Practice before patients

The lab develops and studies simulation-based practice, so that a trainee can work through far more presentations than a caseload of a few patients at a time allows, and an error becomes a teaching moment instead of a clinical event.

Supervision, supercharged

These tools keep the human supervisor in the loop and give supervisors more observable behavior and more immediate feedback to work with. The aim is to strengthen supervision and not to hand it off. Clinics decide how to put the science into practice. The lab’s role is to make sure the science is there to build on.

Why It Matters for Implementation

Evidence-based treatments reach patients through trainees and supervisors. The lab treats training and supervision as implementation strategies, and a strategy cannot be improved unless its effect can be measured. A valid, stable measure of competency lets programs test which training and supervision practices change what trainees and supervisors can do, and it ties feedback to patient outcomes rather than to hours accrued.

References

  1. Al-Jabari, R., Murrell, A. R., Callahan, J. L., Cox, R. J., & Lester, E. G. (2019). Do distress level and waitlists impact termination in a training clinic? Training and Education in Professional Psychology, 13, 127-137. https://doi.org/10.1037/tep0000223
  2. Aubuchon-Endsley, N., & Callahan, J. L. (2009). The hour of departure: Predicting attrition in the training clinic from role expectancies. Training and Education in Professional Psychology, 3, 120-126. https://doi.org/10.1037/a0014455
  3. Callahan, J. L. (in press). What we measure when we measure competency: Progress, challenges, and future directions. Clinical Psychologist.
  4. Callahan, J. L., Almstrom, C. M., Borja, S. E., Swift, J. K., & Heath, C. (2009). Exploring the contribution of supervisors to intervention outcomes. Training and Education in Professional Psychology, 3, 72-77. https://doi.org/10.1037/a0014294
  5. Callahan, J. L., Aubuchon-Endsley, N., Borja, S. E., & Swift, J. K. (2009). Pretreatment expectancies and premature termination in a training clinic environment. Training and Education in Professional Psychology, 3, 111-119. https://doi.org/10.1037/a0012901
  6. Callahan, J. L., & Hynan, M. T. (2005). Models of psychotherapy outcome: Are they applicable in training clinics? Psychological Services, 2, 65-69. https://doi.org/10.1037/1541-1559.2.1.65
  7. Callahan, J. L., Neumann, C. S., Cox, R. J., & Ruggero, C. J. (2017). The Supervisor Trainee Quarterly Evaluation (STQE): Psychometric support for use as a measure of competency. Training and Education in Professional Psychology, 11, 117-127. https://doi.org/10.1037/tep0000140
  8. Cruess, R. L., Cruess, S. R., & Steinert, Y. (2016). Amending Miller’s pyramid to include professional identity formation. Academic Medicine, 91(2), 180-185. https://doi.org/10.1097/ACM.0000000000000913
  9. Dimmick, A. A., Callahan, J. L., & Cox, R. J. (2023). Further validation of competency assessment: The practicum evaluation form. Training and Education in Professional Psychology, 17(3), 269-276. https://doi.org/10.1037/tep0000416
  10. Gonsalvez, C. J., Deane, F. P., Terry, J., & Gooi, C. H. (2020). Anatomy of psychology practitioner competence: Clustering approach reveals the hierarchical organization of competencies. Clinical Psychology: Science and Practice, 27(4), Article e12347. https://doi.org/10.1111/cpsp.12347
  11. Howard, K. I., Kopta, S. M., Krause, M. S., & Orlinsky, D. E. (1986). The dose-effect relationship in psychotherapy. American Psychologist, 41(2), 159-164. https://doi.org/10.1037/0003-066X.41.2.159
  12. Miller, G. E. (1990). The assessment of clinical skills/competence/performance. Academic Medicine, 65(9), S63-S67. https://doi.org/10.1097/00001888-199009000-00045
  13. Price, S. D., Callahan, J. L., & Cox, R. J. (2017). Psychometric investigation of competency benchmarks. Training and Education in Professional Psychology, 11, 128-139. https://doi.org/10.1037/tep0000133
  14. Rieck, T., & Callahan, J. L. (2013). Emotional intelligence and psychotherapy outcomes in the training clinic. Training and Education in Professional Psychology, 7, 42-52. https://doi.org/10.1037/a0031659
  15. Rieck, T., Callahan, J. L., & Watkins, C. E., Jr. (2015). Clinical supervision: An exploration of possible mechanisms of action. Training and Education in Professional Psychology, 9, 187-194. https://doi.org/10.1037/tep0000080
  16. Schuler, K., Kilmer, E. D., Callahan, J. L., Dziurzynski, K., & Swift, J. (2021). The dose-effect model is good enough. Counselling and Psychotherapy Research, 21, 608-620. https://doi.org/10.1002/capr.12337
  17. Swift, J. K., Callahan, J. L., & Levine, J. C. (2009). Using clinically significant change to identify premature termination. Psychotherapy, 46, 328-335. https://doi.org/10.1037/a0017003
  18. Swift, J. K., & Callahan, J. L. (2011). Decreasing treatment dropout by addressing expectations for treatment length. Psychotherapy Research, 21, 193-200. https://doi.org/10.1080/10503307.2010.541294
  19. Swift, J. K., & Greenberg, R. P. (2012). Premature discontinuation in adult psychotherapy: A meta-analysis. Journal of Consulting and Clinical Psychology, 80(4), 547-559. https://doi.org/10.1037/a0028226
  20. ten Cate, O., Carraccio, C., Damodaran, A., Gofton, W., Hamstra, S. J., Hart, D. E., Richardson, D., Ross, S., Schultz, K., Warm, E. J., Whelan, A. J., & Schumacher, D. J. (2021). Entrustment decision making: Extending Miller’s pyramid. Academic Medicine, 96(2), 199-204. https://doi.org/10.1097/ACM.0000000000003800
  21. Wampold, B. E., & Imel, Z. E. (2015). The great psychotherapy debate: The evidence for what makes psychotherapy work (2nd ed.). Lawrence Erlbaum. https://doi.org/10.4324/9780203582015
  22. Wrape, E. R., Callahan, J. L., Ruggero, C. J., & Watkins, C. E., Jr. (2015). An exploration of faculty supervisor variables and their impact on client outcome. Training and Education in Professional Psychology, 9, 35-43. https://doi.org/10.1037/tep0000014
Back to the CARE Lab
×

Texas NABORS Dashboard Data Usage Terms & Conditions / Published by: Texas NABORS / Usage Limitations: None