QUALITY OF COMPETENCY-BASED RESIDENCY PROGRAMS IN LITHUANIA

(1)

LITHUANIAN UNIVERSITY OF HEALTH SCIENCES MEDICAL ACADEMY

Eglė Vaižgėlienė

QUALITY OF COMPETENCY-BASED

RESIDENCY PROGRAMS

IN LITHUANIA

Doctoral Dissertation Biomedical Sciences, Public Health (09B) Kaunas, 2017

(2)

Dissertation has been prepared at the Department of Preventive medicine of Faculty of Public Health of Medical Academy, Lithuanian University of Health Sciences, during the period of 2013–2017.

Scientific Supervisor:

Prof. Habil. Dr. Žilvinas Padaiga (Lithuanian University of Health Sciences, Biome-dical Sciences, Public Health – 09B)

Scientific consultants:

Prof. Habil. Dr. Daiva Rastenytė (Lithuanian University of Health Sciences, Biomedi-cal Sciences, Public Health – 09B)

Prof. Dr. Algimantas Tamelis (Lithuanian University of Health Sciences, Biomedical Sciences, Medicine – 06B)

Dissertation is defended at the Public Health Research Council of the Lithuanian University of Health Sciences:

Chairperson

Prof. Dr. Lina Jaruševičienė (Lithuanian University of Health Sciences, Biomedical Sciences, Public Health – 09B)

Members:

Prof. Dr. Skirmantė Sauliūnė (Lithuanian University of Health Sciences, Biomedical Sciences, Public Health – 09B)

Prof. Dr. Linas Šumskas (Lithuanian University of Health Sciences, Biomedical Sciences, Public Health – 09B)

Prof. Dr. Vytautas Kasiulevičius, (Vilnius University, Biomedical Sciences, Medicine – 06B)

Prof. Dr. Fedde Scheele (VU University of Amsterdam (the Netherlands), Biomedical Sciences, Medicine – 06B)

Dissertation will be defended at the open session of the Public Health Research Council of the Lithuanian University of Health Sciences on the 27th_{of November, 2017 at 2 p.m. in the} prof. Vl. Laso auditorium of Lithuanian University of Health Sciences.

(3)

LIETUVOS SVEIKATOS MOKSLŲ UNIVERSITETAS MEDICINOS AKADEMIJA

Eglė Vaižgėlienė

KOMPETENCIJOMIS GRĮSTŲ

REZIDENTŪROS STUDIJŲ

PROGRAMŲ KOKYBĖ LIETUVOJE

Daktaro disertacija

Biomedicinos mokslai, visuomenės sveikata (09B)

(4)

Disertacija rengta 2013–2017 metais Lietuvos sveikatos mokslų universiteto Medicinos akademijos Visuomenės sveikatos fakulteto Profilaktinės medicinos katedroje.

Mokslinis vadovas

prof. habil. dr. Žilvinas Padaiga (Lietuvos sveikatos mokslų universitetas, biomedicinos mokslai, visuomenės sveikata – 09B)

Konsultantai:

prof. habil. dr. Daiva Rastenytė (Lietuvos sveikatos mokslų universitetas, biomedicinos mokslai, visuomenės sveikata – 09B)

prof. dr. Algimantas Tamelis (Lietuvos sveikatos mokslų universitetas, biomedicinos mokslai, medicina – 06B)

Disertacija ginama Lietuvos sveikatos mokslų universiteto Visuomenės sveikatos mokslo krypties taryboje:

Pirmininkė

prof. dr. Lina Jaruševičienė (Lietuvos sveikatos mokslų universitetas, biomedicinos mokslai, visuomenės sveikata – 09B)

Nariai:

prof. dr. Skirmantė Sauliūnė (Lietuvos sveikatos mokslų universitetas, biomedicinos mokslai, visuomenės sveikata – 09B)

prof. dr. Linas Šumskas (Lietuvos sveikatos mokslų universitetas, biomedicinos mokslai, visuomenės sveikata – 09B)

prof. dr. Vytautas Kasiulevičius, (Vilniaus universitetas, biomedicinos mokslai, medicina – 06B)

prof. dr. Fedde Scheele (VU Amsterdamo universitetas (Nyderlandai), biomedicinos mokslai, medicina – 06B)

Disertacija ginama viešame Visuomenės sveikatos mokslo krypties tarybos posėdyje 2017 m. lapkričio 27 d. 14 val. Lietuvos sveikatos mokslų universiteto Centrinių rūmų prof. Vl. Lašo vardo auditorijoje.

(5)

ABBREVIATIONS

CBME – Competency based medical education CanMEDS – Physician competency framework

CTEI Clinical Teaching Effectiveness Instrument

ECTS – European Credit Transfer and Accumulation System European Higher Education Area

EFFECT – Evaluation and feedback for effective clinical teaching EFFECT-S EFFECT system

EPA – Entrustable professional activities

ESG – The Standards and guidelines for quality assurance in the European Union

EU – European Union

JGA – Lithuanian Junior Doctors Association LSMU – Lithuanian University of Health Sciences

MEDINE Thematic Network for Medical Education in Europe LR – Republic of Lithuania

SETQ Systematic Evaluation of Teaching qualities

SETOC The Student Evaluation of Teaching in Outpatient Clinics SKVC – Centre for Quality Assessment in Higher Education UEMS – European Union of Medical Specialists

VU – Vilnius University MOS – Mean overall scores MSS – Mean subdomains’ scores

(8)

INTRODUCTION

The expectations by patients are always a priority despite economic situation and availability of health human resources in the country. All countries have their own important and unique achievements in medical education, looking for new and more effective training methods within their economic and cultural conditions. Sharing these experiences enriches everyone and builds up more value for a patient – more effective and safe services.

One of most important training periods of physicians is residency training. After it, specialists are supposed to become perfectly skilled members of health care system, providing best quality services.

Until now, different models for training physicians exist – British, American, Canadian, Scandinavian, French, etc. The largest differences within these models are registered in postgraduate education, while undergra-duate training is more or less aligned. While in the EU minimal requirements for training physicians exist, educational process might be quite different between countries’ and even within countries different universities. High quality and safe patient care can only be assured if physicians gain high-quality clinical teaching through residency studies [1–4]. Residency aims at training competent physicians, who are able to provide evidence-based health care services and to acquire necessary clinical skills, knowledge and competencies [5–7]. Therefore, regulatory organizations now require de-monstration of attainment of their expectations in some countries and this requirement now guides accreditation processes. Following these require-ments, medical schools focus on the improvement of clinical teaching through modernizations including the implementation of competency-based medical education (CBME) curricula and assessment of the quality in residency studies following the accreditation and quality assurance standards [8,9].

CBME has been recognized internationally as a system of education and training that holds the best promise of improving learner and patient outcomes [10]. CBME is focused on the achievements and it is a perfect way to disclose the knowledge and abilities that residents obtain to employers, health-care politicians and patients. Competency-based crucially requires demonstration that the learner is competent to progress in training. Transition from a time and process-based system to a competency-based framework claims certain changes in all levels of postgraduate training. Core components of this change involve the development of valid and reliable assessment tools, such as work-based assessment using direct observation, frequent formative feedback, and learner self-directed assessment, active involvement of the learner in the

(9)

educational process and intensive faculty development that addresses curricular design and the assessment of competency [5,11,12]. In the future, programs and institutions will be evaluated by more than their average scores on standardized exams or placement rates into competitive programs. Rather, they will need to demonstrate, through the outcomes of the learners they graduate, that they are meeting the needs of patients and communities. Bedsides the promises offered by CBME, it should be highlighted that practical implementation of CBME curricula is always challenging and requires all stakeholders’ enrolment in this process.

In terms of the methodology designed under „Tuning Lithuania” project, in 2013 Lithuanian University of Health Sciences (LSMU) renewed its residency programs according to the methodologies based on the intended learning outcomes and competencies. These innovations induced urgent need to implement evidence-based quality evaluation system for CBME residency studies in Lithuania.

Scientific novelty

This is the first research study in EU-10 (new members of EU since 2004) evaluating quality of competency-based residency programs and its change applying validated evidence-based instrument.

The practical value of scientific work

Validation and implementation of EFFECT-S system for assessing quality of CBME residency studies created possibility for continuous quality improvement and research in Lithuania.

Author's contribution to research

The author was responsible for whole study: validating the EFFECT questionnaire and assessing quality and its’ changes of CBME residency studies. It involved all organisational aspects results to the departments, statistical analysis, drafting publications, presenting study results in the conferences, preparing and moderating feedback dialogues.

(10)

AIM AND OBJECTIVES

Aim

To examine the quality of competency-based residency programs in Lithuania.

Objectives

1. To validate a selected quality evaluation methodology for compe-tency-based residency programs.

2. To assess quality of competency-based residency programs.

3. To examine quality changes of competency-based residency prog-rams following implementation of the EFFECT system.

(11)

1. LITERATURE REVIEW

1.1. Residency studies and their regulation

in the European Union and Lithuania

1.1.1. Medical training and its regulation in the European Union

Guiding principles for high level medical training were defined by the Council of the European Union of Medical Specialists (UEMS) in the “Charter on training of medical specialists in the European community” issued in October, 1993. The key aim in the following charter, is to assure the requirements for adequate training that prepares specialists for practice of their specialty at an appropriate level in any Member State of the European Union (EU) [13]. The equal level of specialists’ preparation is assured through the definition of the training contents. This charter divides the requirements regarding content of training into a general part, defined by the UEMS, and a specific part for each recognized specialty, defined by the UEMS Specialized Sections. The UEMS made a number of recommendations for effective implementation of various residency training programs. Recommendations were made for the structure of training program, competency-based training standards, standards for training institutions, trainers and supervisors, and quality assurance mechanisms.

UEMS Specialist Sections, European Boards, Divisions and Multidis-ciplinary Joint Committees are working on developing European Standards in Medical training that reflects modern medical practice and current scientific findings. Each EU member country has its National Board that monitors specialists’ training in compliance with general EU requirements in the country. Each country also has the National Authority that regulates the specialists’ training standards in the state and carries a direct responsibility for specialists’ qualification, conduct of study programs and quality research. The solutions made agreeably with this institution have to comply with national rules and the EU legislation as well as considering UEMS European Board recommendations.

On September 7, 2005, the European Parliament and the Council signed the European Parliament and Board Directive 2005/36/EC that turned into the key document defining the rules and regulating the recognition of professional qualifications in the EU. The Directive replaced three general system directives (89/48/EEC, 92/51/EEC and 1999/42/EC) and twelve sectoral directives (93/16/EEC, 77/452/EEC, 77/453/EEC, 78/686/EEC, 78/687/EEC, 78/1026/EEC, 78/1027/EEC, 80/154/EEC, 80/155/EEC, 85/432/EEC, 85/433/EEC, and 85/384/EEC) [14, 15]. This consolidation of

(12)

documentation affected changes which move the system in the direction of greater liberalization of cross border service provision, more automatic recognition of qualifications and increased flexibility in the procedures for updating the legislation [15].

European Union has emphasised the need for harmonisation of medical training across Europe. This is an important issue which imposes a process of standardisation in training programs, competencies and quality assessment for the safety of patients and the security of professionals, whereas respecting existing cultural diversity. In the EU there are only minimal requirements for medical training. Thus, although medical training across the EU is being harmonized, still plenty of differences remain in educational process between countries and even within countries between universities.

1.1.2. Medical training and its regulation in Lithuania

Standards of the EU for medical training turned into compulsory ones after Lithuania entered the EU on May 1, 2004. During recent decade, the significant progress has been made towards harmonisation of EU standards for medical training in Lithuania [16, 17]. New legal environment set new challenges for medical training and opened new possibilities for medical specialists to integrate in the EU professionals’ market.

One of the most significant events in residency training were those affecting the change of the resident’s status. It took place in 2008 when the amendments to the Resolution No. 1359 of 31 October 2003 of the Govern-ment of the Republic of Lithuania on the training of physicians were made. Accordingly, these legal acts regulated the residents’ employment as a phy-sician. This process has ensured social security for residents – being officially employed grants social benefits in case of disease or having children [18]. Newly regulated residents’ status confirmed an approach that a resident is not only a student, but he/she is an employed medical physician. Up to present a resident has a double status in Lithuania: she/he studies in residency and works as a physician-resident [19, 20]. Such a double status of resident is legislated only in Lithuania, Latvia and France. In other EU countries resident has the single resident-physician’s status.

Following the 2005/36/EC Directive requirements, in 2010 the one year internship studies were integrated into the basic 6-year medical training. General practitioners’ training was reoriented into the gaining of family doctors’ competencies. Training of physicians-specialists’ was essentially changed: subspecialties, such as endoscopy, phthysiatry and others, were eliminated, fields of specializations changed following the EU directive, the duration of medical specialist training and the curriculum was focused on

(13)

acquisition of practical skills and competencies. Up to now the duration of the residency study programmes complies only with minimal requirements of 2005/36/EC Directive.

Another important change was made in 2011 when the joint admission into residency studies at Lithuanian University of Health Sciences (LSMU) and Vilnius University (VU) was started. The following strengthened the trust of the admission procedure and facilitated the administration of the admission process into residency studies and its transparency.

At the moment, it takes about nine to twelve years to become a medical specialist in Lithuania. After a six-year undergraduate medical training nearly all students apply for residency study programs. Enrolment into residency is regulated by the Ministry of Health taking into consideration the need for specialists in the country. Following the data of Ministry of Health the enrolment into residency studies since 2006 has doubled (Figure 1.1.2.1).

Figure 1.1.2.1. Enrolment into the residency studies

in Lithuania during 2006–2016.

Source: Ministry of Health.

In Lithuania, residency studies come under regulations jointly establi-shed by the Ministry of Health and the Ministry of Education and Science. The right to conduct residency study programs is granted to Lithuanian University of Health Sciences and Vilnius University. Studies are conducted in compliance with the requirements for residency study programs and are accredited by the Centre for Quality Assessment in Higher Education and registered in the Register of Study programs, Learning programs and

(14)

Qualifications [21]. In 2016, 1821 residents studied in Lithuania (in LSMU – 1027, in VU – 794) [22, 23].

Depending on the specialty the total duration of residency studies varies from 3 to 6 years and on average complies with 66 ECTC credits per year [24]. During residency studies residents complete a theoretical course that is delivered by university lecturers and conduct clinical practice, supervised by physicians working in that clinical setting. Essentially, the residents’ training takes places in university hospitals. However, the possibilities to conduct part of the residency program at the other hospitals are also created. In order to become an official basis for residency program(s) hospitals have to be accredited by the universities. Since 2008 LSMU has accredited – 66, and VU – 60 residency bases [22, 25] .

In the period of three latter years plenty of discussions concerning the development of resident-physicians’ training system have been organized between health politicians, universities and Lithuanian Junior Doctors Association. Following the political will the Government of Lithuania under the resolution No 167 on March 13, 2017 has assigned Ministry of Health to prepare a stepped system for the assessment of residents’ competencies and skills throughout their educational process [26]. This system is believed to improve residents’ training and confer their higher independency during their residency studies and practical work.

The latter trends in health policy have invoked new challenges for both universities and health care institutions (residency bases), also for residents and their supervisors. Implementation of these resolutions into practice will request modern decision-making and collaborative efforts seeking for the new ways to improve quality of residency studies and the assurance of patient safety.

1.2. Competency-based medical education 1.2.1. CBME and its implementation in residency training

The transition in medical education from a process-based system to a competency-based training framework started with the introduction of Tomorrow's Doctors in 1993 [11]. Competence-based medical education (CBME) is an outcomes-based approach to the design, implementation and evaluation of medical education program using an organizing framework of competencies [27]. A comparison of the elements of structure-and process-based versus competency-process-based educational programs is presented in Table 1.2.1.1.

(15)

Table 1.2.1.1. A comparison of the elements of structure- and process-based

versus competency-based educational programs [11] Variable _{Structure- and}Educational program

process-based Competency-based

Driving force for curriculum

Content – knowledge acquisition

Outcome – knowledge acquisition

Driving force for process Teacher Learner Path of learning Hierarchical

(teacher → student) Non-hierarchical (teacher ←→ student) Responsibility for content Teacher Student and teacher Goal of educational

encounter

Knowledge acquisition Knowledge application Typical assessment tool Single subjective measure Multiple objective measures

(“evaluation portfolio”) Assessment tool Proxy Authentic (mimics real tasks

of profession) Setting for evaluation Removed (gestalt) “In the trenches”

(direct observation) Evaluation Norm-referenced Criterion-referenced Timing of assessment Emphasis on summative Emphasis on formative Program completion Fixed time Variable time

Lobst WF et. al. Competency-based medical education in postgraduate medical education. Medical teacher 2010.

The approach to planning CBME in contrasts with process-based curricula is widely described in literature [28]. Traditional program starts with a question “What do learners need to know?” or “How shall we teach our learners?”, while CBME is organized around the question “What abilities are needed of graduates?” [29, 30]. Answers to this question are gained by the program executives through the assessment of social expectations and needs, patient and population health needs, questioning of potential employers and identification of the entrustable professional activities (EPA), devoted to different specialties [31, 32]. The identified requested abilities are grouped and the competencies are described. They are applied as constructive blocks to design the whole study program. Having described the finalizing competences, they are intended for interim learning outcomes. Considering the selected interim and final outcomes, the most relevant learning and assessment methods are selected, which may aid learners at the achievement

(16)

of the best results and disclosure of their abilities [28]. Frank et al. has summarized the following steps for CBME curricula planning [27]:

• identify the abilities needed for graduates;

• explicitly define the required competencies and their components; • define milestones along the development path for the competencies; • select educational activities, experiences, and instructional methods; • select assessment tools to measure progress along the milestones (milestones are descriptors of the expected abilities of physicians at defined stages of expertise development [33]);

• design an outcomes evaluation of the program.

Currently, CBME is being adopted in many countries worldwide [34]. Early evidence suggests that CBME may lead to better residency education, enhanced promotion of continuous medical education once in practice, and ultimately better care for patients. There is still much resistance to its implementation and much confusion as to what exactly CBME entails [35, 36]. The implementation of CBME curricula in practice presents both benefits and challenges that have led subject of extensive debate in the literature [11, 12, 27, 37–41]. Potential advantages and challenges following the adoption of CBME, described by the international CBME collaborators, are presented in Table 1.2.1.2.

Table 1.2.1.2. Potential advantages and challenges following the adoption of

CBME

Potential advantages Potential challenges

A new paradigm of competence – greater

attention paid towards the abilities of those graduated from studies may aid at the development of health care spe-cialists, capable of providing patients and communities with relevant aids.

The threat of reductionism – seeking to describe

and assess competencies relevantly, very detai-led lists of knowdetai-ledge skills, behaviour, attitudes and their constituents are developed, which aggravate both learners and teachers’ daily activities.

New focus on the assessment and deve-lopment of milestones – fosters payment

of more attention to the assessment in the learning process.

Promoting the lowest common denominator –

the threat is that there may be shaped an image for learners that milestones and not excellence are the ultimate pursuit in medicine.

A mechanism to promote a true continuum of medical education – this

system aids at the assurance of the continuity in the learning process from undergraduate to residency training and to continuous professional development.

Logistical chaos – organizational and logistic

issues, combining traditional education system, when a group of residents learn for the certain indicated period in the program, where each learner studies at individual pace until he/she gains competencies, intended in advance.

(17)

Table 1.2.1.2. Continued

Potential advantages Potential challenges

A method to promote learner-cantered curricula – CBME is focused on the

learner, thus it is likely to involve participants, enforce them to learn at the selected pace and infuse responsibility for the final outcome.

Depreciation of traditional values, such as

mentoring, learning and teamwork.

Away to de-emphasize time-based cre-dentialing in medicine – learning

differ-rent subjects, in order to master them, each learner needs different time, thus, theoretically, CMBE curriculum may be more effective and useful.

The tyranny of utility – a pure CBME approach

is inherently utilitarian, and proposes cutting content and experiences that do not directly contribute to defined program outcomes. That can be unacceptable to some stakeholders in the profession.

Potential of portability of training –

CMBE curriculums may facilitate specia-lists’ accreditation in different institu-tions and their employment in different countries.

The need for new educational technologies –

would require new teaching techniques, new modules, and new assessment tools to be practical and effective.

Inertia and lack of resources – would require

significant investments in teaching, infrastruc-ture and assessment, and perhaps augmented workforce.

Source: Frank et al. Competency-based medical education: theory to practice. Medical teacher 2010.

Implementation of CBME requires development of valid and reliable assessment tools, such as work-based assessment using direct observation, frequent formative feedback and a learner’s self-directed assessment, active involvement of the learner in the educational process, and intensive faculty development that addresses curricular design and the assessment of competency [11].

In summary, it can be stated that CBME curricula are individualized-dynamic and by far more focused on the patient and learner. However, implementation of such curricula in practice arise many questions and problems. There are no universal answers – e. g., what instruments have to be applied for the assessment and how often it should take place in order to evaluate learners’ progress and final achievements. In reality, many organizational and logistic problems exist combining traditional learning system with CBME, when a group of residents learns the certain theme during the indicated time in compliance with the programme, where each learner learns at individual pace until he/she gains the competencies, intended in advance. Successful implementation of CBME curricula is considered to need more relevantly trained lecturers, more equipment, new measures and

(18)

methods for learning and assessment as well as financial resources. Nevertheless, the most significant is to have a clear vision – what and how we want to achieve in order to be able to decide if the certain part of the curriculum should be more focused in the duration and structure/framework of learning or on the certain achievements, abilities and on a possibility to learn at the personally selected pace for the learner.

Implementing CBME comes with significant challenges and always faces resistance and it is a slow and complex process, however, various strategies can be used to address these challenges [40]. Major strategies should be addressed to the quality assurance and to the faculty development around the teaching and assessment of the required resident competencies.

1.2.2. CBME and Tuning project on the EU level

In the EU increasing shift from content-based to competence-based higher education has started in 2000 when a group of European universities took up the Bologna challenge collectively and designed a pilot project called “Tuning educational structures in Europe” [42]. Tuning project aided the implementation of the Bologna process in higher education under conditions of diversity which grounded on international and national theory and practice for quality assurance in higher education [43, 44]. Firstly, higher education study programs have been started to be focused on learning aims and outcomes, and European credit transfer and accumulation system which conditioned possibilities for mobility and development of competences was launched [45]. Secondly, it gives special attention to the role of learning, teaching, assessment and performance in relation to quality assurance and evaluation.

In medical education, this process started in 2004 under the auspices of the MEDINE Thematic Network for Medical Education in Europe. The Tuning (Medicine) Task Force has generated a draft set of learning outcomes for primary medical degree qualifications in Europe [46, 47]. It was incorporated into a framework and process, to promote curriculum development and harmonization in EU medical education. The Tuning (Medicine) project estab-lished an EU-wide consensus on core learning outcomes for medical degrees.

The Tuning (Medicine) had a significant impact on the consistency, transparency, quality, and overall standing of medical education in the EU.

1.2.3. CBME and Tuning project on the National level

Seeking to implement the liabilities of the Bologna process, on April 30, 2009, the Law on Education and Studies was issued [48]. It obliged Lithuanian higher schools to implement the new credit concept to achieve a higher degree of consistency with the European higher education institutions and to make

(19)

optimal use of the advantages of the credit system until 2011. The EU “Tuning” experience in Lithuania was overtaken during 2009–2012 via implementation of the EU structural funds project – “Development of the Concept of the European Credit Transfer and Accumulation System (ECTS) at the National Level: Harmonization of the Credit and Implementation of the Learning Outcomes Based Study Program” (No VP1-2.2-ŠM_08_V-01-001)” [49]. During this project, the national concept of ECTS and methodology for its implementation was created. Tuning project outcomes have created a need for significant changes implementing new approaches for the organization and delivery of medical training in Lithuania. According the created guidelines of competence development, undergraduate and postgraduate study programs in the field of medicine were renewed [50].

Getting ready for the external evaluation of the residency programs, in 2013 LSMU renewed all its residency study programs according to the methodologies based on the intended learning outcomes (results), which are the acquisition of the generic and specific competences compulsory to the certain professional qualification. Tuning medicine project used the framework of Canadian Medical Education directives for Specialists – CanMEDS, which is the most widely accepted and applied physician competency framework in the world [51, 52]. This framework identifies and describes the abilities that physicians require to meet. The CanMEDS Roles and their brief description are the following [53]:

Medical expert (the integrating role)

1. Practices medicine within defined scope of practice and expertise. 2. Performs a patient-centred clinical assessment and establish a

management plan.

3. Plans and performs procedures and therapies for the purpose of assessment and/or management.

4. Establishes plans for on-going care and, when appropriate, timely consultation.

5. Actively contributes, as an individual and as a member of a team providing care, to the continuous improvement of health care quality and patient safety.

Communicator

1. Establishes professional therapeutic relationships with patients and their families.

2. Elicits and synthesizes accurate and relevant information, incor-porating the perspectives of patients and their families.

3. Shares health care information and plans with patients and their families.

(20)

4. Engages patients and their families in developing plans that reflect the patients’ health care needs and goals.

5. Documents and shares written and electronic information about the medical encounter to optimize clinical decision-making, patient safety, confidentiality, and privacy.

Collaborator

1. Works effectively with physicians and other colleagues in the health care professions.

2. Works with physicians and other colleagues in the health care profes-sions to promote understanding, manage differences, and resolve conflicts.

3. Hands over the care of a patient to another health care professional to facilitate continuity of safe patient care.

Leader

1. Contributes to the improvement of health care delivery in teams, organisations, and systems.

2. Engages in the stewardship of health care resources. 3. Demonstrates leadership in professional practice.

4. Manages carer planning, finances and health human resources in a practice.

Health advocate

1. Responds to an individual patients’ health needs by advocating with the patient within and beyond the clinical environment.

2. Responds to the needs of the communities or populations they serve by advocating with them for system level change in a socially accountable manner.

Scholar

1. Engages in the continuous enhancement of their professional activities through ongoing learning.

2. Teach students, residents, the public, and other health care profes-sionals.

3. Integrates the best available evidence into practice.

4. Contributes to the creation and dissemination of knowledge and practices applicable to health.

Professional

1. Demonstrates a commitment to patients by applying best practices and adhering to high ethical standards.

2. Demonstrates a commitment to society by recognising and responding to social expectation in health care.

(21)

3. Demonstrates a commitment to the profession by adhering to standards and participating in physician-led regulation.

4. Demonstrates a commitment to physical health and well-being to foster optimal patient care.

After some years of conducting competence-based residency programs in Lithuania is a growing need to develop residents’ supervisors’ educational competencies. It also requires to change the attitude of the whole university society and health/education politicians/advocates towards the physicians’ training both in undergraduate and postgraduate stage. Following the experience of the largest countries, seeking for changes in this field, at least 10 years of consistent and responsible organizational and educational work there is needed. Consistent and defined acquisition and development of the physicians’ competencies is very significant to all participants of the health care system. Physicians, participating in practical trainings and sharing their experience, may better systematize and share their knowledge, see and correct their drawbacks. Subsequently, residents would have a role of physician they are willing to become. Patients will be consulted by the physician expert in his/her field, and also by the one who is able to communicate, collaborate with colleagues, able to apply scientific knowledge into practice, to lead and promote health and has a professional attitude towards his/her work, health and place in the society.

CanMEDS are not a panacea fully solving physicians’ training needs and the development of the resident supervisor’s educational competencies. However, it enables finding of many answers to the question, arising during the training of potential special descriptors.

It should be highlighted that well-developed curricula and accredited study programs do not ensure their qualitative execution in practice. This is stressed in the advancement assessment report (2009) of the Bologna Process, where the Qualifications Frameworks coordination group revealed it concern in terms of possible danger that the institutions of higher education might learn how to draft a technically correct formal description of learning outcomes without actually implementing them in practice [54].

Following the Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG 2015), universities have to review their programs on a regular basis ensuring their compliance with international aims meeting learners’ and social needs, especially on quality assurance [55]. This fosters to implement continuous and evidence-based infrastructure of quality assurance of clinical teaching and to change management strategies that are effective in the transition to competency based training programs [12, 56].

(22)

1.2.4. Teaching and learning in the CBME curricula

In CBME paradigm learning is a collaborative process in which responsibility is shared between clinical teacher and resident [11]. Sustain-able intercourse between clinical teacher and resident facilitates the learning and teaching process. CBME movement prompts considerable amount of literature, identifying specific characteristics of good clinical teachers [7, 57]. Sutkin et al in their literature review of good clinical teachers grouped characteristics into three categories: physician, teacher and human, and described them as cognitive and non-cognitive (Table 1.2.4.1).

Table 1.2.4.1. The most often reported characteristics of good clinical

teacher from the Sutkin et al. (1909–2006) literature review [7] . Category

characteristics Description of characteristics

Physician characteristics

Demonstrates knowledge*, expertise, mastery of subject, thorough

knowledge, breadth of knowledge, knowledge of general medicine, understanding of the multicultural society in which medicine is practiced, intensively trained in medicine.

Demonstrates clinical competence*, clinical acumen, clinical reasoning

skills, can correlate and synthesize, diagnostic competence, technical expertise, clinical atitude; models clinical practice, skills in managing patients, viewing patient as a whole; links book facts with clinical practice.

Demonstrates enthusiasm for medicine**, for a specialty in medicine.

Teacher characteristics

Maintains positive learning climate** respect and support for residents;

creates facilitative and comfortable learning environment, encourages residents, respect for trainees as peers; receptive to residents, genuine interest in residents; positive relationships; cares about residents; shows unconditional positive regard for residents; shows love for youth; residents like him or her and want to work with him or her; interested in residents; ready to hear a residents’ troubles; aware of needs and problems; curious about trainee’s personality, norms, and values; sensitivity and responsiveness to the educational needs of the students and junior doctors, provides safe environment, corrects mistakes without belittling.

Demonstrates enthusiasm for and enjoyment of teaching**, love of

interpreting and expounding ideas, sense of teacher identity. Human

characteristics

Good communicator** open communication, good interpersonal

commu-nication skills, listens well, capable of lucid expression, persuasive.

Is an enthusiastic person in general**cheerful, eager.

(23)

Sutkin et al. in their literature review identified that 63% of descriptions of good clinical teaching are related to non-cognitive characteristics such as good role models, enthusiasm, honesty, sincerity, compassion and empathy as opposed to the skills referred to medical professional knowledge [7]. Other researchers also emphasize that good clinical teacher has to demonstrate good professional and teaching skills, and humanistic qualities as well [58–61]. In compliance with this they are described as effective supervisors, dynamic teachers, supportive individuals and role models of CanMEDS competencies [52, 62–65]. It is evident that clinical teachers who demonstrate these qualities are reportedly more efficient at creating an optimal learning environ-ment that fosters residents to identify their strengths and improve weaknesses in their residency training.

It should be highlighted that up to now in most of clinical teachers LSMU were not specifically trained to supervise residents in a competency-based curriculum. The majority of clinical teachers improved their supervisory skills through self-education, practice, experience and social interaction. However, there was no relevant deliberate reflection revealed on their own or others’ performance. There is evidence that feedback on clinical teaching skills has the power to change teachers’ behaviour and improve teaching skills and can also function as a source of support and motivation for clinical teachers [66]. All stakeholders need to understand that the success of curricular reforms is dependent on the teachers who put the reforms into practice [67,68]. Knowledge about CBME, should be provided, training for effective teaching techniques and new strategies for providing the authentic and regular assessment that is an essential aspect of effective CBME curricula development should be established [40].

1.3. Evaluation of quality of residency training

The discussion on the development need of the quality in medicine studies has been taking place for more than 100 years [69]. Currently, the interest in the quality of medicine studies and results is growing and a lot of attention is paid to the patients’ safety, rational usage of resources and the quality of aid. In the last decade, medical schools worldwide have increa-singly focus on the quality improvement through the curricula modernization according to the accreditation and quality assurance standards [9, 55].

The quality assurance of the EU higher education is based on the regulations and guidelines as well as on the national state legal acts. Following this, higher schools must have an internal quality assurance system of studies and take control of its implementation [55, 70].

(24)

In the literature on quality in residency studies, quality assurance efforts which previously largely focused on the accounting of clinical teaching quality, have shifted towards a focus on the improvement of clinical teaching quality [5, 59, 71].

1.3.1. Instruments for teaching quality assessment

Learning and teaching in a clinical environment depending on formal educational settings, providing specific challenges for clinicians who are teachers [5]. Thus, assurance of clinical teaching quality turns to be a prevailing aim, shifting to the conduct of CBME residency studies. Recently high significance that there was applied evidence based and valid instruments for the quality assessment of clinical teaching were developed [5, 59, 72]. These instruments are requested to evaluate the quality of clinical teaching and empower improvements of residency studies quality.

Several literature reviews disclosed that the most frequent instrument applied for the teaching quality assessment was the questionnaire [5, 73]. Scientists, examining the quality of clinical teaching, face a problem involving the selection of an instrument - the discovery of the most relevant one for a certain study and interpret outcomes [74]. There studies are described which used adapted instruments being designed and validated by others, and also studies, where the authors described the reliability and validity of instruments designed for teaching quality assessment designed by themselves. For example, Fluit et al., (2010) in their systematic review (1976– 2010) of content and quality of questionnaires for assessing clinical teachers, identified 32 instruments. These instruments contained from 1 to 46 items, with Likert scale points ranging from 4 to 9. For instruments content analysis, the authors categorized teaching activities into 7 domains in the process of clinical teaching: (1) physician role modelling, (2) task allocation, (3) provi-ding feedback, (4) planning/organising teaching, (5) teaching methodology, (6) assessing trainees, and (7) creating a supportive environment. Firstly, the researchers' findings disclosed that most frequently assessed domains were “teaching strategies” (30 questionnaires), “supporter role” (29 res), “role modelling” (27 questionnaires) and “feedback” (26 questionnai-res). The most rarely assessed domains were “assessor role” (5 instruments) and “assigning work” (13 instruments). Secondly, researchers identified that only one-third of all the analysed instrument-items were assessing the way clinical teachers teach CanMEDS competencies, which were mostly related to the Medical Expert and Scholar roles. Authors stated that questionnaires involved in review differ in terms of content and there is no any instrument,

(25)

which covers all clinical teaching domains that have been emphasized in the literature of clinical teaching [5].

Validity of instruments involved in Fluit et al. systematic analysis assessed following five aspects, indicated by the American Psychological and Education Research associations: (1) contents, (2) response process, (3) internal struc-ture, (4) relation to other variables, and (5) consequences. Their findings showed that most questionnaires were limited to dimensionality and internal consistency confirmation with applying factor analysis and/or Cronbach’s Alpha calculation [5]. However, more or less commonly used methods for validation showed the convergence between new and existing instruments and the faculty assessment correlation with educationally relevant outcomes [66]. Most of these instruments were used in inpatient setting or/and in one discipline, and instruments were usually completed only by residents. In general, instruments served to provide formative feedback, but were also used for resource allocation, promotion, and annual performance review [5].

Fluit et al. drew a conclusion that there is no single instrument covering all relevant aspects of clinical teaching comprehensively. Most studies involved in review were limited by context (inpatient), specialty (mostly a single clinical domain), breadth (not comprehensive of all of the roles of a clinical teacher), and validation of instruments was often limited to assessment of internal consistency and reliability. This suggest that the use of any of these instruments for assessment of CBME teaching quality will be limited [5].

Lombarts et al. stressed that systematic analysis conducted by Fluit and co-authors (2010) failed to detect instrument SETQ (Systematic Evaluation of Teaching Qualities) for evaluation teaching qualities. Authors also highlighted that SETQ is the most widely used clinical teaching assessment instrument in the Netherlands and it is reliable and valid for the use within residency training programs [75]. SETQ questionnaire was developed in the US which was adopted and validated for Netherlands situation. SETQ (consists of 25 items) covers physician role modelling and teaching strategies domains, such as creating positive learning climates and feedback, and psychometric testing of instrument covers five sources of validity evidence listed by Fluit et al. [75–77]. The authors claim that SETQ is relevant for the assessment of the quality in residency training and for the identification of differences among the subdivisions of the faculty departments and it is capable of providing with valuable information for the development of quality in residency training [76, 77]. Although SETQ covers most roles of effective clinical teaching, nevertheless it excluded roles mentioned by Fluit et al., i. e. “task allocation” and “planning”.

(26)

Zibrowski et al. (2011) decided to adopt two questionnaires developed and validated in US: CTEI (Clinical Teaching Effectiveness Instrument) and SETOC (The Student Evaluation of Teaching in Outpatient Clinics). For the evaluation of quality in residency studies in EU. It should be noted that these questionnaires include only 15 items, and do not cover all important roles of clinical teaching, e. g. “role modelling” and “assessment” in CTEI is exclu-ded”, while “task allocation”, “supporter role” and “assessment” is excluded in CETOC. Although validation and feasibility of CTEI and SETOC in published original studies seem to be very good [78, 79], however, having completed the adoption study, it came out that the questionnaires did not enable relevant evaluation of the researched fields [80]. Zibrowski et al. asked a question: why their findings did not comply with the published study results by the authors Copeland and Zuberi, who designed the instruments. Thus, a conclusion may be drawn that before starting to use the designed instrument, it is a necessary to conduct the pilot studies, assessing if the questionnaire was selected reasonably for the certain research [80, 81].

There is a growing body of literature relating to the development and adoption of these questionnaires [61, 80–82]. Recently, Fluit et al. have created and developed an instrument EFFECT (Evaluation and feedback for effective clinical teaching) based on the theoretical constructs of workplace learning and teaching and covering all seven key-domains for effective clinical teaching: (1) physician role modelling, (2) task allocation, (3) provi-ding feedback, (4) planning/organizing teaching, (5) teaching methodology, (6) assessing trainees, and (7) creating a supportive environment [61, 81, 83]. The reliability and validity of EFFECT questionnaire was empirically proved following five sources of validity evidence listed by The American Psychological and Education Research Associations [61]. Furthermore, this questionnaire incorporates the CanMEDS competencies and consequently, it could be used as a learning tool for important aspects of clinical teaching involving both teachers and residents.

Multiple quantitative instruments for measuring clinical teachers’ effectiveness or quality exist [5, 61, 77]. However, there is no unified methodology or the best standardized instrument (questionnaire) for the assessment of clinical teaching quality. In choosing an evaluation instrument, it is important that the questionnaire should meet content and psychometric quality standards, that it should be relevant to actual practice, that the instrument should be acceptable, and that the information serves a defined purpose [66, 74]. Selecting an instrument, researches must take into consideration the cultural context, educational system and the desirable study findings for the assessment. Currently, every institution chooses or creates an instrument for the assessment of residency quality following its needs.

(27)

1.3.2. Factors affecting clinical teaching quality

Having a good instrument for the quality assessment in clinical teaching is not sufficient. To enable such assessments to be properly interpreted and used for the improvement of clinical teaching quality, it is essential to understand the factors that might influence the outcomes of these evaluations [83, 84]. Arach at al. indicated that evaluations on clinical teaching quality are determined by such factors as resident’s year of training, the number of years since the faculty member’s board certification, and both faculty members’ and residents’ gender. The researchers concluded that younger faculty members, who dedicated more time to teaching and were evaluated by male residents in the early years of residency, were more likely to receive higher scores for teaching performance [84]. Lombarts et al. reported that residents from different disciplines rank specific teaching qualities differently [85]. Fluit et al. indicated that differences in residents’ ratings are influenced by the type of hospital, the clinical teachers’ gender and supervisory position, and the year of residency training [83]. Similar to Fluit et al. results other studies also reported that teaching in university hospitals were evaluated lower [86]. Same studies identified that lack of time for teaching was perceived as responsible for poor residents’ supervision [87, 88]. The understanding how following factors are related to the quality of clinical teaching could help to improve the interpretation of evaluation outcomes.

1.3.3. Feedback on teaching

Seeking to improve teaching, it turns to be crucial to provide a clinical teacher with feedback involving their teaching performance [59]. The efforts were described to provide effective feedback based on residents’ ratings to teachers in workplace settings and the principles how to do this [87]. This process requires instruments based on workplace learning and an effective procedure to provide this feedback [88]. The feedback based on residents’ ratings can aid teachers to identify areas for improvement and stimulate them to reflect on their teaching, and finally foster them to make changes in teaching practice [89]. There are some studies which examined the effective-ness of feedback through variable interventions [89, 90]. It should be highlighted that feedback used to improve performance is influenced by the recipient's interpretation of and responsiveness to feedback, which are, in turn, influenced by that individual's personal attributes, tensions, fear, confidence and reasoning processes [91]. For providing upward feedback a safe environment is crucial both for teacher and resident [90]. Boerboom et al. stated that it is necessary to facilitate teachers’ acceptance of feedback and

(28)

to help them to find their tipping point. Authors indicated the following objectives to be assisting clinical teacher for feedback facilitation [87]:

• to overcome negative emotions caused by the feedback; • with interpreting the feedback (report);

• to engage in critical and constructive reflection;

• to define (learning) goals and develop concrete plans to put the feedback into action.

The EFFECT system has more comprehensive strategy for feedback facilitation comprising planned sessions to stimulate discussion around feedback [66]. Individual teachers’ feedback report is constructed after an analysis (EFFECT questionnaire) of the resident’s and teacher’s self-evaluation ratings. A face-to-face feedback dialogue was organized involving the teachers who received reports and two residents based on the discussion of feedback. During dialogues residents are able to focus on the most important aspects of the feedback. They give additional specific comments, clarify results and formulate concrete suggestions for improvement. In order to stimulate constructive dialogue and assure safe environment, a moderator guides these dialogues. The EFFECT system looks promising to make teachers aware of their own strengths and weaknesses regarding their clinical teaching [90]. Authors concluded that EFFECT system is a feasible system that can lead to improvements in clinical teaching and it could contribute to a more open working climate in which it is considered normal to give feedback to each other, regardless to hierarchical position [66].

Research shows that in the Netherlands EFFECT system has been implemented successfully. Most appreciated element is the so-called upward feedback during the dialogue, although the other elements are essential too. Although EFFECT relies on an international literature study and is based on the theory that is internationally recognized as highly relevant to medical education, the authors claim the caution that is warranted in extrapolating their findings to other countries with different residency training programs and different feedback cultures as one of its possible limitations [83].

Taking in to consideration an organizational model of the residency studies in Lithuania and institutional needs of LSMU, for quality evaluation of clinical teaching in residency programmes we chose the EFFECT – theory-based, reliable, and valid instrument designed and validated by Fluit et al. in the Netherlands.

(29)

2. MATERIALS AND METHODS

2.1. The system for Evaluation and Feedback

For Effective Clinical Teaching

The system for Evaluation and Feedback For Effective Clinical Teaching (EFFECT-S) starts with an introduction meeting with staff and residents at the department responsible for that particular residency programme to inform (and involve) them about the formative purpose of the evaluation procedure at their department and to make tailor-made appointments (Figure 2.1.1.). EFFECT-S includes careful planning agreement on who fills in the questionnaires (residents on a voluntary basis, anonymous ratings), how the feedback procedure is organised, and who has access to the results (Annex 4). The evaluation itself consists of an internet-based self-evaluation question-naire for supervisors (Annex 3) and a questionquestion-naire to be completed by residents (Annex 2), a feedback report (is constructed in case supervisor has received three or more residents evaluations), including the mean scores per item and domain, a group score (the mean scores of all evaluated residency programs) and the written comments, and a face-to-face meeting (feedback dialogue) between the supervisor and two residents (representing their group), guided by a moderator from outside the department (Annex 5) [90].

(30)

Figure 1.1.1. EFFECT system (EFFECT-S)

Fluit et al. Residents provide feedback to their teachers: reflection through dialogue. Medical teacher 2013.

The EFFECT questionnaire

The EFFECT questionnaire was validated for the Netherlands and is based on theories of workplace learning and clinical teaching and incor-porates CanMEDS (Canadian Medical Education directives for Specialists) competencies. EFFECT contains 58 items grouped into 11 (sub)domains of clinical teaching: (1) role modelling clinical skills, (2) role modelling CanMEDS competencies, (3) role modelling academic research, (4) role modelling reflection, (5) task allocation, (6) planning, (7) feedback process, (8) feedback content, (9) teaching methodology, (10) personal support, and (11) assessment [81]. Items are scored on a six-point Likert scale (1 – critical, can’t proceed this way, 2 – unsatisfactory, large improvements required, 3 – average, needs to improve at some points, 4 – satisfactory, can improve on details, 5 – good, proceed this way, 6 – excellent, example to others and “not (yet) able to evaluate”). The option “not (yet) able to evaluate” was chosen if

(31)

a specific item did not (yet) occur during clinical teaching. Having obtained the authors’ agreement to use the questionnaire, we made a double-translation from Dutch to Lithuanian by 2 professional translators. In addition to the original items, information on gender, residency program, and the year of training was included.

2.2. Participants and research plan

This study was carried out in 7 residency programs which are regularly conducted at the Hospital of Lithuanian University of Health Sciences Kauno klinikos. Residents (2015/2016 year; n=182) (2017; n=218) and their clinical teachers (2015/2016 year; n=284) (2017; n=341) were invited by e-mail to fill-in the online EFFECT questionnaire (Figure 2.2.1). Residents were asked to evaluate those supervisors whom they have actually been working with during their residency training. Residents are supervised by clinical teachers who can either have an academic position (professor, associate professor, lecturer or assistant) or non-academic position (i. e., non-academic teacher, who is hospital employee having no work contract with the University). The residents could decide how many teachers they wanted to evaluate, not necessarily filling in the questionnaire for every teacher they worked with. Residents could indicate if the supervisor performs (portfolio) assessments. If not, items of the assessment domain were skipped. Data collection for the 1st_evaluationtook place during 2015−2016, while for the 2nd_{– in 2017. The}

residency study programs of anaesthesiology reanimathology, dermatovene-rology, emergency medicine, cardiology, neudermatovene-rology, physical medicine and rehabilitation, obstetrics and gynaecology were included into both evalua-tions.

(32)

Figure 2.2.1. Research plan

2.3. Description of study population

During the 1st evaluation 333 questionnaires to assess 146 clinical teachers were filled-in, during in the 2nd_{evaluation – 387 questionnaires}

assessing 129 teachers were filled in. As the evaluations were anonymous, we could not calculate the number of questionnaires filled in by every resident. During the 1st_{evaluation we received 143, while during the 2}nd_{- 137}

self-evaluations filled-in by clinical teachers (42 males and 101 female in 2015/2016; 46 males and 91 female in 2017). Out of 333 questionnaires from the 1st_{evaluation 36.9% were filled in by the first-year residents, 24.3% - by}

the second-year, 25.2% – by the third-year and 13.5% – by the fourth-year residents. During the 2nd evaluation the figures were 23.3%; 25.6%; 28.2% and 23.0%, respectively. Table 2.3.1 shows residency programs, their duration, number of residents per program, number of questionnaires filled-in by residents evaluatfilled-ing their clfilled-inical teachers, number of teachers filled-in the program, number of teachers evaluated in the program and number of self-evaluations of clinical teachers, for both 1st_{and 2}nd_evaluation.

(33)

Table 2.3.1. Number of residents in the program, number of questionnaires

filled in by residents, number of teachers in the program, number of teachers evaluated in the program and their self-evaluations per residency programme for 1st_{and 2}nd_evaluation

Residency program, (duration in years) Residents in the program, n Question-naires filled in by residents, n Teachers in the program, n Teachers evaluated in the program, n No. of self-evaluations by teachers Evaluation No 1st 2nd 1st 2nd 1st 2nd 1st 2nd 1st 2nd Anaesthesiology reanimathology (4) 53 48 57 71 70 86 21 28 30 36 Cardiology (4) ₃₀ ₃₁ ₄₅ ₂₀ ₅₃ ₅₆ ₂₂ ₁₂ ₂₂ ₂₆ Dermatovenerology (4) 19 17 37 35 10 22 8 10 8 13 Emergency medicine* (5) 17 37 115 62 74 76 56 17 24 10 Neurology (4) 11 29 20 24 22 37 12 13 14 9 Obstetrics and gynaecology (4) 34 33 30 136 37 51 16 39 34 32 Physical medicine and rehabilitation (3) 18 23 29 39 18 13 11 10 11 11 Total 182 218 333 387 284 341 146 129 143 137

*New residency program which started in 2013.

2.4. Data analysis

For assessing the internal consistency and reliability of the Lithuanian version of EFFECT questionnaire, the Cronbach's alpha was calculated. Structural equation modelling was applied to determine the amount of interdependency between items and constructs using the existing factorial solution as a model for maximum-likelihood estimation. In addition, common incremental measures of the scale fit in structural and equation modelling – the Comparative Fit Index (CFI) and Root Mean Square Error of Approxi-mation (RMSEA) – were calculated [92, 93]. Correlations between the dimensions were determined by correlation coefficients from the estimated covariance matrix. Correlation coefficients with a magnitude of 0.7 to 1.0 indicated interdependency of the factors.

To determine item characteristics, item means (M), standard deviations (SD), and percentages were calculated. For the EFFECT questionnaire and its

(34)

(sub)domains the mean overall scores (MOS) and mean (sub)domains scores (MSS) were calculated, averaging scores of all items with all responses except “not yet able to evaluate”. In order to see how residents’ evaluations match with the self-evaluation for every particular teacher, the intra-class correlation coefficient (ICC) was calculated. The overall comparisons of MOS and MSS across study groups (residents and teachers) and subgroups (by age, gender, year of training, academic position) were conducted using Student t-test and ANOVA for trend. In addition, the items were analysed subtracting their mean score from respective (sub)domains mean score. Items with negative values (≥0.20) we considered as areas for systematic quality improvement, while items with positive values (≥0.20) – as a good quality teaching areas within residency studies.

For assessment of quality changes of competency-based residency studies in 2017 as compared with 2015/2016 evaluation differences between items and (sub)domain means and adjusted means (for study year, teacher age, teacher gender and teacher academic position) while comparing overall samples were used. To evaluate influence of EFFECT system on quality changes we evaluated differences between items and (sub)domain means being assessed in 2017 of three separate groups: (1) teachers who were evaluated by at least three residents and participated in feedback dialogues in 2015/2016; (2) teachers who were evaluated by one or two residents in 2015/2016; (3) teachers who were not evaluated in 2015/2016 at all. To evaluate the highest possible influence of EFFECT system on quality changes differences between items and (sub)domain means where compared only for those teachers who were evaluated by residents (any number) and participated in feedback dialogue during the 1st evaluation in 2015/2016 and were evaluated by residents during the 2nd evaluation in 2017. For differences between items and (sub)domain means Student t-test, Wilcoxon Signed Rank Test and ANOVA for trend, while for differences between adjusted means - ANCOVA test were used.

The statistical significance was set at p<0.05. All the calculations were run with SPSS 20 and AMOS 20.

2.5. Ethics

The study was approved by Bioethics Centre and Study Quality Moni-toring and Improvement Commission of LSMU. The heads in charge of each residency program decided to take part in the study on voluntary basis.

(35)

3. RESULTS

3.1. Validation of Lithuanian version of EFFECT questionnaire

For the validation of the questionnaire we used data of residents’ questionnaires from the 1st survey. The results of the item characteristics are provided in Table 3.1.1. Mean scores ranged from 4.58 (item 29, “reminds me of previously given feedback”, and item 50, “helps and advises me on how to maintain a good work-home balance”) up to 5.40 (item 9, “applies to guidelines and protocols”). More than 20% of the answers in item 12 “have a bad news conversation”, item 40 “reviews my reports”, item 50 “helps and advises me on how to maintain a good work-home balance” were scored as “not (yet) able to evaluate”, while this proportion was over 70% for all the assessment domain items (items from 51 to 58). Factor loadings varied from 0.788 (item 30) to 0.957 (item 47).

Table 3.1.1. Item characteristics of the EFFECT questionnaire (n=333)

Domain Mean SD _loadingFactor NAE, _n NAE, _% ROLE MODELLING

Role modelling clinical skills

1. ask for a patient history 5.21 1.029 0.900 30 9.0 2. examine a patient 5.22 1.027 0.932 30 9.0 3. perform clinical actions 5.34 0.981 0.878 20 6.0

Role modelling general CanMEDS roles

4. cooperate with other health professionals while providing care to patients and relatives

5.25 1.047 0.855 8 2.4 5. communicate with patients 5.20 1.110 0.923 4 1.2 6. cooperate with colleagues 5.02 1.249 0.850 0 0.0 8. organize his or hers’ own work adequate 5.20 1.079 0.795 5 1.2 9. apply guidelines and protocols 5.40 0.990 0.804 6 1.5 10. treat patients respectfully 5.31 1.080 0.869 4 1.2 11. handle complaints and incidents 5.24 1.185 0.850 62 18.6 12. have a bad news conversation 5.26 1.071 0.860 76 22.8

Role modelling scholarship (academic research)

QUALITY OF COMPETENCY-BASED RESIDENCY PROGRAMS IN LITHUANIA

Eglė Vaižgėlienė