Operationalising the Framework

Assessment, curriculum, education program design and policies, are all interconnected and inform each other. The context of all RACGP education and assessment are underpinned by the RACGP Educational Framework and the Aboriginal and Torres Strait Islander Culture and Health Training Framework. Assessment occurs in the context of the broader environment, which includes:

Aboriginal and Torres Strait Islander Culture and Health Training Framework.

The Aboriginal and Torres Strait Islander Cultural and Health Training Framework is crucial to embedding Aboriginal and Torres Strait Islander values, approaches to learning and ways of thinking into education and training at the RACGP. It includes twelve guiding principles that are mutually reinforcing and help to provide a way to navigate complex and dynamic areas of work. The principles can respond to circumstances, overlap, and evolve. This enables them to be locally contextualised and situationally interpreted. 

Assessment blueprint 

Purposeful blueprinting is an integral part of an assessment system.^{10, 15} It ensures alignment of educational objectives with assessment and reduces risks to validity.¹⁵ The RACGP assessment blueprint demonstrates the alignment of assessment content to the Progressive capability profile and the curriculum. 

Curriculum and Syllabus for Australian General Practice (the curriculum). 

The curriculum and syllabus describe the core competencies and learning outcomes of GP education. It informs the development and delivery of training programs. 

Educational Framework 

The Educational Framework declares the RACGP values in relation to education and how things connect to allow coordination and a clear articulation of approach. It includes imperatives and education guiding principles. 

Progressive capability profile of the specialist general practitioner (The Profile). 

The Profile is structured around the roles of a specialist general practitioner’s work. It defines the progressive development and expansion of capabilities from entry to general practice terms, through to achievement of the Fellowship of the Royal Australian College of General Practitioners (RACGP). The capabilities are linked to four key milestones and levels of supervision.  

Standards for general practice training (Training Standards) and educational policies. 

The Training Standards define the expectations of training program design and delivery in order to meet the AMC Standards. This includes assessments as well as learning and training environments. 

Australian Medical Council (AMC) Standards for Assessment and Accreditation of Specialist Medical Programs.  

The AMC Standards outline the expectations for all specialist medical colleges in regard to education, assessment and training. 

An effective assessment program is a multi-faceted approach that allows for a comprehensive and longitudinal evaluation of abilities that supports and guides educators in measuring performance while at the same time, having a positive educational impact on registrars.

Assessment programs define the assessment instruments and inform curriculum development as well as drive continuous quality improvement in medical education.

Good assessment design enables training programs to identify and support those registrars struggling to meet the program requirements early and instil a culture of positive growth. For those performing well, assessment outcomes can provide stimulus and encouragement for them to extend themselves and take on roles in leadership, medical education and research.

Multiple tensions need to be balanced during assessment design to achieve the overall optimal program. The purpose of the assessment should guide what is foregrounded as a consideration, as well as strategic decisions about the weighting of competing priorities.⁸  As an example, the provision of individualised feedback for candidates to support their learning is an important principle. While this is possible in WBA where there is direct observation of a candidate by an assessor who is able to provide individualised feedback for ongoing improvement, it is less viable to do this individually for higher stakes standardised assessments.

For example, utility considers areas such as validity, reliability, educational impact, practicality, cost-effectiveness and acceptability. These will vary for individual assessment instruments but need to work overall as an assessment system.¹⁶

Engagement at all stages of assessment design with Cultural Educators and Aboriginal and Torres Strait Islander Medical Educators can ensure they are relevant, holistic, culturally safe and supported.¹⁷

Whilst the principles may be the same across assessments, it is relevant to note that the experience may vary by candidate and participant (participants includes all those involved in the development and delivery of assessments).

Safe, inclusive and fair assessment design should allow candidates to be able to demonstrate their actual level of competence.

Assessors, including supervisors and medical educators, may experience the practicality and cost-effectiveness of an assessment differently to a candidate or a program. A candidate may perceive the acceptability of an assessment differently to a patient who is a participant.

Wellbeing

Assessments can inherently impact wellbeing, both positively and negatively, depending on how they are designed, implemented, and perceived by individuals. Assessments involve being observed and judged, as well as judging others. As such, consideration must be given to the potential for impact on participants’ mental health, including candidates, assessors and patients. Balancing the needs of all involved can be challenging for those designing and delivering assessments as interventions that make assessments more supportive for one group may impact the wellbeing of others.¹⁸ 

The RACGP has signed up to the Every Doctor, Every Setting Framework¹⁹ which has a key focus on safe supported workplaces. The development of all College activities, not just assessment, need to be viewed through the lens of wellbeing to ensure that standards that have been agreed to are being met. Therefore, it is critical that the organisation appropriately resource support for all participants, including candidates, assessors, patients, staff and role players before, during and after assessments.   

The principles within the framework cover multiple elements that may impact wellbeing through an individual’s assessment experience. It seeks to provide an optimal overall assessment program that meets its required outcomes.  For example, the principles of alignment, fairness and validity are actioned through the provision of clarity and transparency of expectations, goals and standards. In addition, the impactful principle can be actioned by providing²⁰ constructive feedback which empowers students through catalytic learning²¹ which can be particularly powerful in workplace-based assessments.

Inclusive design includes both the tools used for assessment and the logistical elements of assessments.²² It supports the wellbeing of candidates through providing equal opportunities to demonstrate their learning and in the management of workload for all parties.²²

Wellbeing needs may vary between assessment participants, for example the needs of assessors will vary from those of candidates. Assessment design should take into consideration the need for appropriate training and support of assessors, such as managing bias and fairness. It may also include acknowledging the impact of being an assessor in workplace-based assessment specifically, which is a complex environment and therefore may require appropriate resourcing. One of the tools the RACGP has to assist in protecting wellbeing is through setting appropriate safeguards and guidance in the RACGP Standards for general practice training. The standards outline the expectations and responsibilities for all involved in training and as the workplace is the context for workplace-based assessment, the training standards are a mechanism to support safe and ethical environments.

The RACGP currently employs a suite of assessment tools for assessment, the principle being that assessments are designed ‘for’ learning and/or ‘of’ learning. Combined, they provide a rational system of assessment.

Assessment for learning

Assessment for learning is an approach that uses feedback to allow the learners to reflect on their performance and identify areas for improvement. It is usually conducted in the workplace and also provides educators information to support learners in meeting their individual and the programs goals.

Each element of these assessments are low stakes, however based on the picture of multiple low stakes assessments it is possible to triangulate information to provide a fuller picture of performance. This can involve multiple assessments, multiple assessment tools and multiple assessors to provide a foundation for programs to make higher stakes decisions.^{16, 23}

Assessments for learning may foreground principles such as being impactful and validity. Reliability may be achieved by triangulation, rather than on a single part of an assessment. Feasibility needs to consider the hundreds of locations and assessors when considering practicality.

Assessments of learning

The assessment of learning ascertains whether participants have met defined outcomes or benchmarks. They are known as high-stakes or hurdle assessments. RACGP high-stakes assessments of learning include Selection and the Fellowship examinations.

As a group, the Fellowship exams utilise a range of formats to assess across the breadth of the curriculum. Selection is a high-stakes assessment that assesses professional attributes and aptitude for general practice rather than being an assessment of learning.

Higher stakes assessments always foreground principles of fairness, validity, reliability and inclusivity when making decisions about assessment design.

Purposes of RACGP assessments 

Selection Assessment

The purpose of the RACGP selection assessment is to:

• select doctors who are suitable for the training program and meet the current and future needs of the Australian community
• allow applicants to demonstrate they have attained the required competencies for training program entry.

Workplace-based Assessments

The purpose of an early assessment for the:

• registrar to begin to identify learning needs, potential knowledge gaps and understand current level of competence. It should also support patient and registrar safety by identification of what oversight and assistance is needed for safety.
• supervisor and practice to understand the registrar's level of competence to be able to match supervision and to understand the registrar’s learning needs.
• training program to obtain evidence of the registrar’s performance and current level of competence, compared to that expected to enable appropriate support.²⁴

The purpose of a Workplace-based Assessment program as a whole is for the:

• registrar to receive feedback to support self-directed learning and to develop self-reflective skills and critical thinking. It should allow the registrar to understand their progress towards being a safe practitioner suitable for independent practice as a general practitioner in Australia.
• supervisor and practice to inform ongoing decisions on the appropriate supervision and education needed to support patient safety and registrar development
• training program to understand a registrar's progress, identify registrars in difficulty and inform decisions about extra support. It will also evaluate professional practice and behaviour and determine eligibility to sit the Fellowship exams.
• College to understand the professional practice and behaviour of registrars which can inform eligibility for the award of Fellowship of the RACGP.²⁵

Fellowship exams

The purpose of the Fellowship examinations is to assess for competency for unsupervised general practice anywhere in Australia. They are hurdle assessments that contribute to the RACGPs understanding of candidates competence and readiness for unsupervised practice. This assures registrars, practices, programs, and the community that the doctor has the required knowledge and skills for the role.²⁶ 

Recognition of Extended Skills Assessments

The purpose of RES-FRACGP assessment is to allow GPs to demonstrate their achievements against a prescribed set of extended skills outcomes.

Pre-Employment Structured Clinical Interview (PESCI)

The purpose of a PESCI is to evaluate an international medical graduate’s suitability for a particular general practice position. It assesses whether the doctor has the clinical knowledge, skills and experience to practise safely and effectively in the supervised position.²⁷

The RACGP Education and Training Monitoring and Evaluation Framework outlines an approach to evaluation to facilitate continuous improvement and quality.   Evaluation will measure whether the intended purpose of the assessments are being met and look beyond satisfaction ratings and processes. It will also look at whether the intended purposes of assessment are being met.