COVER STORY: JOURNAL CLUB

Journal club: expert opinion

Jennifer Reid introduces our new series to help you access the speech and language therapy literature, assess its credibility and decide how to act on your findings. Each instalment will take the mystery out of critically appraising a different type of journal article, starting with Expert Opinion.
ow confident are you that you can read, digest and evaluate the sorts of research articles that count as evidence? Check the expectations of the Royal College of Speech & Language Therapists (RCSLT) in figure 1. Last year I conducted a short online survey on evidence-based practice among speech and language therapy colleagues in NHS Fife in Scotland. These were some of the survey items, in case you wish to do a quick self-evaluation: hh I can find a research article that might be relevant to my work hh I can recognise whether a research article is of good quality hh I know which areas of my work are supported by systematic evidence hh I know how to evaluate my own practice in the light of new research findings hh I can change my practice to ensure it is more evidence-based hh I generally rely on other people summarising evidence for me hh My strategy for keeping up to date with developments in my field is hh I prefer to update myself by yy Reading journal articles yy Reading textbooks yy Reading whatever I can find on a topic on the internet yy Going to a SIG day, course, conference or lecture yy Talking to my colleagues hh Two areas of my practice that I would like to become more systematic and / or evidence-based are The results from the 50 Fife respondents suggested that there was room for improvement in the skills and confidence levels reported across most of the key areas highlighted in the RCSLTs minimum skills set (figure 1, no.3). Consequently, improvement in awareness and use of evidence-based practice is now embedded in the NHS Fife speech and language therapy services quality improvement strategy, and a programme of activity is underway. During

READ THIS SERIES IF YOU WANT TO yy BE MORE EVIDENCE-BASED IN YOUR PRACTICE yy FEEL MOTIVATED TO READ JOURNAL ARTICLES yy INFLUENCE THE DEVELOPMENT OF YOUR SERVICE

the past year, this programme has focused primarily on development opportunities for staff in electronic searching skills and critical appraisal. Applications of research findings to practice and evaluation of own practice will be a focus in future years. In Fife, our critical appraisal education is delivered through a series of small group journal clubs within each of our three client care groups (adult learning disability; adults with acquired disorders; paediatrics).

Figure 1 RCSLT expectations The Royal College of Speech and Language Therapists (RCSLT), in its current Research Strategy, lists the following underpinning principles: 1. All practitioners engaged in meeting the speech, language, communication and swallowing needs and disorders in the population / of their clients must use the evidence base to inform and support their clinical decision-making and as part of judging the safety, efficacy and appropriateness of their clinical practice. 2. The RCSLT expects all members will engage in a range of research related activities, (including self-directed and work-based learning) that will enable them to continue to develop their skills and knowledge throughout their careers. Speech and language therapists must demonstrate a personal commitment to ongoing education in order to continue developing their knowledge and skills when undertaking research. 3. The RCSLT expects every practitioner will have a minimum 'skills set' which will allow them to be evidence based practitioners. These skills should include searching the evidence, critical appraisal, applying research findings to practice and methods for evaluating their own practice.

Appraisal help

There are quite a lot of appraisal tools available now to help practitioners evaluate healthrelated research literature, for example the Critical Appraisal Skills Programme (CASP) tools published in 2006 by the Public Health Resource Unit in England. RCSLT Clinical Guidelines (2005: Appendix 2) also provide the very detailed set of checklists used during development of the guidelines. With these sorts of tools, you choose a specific checklist according to the methodology of the research article in question. Each checklist is a structured set of questions with points for consideration and room to record your appraisal notes. I encountered some problems when I tried to use the currently available appraisal tools in our journal clubs. Most were initially developed for appraisal of medical literature, so there is a strong emphasis on clinical trials of medical treatments. Novice users may be put off by the arcane terminology of the randomised controlled trial. By and large, current evaluation tools also favour quantitative over qualitative study designs (although the CASP toolkit does include a checklist for qualitative research). This is a problem given that qualitative methods are the ones of choice when the research either involves an under-explored area lots of those in our practice! or seeks to understand clients experiences, attitudes or beliefs.

SPEECH & LANGUAGE THERAPY IN PRACTICE AUTUMN 2010

17

COVER STORY: JOURNAL CLUB

There are many areas of current practice for which evidence is absent, scanty or of poor quality. For some topics we may need to rely on professional consensus or expert guidance, so we also need ways of evaluating such expert knowledge. There are appraisal tools for clinical guidelines but these are not really appropriate for appraising articles such as small scale narrative reviews of an area of speech and language therapy. Nor could I find a ready-made framework for appraising the sort of information provided by, for example, single-case study designs, which can be particularly useful for providing preliminary evidence or promising avenues for further study where no other more robust research evidence exists. For our journal clubs, I have developed a set of structured appraisal frameworks. For the overall framework, as well as much of the specific content, I acknowledge a strong debt to Greenhalgh (2006) as well as to the CASP tools. However, I have extended beyond the CASP set to include, for example, appraisal of expert opinion and of single-case design studies. We also, with tongue firmly in cheek, added graphics to our worksheets to support their imageability. Theres nothing like following your own advice! For this series, these have been replaced by cartoon drawings specially commissioned by Speech & Language Therapy in Practice. (Now, just how strong is the evidence for the impact of witty graphics on the transparency of abstract vocabulary?)
Primary studies report research first hand Experiment Clinical trial (intervention study) Survey A manoeuvre is performed on a volunteer in controlled surroundings An intervention is offered to a group of patients who are followed up to see what happens to them Something is measured in a group of patients, health professionals or some other sample of individuals Summarise primary studies Draw conclusions from primary studies about how clinicians should behave Use the results of primary studies to generate decision trees to be used by health professionals and patients in making choices about clinical management Use the results of primary studies to say whether a particular course of action is a good use of resources

Secondary or integrative studies summarise and draw conclusions from primary studies

Overviews Guidelines Decision analyses

Economic analyses Table 1 Primary or secondary study? Level 1 Systematic reviews and metaanalyses Randomised controlled trials (RCTs) with (statistically) definitive results RCTs with nondefinitive results Cohort studies

All primary studies on a particular subject hunted out and critically appraised according to rigorous criteria. Participants randomly allocated to one intervention or another. Both groups followed up for a specific time period and analysed in terms of specific outcomes defined at the outset of the study. Because, on average, the groups are identical apart from the intervention, any differences in outcomes are, in theory, attributable to the intervention. Two or more groups selected on the basis of differences in their exposure to a particular agent (for example, prematurity) and followed up to see how many in each group develop a particular outcome (for example, language impairment). People with a particular condition are identified and matched with controls. Data then collected for both groups on their past exposure to possible causal agents. Representative sample of people are interviewed, examined or otherwise studied to gain answers to a particular clinical question. Considered relatively weak scientific evidence but they have the advantage of being richer in information and easier to understand and remember!

Level 2

Level 3 Level 4

Getting your bearings

Once you have found your article (more on this later in the series), your first task is, in Greenhalghs words, getting your bearings (2006, p.40). The following three questions may be helpful. 1. Why was the study done? (What clinical questions did it address?) Sometimes the authors will explicitly tell you their clinical questions. If they dont, we have found it useful to try to reformulate the authors aims as one or more questions. The PICO framework can be helpful with this: P I C O population, problem intervention control, comparison outcome

Level 5

Case-controlled studies

Level 6

Cross-sectional surveys Case reports

Level 7

Table 3 Rating a study for level of evidence

For example: In children aged under 6 years with speech sound disorder (P), is there any difference in rate of speech sound development (O) when intervention includes non-oral motor exercises (I) compared to speech sound intervention alone (C)? When you reword it like this, it is immediately obvious what things the authors needed to define and measure. So, in

the example above, their definition of speech sound disorder should link explicitly to their participant selection criteria for the study. How are they defining and measuring rate of speech sound development and does this accord with your expectations? For some research designs, you only need part of the PICO. For example: Are teenagers with a history of specific (i.e. primary) language impairment (P) more at risk of negative mental health (O) than their peers (C)? At this point, you may decide to discontinue if the question(s) posed are really not what you are looking for answers to.

2. What type of study was done? This is a crucial step as the choice of an appraisal framework is largely driven by the design and methods used in the study. Tables 1, 2 and 3 provide some definitions to help you. 3. Was the study design appropriate to the broad field of research addressed? This question gets easier with experience - the most important issues are covered within the individual appraisal tools. Broadly speaking, you are checking how well the study has been designed so as to minimise the possibility that the results are untrue, biased, misleading or unreliable. An intervention study needs to demonstrate that the outcomes resulted

18

SPEECH & LANGUAGE THERAPY IN PRACTICE AUTUMN 2010

COVER STORY: JOURNAL CLUB

Quantitative designs involve measurement via data collection; stronger on reliability (same results every time) Therapy Testing the efficacy or effectiveness of interventions. Randomised controlled trial (RCT) is the preferred study design. Is a new diagnostic test valid (trustworthy) and reliable (gives same results all the time)? Cross-sectional survey is the preferred study design. Can a test pick up a condition at early or presymptomatic stage? Cross-sectional survey is the preferred study design. What is likely to happen to a person with a particular condition? Longitudinal survey is the preferred study design. Determining whether an agent is related to the development of a condition. Cohort or case-control study is the preferred study design, depending on rarity of the condition. Case reports may also provide crucial information. Measuring aptitudes, abilities, attitudes, beliefs or preferences. Study of documents produced by real people in real situations (for example, casenotes). Systematic recording of behaviour and talk in naturally occurring settings. The researcher takes part in the setting as well as observing. Face-to-face (or telephone) conversation with the purpose of exploring issues or topics in detail. Uses a pre-set list of questions or topics but is not restricted to these. Interview undertaken in a less structured fashion, with the purpose of getting a long story from the interviewee (typically a life story or the story of how a condition has unfolded over time). The interviewer uses only general prompts to tell me more. Method of group interview which explicitly includes and uses the group interactions to generate data.

Diagnosis

Screening

Prognosis

Causation

Psychometric studies Qualitative design methods involve exploration and interpretation via data generation; stronger on validity (closeness to the truth): preferred methods for poorly understood or relatively unexplored phenomena Documents Passive observation Participantobservation Semi-structured interview

Narrative interview

Focus Groups

Table 2 Quantitative, qualitative or mixed methods? Delegates at the 3rd East African Speech and Language Therapy Conference

from specific aspects of the intervention and not from, for example, the passage of time or receiving general attention from a nice therapist (the speech and language therapy equivalent of the placebo effect, which I think I once heard Professor Pam Enderby describe as random niceness!) You also need to have a think about whether the design of an intervention study was more about efficacy (Does it work in ideal conditions with carefully selected participants?) or effectiveness (Does it work under typical clinical conditions with a range of clients?) Logically, youre supposed to do efficacy first but, well, the path of research is sometimes more about serendipity than logic. However, if you cant work out what the researchers thought they were doing in this respect, be suspicious. There is an established pecking order within study designs in terms of weight and quality of evidence (see the Hierarchy of Evidence in table 3). However, how well the research was conducted (methodological quality) should influence how you rate it just as much as the level of evidence of the study

An intervention study needs to demonstrate that the outcomes resulted from specific aspects of the intervention and not from, for example, the passage of time or receiving general attention from a nice therapist
design. Common sense judgement is needed as well as hierarchies of study design when assessing a studys relative contribution to clinical evidence. If the study design is very wide of the mark, you may wish to conclude that it is not worth the effort of continuing with the appraisal. If on the other hand you wish to continue, you can select from your appraisal toolkit the framework that appears to fit best with the

study design. (Be aware that a mixed methods study may need you to use bits of more than one tool.)

EXPERT OPINION FRAMEWORK

I shall be presenting a range of critical appraisal tools over coming issues of the magazine. For this issue, here is an appraisal framework for expert opinion articles which are not based on systematic research or which go beyond the evidence base. It can be applied to a narrative or simple (non-systematic) review of intervention, management or decisionmaking for a specific clinical population, problem or issue, as well as for the sorts of articles that offer overviews of or advice on specific areas of clinical practice. You may download this tool as a document set up for you to print off and use as an individual or with colleagues in a journal club from www.speechmag.com/Members/CASLT. The original set of questions came from a very useful recent article in an American journal (Lass & Pannbacker, 2008):

SPEECH & LANGUAGE THERAPY IN PRACTICE AUTUMN 2010

19

COVER STORY: JOURNAL CLUB

Question 1: What is the experts training and experience? This information should help you decide how credible this author is. However, you also need to take into account what sort of advice is being offered. Factual information needs the support of scientific evidence, so the person offering this should be able to demonstrate an understanding of research methods. Speech and language therapy craft knowledge may be more credible coming from someone with a strong background in clinical practice and / or service development. Clients themselves are the ones to offer insights into what its like to receive our services, but the interpretation of clients experiences may be most credible coming from someone with a track record in qualitative research or patient engagement for health service evaluation. Here are some ideas to explore. Do you or any of your colleagues know this person? What is their background, their institutional affiliation and status? Where in the world are they, with what sort of healthcare or other institutional system? Are they a practitioner or an academic, or do they have a foot in both camps? You can try googling them! Question 2: Is the expert trained in Evidence-Based Practice? If authors are experienced in evidence based practice methods, this will influence how they have written the article. They may mention evidence based practice principles explicitly, or the structure and content of the article may reflect this knowledge. They will also be more likely to adopt a cautious, measured approach to expressing their recommendations or opinions. Question 3: Has the treatment or other practice been published in peer-reviewed journals? Question 4b: Does the expert consider counter-evidence? It is not acceptable for authors to present only the evidence that appears to support their interpretations or recommendations. Even if counter-evidence is not available, they need to show that they have considered alternative interpretations. Question 5: Does the expert provide up-to-date information? Dont be impressed with authors who appear to be stuck in a time warp, no matter how innovative they were in the past. Knowledge moves on, as do the contexts in which our clients are living and the constraints of service provision. Check the dates of the references at the end of the article and / or within the text. Are there references or authors missing that you might have expected to see? Question 6: Is the experts opinion consistent with known facts, previous research, and theory?

Check in the references, though you may need to follow this up with a literature search. Most academic journals claim to be peer-reviewed these days. It is tempting to think that an article must surely be credible if it has got through peer review into print. Unfortunately, it seems the process is less than watertight and reviewers may not share a clinicians main concerns. It might help to understand that there is a pecking order amongst academic journals. People whose careers hinge on the rankings of their publications (that is, any current or aspiring academic) want to get their articles published in journals that are highly ranked and cited by other academics in their field, such as the Journal of Speech, Language and Hearing Research. The journal Child Language Teaching and Therapy may not be so highly ranked by academics but I bet that it is considerably more widely read by speech and language therapy practitioners. Aspiring authors have to play the game by the rules of the academic journals; this has an impact on perceptions of scientific rigour and credibility, as well as on readability and impact on practitioners. Question 4a: Does the expert consider the quality of the quoted evidence? They should be considering methodological quality and not just its level in the evidence hierarchy (see table 3).

This is where you need your speech and language therapy craft knowledge to help you as well as any awareness of what research has been undertaken in this area. Of course, there may be no relevant research, and thats why you may be reviewing an expert opinion article in the first place! Question 7: Did the expert make full disclosure of any financial interests related to products such as materials and publications?

There are lots of definitions of evidence-based healthcare but Justice (2006) stresses the importance for speech and language therapy of four key types of knowledge: 1. Information from high quality research studies and systematic literature reviews. 2. Clinicians expertise (speech and language therapy craft knowledge) because of their theoretical knowledge and clinical experience. 3. Understanding of client preferences to allow us to work effectively with people and their communities. 4. Institutional norms and policies, which constrain the scope of clinical decision-making.

20

SPEECH & LANGUAGE THERAPY IN PRACTICE AUTUMN 2010

COVER STORY: JOURNAL CLUB

Its not that we dont want those who invent programmes to write about their own inventions; its just more credible when its done by those who have no commercial gains to make. A lot of the journals are now insisting that authors include a statement about any potential conflicts of interest within the text of the article. Question 8: Is the expert objective and free of bias? We have used this expert opinion framework successfully in several journal clubs. For example, here is an extract from one journal club summary, in which the central article was a research study of ParentChild Interaction Therapy (PCI) for children who stammer: We found it helpful to read about PCI in the context of a recent overview The aetiology and treatment of developmental stammering in childhood by David Ward from the University of Reading. We felt Wards overview was unbiased, objective, evidence-informed, presented from a UK standpoint and free of commercial interests. Wards overview (2008) is a succinct 5-page summary of the current state of the field. It outlines all the important issues around prevalence, diagnosis, prognosis, the developmental features of the disorder, theories of causation, intervention and speech and language therapy practice. He manages to benchmark the issues with the available evidence while not neglecting aspects of the theoretical and practice-based knowledge that contribute to speech and language therapists decision-making. You feel he understands both the academic and the clinical perspectives. The members of this journal club were community paediatric speech and language therapists who worked in a relatively sparsely populated area; they felt that they needed to keep on top of dysfluency work but never had enough clients to feel confident in their skills in this area. The Ward overview was ideal, really, because it provided a context for the focus on one particular therapy for children who stammer. Moreover, it gave them a common framework for beginning to evaluate the care pathways for children who stammer in their local population. In a different journal club, another author was less favourably rated in terms of her attention to the scientific knowledge base: This article was an easy read and the guidance relevant for our clinical practice It gave us some ideas for managing barriers and conflict in our work with parents. There was much that rang true in the family-centred approach described, and we felt that there were strong links to Care Aims and getting the patients story emphasising yet again the need for therapists to have strong emotional and negotiation skills. However, we were not convinced that the guidance had any real scientific basis. We did some googling of said author the group had no previous knowledge of her and according to the entry in her university website she had an impressive curriculum vitae, including experience of relevant clinical practice and teaching, funded research in the area, associate editorship of a peer-reviewed journal, PhD supervision and so on. However, her references were out of date, very few were research-based and she did not discuss the methodological quality or strength of the rather flimsy set of evidence she did present. The article had been published in a peer-reviewed journal with an international readership. So you really cannot make assumptions about credibility just from the SLTP fact the stuff has got into publication! Jennifer Reid is a consultant speech and language therapist with NHS Fife, email jenniferreid@nhs.net.

References

Read between the lines and look for sources of bias or unspoken influence, for example from institutional affiliations or personal beliefs. Question 9: Does the expert provide a comprehensive overview (both sides)?

Greenhalgh, T. (2006) How to read a paper: the basics of evidence-based medicine (3rd edn). Oxford: Blackwell Publishing Ltd. Justice, L. (2006) Evidence-based practice briefs: an introduction, EBP Briefs. Available at: http://www.speechandlanguage.com/ebp/ justice-intro.asp (Accessed: 19 July 2010). Lass, N.J. & Pannbacker, M. (2008) The application of evidence-based practice to nonspeech oral motor treatments, Language, Speech and Hearing Services in Schools 39(3), pp.408-421. Public Health Resource Unit (2006) Critical Appraisal Skills Programme (CASP). Oxford: PHRU. Available at: www.phru.nhs.uk/Pages/ PHD/CASP.htm (Accessed: 19 July, 2010). Taylor-Goh, S. (2005) RCSLT Clinical Guidelines. Milton Keynes: Speechmark. Ward, D. (2008) The aetiology and treatment of developmental stammering in childhood, Archives of Disease in Childhood 93(1), pp.68-71.

Is there another interpretation or approach that the expert appears to have overlooked or ignored? Question 10: Does the expert mainly cite his or her own work?

Download the expert opinion framework document from w w w. s p e e c h m a g . c o m / Members/CASLT. Use it yourself or with colleagues in a journal club, and let us know how you get on (email avrilnicoll@ speechmag.com). If you are a member of the Royal College of Speech & Language Therapists you can also see information to help you start and make the best of journal clubs at www.rcslt.org/ members/cpd/journal_clubs.
21

Check the references and the text. This can be a bit of a giveaway. Its okay to cite yourself (racks up citation points for your academic ratings!) so long as you cite other authorities as well.

SPEECH & LANGUAGE THERAPY IN PRACTICE AUTUMN 2010