Evaluating texts is the toughest part of critical reading. It’s where we get really down and dirty with the piece and pay attention to all the little details. Our aim is to identify the paper’s strengths and weaknesses, figure out how it applies to our own research and understand how it fits within the field. Unfortunately, even if a paper is peer-reviewed, it doesn’t mean it’s perfect, or that it’s relevant to our purposes. Part of engaging in the university means being able to determine the context, reliability and relevance of published research (or any kind of text!)

But in order to be able to evaluate a text effectively, we must be able to engage our critical thinking skills at a deep level. Critical thinking is not just a skill that’s vital for reading, but for writing academically, participating actively in class, and in everyday life.

You have probably seen ‘critical thinking’ in your rubrics and assignment criteria, and tutors and lecturers love talking about how important it is. But do you feel like you actually know? It’s another one of those concepts that lecturers often assume students have, but it isn’t necessarily so. So before we dive into how we evaluate texts, let’s explore what critical thinking really is.

Critical Thinking

It’s important not to simply take things at face value. We want to bring a healthy level of scepticism to ideas and think about whether we have good reason to trust what we see. Critical thinking gives us the tools to be able to do this.

So, critical thinking is about learning to evaluate, analyse, categorise and select information. It is a complex process that takes some time to develop and employs a wide range of skills.

It includes:

Identifying other people’s positions, arguments and conclusions
Weighing up opposing arguments and evidence fairly
Being able to read between the lines, seeing behind surfaces, and identifying false or unfair assumptions
Recognising techniques used to make certain positions more appealing than others, such as false logic and persuasive devices
Reflecting on issues in a structured way, bringing logic and insight to bear
Drawing conclusions about whether arguments are valid and justifiable, based on good evidence and sensible assumptions
Synthesising information: drawing together your judgements of the evidence, synthesising these to form your own new position
Presenting a point of view in a structured, clear, a well-reasoned way that convinces others

Critical thinking is associated with reasoning, or our ability to employ rational thinking. We need to be able to consider what reasons we have to believe something is true, be able to critically evaluate our own beliefs and actions, and be able to understand and evaluate the reasoning of others.

Basically: why do you believe what you believe? Why does someone else believe what they believe?

It’s also important to remember that being critical doesn’t necessarily mean being negative. It’s easy to conflate the two because we tend to think of criticism as identifying flaws or areas for improvement. Critical evaluation means identifying both strengths and weaknesses in information.

We employ critical thinking in our academic writing, in class discussions, and, of course, in how we read and understand texts. Being able to critically evaluate texts, whether they are set readings or found in our own research, is incredibly important.

So how do we do it?

Critical Reading

When reading critically, we apply our critical thinking skills in order to understand and evaluate a text. In order to do this properly, we must first understand a text’s context – why was it created, by whom, where does it fit within the discipline? Therefore, it’s really important to engage in the preparatory reading strategies I outlined in my previous posts such as planning your reading, skimming and scanning.

But, while these methods are useful for finding information, they provide only a superficial understanding of the text. Once we’ve used them to locate what we need, we must read much more closely. For this reason, it’s important to understand how a typical research paper is structured. So, if you haven’t already, have a look at my post on Understanding Academic Texts.

Introduction

In evaluating and critiquing a paper, we need to get an idea of its overall aim, the research question and its significance. We should be able to find this in the introduction.

Does the author clearly state the research question?
Is the significance of the research justified?
Does the author provide sufficient background to understand the significance of the question?
Have they clearly outlined the structure of the paper?

In some papers, the strength of the research may rely on the author making an argument or applying a theory to an object of inquiry rather than, or in addition to collecting data. Therefore, in order to evaluate all kinds of papers, we must be able to identify the author’s argument and theoretical position.

Identifying Argument

To help us identify an argument, it is useful to first outline what an argument is not.

An argument is not the statement of a fact, eg: ‘The average normal body temperature is 37°C.’

An argument is not an unsubstantiated opinion or belief. ‘University education should be free.’

An argument is not a recommendation: ‘To prevent infections, hand-washing stations should be placed in every hospital waiting room.’

However, we can have an argument when these elements come together. Our most basic argument is made up of an opinion + fact. An argument can also be made up of an opinion + fact + recommendation.

Eg: ‘Due to low rainfall and dry soil conditions (facts), this will be a high-risk fire season (opinion). Therefore, existing fire services should be supplemented (recommendation).

Identifying the argument of the text is all about figuring out what the author’s position is. This means figuring out what the author wants you to do, think, accept or believe. In the example above, the author wants us to accept that conditions are dangerous and it would be a good idea to properly prepare for the fire season. Identifying an argument also includes looking for the reasons that are given to support this position. These are generally the key ideas that will make up the paper, and they should be logically structured and evidence based.

Sometimes the author will be quite clear about their argument and may include statements such as:

‘This paper argues that …’,
‘It is hypothesised that …’
‘I suggest …’.

Sometimes, however, the argument may be presented as a statement without an introductory phrase that gives it away. Look for statements that come at the end of introductory paragraphs and seem to contain an idea that needs qualifying or supporting. For example, ‘A good essay must include a strong thesis statement.’ This type of statement is not necessarily a fact, and others may disagree with it. Therefore, we’d expect that the author is going to support this statement with evidence. We can therefore conclude that this statement and the supporting reasons make up the author’s argument.

Some critical questions to ask yourself about the argument include:

Does the author clearly state their argument or thesis?
Does the author provide premises that will support their argument?
Does the author acknowledge assumptions or bias in their argument? If not, can you identify bias or assumptions?
Are the author’s claims reasonable?
Is there an indication that the author will support their claim with evidence?

If the paper does not follow a typical report format, and is instead made up of a essay-style argument, you will need to continue to critique how that argument is made. These types of papers are more common in the humanities and social sciences and are often written with a strong theoretical position.

Some critical questions to consider for an argumentative paper include:

Is the paper logically structured?
Is each supporting premise (key idea) supported with credible and viable evidence?
Does the author use problematic argument techniques such as slippery slopes or logical fallacies?
Are examples contextualised and clearly linked to the claim or idea?
Are any key issues overlooked?
Does the author make any assumptions or generalisations?
Does each key idea clearly and logically lead to the author’s conclusion?

If you haven’t already, it might be worth checking out my previous post about writing an argument, as understanding the elements of an argument will help you to be able to identify it.

Theoretical Position

In some articles, an argument may rely on a theoretical position, which basically means the author is approaching a particular topic from an already established ‘school’ of ideas.

A theory is a set of ideas that attempts to explain or predict phenomena. You might be familiar with some famous theories such as the theory of evolution, or the theory of relativity. Theories can also be collections or ‘schools’ of thought, whereby existing knowledge is categorised and systematised, for example Feminist Theory or Marxist Theory. In this case, it may not be one particular phenomenon that academics are setting out to explain, but rather the theory provides a specific lens through which to examine phenomena or analyse something. While theories are based on reasoning and evidence, they have not been conclusively proven, and in some disciplines, can never really be proven. For this reason, theories can often seem quite abstract and are difficult to pin down.

Most research is conducted with a theoretical framework. This is the structure that supports and organises whatever theory the author is using. They are quite complex, and so I won’t confuse you with all the details, but basically, this is where the author will define and contextualise the theoretical position that they have chosen. In the best research, the theoretical position is made quite clear by the author, usually in the introduction. They may state the name of the theory, or the key authors that they have drawn on, and will usually include a literature review that will give you an indication of the body of knowledge upon which they are building.

Identifying the author’s theoretical position will help us to:

Understand the position or ‘lens’ through which they are approaching the topic
Identify what prior research or ideas they might be drawing on
Sort the information in the literature review
Understand how this research builds on prior knowledge and fits with the bigger picture

So, once you’ve identified the theoretical position, ask yourself some critical questions:

What is the author’s position?
How does it fit within the field?
What literature are they drawing on? Have they overlooked anything?
Have they made their theoretical position clear?
Is the theoretical position justified and does it seem appropriate?

It’s important to know that not all research will have a theoretical position or use a theoretical framework, and not all theories are arguments in and of themselves.

Literature Review

The literature review may be included in the introduction, or as a separate section of the paper. Either way, when evaluating a paper, we need to pay attention to how the author grounds their own research. As well as helping us understand the author’s theoretical position, the literature also reveals what work has come before them, and what gaps this research attempts to fill.

Some critical questions to ask of the literature review include:

Is the literature review logically organised?
Is the literature recent, and if not, is it for good reason (eg. a key text in the field)?
Is the literature relevant (eg. is it from a comparable context)
Does the author offer a fair analysis of the literature?
Does the author fairly represent opposing views?
Does the literature review reveal a gap?

Methodology

As well as identifying the author’s theoretical position, it is important to understand the methodology they are using to conduct the research. The methodology tells us how the research was designed and conducted and ensures that data is collected systematically.

First, we need to identify the research design. Common research design methodologies include:

Quantitative – research that is measurable and collects numerical and statistical data. This can include surveys, experiments, or randomised control studies.
Qualitative – research that descriptive and collects data based on words, feelings, experiences and other non-quantifiable elements. This can include interviews, observations, focus groups or case studies.
Mixed Methods – research that includes a combination of both quantitative and qualitative methods.

The research design ensures that evidence is collected in a logical, unambiguous and reliable way. Within each broad category – qualitative, quantitative and mixed methods – there are various kinds of designs and their application depends on the type of research the author is conducting. Different disciplines have different kinds of designs, and you will become more familiar with these as you progress in your degree.

Once you’ve identified the research design, consider the specifics of the data collection:

What is the sample size or population?
Who are the participants?
What is the setting?
What are the limitations?

Part of the data collection might include the use of an instrument (or apparatus), which is a tool that researchers use to measure the concepts under study. These can include things like standardised tests, survey templates, or scales. The author/s might use an instrument that has been used in previous research, or they might develop a new one. Either way, the author should justify why they have chosen this particular instrument and why it is the most reliable for this study.

Some critical questions you might ask of the research methodology include:

Is the research design appropriate for the research question?
Is the data collection method appropriate for the research question?
Is the use of instrument described and justified?
Do the research methods and instrument seem reliable and valid?
Is the sample size appropriate?
Are ethical considerations addressed appropriately?
Are the research limitations discussed?

Data Analysis and Discussion

Knowing the research design and methodology will then help us understand the results. In the data analysis section, the author will summarise the results with enough detail so that the reader can understand how the conclusions were made.

Some critical questions to ask of the data analysis include:

Are the results clear and unambiguous?
Are the results relevant to the research question?
Did the author explain how they conducted the data analysis?
Were all the data taken into account? If not, did the author explain why?
Are graphs or tables presented in a clear and relevant way?

Finally, we have the discussion. In this section, the author should clearly link the results from the previous section back to the research question or hypothesis and draw out the significance and implications. Basically, why is this important? If the paper has a theoretical position, it should also make us aware of how the results help develop or support the theory.

Some critical questions to ask of the discussion include:

Do the results clearly demonstrate the question or aim?
Does the author make clear links between the results, research question and theoretical framework?
Are any recommendations drawn clearly from the data?
Are the findings linked back to the literature review?
Does the author discuss the strengths and limitations of the paper?

Phew – there’s a lot of information in this post! It is a big one because, frankly, evaluating a text is hard work! It requires a lot of in-depth critical reading and it means asking yourself lots of questions. But it is something that you’ll get better at the more you do it and, hopefully, eventually you start automatically asking yourself such questions and identifying flaws not just in research papers, but in all kinds of mediums.

Good luck! And if you want more posts like this, don’t forget to subscribe 🙂

References

Cottrell, S. (2017). Critical thinking skills: Effective analysis, argument and reflection. Macmillan International Higher Education.

Coughlan, M., Cronin, P., & Ryan, F. (2007). Step-by-step guide to critiquing research. Part 1: quantitative research. British journal of nursing, 16(11), 658-663.

Student Learning centre (2012). Critiquing Research Articles. Retrieved from https://students.flinders.edu.au/content/dam/student/slc/critiquing-research-articles.pdf

Yunggorendi study skills

Reading Effectively #5 – Evaluating Texts