Discussion, criticism and advice: content analysis of health blogs.
My PhD focused primarily on research blogs from ResearchBlogging.org (RB), a blog post aggregator covering peer-reviewed research. In this article, we (Prof. Judit Bar-Ilan, Prof. Mike Thelwall and myself) describe the study of blogger motivations using a content analysis approach. That is to say that we read 10% of the RB “Health” category between 2010-2012, or 391 messages in total. The result was an article titled “How is research blogged? A content analysis approach ” (green OA version here). At first we tried to analyze the content of each quote, but it turned out to be extremely difficult – although bloggers cite at least one source formally, they don’t always cite all of them, and even when they do. do, they don’t quote them. don’t always attribute every piece of information in the message to its source.
We could only roughly classify the citations into the categories “confirmatory”, “partly negative” and “negative”. A citation was “confirmatory” by default, unless the blogger specified otherwise, as we assumed that a decision to take a post without criticizing it was a way of approving its content. We generally did not view the mention of limitations of a study (eg a pilot study) as negative, as studies always have certain limitations. If they were only mentioned in passing, the quote was always considered “confirmatory”.
Referrals were “partially negative” in cases where the blogger rejected certain elements of the article but accepted others. For example:
But why look at the use of “social drugs” in depressed people? Are these individuals not less inclined to be sociable? And aren’t they likely to present with anhedonia (loss of interest in pleasure) according to the DSM IV criteria? (Le Neurocritique, 2011).
The Neurocritic pointed out that the article is flawed, but did not completely dismiss the results of the article.
A “negative” quote, however, meant that bloggers were only printing the article for the sake of burning it. Here is a quote from Rogue Medic, expressing his very honest opinion on an article:
Is survival to admission a clear benefit? Many more patients are dying in hospital, is that a net benefit? Survival until discharge is a real advantage. Imagine telling the family of a patient, “She passed away, but she lived long enough to rack up thousands of dollars in hospital bills.” We see this as a net benefit. A substitution endpoint is not a real advantage! ”
As fun as negative quotes were, they were rare, as you can see in Table 1. This is in line with traditional quote content / context analysis studies, which have shown, for example, 14% quotes negative (Moravcsik and Murugesan, 1975) and 5% partially negative (none were completely negative) (Chubin and Moitra, 1975).
Table 1. Classification of the extent of negation in the blog references sampled according to their publication context.
We have created a motivation classification into 10 categories, most with subcategories, for the most popular reasons we have coded. All the motivations could relate to questions concerning the general public, experts only, or both. For example, an ethical issue may be general (eg, should dogs’ tails be cut?) Or discipline-specific (eg, is it ethical to prescribe a placebo to patients?). A post can have more than one motivation (the average was 2.8), hence the following percentages exceeding 100%.
- Discussion, consideration and examination of a question (89.3%).
- Critical, finding fault with a research-related problem (29.9%).
- advice, recommending actions for readers (27.1%).
- Trigger, a direct stimulus for writing articles that was mentioned in the article (17.9%).
- Extensions, suggesting possibilities beyond the scope of the position (18.2%)
- Controversial, discuss the controversy; explain and / or discuss disagreements (eg, is BMI an appropriate index to use in weight management in older adults?) (11.5%)
- Self, bloggers added post content that was speci ﬁ cally related to them (8.2%).
- Data, providing data and facts with practical implications; basic factual information about the topic of the blog post (5.4%)
- Ethics, discussion of ethical questions (questions of morality) (4.6%).
- Other, grouping of positions classified as “other” in another category (7.9%).
I will focus on the categories that I find most interesting: Criticism, Advice and Controversy.
Bloggers wanted to disseminate useful information; the “Advice” category consisted of “providing practical advice and recommendations” (in 25.6% of messages) and “advocacy against certain treatments / lifestyles / interventions” (in 2.0% of messages). It seems bloggers would rather give advice than argue against a practice, lifestyle, or treatment.
About 30% included some sort of criticism. This may sound strange, since I just mentioned that most of the references in academic articles and in our study are con ﬁ ratory, but bloggers have not necessarily criticized the articles they have covered. Yoni Freedhoff of Weighty Matters, for example, used an article to challenge the president of the Canadian Beverage Association’s sugar claims.
“Mr. Sherwood is the president of the Canadian Beverage Association and this week he was tasked with championing sugar as a contributing factor in obesity, diabetes and heart disease. Let’s review Mr. Sherwood, but seen through the lens of Kelly Brownell and Kenneth Warner’s Big Tobacco Playbook – in which they put together a list of pieces the food industry co-opted from the early days of the fight to prove tobacco is harmful. (Freedhoff, 2012) ”
Only 16 articles (4.1%) criticized the conclusions or recommendations of an academic article. According to Cole and Cole (1971), researchers generally ignore and do not bother to cite work that they deem of poor quality. To be worthy of criticism, the work must be influential. Cole and Cole’s conclusions appear to be consistent with our findings.
The “Controversy” category could represent either internal or public disciplinary controversy, and was found in 11.5% of messages. One of the controversies discussed was that of food labeling:
“Recently there has been a push to make labeling mandatory in restaurants and fast food stores. In the United States, this is a huge initiative, passed as part of the 2010 Health Reform Bill ”(De Winter, 2012)
Of course, the study has many limitations; we only ranked motivations using content analysis, which means it’s our take on blogger motivations rather than necessarily their real motivations; Results are based on 10% of a specific category in the RB aggregator and may not work for other disciplines (some of the subcategories related specifically to health).
The tendency of bloggers to give advice could be particularly important, given that many members of the public search for health information online. As for the review, we found a fair amount of it, but most of it was not aimed at the articles covered in the post but at the media, common beliefs, etc. This could be because bloggers, like academic writers, tend to cover articles that they consider to be valid. We also suggest that this could be because bloggers mainly post under their own name and could be held responsible for criticism from colleagues, for example.
Blog posts are already a source of alternative metrics (altmetrics), but to be accepted as a valid source of impact in the scientific community, they (and other altmetrics) need to be better understood.
Chubin, DE, & Moitra, SD (1975). Analysis of the content of references: Complementary or alternative to counting citations? Social Studies of Science, 5 423-441.
Cole, J., & Cole, S. (1971). Measuring the quality of sociological research Problems in using the Science Citation Index. American sociologist, 6 (1), 23-29.
De Winter, G. (2012, June 22). Mutant influenza, study two [Blog Post]. Recovered from
Freedhoff, Y. (2012). Justin Sherwood, President of Refreshments Canada,
defends sugar [blog post]. Retrieved from http: //www.weightymatters.
ca / 2012/04 / justin-sherwood-refreshments-canada.html
Moravcsik, M., & Murugesan, P. (1975). Some results on the function and the quality of the quotes. Social Studies of Science, 5, 86-92.
Noonan, T. (2010). Amiodarone for Cardiac Arrest in ACLS 2010 – Part
III [blog post]. Retrieved from http://roguemedic.com/2010/12/
The Neurocritic (2011). Chocolate abuse and bipolar diagnoses [blog
post]. Retrieved from http://neurocritic.blogspot.co.il/2011/05/abusing-
Shema, H., Bar-Ilan, J., Thelwall, M. (2014). How is research blogged? A content analysis approach JASIST : 10.1002 / asi.23239