Perinatal care

August 2016

Research

A diabetes-specific e-mental health tool: Development, acceptability and outcomes of a feasibility study

Volume 45, No.8, August 2016 Pages 600-605

Jan Orman

Janine Clarke

Erin Whittle

Cesar Anonuevo

Judith Proudfoot

Background

Psychological problems are frequently comorbid with diabetes and can be complex and time consuming to manage in general practice. They can also complicate diabetes management.

Objective/s

This pre-post, mixed-methods study examined whether a diabetes-specific online module, integrated into an existing e-mental health program, is acceptable to patients and helps to improve psychological wellbeing.

Methods

The study was conducted in three stages. First, qualitative information was obtained from consumers and practitioners to inform module content. Second, clinical content for the module was developed, including psychoeducational material and clinical content. In the final stage, acceptability and feasibility was evaluated in a group of patients with diabetes.

Results

Patients’ satisfaction with and acceptance of the module was high. Significant improvement in mental health and diabetes outcomes was also observed.

Discussion

Preliminary evidence suggests that the online diabetes module may be a useful psychological support for patients with diabetes. Further controlled investigation is warranted.

The health consequences of poorly managed diabetes are significant, and diabetes takes a toll on patients psychologically as well as physically. The scientific literature now distinguishes between general depressive symptoms, the rates of which are two to three times higher in patients with diabetes,1 and ‘diabetes-related distress’, which arises as a consequence of the significant negative impact of living with diabetes. Depressive symptoms and diabetes-related distress have been found to correlate with reduced self-care and poorer health outcomes in patients with diabetes;2–4 diabetes-related distress also contributes to poorer glycaemic control.5

Treatment for diabetes is typically delivered in the primary care setting.6 Current primary care guidelines advocate routine screening for psychological problems as a way of identifying those patients whose self-management efforts and diabetes outcomes might benefit from psychotherapeutic intervention.7 Nevertheless, rates of identification and, consequently, treatment of psychological distress in people with diabetes are low.8

A major contributor to under-treatment of depression and diabetes-related distress is the reluctance of many patients with these problems to accept a formal referral for face-to-face psychological care from their general practitioner (GP).9 This is likely to be due to a complex interplay of barriers to help-seeking, including economic, geographical and time constraints, and concerns about privacy, confidentiality and stigmatisation. Many of these issues can be addressed by the delivery of mental health interventions via the internet.

Effective online interventions exist for the management of mild-to-moderate depression, anxiety and stress. These tools are evidence-based, affordable and accessible, and can take some of the burden of care away from the GP.10 Meta-analyses indicate that online psychological programs for depression and anxiety are as effective as face-to-face therapy.11 Furthermore, data from a Dutch randomised controlled trial (RCT) suggests that internet-delivered psychotherapy programs may be effective in reducing depressive symptoms in patients with type 1 and type 2 diabetes.12

One such tool is Black Dog Institute’s myCompass program. Launched in 2012, myCompass consists of interactive self‑help learning modules based on cognitive behaviour therapy (CBT), interpersonal psychotherapy, positive psychology and problem-solving therapy. The program also contains a helpful smartphone-based symptom-monitoring facility, which can be adapted to tracking the specific needs of people with diabetes. myCompass is broadly available, free of charge and does not require a referral. The program has been shown to effectively treat symptoms of mild-to-moderate depression, anxiety and stress, and improve work and social functioning in a large community-based sample.13

The purpose of this study was to develop and pilot-test a diabetes-specific module for inclusion in the myCompass program, aimed specifically at reducing diabetes-related distress in people with the condition. In addition, we assessed the acceptability and utility of the module in a small group of patients with diabetes.

Methods

The project was conducted in three stages.

First stage

Qualitative data were collected from focus groups and interviews conducted with GPs and consumers to determine the perceived need for (GPs and consumers) and desirable content of (consumers) a diabetes-specific online module. Practitioners responded to an invitation placed in the Black Dog Institute’s GP newsletter. Consumers responded to an advertisement placed in online news distributed by Diabetes NSW. Focus groups and interviews used a semi-structured format and were audio-recorded and later transcribed. Participants received nominal financial recompense for their time and travel.

The data collected were then analysed using NVivo v10 qualitative data analysis software. Data derived from GPs and consumers were analysed separately using Braun and Clarke’s14 thematic analysis framework, which resulted in identification of the major themes and sub-themes in our participants’ responses.

Second stage

Clinical content for the online module was developed by a team of mental health researchers at the Black Dog Institute in Sydney, including the authors, using information from the first stage. Technical specifications and software development were carried out by the Black Dog Institute’s information technology systems team.

The module blends skills from traditional CBT and acceptance and commitment therapy (ACT), both of which are evidence-based treatments for mental health symptoms generally, and in diabetes patients, specifically.15,16 CBT prioritises the restructuring of unhelpful thoughts, whereas ACT uses mindfulness and defusion techniques to change the way people relate to problem thoughts and feelings. ACT also encourages commitment to meaningful activity to improve quality of life. The development team felt this approach was best able to balance the issues raised by patients with the inevitable difficulties they face as a consequence of their illness.

Consistent with existing myCompass modules, the new module titled Doing what really counts consists of three 10-minute sessions, each of which includes written information, interactive skill-building exercises, vignettes and home tasks (for in-between sessions). The module steps users through identification of important life domains and values, goal-setting techniques and identification and management of obstacles to goal achievement. Throughout the module, users are encouraged to consider diabetes and physical health as one of several important life domains and to pursue valued activities that create rich and fulfilling lives. Components of each session of the module are shown in Table 1.

Third stage

The new module was pilot-tested in a group of 35 patients with diabetes, who met the following eligibility criteria:

  • diagnosed with type 1 or type 2 diabetes by a GP or endocrinologist
  • experiencing at least mild depressive symptoms (as measured by the patient health questionnaire [PHQ-9])17
  • aged 18–75 years
  • an Australian resident
  • have access to the internet via both mobile phone and computer
  • have a valid email address.

People experiencing self-reported psychotic symptoms (as assessed by the psychosis screening questionnaire)18 or severe suicidal ideation (item 9 on the PHQ-9) were ineligible to participate.

The pilot study used a single-group, repeated measures design. Participants completed the standardised outcome measures summarised in Table 2 at baseline (before using the online module) and at four weeks. These measures assessed:

  • depressive and anxiety symptoms
  • diabetes-related distress
  • perceived confidence in managing diabetes and mental health
  • daily self-management of diabetes
  • self-reported glycaemic control
  • work and social functioning.

To evaluate change in the outcome measures, we applied the non-parametric Wilcoxon test. Missing data at post-intervention were handled using a last-observation-carried-forward (LOCF) method (ie missing post-intervention data were replaced with baseline scores on that measure).

At four weeks, participants also rated their experience with the module using a set of eight study-specific items. The items asked users to rate how strongly they agreed with statements about the module’s usability, content, flexibility and functionality using a scale of 1 (‘Strongly disagree’) to 5 (‘Strongly agree’).

Ethics approval for the study was obtained from the University of New South Wales Human Research Ethics Committee (reference number HC13375).

Results

Qualitative data

The GP sample consisted of 11 Sydney-based GPs (females, n = 9) with experience treating patients with diabetes. GPs participated in one of two focus groups led by the first author (JO). The consumer sample consisted of 18 patients with diabetes, who either participated in a focus group led by the second author (JC) or an interview with the third author (EW). The majority of consumer participants were women (n = 10; 55%) aged 35–80 years (mean = 63.76; standard deviation [SD] = 14.3) and had type 2 diabetes (n = 15; 83%).

GP participants indicated that the online program was likely to be helpful and potentially a useful alternative to antidepressant medication:

One of the other barriers is that [when] we do identify people [who] have mental health issues, we then put them on pills. Pretty much all of them have a side effect of making you eat more and gain weight. So maybe we need to address the issue that if you get someone early and put them down a different path like this, instead of onto pills, because putting them onto pills is going to make their diabetes worse. – Female GP

However, they did express concern about uptake (especially among older and less literate patients). They identified self-care issues as major concerns for patients with diabetes, in particular diet and weight loss. Related to this, they felt that insufficient diabetes education was a major obstacle to effective patient self-management, and specified clinical aspects of diabetes treatment (including management of complications and specific regimen components) as possible program targets.

But if you’re going to have a website with stuff about diabetes, you do have to ask the questions [about dietary knowledge] somewhere, and make sure that they have, you know, addressed that. – Male GP

Let’s try and get something constructive and real to the people whose weight is marshmallowing, to the people who use food to cope, instead of coping with exercise and supports. – Female GP

You know, Chinese-speaking patients and Arabic-speaking patients might not necessarily find that this is going to be useful. – Male GP

A majority of GPs agreed that mental health issues were barriers to adherence for some patients with diabetes, but they were less clear about how patients’ needs in this area could be best met.

By contrast, patient participants indicated that online self-help for diabetes was likely to be useful, but suggested that educational resources were currently in abundance. They considered psychosocial support as essential for self-management, and their responses suggested that key areas of struggle included worry about the future (especially diabetes complications), feelings of helplessness and of being ‘controlled’ by their diabetes, and frustration about the limitations imposed by their conditions on their daily lives.

I try to keep on working and then I – I can feel I’m getting irritable and then I’m not attaining anything and not achieving anything then I go in and take my sugar and – and invariably it’s always low. – Male consumer

If I walked into a restaurant, the job of ordering a meal from the menu that had enough carbohydrate in it to be all right was just – it set my stress levels up so that it became an unpleasant experience. – Male consumer

Ah – the side effects concern me. I do worry about the side effects and my eyes have been affected. It makes reading difficult and crosswords, and I’m a reader and a crossword fanatic. So that’s a little bit of, you know, that’s one of the stresses. – Female consumer

The biggest thing that I find difficult is probably the consequences of not managing my diabetes in the future. – Female consumer

So I never usually get angry with it but I think there’s other things in my life that can, you know, fall down a little because I’m concentrating on my diabetes. – Female consumer

Table 1. Session content for the ‘Doing what really counts’ module

Session 1

Session 2

Session 3

  • The life wheel – Identifying important life domains
  • Planning to make changes, Part 1
  • Identifying reasons and resources
  • Learning about values
  • Planning to make changes, Part 2
  • Learning to identify and deal with barriers and obstacles
  • Setting goals that match values
  • Committed action
  • Maintaining a life in balance
  • Recognising warning signs
  • Dealing with emergency situations
  • Rewarding goal achievement
  • Developing a well-being plan
Table 2. Outcome measures

Outcome

Standardised measure

Depressive symptoms

The Patient Health Questionnaire (PHQ-9)17

Generalised anxiety symptoms

Generalised Anxiety Disorder 7 item scale (GAD-7)19

Diabetes-related distress

Problem Areas in Diabetes Questionnaire (PAID)20

Daily functioning

Work and Social Adjustment Scale (WSAS)21

Confidence with diabetes management

Self-Efficacy for Diabetes Scale (SEDS)22

Confidence in mental health management

Mental Health Self-Efficacy Scale (MHSES)23

Diabetes self-management

Diabetes Self-Care Activities measure (DSCA)24

Self-reported glycemic control

Hyperglycaemia and Hypoglycaemia Scale (HHS)25

Quantitative data

Mental health and diabetes outcomes

The pilot sample (n = 35) had an average age of 47.6 years (range 26–74 years), and most participants were women (n = 21; 60%), employed at least part time (n = 18; 51%), with type 1 diabetes (n = 22; 63%).

At baseline, mean depression and anxiety scores for the total sample fell in the moderate (mean = 11.69; SD = 5.6) and mild (mean = 9.09; SD = 5.47) ranges, respectively. At the same time, participants reported moderate levels of diabetes-related distress (mean = 41.93, SD = 22.31), and work and social impairment (mean = 15.29; SD = 10.57). Self-reported adherence to daily self-care requirements was generally poor. On average, participants reported following dietary recommendations on half the days of the week and exercising two days a week. Adherence to blood sugar testing was slightly better – on average, participants tested five days a week.

Twenty-two participants (63%) provided data at four weeks post-intervention. There were no significant differences (all P values >0.05) in mental health symptoms (eg depression, anxiety and diabetes distress), daily functioning, diabetes self-care and diabetes symptoms between those who did and did not complete post-intervention assessments. However, dropouts were more likely to be patients who smoked.

At post-intervention, significant improvements (all P values <0.05) were found on the PHQ-9, PAID, GAD-7 and general diet subscales of the DSCA (assesses adherence to a healthy eating plan as opposed to consumption-specific food groups; Table 3).

Table 3. Observed means and standard deviations for the outcome measures

Outcome

Baseline
Mean (SD)
(n = 35)

Post
Mean (SD)
(n = 35)

Depressive symptoms (PHQ-9)

11.69 (5.61)

7.55 (5.53)

Generalised anxiety symptoms (GAD-7)

9.09 (5.47)

5.71 (4.20)

Diabetes-related distress (PAID)

41.93 (22.31)

33.35 (24.29)

Daily functioning (WSAS)

15.29 (10.57)

13.86 (11.87)

Confidence with diabetes management (SEDS)

5.92 (1.97)

6.82 (1.59)

Confidence in mental health management (MHSES)

33.94 (15.09)

39.36 (15.68)

Diabetes self-management (DSCA)

General diet

3.67 (2.31)

4.75 (2.06)

 Specific diet

3.6 (1.72)

3.93 (1.67)

 Exercise

2.19 (1.75)

2.89 (2.04)

 Blood glucose testing

5.40 (2.49)

5.98 (1.50)

Hyperglycaemic symptoms

2.77 (1.68)

2.5 (1.37)

Hypoglycaemic symptoms

2.26 (1.54)

2.14 (1.58)

Table 4. User satisfaction ratings

Question (n = 34)

Mean

Standard deviation

1.

The module was easy to use

4.05

0.88

2.

The information was easy to understand

4.14

0.96

3.

The information was relevant to me

3.90

0.89

4.

The module kept my interest and attention

3.60

1.11

5.

The module helped improve my stress and low mood

3.38

1.07

6.

The module taught me skills that will help me handle future problems

3.42

1.12

7.

I would be happy to use the module again

3.61

1.02

8.

I would recommend the module to other people with diabetes

3.70

0.90

Feasibility and acceptability

Participants logged on to use the module an average of 10.94 times, most frequently from their mobile phone. On average, participants used the module for 15.23 days. Overall, mean ratings for scale items were above the mid-point, suggesting that participants felt the module was easy to use, clear and relevant (Table 4). Importantly, participants generally agreed that they would recommend the module to other people with diabetes.

Discussion

The results of this feasibility study show that GPs and patients with diabetes consider the internet as a potentially valuable tool for improving access to psychosocial support for those with diabetes. Furthermore, our findings suggest that the new online module holds promise as an acceptable and effective tool for meeting the mental health needs of patients with the disease.

As the participants in this study acknowledged, it is not unusual for patients with diabetes to feel overwhelmed by the demands of their condition and, at the same time, experience a lack of appropriate emotional support. As a consequence, many patients reported feeling ‘defined’ and ‘controlled’ by their illness, with adverse consequences for their overall wellbeing and diabetes management.

GPs play a central role in diabetes management, yet chances to discuss emotional issues in depth within a consultation are often scarce. At the same time, many patients have difficulty and/or avoid discussing mental health issues with their treating doctors for fear that their problems might be misunderstood.26

The new diabetes-specific module on myCompass has the potential to break this impasse. Recommending this freely available and fully automated module to general practice patients may assist those who report feeling distressed and overwhelmed by their disease, including those experiencing difficulty managing their self-care demands and meeting their treatment targets.

Findings elsewhere show that adherence to online programs and treatment gains are maximised for patients whose GPs and/or other health professionals (eg practice nurses) take a supportive role in their online activities.27 ‘Support’ may include:

  • monitoring or supervising program use
  • technical and emotional support
  • motivation and encouragement
  • ‘setting the pace’ for a patients’ progression through a program (ie based on their symptom needs and motivation levels).28

Because improvements in mental health are linked with better diabetes self-care and health outcomes, it may be of benefit to patients and practitioners for GPs to adopt a supportive stance with patients who choose to engage with or are referred to the new diabetes module (and myCompass generally). Used in this way, the new module may be a valuable adjunct to primary healthcare management of diabetes. Reynolds et al28 have previously described the ways in which GPs can help motivate and encourage their patients to make use of online mental health programs such as myCompass.

The exploratory nature of this study and high dropout out rate mean that our conclusions are tentative. However, it is worth noting that the attrition rate documented here is consistent with other studies of online self-guided treatments,13 and that the symptom improvements observed occurred after only four weeks of module use. The uncontrolled design of the study also complicates the making of casual inferences (ie attributing symptom improvement to the intervention). Nevertheless, preliminary work such as this is a useful and necessary precursor to more resource-intensive and controlled investigations of online tools in the primary care setting. Our findings justify more rigorous examination of the new module in a larger and more representative group of patients with diabetes.

In the meantime, the new module has been freely and publically available via the myCompass program (www.mycompass.org.au) since August 2015. Program algorithms ensure that the module is recommended to all users who self-report at registration as having both diabetes and difficulties with low mood and/or stress. However, users are also able to freely select modules of their choice from the module list. This provides patients with the opportunity to engage with a range of helpful, evidence-based psychological tools. We are currently investigating ways to more closely monitor how people with diabetes are engaging with the new module, and with myCompass generally.

Conclusion

Integrating online self-help tools into patient care may assist in reducing the demands on busy GPs and other health professionals working with patients with diabetes, and add value to overall patient care for people with the disease. This is because psychologically healthy patients are better able to manage the cognitive and behavioural challenges of their condition and achieve their treatment targets, thereby reducing their reliance on health services.

Implications for general practice

  • GPs in the study appreciated the complexity of diabetes self-care and the limited access to appropriate supports/resources.
  • Patients in the study confirmed this, reporting a lack of supports/resources to assist them in coping with the distress of feeling controlled and defined by their illness.
  • Online mental health tools add to the repertoire of options available to patients and practitioners, and reduce the barriers to psychological support for patients struggling with their disorder.
  • The diabetes-specific myCompass module is able to help break the impasse by empowering patients through improved psychological wellbeing and improved self-care. Participants found myCompass to be acceptable and would recommend it to others.
  • GPs and diabetes educators can confidently recommend the myCompass online mental health tool as an adjunct to the care of patients with diabetes who are having difficulty meeting their treatment targets.

Authors

Jan Orman MBBS, MPsychMed, GP Services Consultant, Black Dog Institute, Randwick, NSW. j.orman@blackdog.org.au

Janine Clarke BA (Hons), MClin Psych, PhD, Research Psychologist, Black Dog Institute, Randwick, NSW

Erin Whittle BA, MPH, Research Officer, Black Dog Institute, Randwick, NSW

Cesar Anonuevo, Technical Support Analyst, Black Dog Institute, Randwick, NSW

Judith Proudfoot BEd (Hons), GradDipSpEd, MA (Psych), PhD, Associate Professor, School of Psychiatry, University of New South Wales, Sydney, NSW

Competing interests: None
Provenance and peer review: Not commissioned, externally peer reviewed.

Acknowledgements

The authors gratefully acknowledge the participants for their involvement in the research, Diabetes NSW for assisting with participant recruitment, and the RACGP for funding the project. Gratitude is also extended to Dr Margot Woods, Associate Professor Jane Overland, Mr Joel Tuccia, and Mr Michael Sluis for their input into development of the research protocol and constructive feedback on earlier versions of the module.

References

  1. Roy T, Lloyd CE. Epidemiology of depression and diabetes: A systematic review. J Affect Disord 2012;142:S8–21.
  2. Gonzales JS, Peyrot M, McCarl LA, et al. Depression and diabetes treatment non-adherence: A meta-analysis. Diabetes Care 2008;31(12):2398–403.
  3. Lin EH, Katon W, Von Korff M, et al. Relationship of depression and diabetes self-care, medication adherence and preventative care. Diabetes Care 2004;27(9):2154–60.
  4. Katon WJ. The comorbidity of diabetes mellitus and depression. Am J Med 2008;121(11 Suppl 2):S8–15.
  5. Reddy J, Wilhelm K, Campbell L. Putting PAID to diabetes-related distress: The potential utility of the Problem Areas in Diabetes (PAID) Scale in patients with diabetes. Psychosomatics 2013;54:44–51.
  6. The Royal Australian College of General Practitioners. General practice guidelines for the management of type 2 diabetes – 2014–15. East Melbourne, Vic: RACGP, 2014.
  7. Speight J. Managing diabetes and preventing complications. Med J Aust 2013;198:206–12.
  8. Katon WJ. The comorbidity of diabetes mellitus and depression. American J Med 2008;121(11):S8–15.
  9. Fleer J, Tovote K, Keers J, et al. Screening for depression and diabetes related distress in a diabetes outpatient clinic. Diabetic Med 2012;30(1):88–94.
  10. Andrews G, Cuijpers P, Craske MG, McEvoy P, Titov N. Computer therapy for the anxiety and depressive disorders is effective, acceptable and practical health care: A meta-analysis. PloS One 2010;5(10):e13196.
  11. Cuijpers P, Donker T, van Straten A, Andersson G. Is guided self-help as effective as face-to-face psychotherapy for depression and anxiety disorders? A systematic review and meta-analysis of comparative outcome studies. Psychol Med 2010;40(12):1943–57.
  12. Van Bastelaar KM, Pouwer FO, Cuijpers P, Riper H, Snoek FJ. Web based depression treatment for type 1 and type 2 diabetic patients: A randomised controlled trial. Diabetes Care 2011;34:320–25.
  13. Proudfoot J, Clarke J, Birch M, et al. Impact of a mobile phone and web program on symptom and functional outcomes for people with mild-to-moderate depression, anxiety and stress: A randomised controlled trial. BMC Psychiatry 2013;13:312.
  14. Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology 2006;3(2):77–101
  15. Lustman PJ, Clouse RE, Nix BD, et al. Sertraline for prevention of depression recurrence in diabetes mellitus: A randomized, double-blind, placebo-controlled trial. Arch Gen Psychiatry 2006;63(5):521–29.
  16. Gregg JA, Callaghan GM, Hayes SC, Glenn-Lawson JL. Improving diabetes self-management through acceptance, mindfulness, and values: A randomized controlled trial. J Consult Clin Psych 2007;75(2):336–43.
  17. Kroenke K, Spitzer RL. The PHQ-9: A new depression diagnostic and severity measure. Psychiat Ann 2002;32:509–15.
  18. Bebbington P, Nayani T. The psychosis screening questionnaire. Int J Methods Psychiatr Res 1995;5:11–19.
  19. Spitzer R, Kroenke K, Williams, J, Lowe, B. A brief measure for assessing generalised anxiety disorder. Arch Intern Med 2006;166(10):1092–97.
  20. Welch GW, Jacobson AM, Polonsky WH. The Problem Areas in Diabetes scale: An evaluation of its clinical utility. Diabetes Care 1997;20(5):760–66.
  21. Mundt JC, Marks IM, Shear MK, Greist JH. The Work and Social Adjustment Scale: A simple measure of impairment in functioning. Br J Psychiatry 2002;180:461–64.
  22. Lorig K, Stewart A, Ritter P, Gonzalez V, Laurent D, Lynch, J. Outcome measures for health education and other health care interventions. Thousand Oaks, CA: Sage Publications, 1996.
  23. Clarke J, Proudfoot J, Birch M-R, et al. Effects of mental health self-efficacy on outcomes of a mobile phone and web intervention for mild-to-moderate depression, anxiety and stress: Secondary analysis of a randomised controlled trial. BMC Psychiatry 2014;14:272.
  24. Toobert DJ, Hampson SE, Glasgow RE. The summary of diabetes self-care activities measure: Results from 7 studies and a revised scale. Diabetes Care 2000;23(7):943–50.
  25. Piette JD. A study of English and Spanish speakers with diabetes. Am J Prev Med 1999;17(2):138–41.
  26. Broom D, Whittaker A. Controlling diabetes, controlling diabetics: Moral language in the management of diabetes type 2. Soc Sci Med 2004;58(11):2371–82.
  27. Spek V, Cuijpers P, Nyklicek I, Riper H, Keyzer J, Pop V. Internet-based cognitive behaviour therapy for symptoms of depression and anxiety: A meta-analysis. Psychol Med 2007;37(3):319–28.
  28. Reynolds J, Griffiths KM, Cunningham JA, Bennett K, Bennett A. Clinical practice models for the use of e-mental health resources in primary health care by health professionals and peer workers: A conceptual framework. JMIR Ment Health 2015;2(1):e6. 

Correspondence afp@racgp.org.au

Yes     No

Declaration of competing interests *

Yes No

Additional Author (remove)

Yes No

    

 

 

 

 

Competing Interests: 

Your comment is being submitted, please wait

 

Download citation in RIS format (EndNote, Zotero, RefMan, RefWorks)

Download citation in BIBTEX format (RefMan)

Download citation in REFER format (EndNote, Zotero, RefMan, RefWorks)

For more information see Wikipedia: Comparison of reference management software
Advertisement loading...

Advertisement

Type

Research