• Users Online: 2758
  • Home
  • Print this page
  • Email this page
Home About us Editorial board Ahead of print Current issue Search Archives Submit article Instructions Subscribe Contacts Login 


 
 Table of Contents  
ORIGINAL ARTICLE
Year : 2017  |  Volume : 11  |  Issue : 2  |  Page : 84-89

Validity of the periodontal disease surveillance self-report questionnaire in a Nigerian population


Department of Periodontology and Community Dentistry, Faculty of Dentistry, College of Medicine, University of Ibadan, Ibadan, Nigeria

Date of Submission10-Jan-2017
Date of Acceptance12-Apr-2017
Date of Web Publication15-Sep-2017

Correspondence Address:
Juliana Obontu Taiwo
Department of Periodontology and Community Dentistry, Faculty of Dentistry, College of Medicine, University of Ibadan, P.O. Box 5116, Ibadan
Nigeria
Login to access the Email id

Source of Support: None, Conflict of Interest: None


DOI: 10.4103/phmj.phmj_7_16

Rights and Permissions
  Abstract 

Background: Chronic periodontal disease is important in the aetiology of various life-threatening systemic diseases. Its surveillance using self-report questionnaire has become necessary due to the disadvantages associated with its surveillance using clinical examination methods.
Aim: The aim is to determine the validity of periodontal disease surveillance self-report questionnaire in a Nigerian population.
Methods: A representative sample of 250 consecutive adults, attending the dental clinic, University College Hospital, Ibadan, were interviewed and clinically examined using a periodontal disease self-report questionnaire, and community periodontal index of treatment need (CPITN), index as a gold standard. Tooth mobility and gingival recession were also assessed. The data were analysed using SPSS version 15 and OpenEpi module for performance evaluation of a diagnostic test. Validity was determined using receiver operating characteristic curve, the sum of sensitivity and specificity as well as the sum of positive and negative predictive values.
Results: The mean age of the study participants was 40.9 ± 16.3 years where the males accounted for 48.9%. It demonstrated low sensitivity, high specificity and low diagnostic accuracy with receiver operating characteristic curve of 0.6. The sum of sensitivity and specificity as well as the sum of positive and negative predictive values were below 120. Only 21.7% reported they had the periodontal disease as compared to 87% diagnosed cases using the clinically based examination method (CPITN index).
Conclusion: The periodontal disease self-report questionnaire demonstrated low validity in the study group as many people who had periodontal disease were missed. Development of a questionnaire tool adapted specifically for this environment is hereby recommended.

Keywords: Periodontal disease, self-report questionnaire, surveillance, validity


How to cite this article:
Taiwo JO, Ibiyemi O, Osuh ME, Lawal FB. Validity of the periodontal disease surveillance self-report questionnaire in a Nigerian population. Port Harcourt Med J 2017;11:84-9

How to cite this URL:
Taiwo JO, Ibiyemi O, Osuh ME, Lawal FB. Validity of the periodontal disease surveillance self-report questionnaire in a Nigerian population. Port Harcourt Med J [serial online] 2017 [cited 2024 Mar 29];11:84-9. Available from: https://www.phmj.org/text.asp?2017/11/2/84/214869


  Introduction Top


Chronic periodontal disease is the most common chronic oral disease in adults affecting more than 50% of the world's population.[1] It is highly prevalent in the adult Nigerian population, ranging from 50% in adults to 94.4% in the elderly population.[2],[3] Various reasons make it necessary to monitor trends of periodontal diseases. The life expectancy of man in many countries, especially the developed countries, has increased and the population is becoming older.[4] Furthermore, more people are retaining their teeth into old age.[4] Various studies demonstrated a steady increase of chronic periodontal disease with age.[5],[6],[7] Chronic periodontal disease is of great public health concern as it is highly implicated in the aetiology of many life-threatening systemic diseases such as cardiovascular disease, diabetes mellitus and has an effect on reproduction.[8] It is a preventable disease, but its surveillance in Nigeria is practically non-existent due to its prohibitive cost and its demand on workforce. Surveillance of periodontal disease using clinical examination methods generated a lot of discussions and reservations, and researchers are beginning to look at alternatives to these clinical examination methods, hence the introduction of periodontal disease surveillance self-report questionnaire.[9]

This questionnaire includes a multivariable model for population estimates using a combination of self-reported measures and risk factors.[9] It was tested in Australia and in the United States with tremendous success.[9] The use of a periodontal disease surveillance self-report questionnaire is a welcomed idea in a developing country like Nigeria where the prevalence of the chronic periodontal disease is high, and resources are scarce. However, the validity of an instrument is influenced by environmental factors such as culture and language.[10],[11] This questionnaire to the best of our knowledge has not been tested in Nigeria. Although its use is welcomed in this environment, it is pertinent to determine its validity before its use on a large scale.

Objective

The objective of this study was to determine the validity of periodontal disease surveillance self-report questionnaire in a Nigerian population.


  Methods Top


A comparative study was conducted at the Dental Centre of University College Hospital, Ibadan, on a representative sample of 250 consecutive newly registered, dentate, adult patients who had come for dental treatment and were willing to participate in the study. An individual must be 18 years and above and have at least twenty teeth to be included in the study. The sample size was determined using the WINPEPI computer programme for Epidemiologists software [12] for sample size determination for comparing proportions. The values used included prevalence (P) of 60%,[13] power of 95%, the level of significance of 5% and allowance for non-inclusion of 0%. This gave a required sample size of 230 (continuity corrected) in each category. However, the values were increased by 20 in each category to make up for missing data. The 250 patients were interviewed using the periodontal disease surveillance questionnaire as one category and were later examined using the community periodontal index of treatment need (CPITN) index as the second category.

Instruments and data collection

The instrument of measurement included a periodontal disease self-report surveillance questionnaire, designed by the periodontal disease surveillance workgroup, convened by Division of Oral Health, Center for Disease Control and American Academy of Periodontology.[9] Periodontal disease surveillance self-report questionnaire is a validated multivariable instrument for population estimates of the burden of periodontal disease, which uses a combination of self-reported measures and risk factors. It includes eight items which are predictive of periodontal disease.[9] The questionnaire was translated into the local language (Yoruba) and after data collection, it was back translated into English during data handling and management. It was pretested on 10 patients for content validity before it was administered to the study group both in English and Yoruba by a trained interviewer who was a resident doctor in community dentistry and could read, write and speak both languages.

The study group was assessed for periodontal disease using the surveillance self-report questionnaire, and the CPITN index,[14] a gold standard for periodontal disease assessment. CPITN is highly recognised and universally tested and accepted as a standard index for periodontal disease by the WHO and all countries of the world.[15] The principal features of the CPITN index include the use of a specially designed probe (CPITN probe) and the recording of the highest score for each index tooth. Index teeth missing were excluded from the study. Variables measured by the CPITN index were healthy periodontium (coded 0), bleeding gingiva (coded 1), calculus (coded 2), shallow pockets (coded 3), deep pockets (coded 4) and excluded sextant (coded X). CPITN index uses a partial recording system using teeth: 11, 16, 17, 26, 27, 31, 36, 37, 46 and 47 as index teeth. In various studies, clinical attachment loss measurement was used as a gold standard, which only measures periodontitis. However, the authors are of the opinion that the first part of this questionnaire reflects symptoms which are also consistent with gingivitis hence the decision to use CPITN index as a gold standard. This index reflects the full spectrum of periodontal disease.

Recession, as well as mobility, was assessed for each patient. A WHO clinical assessment form,[15] which was modified to include occupation, educational status, Grace and Smale [16]tooth mobility index as well as Miller's index [17] for the gingival recession were used to record the findings of the examination and demographic information.

For the purpose of this study, a periodontal disease patient was defined as any individual who has one or more teeth with gingivitis or periodontitis, defined by signs of inflammation of the gum and probing depth more than 3.5 mm, recession, defined by displacement of the marginal gingiva from the cementoenamel junction, in the apical direction as well as mobility as classified according to Grace and Smale.[16]

The social class of each patient was determined using the modified British Registrar Generals classification as adapted to the Nigerian system by Olojugba.[18] Both the periodontal disease self-report questionnaire and the clinical assessment form were numbered serially.

Patients were recruited at the reception after they have been registered and were interviewed as well as examined in periodontology clinic before they were dispatched to oral diagnoses/oral medicine specialist clinic where they were screened and dispatched to other specialist clinics.

The patients were first interviewed by a senior resident doctor in one cubicle using the periodontal disease surveillance self-report questionnaire and were later examined in another cubicle by another senior resident doctor. Each patient was examined on a dental chair under fluorescent light using a dental mirror and CPITN probe. The findings of the clinical examination were recorded by the same resident doctor, on the modified WHO clinical assessment form.

Data analysis

Collation and verification of data were done on a daily basis while computer data entry and analysis were carried out after the required number of patients had been seen, using the SPSS for Windows 15.0 (SPSS Inc., IL, USA) and OpenEpi [19] for performance evaluation of a diagnostic test. The responses to the questions were dichotomised. A Yes response was considered positive while No and I don't know, negative. Each yes response carried 1 mark while a negative response was scored zero.

Question two which had sub items was dichotomised; good, very good, excellent were considered positive responses while fair, poor, don't know were regarded as negative responses. Each sub-item in question two was considered individually, and 1 mark each was given to the positive responses while a negative response was scored zero. Questions 7 and 8 were also dichotomised; use of dental floss and use of mouth rinse once and more were considered positive while 0 times was considered negative. All responses categorised as negative were scored zero. Results of the CPITN was dichotomised into no periodontal disease which refers to code 0 and periodontal disease present which included codes 1, 2, 3, 4. Overall, these questions culminated in an 11-point scale, indicating possible scores of 0–11. Scores were then converted to levels 1–11 and cut-off points between two subsequent levels were selected. A receiver operating characteristic (ROC) curve was plotted [Figure 1] and sensitivity, specificity, predictive values, diagnostic accuracy, were calculated. The sum of specificity and sensitivity as well as positive and negative predictive values were determined [Table 1]. The value of the sum of sensitivity and specificity or the sum of positive and negative predictive values was considered indicative of poor validity when <120.
Figure 1: Receiver operating characteristic curve for validity of periodontal disease surveillance self-report questionnaire

Click here to view
Table 1: Diagnostic power (accuracy) of the periodontal disease surveillance self-report questionnaire at different cut off levels

Click here to view


The study protocol (UI/EC/11/0061) was reviewed and approved by the University of Ibadan and University College Hospital Ethics Committee.


  Results Top


The sample included 250 patients mainly of Yoruba ethnic group 205 (82%), 16 (6.4%) from Hausa, 8 (3.2%) from Ibo and 21 (8.4%) from other ethnic groups. The participants included 48.9% (122) males and 59.1% (128) females with age range of 18–80 years and mean/(standard deviation) age of 40.9 ± 16.3 years. [Table 2] illustrates the social class distribution of the study group. A high percentage of participants were from social class 4 (lower class). Amongst the participants 61.2% had tertiary, 14.5% postsecondary, 14.6% secondary and 6.2% primary education while 3.2% were illiterate.
Table 2: Social class distribution of the participants

Click here to view


The prevalence of periodontal disease as determined by the CPITN index was high among the study group (87%). Furthermore, 19 (7.6%) had recession of the gingivae while 110 (44%) had one or more mobile teeth. However, using the periodontal disease surveillance self-report questionnaire for question 1, 68% reported that they had no periodontal disease while 10.3% did not know. Only 21.7% reported that they had periodontal disease.

The sensitivity, specificity, positive and negative predictive values, diagnostic accuracy are illustrated in [Table 1]. The ROC curve which demonstrates the ability of the self-report questionnaire to predict periodontal disease in the study group is depicted in [Figure 1]. In general, the questionnaire had a low sensitivity which decreased further when cut-off level increased and high specificity which increased with an increase in the cut-off level. The diagnostic power also decreased with an increase in cut-off level [Table 1]. The area under the curve for the ROC was 0.6 [Figure 1]


  Discussion Top


Public health surveillance is defined as a continuous systematic collection, analysis, interpretation and dissemination of data on a health-related event, the purpose for which include the following: Use in the control of diseases to reduce morbidity and mortality, improving health, measure trends of diseases, guide planning, implementation and evaluation of health programmes, identify population at risk and other various public health activities.[20] In medicine, public health surveillance of diseases using self-report measures has been in use for decades.[21] A very good example is the Behavioral Risk Factor Surveillance System, which has demonstrated much success and support.[22]

However, surveillance of oral diseases using self-report measures is a recent development.[21] The burden of periodontal disease and its repercussions on general health has long been recognised and it is pertinent to monitor its trend and include its control and management in public health policies. Hence, its surveillance is imperative. Attempts have been made at surveillance of this disease using clinically based examination methods by the United States government during the National Health and Nutrition Examination Survey (NHANES) in 2003/2004.[9] However, clinically based examination methods used for periodontal disease surveillance in previous studies are not only cost intensive and have high demand on workforce but are tedious.[9],[23]

The majority of the participants were of the Yoruba ethnic group and 96.8% of the participants were literate enabling good understanding and easy administration of the questionnaire. Furthermore, many of them (62%) were from social class I to III. Periodontal disease increases with increase in age.[7] The mean age of the study group was 40.9 years and established periodontal disease is quite common around this age group.

Slightly more females than male participated in the study; however, this did not affect the results in any way because the two instruments were tested on the same participants.

The periodontal disease surveillance self-report questionnaire was less able to detect chronic periodontal disease when compared with CPITN. Using the self-report questionnaire many people who had periodontal disease were missed because only 21.7% reported they had the periodontal disease as compared to 87% diagnosed cases using the clinically based examination method (CPITN index). The mobility of a tooth is an indication of advanced chronic periodontal disease reflecting the loss of supporting tissues of the tooth. A high percentage 44% of the participants had at least one or more mobile teeth as diagnosed using the Grace and Smale index.[16] This was not reflected in the findings of the questionnaire. The inability of the periodontal disease surveillance self-report questionnaire to have good validity was demonstrated in this study with high specificity, low sensitivity and moderately low diagnostic accuracy confirming previous studies.[24],[25]

At low cut-off levels, it had higher sensitivity and lower specificity. At all levels the sum of sensitivity and specificity as well as the sum of positive and negative predictive values were <120, indicating that it had poor validity in the study group.

ROC curve value of 0.6 is just above average and falls short of the useful values of 0.7–0.9 and the excellent value of ≥0.9.[26]

NHANES having observed the shortcomings of the periodontal disease surveillance self-report questionnaire, recommended and supported funding of the full periodontal clinical examination in NHANES 2009–2012. This was with the view that it would generate accurate estimates of the prevalence of periodontitis in the US adult population and provide a superior dataset for research and surveillance. The database would also generate necessary coefficients for the US self-report questionnaire for use in subsets of the total US population.[27] Various studies have evaluated the accuracy of the self-report questionnaire on the periodontal disease with various outcomes. While in some studies, there was no question with satisfactory validity,[24] a few studies recorded useful levels of validity.[28],[29]

The failure of the above periodontal disease surveillance self-report questionnaire to have high diagnostic accuracy in the participants may be because 50% of the questions were completely not consistent with the lifestyle of the Nigerian population. Furthermore, periodontal disease is quiescent. It is quite often asymptomatic until it has advanced appreciably, so many people with this disease are not aware of their periodontal status. The majority of the Nigerian population have very little knowledge of the periodontal disease. Materials like dental floss and mouth rinses are not part of the oral cleaning habits of the average Nigerian. Likewise, regular dental clinic attendance is not a part of the lifestyle in this environment.[30] Questions based on these will likely underestimate the prevalence of disease in this population.

In Nigeria, surveillance of oral diseases is not a government planned activity and surveillance of oral diseases on a national basis is non-existent, although there have been sporadic surveys carried out by individuals. This may be because much importance is not accorded oral health as is accorded general health. Furthermore, the high cost of oral health surveillance and its demand on workforce could be a discouraging factor. The need to change to a more favourable method of surveillance for oral diseases is long overdue, especially periodontal disease which is highly prevalent in developing countries including Nigeria. The periodontal disease surveillance self-report questionnaire will, therefore, be a great asset to developing countries like Nigeria which have to manage health within finite resources.

Even though periodontal disease surveillance using self-report questionnaire is a laudable idea, health authorities in this country will need to design a questionnaire specifically for this environment, taking language, culture and lifestyle into consideration.

There is great need to intensify oral health education efforts in Nigeria. Oral health education should be directed at educating people to identify signs and symptoms of periodontal disease. Self-examination of the mouth using an ordinary mirror to identify simple signs such as swollen gums, gingival recession, calculus, teeth which do not look right, loose teeth, should be encouraged.


  Conclusion Top


Periodontal disease self-report questionnaire had low validity and was not a good predictor of the actual periodontal status of the study group.

Fifty percent of the questions were completely not consistent with the lifestyle of the Nigerian population and this might have contributed to the low validity.

Recommendation

The development of a periodontal disease surveillance questionnaire tool adapted specifically for Nigerian socio-cultural environment is highly recommended.

Limitations

The limitation of this study is based on the fact that some of the questions in the self-reported questionnaire were not consistent with the lifestyle of the Nigerian population.

Acknowledgement

The statistician, Dr. A E Orimadegun is well appreciated for his professional contribution towards the success of this study. The University College Hospital is also appreciated for providing the facility utilised in conducting the study.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.

 
  References Top

1.
Loesche WJ, Grossman NS. Periodontal disease as a specific, albeit chronic, infection: Diagnosis and treatment. Clin Microbiol Rev 2001;14:727-52.  Back to cited text no. 1
[PUBMED]    
2.
Macgregor ID. Radiographic survey of periodontal disease in 264 adolescent schoolboys in Lagos, Nigeria. Community Dent Oral Epidemiol 1980;8:56-60.  Back to cited text no. 2
[PUBMED]    
3.
Taiwo JO, Jeboda SO, Motayo TO, Obiechina AE. Periodontal health of the elderly people in South East local government area in Ibadan, Nigeria. Afr J Med Med Sci 2004;33:285-91.  Back to cited text no. 3
[PUBMED]    
4.
Locker D. Ageing of a modern population. Ageing and oral health. An Introduction to Behavioural Science and Dentistry. 1st ed. London: Tavistock, Routledge, 1989; 60-1.  Back to cited text no. 4
    
5.
Brown LJ, Brunelle JA, Kingman A. Periodontal status in the United States, 1988-1991: Prevalence, extent, and demographic variation. J Dent Res 1996;75:672-83.  Back to cited text no. 5
[PUBMED]    
6.
Dye BA, Tan S, Smith V, Lewis BG, Barker LK, Thornton-Evans G, et al. Trends in oral health status: United States, 1988-1994 and 1999-2004. Vital Health Stat 11 2007;248:1-92.  Back to cited text no. 6
    
7.
Taiwo JO, Omokhodion F. Pattern of tooth loss in an elderly population from Ibadan, Nigeria. Gerodontology 2006;23:117-22.  Back to cited text no. 7
[PUBMED]    
8.
Seymour GJ, Ford PJ, Cullinan MP, Leishman S, Yamazaki K. Relationship between periodontal infections and systemic disease. Clin Microbiol Infect 2007;13(Suppl 4):3-10.  Back to cited text no. 8
[PUBMED]    
9.
Eke PI, Genco RJ. CDC Periodontal Disease Surveillance Project: Background, objectives, and progress report. J Periodontol 2007;78(7 Suppl):1366-71.  Back to cited text no. 9
    
10.
Hägglin C, Berggren U, Hakeberg M, Edvardsson A, Eriksson M. Evaluation of a Swedish version of the OHIP-14 among patients in general and specialist dental care. Swed Dent J 2007;31:91-101.  Back to cited text no. 10
    
11.
Montero J, López JF, Vicente MP, Galindo MP, Albaladejo A, Bravo M. Comparative validity of the OIDP and OHIP-14 in describing the impact of oral health on quality of life in a cross-sectional study performed in Spanish adults. Med Oral Patol Oral Cir Bucal 2011;16:e816-21.  Back to cited text no. 11
    
12.
Abrahamson JH. WINPEPI: Computer programmes for epidemiologist and their teaching potential. Epidemiol Prospect Innov 2011;8:1.  Back to cited text no. 12
    
13.
Akpata ES. Oral health in Nigeria. Int Dent J 2004;54(6 Suppl):361-6.  Back to cited text no. 13
    
14.
Ainamo J, Barmes D, Beagrie G, Cutress T, Martin J, Sardo-Infirri J. Development of the World Health Organization (WHO) community periodontal index of treatment needs (CPITN). Int Dent J 1982;32:281-91.  Back to cited text no. 14
[PUBMED]    
15.
World Health Organization (WHO). Oral Health Assessment Document 5434 ORH (41, 86). Geneva: World Health Organization, 1986.  Back to cited text no. 15
    
16.
Grace AM, Smales FC. System for diagnosis, selection, control and treatment planning in general practice. New Malden: Quintessence, 1989; 144.  Back to cited text no. 16
    
17.
Miller PD Jr. A classification of marginal tissue recession. Int J Periodontics Restorative Dent 1985;5:8-13.  Back to cited text no. 17
    
18.
Olojugba OO, Lennon MA. Dental caries experience in 5- and 12-year-old school children in Ondo State, Nigeria in 1977 and 1983. Community Dent Health 1987;4:129-35.  Back to cited text no. 18
[PUBMED]    
19.
Dean AG, Sullivan KM, Soe MM. OpenEpi: Open Source Epidemiologic Statistics for Public Health, Ver. 2.3. Available from: http://www. OpenEpi.com. [Last updated on 2009 May 20; Last accessed on 2013 Jul 02].  Back to cited text no. 19
    
20.
Buehler JW. Surveillance. In: Rothman KJ, Greenland S, editors. Modern Epidemiology. 2nd ed.. Philadelphia: Lippincott-Raven, 1998; 435-57.  Back to cited text no. 20
    
21.
Dye BA, Thornton-Evans G. A brief history of national surveillance efforts for periodontal disease in the United States. J Periodontol 2007;78(7 Suppl):1373-9.  Back to cited text no. 21
    
22.
Illinois Department of Public Health. Illinois Behavioral Risk Factor Surveillance System. BRFSS Home Strata Map, State and Strata Data, County Data. Available from: http://www.cdc.gov/brfss. [Last cited on 2017 Jun 7].  Back to cited text no. 22
    
23.
Tomar SL. Public health perspectives on surveillance for periodontal diseases. J Periodontol 2007;78(7 Suppl):1380-6.  Back to cited text no. 23
    
24.
Dietrich T, Stosch U, Dietrich D, Schamberger D, Bernimoulin JP, Joshipura K. The accuracy of individual self-reported items to determine periodontal disease history. Eur J Oral Sci 2005;113:135-40.  Back to cited text no. 24
[PUBMED]    
25.
Vered Y, Sgan-Cohen HD. Self-perceived and clinically diagnosed dental and periodontal health status among young adults and their implications for epidemiological surveys. BMC Oral Health 2003;3:3.  Back to cited text no. 25
[PUBMED]    
26.
Swets JA. Measuring the accuracy of diagnostic systems. Science 1988;240:1285-93.  Back to cited text no. 26
[PUBMED]    
27.
Eke PI, Thornton-Evans G, Dye B, Genco R. Advances in surveillance of periodontitis: The Centers for Disease Control and Prevention periodontal disease surveillance project. J Periodontol 2012;83:1337-42.  Back to cited text no. 27
[PUBMED]    
28.
Slade GD. Interim analysis of validity of periodontitis screening questions in the Australian population. J Periodontol 2007;78(7 Suppl):1463-70.  Back to cited text no. 28
    
29.
Yamamoto T, Koyama R, Tamaki N, Maruyama T, Tomofuji T, Ekuni D, et al. Validity of a questionnaire for periodontitis screening of Japanese employees. J Occup Health 2009;51:137-43.  Back to cited text no. 29
[PUBMED]    
30.
Taiwo JO, Noah M. Pattern of dental clinic attendance of registered diabetic patients in Ibadan. JMBR 2006;5:36-43.  Back to cited text no. 30
    


    Figures

  [Figure 1]
 
 
    Tables

  [Table 1], [Table 2]


This article has been cited by
1 Self-Reported Periodontal Disease and its Association with Dental Anxiety in Lagos, Nigeria
Modupeore Ekua Sorunke, Olubunmi O. Onigbinde, Afolabi Oyapero, Olurotimi A. Coker
Pesquisa Brasileira em Odontopediatria e Clínica Integrada. 2022; 22
[Pubmed] | [DOI]



 

Top
 
 
  Search
 
Similar in PUBMED
   Search Pubmed for
   Search in Google Scholar for
 Related articles
Access Statistics
Email Alert *
Add to My List *
* Registration required (free)

 
  In this article
Abstract
Introduction
Methods
Results
Discussion
Conclusion
References
Article Figures
Article Tables

 Article Access Statistics
    Viewed5725    
    Printed396    
    Emailed0    
    PDF Downloaded363    
    Comments [Add]    
    Cited by others 1    

Recommend this journal


[TAG2]
[TAG3]
[TAG4]