ContentslistsavailableatInternationalJournalofInformationManagementjou - PDF document

1 ContentslistsavailableatInternationalJou
ContentslistsavailableatInternationalJournalofInformationManagementjournalhomepage:www.elsevier.com/locate/ijinfomgt Assessingwebsitesquality:AsystematicliteraturereviewbytextandassociationrulesminingRimRekik,IlhemKallel,JorgeCasillas,AdelM.Alimi Correspondingauthor.E-mailaddresses:(R.Rekik),(I.Kallel),(J.Casillas),(A.M.Alimi). dependencieswiththecategoryofthewebsiteandtoknowthemostreliableones.Moreover,anothermotivationistosearchforapproachesthatusesoftcomputingtechniquestoresolvetheproblem.1.2.ResearchmethodInthiswork,weexercisetheSLRapproachaccordingtoKitchenhamandChartersguidelines(KitchenhamandCharters,2007)supervisedbyFig.1.Theprocessiscomposedofthreephases:planning,conductingandreportingthereview.TheaimoftherstphaseplanningthereviewistonetheobjectiveoftheSLRandaclearreviewprotocol.Itspeciesthemainraisedresearchquestions,theadoptedsearchstrategyandasetofestablishedinclusionandexclusioncriteriatoselectapublication.Thesecondphaseconductingthereviewisforexecutingtheprotocol.Inthethirdphase,wereporttheobtainedresults.1.3.ContentoverviewTheremainingpartofthepaperisorganizedasfollows:Sectionintroducesadetaileddescriptionofthesystematicliteraturereviewprotocol.Sectionpresentsconductingthereviewwhenapplyingthedevelopedprotocol.InSection,areportoftheextracteddataandrelatedworksfromthepreviousreviewisdiscussed.Finally,somendingsaredrawninSectionfordevelopingfutureworkandthemainconclusionsaregiveninSection2.PlanningthereviewInplanningthereviewphase,theobjectiveoftheSLRandthere-searchquestionsaredened.Moreover,aclearreviewprotocolisde-veloped.Itconsistsofdeningasearchprocessstrategyandthein-clusion/exclusioncriteriaconsideredintheresearch.2.1.ObjectiveandresearchquestionsTheobjectiveoftheSLRconsistsofexploringdierentaspectsoftheassessmentofwebsitesqualityexistingintheliterature.Accordingtotheobjectiveandmotivationofthereviewasdescribedinsection1.1,weformulateasetofsixresearchquestionsasfollows:Whatarethepurposesoftherecentresearchfromtheas-sessmentofwebsitesquality?Whatarethemostcommoncategoriesofwebsitesconsideredfortheassessment?Whatarethecriteriathatcharacterizeawebsite?Whataretheirsemanticgroups?Canweextractassociationrulesbetweenthecriteria?WhichWhatarethefrequentcriteriaconsideredinanassessmentDuetothesubjectivityandimprecisionofthatMCDMpro-blem,aretherestudiesthatperformedtheassessmentusingsoftcom-putingorhybridmethods?Ifso,inwhichphaseoftheevaluationandwhatisthemotivationofapplyingsuchintelligentmethods?2.2.SearchprocessItisimportanttofollowasearchstrategyinordertoensureaconvincingreviewconductedinphase2(seeFig.1).Infact,aphaseisneededforexploringscienticpublicationsfromrelatedjournalsandconferencesinrelevantelectroni

2 csourcessuchasElsevier'sScopus,sScienceD
csourcessuchasElsevier'sScopus,sScienceDirect,IEEExplore,ACMDigitalLibrary,SpringerLinkorGoogleScholar.Itisnecessarytodenesomekeyconceptsasse-lectionwords.Indeed,weconsidersomewordssuchas,etc.Acombinationofthesetermsshouldbemadetoenlargethescopeofsearchingforbetter2.3.InclusionandexclusioncriteriaSincewecannotincludeallcollectedpapers,weintroducesomeinclusionandexclusioncriteria.Inordertoselectthemostrelevantones,wedeterminethecriteriathatspecifywhetherastudywillbeincludedorexcluded.Therstinclusioncriterionbasedontermswhichappearedinthetitles,abstractsandkeywordsinstudiesbybrowsingthecomputersciencediscipline;anidenticationofrelevantoneswasestablished.However,paperspublishedbefore2009andnon-Englishwrittenstudieshavetobeexcluded.Inaddition,weexcludesomesub-disciplinessuchasWebservicesnotrelatedexactlytothetopicofFig.2presentsabooleanexpressionqueryperformedusingtheScopusdatabase.Afterobtainingalargesetofpapers,asteptoeliminateshortones(upto4pages)andtomakealteronthesetofpapersisrequired Fig.1.SummaryoftheSLRprocessphasesaccordingto(KitchenhamandCharters, Fig.2.TheperformedsearchqueryinScopus. Fig.3.ExampleofaLucenesearchexpression.R.Rekiketal. usingLucenesearchenginedescribedinthenextsection(seeformoredetails).Fig.3presentsoneexampleofaLucenesearch3.ConductingthereviewFig.4depictsaowchartmethodologycontainingallstepsfollowedinthisreview.3.1.StudyselectionFollowingtheabovesearchstrategy,weconductedapaperselectionprocessillustratedbyFig.5.Indeed,4321documentresultsarere-turnedintherstsearchonScopusdatabasebyapplyingtheinitialinclusion/exclusioncriteriabasedontermswhichappearedinarticletitle,abstractandkeywords.Then,instep2welimitedthesearchtothesubjectareaComputerScience,tostudieswritteninEnglishandpublishedbetween2009and2015.Itreturns1041articles.Step3alsoincludesarenementbyexcludingkeywordsthatdonotrespondtothetopicsuchasWebservices;thenumberwasreducedto758.Step4isbasedonaccessiblePDFdocumentsPDF(578).Instep5;weparseallthesedocumentstakingintoconsiderationtheotherinclusion/exclu-sioncriteriawithoutshortpapersandwithapplyingalterusingLucenebysearchingthefrequencyofdenedtermsinarticles.Finally;532papersisthetotalnumberofdistinctpapersincludedintheSLR.3.2.StudyqualityassessmentTheevaluationconcernseachstudyinordertobesurethatitsa-estheobjectiveoftheresearchandallinclusion/exclusioncriteria.Astepofskimmingandreadingpartsfromthearticlesisnecessary.Inthisstep,wealsorefertogures,tables,appendixesandotherindicesforanalyzingthem. Fig.4.Flowchartofourreviewmethodology. Fig.5.Ourpaperselectionprocessbyapplyingthesearchstrategylteringcriteria.R.Rekiketal. 3.3.DataextractionThedataextractionprocesswaselaboratedinordertopublishanddiss

3 eminateresultsinthereportingphaseoftheSL
eminateresultsinthereportingphaseoftheSLR.3.3.1.TextminingforpapersincludedintheSLRtobuildadatasetTextminingisappliedtothe532papersincludedintheSLR.Itisthesolutiontoparseahugesetofpaperstoextractneededinformationandbuildthedatasetofcriteria.Luceneisanopen-sourceJavaAPIandapowerfultextsearchen-ginetool().Itspurposeistosearchfordocumentswithspeciedterms.So,thefollowedstepsare:1.AllpapersaredownloadedinPDFformat.2.ApacheLucenedoesnothavetheabilitytoextracttextfromPDFles.AllofthemareparsedbyApacheTika()whichisalibraryinJavathatextractstextfromditypes.Then,alibraryofdocumentsisbuiltinTextformat.3.Lucenelooksforstringonly,itanalyzesTextlesandsocreatesanindexfromthem.Then,itenablesustoqueryagainsttheindicestoretrievethematchingresults.IndexingandsearchingstepsarepresentedinFig.6andtheprocessisdetailedinAppendixB4.TobrowsethecontentsofLuceneindex,weusedtheLukeGUItoolwritteninJava(https://github.com/DmitryKey/luke/releases).Ithelpsinrunningasearchexpressionbycriterionandpresentingresults.Foreachcriterion,Lukeisusedtosearchthedocumentsthathaveatermandbuildthetransactionforthedatasetwiththesetoftermsincludedineachdocument.Then,todecideifadocumenthasaterm,insteadofconsideringjusttheuseofthewordonce,weusedthescoringformulainEq.(availableinLucene): scoreqdcoordqdqueryNormqtftindidfttgetBoostnormtd(,)(,)·()·(()·()·.()·(,) ) tinq tf(tind)isterm'sfrequencytindocumentd.Documentsthathavemoreoccurrencesofagiventermreceiveahigherscore.isInverseDocumentFrequency.   =++ i dftnumDocs()1log isthetotalnumberofdocumentsandisthenumberofdocumentswiththeterm.isacoordinationfactorusedtorewarddocumentsthatcontainahigherpercentageofthequeryterms.isaquerynormalizationfactorusedtonormalizeaquerysoscores;Itplaystheroleofaweightingfactor.isasearchtimeboostoftermtinthequeryq,oragivenhighscoretosomeparticularthing.Thehighertheboostfactoris,themorerelevantthetermwillbe,andthereforethehigherthecorrespondingdocumentsscore.isacombinationofthelengthfactorwiththeindexingAnexampleofascoringexplanationforadocumentconsideringcriterionasatermthroughaprintoutfromLukeGUIisgiveninFig.7.ThenalscoreinDoc.Id414is0.2921whichistheproductoftf,idfandAnotherexampleofascoringexplanationforadocumentwhenconsideringmultiplequeriescombinedusingBooleanoperatorsascontentORinformationisgiveninFig.8xedathresholdtoconsiderifthetermisrelevantenoughinthedocumentastoconsiderthatthepaperusesthiscriterion.3.3.2.CollectionofdataSomeexamplesofcollecteddataareshowninTable1accordingtothepurposeofassessment,categoryandreferencesfromstudiesin-cludedintheSLR.AccordingtoTable1,wenoticedierentcategoriesareimpliedinerentpurposesofassessment.Allextractedcategoriesarepresentedinth

4 eresultsdisseminationofRQ2. Fig.7.Scoree
eresultsdisseminationofRQ2. Fig.7.ScoreexplanationofterminDoc.Id414. Fig.8.ScoreexplanationofcontentORinformationtermsinDoc.Id446. Fig.6.TheindexingandthesearchingstepswithLuceneadaptedfrom(Hatcher,&Gospodnetic,2010R.Rekiketal. 4.ReportingthereviewnalphaseoftheSLRisaresultreportoftheresearchques-tions.Indeed,thendingsarebasedonstudiesretrievedbythecon-ductedreviewphaseandbasedontheresearchquestionssetoutpre-Fig.9recallsandgraphicallypresentsthenumberofselectedpublicationsperyear.4.1.RQ1:whatistheaimoftherecentresearchfromtheassessmentofwebsitesquality?Inthefollowing,wefocusonidentifyingtheaimoftheselectedworkstoassessawebsite.AspresentedinTable1,examplesofstudiesareregroupedaccordingtotheirpurpose;theyprovidemodels,resultsorrecommendationstoreaders(i.e.administrators,users,customers,developers,enterprises)intermsofimprovementofawebsitequality,witheventuallyitsclassicationasgoodorbad.Moreover,theaimofotherstudiesistoevaluatetheelectronicservicetoattractcustomersandenhancethenumberoftransactions.Forexample,theE-commercecategoryisassessedforelectronicservicequality(Hsu,Hung,&Tang,2012)thenaninterdependenceperspectivebetweenmultiplecriteriaandsub-criteriaisstudied.Anotherpurposeistosearchaboutthequalityanalysisofonlineinformation.Forexample,inhealthcarewebsitespatientsaresearchingforcredibleinformationtoselectphysicians(Yang,Guo,Wu,&Ju,2015).Inaddition,someresearchesimplementasetofcriteriafortheevaluationprocess.Amultitudeofassessingpurposesexistsaccordingtousersneeds;weareherefocusingonthewellknownones.4.2.RQ2:whatarethemostcommoncategoriesofwebsitesconsideredfortheassessment?Inordertoextracttheassessedcategoryfromstudies,textminingisappliedfortheextractionandanalysisofthisinformation.Fig.10onshowthedistributionofwebsitescategoriesaccordingtothese-lectedstudies.Italsoindicatesascopeofinterestinevaluatingcate-goriessuchasAnytype,SocialMedia(i.e.socialnetworking,socialbookmarking,forums,micro-blogging,wikis,etc.),E-commerce,Health,Educational,E-government,Service(i.e.travel,hotel,tourism,airlines,etc.),InstitutionalandCorporate. Fig.9.Thenumberofselectedpublicationsperyear. Fig.10.Thedistributionofwebsitescategoriesconsideredforthe Table1SomeexamplesofdataextractionfromselectedstudiesintheSLR.PurposeofassessmentCategoryReferencesProvidemodelsorresultsandrecommendationsforadministratorsand/orusers/customersand/ordevelopers/enterprisesforimprovementofthewebsitequalityand/orclassifyingitasgoodorInstitutional(RekikandKallel,2011Educational(SilambannanandSrinath,2013Health(EstevesandLopez,2010Esteban,Porcel,Moral-Muñoz,&Herrera-Viedma,2014Evaluatetheelectronicservicetoattractandincreasethecustomersformakingtransactionsoraccessin

5 gtoneededinformation.E-commerce(Hsuetal.
gtoneededinformation.E-commerce(Hsuetal.,2012;Lin,2011E-government(Alanezi,Mahmood,&Basri,2012Implementasetofcriteriatoinsuretheassessment.Anytype(Rekik,Kallel,&Alimi,2014E-commerce(Hernández,Jiménez,&Martín,2009Hsuetal.,2012Searchaboutthequalityofonlineinformationandanalyzeit.Health(Yangetal.,2015Anytype(Kotenko,Chechulin,Shorov,&Komashinsky,2014Socialmedia(Vosecky,Leung,&Ng,2012R.Rekiketal. Table2Initialcriteriaandattributes.CriteriaorattributesNumberofstudiesExamplesofreferencesAccessibility92(DeLimaetal.,2009;EidaroosandAlkraiji,2015Accuracy/Correctness/Trustworthiness62(Leite,Gonçalves,Teixeira,&Rocha,2014Ozmen-ErtekinandOzbay,2012Adequacy9(Schäfer,Kummer,&Günther,2011Advertising27(Vatankhahetal.,2014Aesthetics/Visualappeal54(PengnateandAntonenko,2013Animation14(Vatankhahetal.,2014Attractiveness23(Leiteetal.,2014Audio/Sound28(Cha,2014Authority19(RafeandMonfaredzadeh,2012Availability31(Chenetal.,2013Clarity/Simplicity22(SilambannanandSrinath,2013Color42(RafeandMonfaredzadeh,2012Communication39(RafeandMonfaredzadeh,2012Compatibility/Interoperability17(ChatzopoulosandEconomides,2009Completeness30(RafeandMonfaredzadeh,2012Comprehensiveness13(Leiteetal.,2014Conciseness11(Leiteetal.,2014Consistency/Coherence30(EidaroosandAlkraiji,2015;Leiteetal.,2014Content/Information320(Voseckyetal.,2012Credibility/Believability44(Leiteetal.,2014Currency26(Leiteetal.,2014Customersupport/Supportability14(Luo,Ba,&Zhang,2012Design/Layout/Organization/Structure111(EidaroosandAlkraiji,2015;PengnateandAntonenko,2013Easeofuse/User-friendliness/Easeofoperation/Operability60(Cha,2014ectiveness30(Vatankhahetal.,2014ciency55(FogliandGuida,2013;Vatankhahetal.,2014Feedback56(Mavlanova,Benbunan-Fich,&Koufaris,2012Form14(ChatzopoulosandEconomides,2009;EidaroosandAlkraiji,2015Functionality34(FogliandGuida,2013Image/Graphic65(RafeandMonfaredzadeh,2012Interactivity72(SilambannanandSrinath,2013RafeandMonfaredzadeh,2012FogliandGuida,2013EidaroosandAlkraiji,2015;Voseckyetal.,2012LiuandWang,2013FogliandGuida,2013EidaroosandAlkraiji,2015;Voseckyetal.,2012Leiteetal.,2014ChatzopoulosandEconomides,2009Mavlanovaetal.,2012RafeandMonfaredzadeh,2012RafeandMonfaredzadeh,2012Chenetal.,2013;Lin,2010Luoetal.,2012;MurakataandMatsuo,2011EidaroosandAlkraiji,2015Muñoz-Leiva,Luque-Martínez,&Sánchez-Fernández,2010Luoetal.,2012;Mavlanovaetal.,2012Purchaseintention/PurchaseMavlanovaetal.,2012Vatankhahetal.,2014Vatankhahetal.,2014Demeester,Nguyen,Trieschnigg,Develder,&Hiemstra,2012FogliandGuida,2013;PengnateandAntonenko,2013RafeandMonfaredzadeh,2012MurakataandMatsuo,2011Chen,Tzeng,&Chang,2015Fuertes-Callén,Cuellar-Fernández,&Pelayo-Velázquez,2014RafeandMonfaredzade

6 h,2012Chenetal.,2013;Luoetal.,2012Demees
h,2012Chenetal.,2013;Luoetal.,2012Demeesteretal.,2012;Voseckyetal.,2012EidaroosandAlkraiji,2015;Muñoz-Leivaetal.,2010Chenetal.,2015;Muñoz-Leivaetal.,2010LiuandWang,2013Speed(ofloadingand/ordownload)Cha,2014;Chenetal.,2015SilambannanandSrinath,2013RafeandMonfaredzadeh,2012Ozmen-ErtekinandOzbay,2012Leiteetal.,2014Mavlanovaetal.,2012Muñoz-Leivaetal.,2010;PengnateandAntonenko,2013FogliandGuida,2013Uniqueness/ValueaddedOzmen-ErtekinandOzbay,2012;SilambannanandSrinath,2013RafeandMonfaredzadeh,2012FogliandGuida,2013Cha,2014Leiteetal.,2014Cha,2014Fuertes-Callénetal.,2014R.Rekiketal. Indeed,thisdistributionshowsthemostimportantcategoriesin-cludedintheevaluation.ThisreectsthatSocialmediaandE-com-mercecategorieshavecompetitiveenvironmentscomparedtootheronesandshouldprovideagoodshowcasetosatisfyusersneeds.Thebiggestpartisalsodevotedtoanytypecategorythatreectsthein-terestofuserswithusablewebsitesingeneral.4.3.RQ3:whatarethecriteriathatcharacterizeawebsite?whataretheirsemanticgroups?Anyprocessofassessingwebsitequalitygoesthroughthestepofspecifyingcertaincriteria.Itisadecisionproblemcharacterizedbymul-tiplecriteriathatcanbesolvedbyMCDMmethods.Somecriteriapresentaictwithothersastheconfusionofmeaning.Trustworthiness,forexample,canmeanthereliabilityortheaccuracyofawebsiteortheloyaltyofcustomers.Somefactorsaresemanticallysimilarandarere-groupedtogethersuchasContent/Information,Design/Layout/Organization/Structure,Satisfaction/Fulllment,Easeofuse/User-friend-liness/Easeofoperation/Operability,credibility/believability,Aesthetics/Visualappeal,Personalization/Customization,Understandability/Comprehension,Playfulness/Enjoyment/Entertainment,etc.Inthedataextractionstep,wecollectedintitaldierentcriteriaandattributesfromstudiesincludedintheSLRasdescribedinTable2Thenumberofinitialcriteriaandattributesishigh.Inordertoreducethem,thenextcontributionistostudytherelationsbetweenthemandtosearchfortheirdependencieswiththecategoryofthewebsiteusingassociationrulesmining.4.4.RQ4:canweextractassociationrulesbetweenthecriteria?WhichByapplyingatextminingprocessforsudiesincludedintheSLRpresentedinthedataextractionstep,adatasetofcriteriaisim-plementedfortheassessment.Extractingasetofassociationrulestodiscoverinterdependenciesbetweencriteriaandtheirimportanceisthemajorpurposeofthiscontribution.BeforeansweringRQ4,itisneces-sarytoelucidatesomenotionsrelatedtoassociationrulessuchasdence,support,liftandtheApriorialgorithmappliedtogeneratethestrongassociationrules(Orriols-PuigandCasillas,2010presentssomefundamentalnotions.4.4.1.ApriorialgorithmApriorialgorithmwasintroducedmorethantwentyyearsagobyAgrawalandSrikant,1994).Itisconsideredasthemostpowerfulas-sociationr

7 uleminer.Itperformsintwostepsillustrated
uleminer.ItperformsintwostepsillustratedbyFig.11DiscoverallfrequentitemsetsScanthedatasettondfrequentitemswithanoccurrencewhichisgreaterthanorequaltothesupportthresholddenedbytheuser.Itiscalledtheminimumsupportintheliterature.Generatecandidatesfromfrequentitemsandthenndthefrequentitemsets.GenerateasetofstrongassociationrulesfromfrequentitemsetsWhenapplyingthisalgorithm,astudyisconsideredasanitemset;itrepresentsthesetofcriteriafortheassessmentofwebsitesquality.Consequently,anitemisconsideredasacriterion.Allcollectedcriteriafromdierentstudiesarestoredinadataset.4.4.2.GenerationofassociationrulesTotally,2054isthenumberofgeneratedassociationruleswithminimumsupport5%andcondencemorethan25%.Alteristhenappliedbyxingathresholdtotheliftofarulegreaterthanorequalto2.15.Thenumberisreducedto1405.Theserulesarereliableandstrongenoughastheliftsuperiorto1.Finally,weanalyzethemtosearchforthemostusefulassociationrules.Theanalysisgenerates632rules.ApresentationofrelationsbetweenthemisgivenbyanetworkgraphasshowninFig.12usingGephisoftware().Thenodescharacterizetheantecedentortheconsequentofrule.Astepofpartitioningthenodesintodierentgroupsisnecessary.Theltermodularityclass(Blondel,Guillaume,Lambiotte,&Lefebvre,)isappliedtoidentifyrelationsbetweennodes;itdeterminessetsofverticesstronglyconnectedbetweenthem.Thepartitioninthisgraphhas6classeswithdierentcolours.Anotherstepistoweighthenodesinordertoresizethemaccordingtotheirdegreeofconnectionwitheachothercalculatedbasedonthenumberofincomingandout-goinglinks.Arrowsarecolouredwiththesamecolourofsourcenodeandtheirthicknessespresenttheirweightswhicharetheliftsofrules.Amongthepurposesoftheresearchistoreducethemassivenumberofcriteria.Consequently,weextractausefulsetofassociationrulesaccordingtothewebsitecategory.Criteriathatinuencethecategoryaredetermined.InTable3,anexampleofextractedassociationrulesusefulforE-commercecategory.Theyaresortedbyrulelift.Thehigheritsvaluethemoretheruleisinteresting.Fromthistable,adetailedexplanationofinterestingassociationresultscanbeasfollows:{E-commerce,Product}{Purchaseintention}:IfthewebsiteisE-commerceandtheevaluationconcernsaproduct,thenitishighlyprobableitinuencesthepurchaseintentionofcustomers.Itsitemsetsupportis13withrulecondence36.1%andlift10.69.{Content,Product}{Purchaseintention}:Thecontentdescribingaproductinuencesthepurchaseintentionofcustomers.Thesup-portoftheruleis11,condence31.4%andlift9.31. Fig.11.TheApriorialgorithmdiagram(AgrawalandSrikant,1994R.Rekiketal. {Transaction}:Thepriceinuencesmakingtransactionsornot.Therulehas8asitemsetsupport,29.6%condenceand6.32lift.{Transaction}:Thewebsiteshouldbere-sponsivetoallowmakingtransactionseasily.Thesupportoftherulei

8 s9,condence29%andlift6.19.{Design,Securi
s9,condence29%andlift6.19.{Design,Security}{Transaction}:Awelldesignedwebsiteandsecurityareveryimportantfactorsformakingatransactionac-cordingtotherulelift5.50.Thecorrespondingsupportandcon-denceare8and25.8%.{Security}:IfthewebsiteisE-commerce,thenevaluatingsecurityisneeded.Therulehas20studiesthatsupportitwith30.3%condenceand2.15lift.FordeeperanalysisoftherelationswiththeE-commercecategory,adependencywheelgraph()isusedasdepictedbyFig.13.Itshowsnotonlyanimportantdependenceofcontentandproductwithpurchaseintentionbutalsoitexposesthefactorsinre-lationwithmakingatransactionwhichareprice,satisfactionofusers,theservicesprovided,security,designandresponsivenessofthewebsite.TheE-commercecategorydependsonothercriteriasuchasaes-theticsandtrust.Criteriarelationsforothercategoriesarealsodetermined.Figs.14representdependencywheelgraphsrespectivelytoEdu-cational,E-government,HealthandSocialmediacategories.ThegraphinFig.14showsahighinterdependencebetweentheevaluationofcontentforEducationalwebsitesandlearnability.Thesiteshouldprovidemeansinordertoassistlearningtousers.Moreover, Fig.12.Networkgraphtovisualizetherelationsbetweenthetopuseful632associationrules.R.Rekiketal. theassessmentofthiscategoryisbasedonevaluatinginteractivity,designandaudio.Awelldesignedcoursewebsiteisrelatedtoeasynavigationwhichleadstointeractivity.Inordertosupportinteractivecontentforusers,communicationishighlyneeded.Audioiscorrelatedtothequalityofvideoandimage.Thefeedbackoflearnersisalsoes-sentialforcontinuousimprovement.TheE-governmentcategorycanbeassessedaccordingtothefol-lowingcriteriaaccessibility,privacy,servicefurnishedtocitizens,ef-ciency,security,satisfactionandeaseofuse.ThecirclegraphinFig.15indicatesanimportantlinkbetweencontent,privacyandpersonalization.Thewebsiteshouldbecustomizedforuserstoensurepersonalization.Itcanoerregistration,congurationofservicestoindividualuserrequirements,etc.TheHealthcategoryismostlyevaluatedaccordingtotherelevancyandlanguagecriteria.Thewebsitesshouldproviderelevantcontenttousersanddierentlanguages.Relevantcontentshouldbecredible.AveryimportantdependencebetweencrediblecontentandobjectivityispresentedinFig.16.Itmeansthatcrediblecontentshouldbeunbiased,accurate,believable,completeandreliable.Accuratecontentisin-tendedtobeconsistent,credible,complete,up-to-dateandrelevantaccordingtotheprovidedresultsinthecirclegraph.TheassessmentofSocialmediawebsitesischaracterizedbyeval-uatingvideoandreputationaccordingtotheassociationsinFig.17Videoisrelatedtothequalityofaudioandimage.Reputationisrelatedtoreliability,relevancyandsecuritycriteria.Responsivenessandac-cessibilityareamongimportantcriteriaassociatedtoreliability.Thesocialmediacircle

9 graphshowsthatthesearchresultsshouldbere
graphshowsthatthesearchresultsshouldbere-levanttousers4.5.RQ5:whatarethefrequentcriteriaconsideredinanassessmentCriteriadiversityandmultiplicityinvolvemanyissuesininforma-tionspacerendering,andthenintheassessmentprocess.ThatiswhyoneofthepurposesofthisSLRistoidentifythefrequentcriteriausedfortheassessmentofwebsitesfromtheselectedstudies.Moreover,identifyingthefrequencyofcriteriacanbeconsideredassearchingthefrequentitemsetsfromalargesetofitems(Orriols-PuigandCasillas,Givenasupportthreshold54meansitemsetsthatappearinatleast54studiesarecalledfrequentitemsets.Toconclude,theplotinFig.18showssomefrequentcriteriareferredbythedierentstudies;theyarefrequentitemsetswithsizeequalto1.RQ6:DuetothesubjectivityandimprecisionofthatMCDMpro-blem,aretherestudiesthatperformedtheassessmentusingsoftcom-putingorhybridmethods?Ifso,inwhichphaseoftheevaluationandwhatisthemotivationofapplyingsuchintelligentmethods?Table4presentsasyntheticoverviewofsoftcomputingandhybridmethodsintheeldofwebsitesassessmentwhenfocusingonthemotivationsofapplyingmethodsbasedonintelligenttechniques.Severalstudiesdevoteinterestinestablishingtheevaluationwithsoftcomputingtechniques.Thefuzzytechniquewaslargelyusedinerentphasesintheevaluationprocess.Thiscanbeexplainedbysubjectivityandimprecisionofsuchdecisionmakingproblems.Infact,fuzzyreasoningisappliedtoobtainawebsitesranking(RekikandKallel,2011)usinglinguistictermssuchas{poor,average,good,ex-cellent}.Italsohelpsdecisionmakerstodistributeweightsforcriteriaandtolimitsubjectivehumanjudgment(Lin,2010;Xing,20105.DiscussionandTheadoptedstrategyandobtainedresultsbytheSLRarecomparedtoareviewofliterature(Chiou,Lin,&Perng,2010)thatproposesastrategicframeworkforwebsiteevaluationinTable5Thissectionalsohighlightsthemainndingsinapplyingthesys-tematicliteraturereviewintheassessmentofwebsitesquality.Itsmainobjectiveistoexplorestudiesinthedomain,bylteringtheessentialonesandextractingdataneededfromthem.Moreover,thisSLRpro-aclearanswerforsomefundamentalconsiderations,essentially,whattobeassessed,andhowtoassessawebsite?First,byansweringtheresearchquestionsinreportingthereviewphasewenoticethatselectingandgatheringcriteriawasacriticaland Table3AssociationrulesrelatedtoE-commercecategory.ConsequentAntecedentItemSetRuleLift{Purchaseintention}{E-commerce,1336.110.69{Purchaseintention}{Content,Product}1131.49.31{Purchaseintention}{Content,E-1230.89.11{Purchaseintention}{E-commerce}1725.87.63{Transaction}{Security,Service}1331.06.60{Transaction}{Satisfaction,1030.36.46{Transaction}{Price}829.66.32{Transaction}{Responsiveness}929.06.19{Price}{E-commerce,1130.66.03{Product}{Content,E-2359.05.51{Transaction}{Design,Security}825.85.50{Responsiveness}{S

10 ecurity,Service}1331.05.32{Responsivenes
ecurity,Service}1331.05.32{Responsiveness}{Satisfaction,1030.35.21{Product}{E-commerce}3654.55.10{Product}{Price}1451.94.85{Price}{E-commerce}1555.64.49{Service}{Responsiveness}2477.43.89{Security}{Responsiveness}1651.63.67{Aesthetics}{Responsiveness}1032.33.18{Security}{Satisfaction,1442.43.01{Trust}{E-commerce}1536.62.95{Security}{Price}1140.72.90{Satisfaction}{Price}1037.02.86{Product}{Satisfaction,1030.32.83{Security}{Service}4239.62.82{Product}{Responsiveness}929.02.71{Satisfaction}{Content,Product}1234.32.65{Service}{Price}1451.92.61{Security}{Trust}1536.62.60{Satisfaction}{Security,Service}1433.32.57{Product}{Security,Service}1126.22.45{Satisfaction}{Service}3331.12.40{Responsiveness}{E-commerce}929.02.34{Product}{Content,1225.02.34{Satisfaction}{Product}1729.82.30{Service}{Content,2245.82.30{Satisfaction}{Trust}1229.32.26{Aesthetics}{E-commerce}1527.82.24{Security}{Content,E-1230.82.19{Security}{E-commerce,1130.62.17{Security}{E-commerce}2030.32.15R.Rekiketal. essentialphaseintheprocessofevaluation.Infact,anotherndingisthesetofassociationrulesthatrevealstheinteractionandrelationbetweencriteriaandtheirimportance.AnotherissueisextractingasetoffrequentcriteriaasaphaseofreducingthemassivenumberofAmongthemostrelevantndingsrelatedtotheevaluationmethodsandtechniquesishowtoassessthequalityfromtheselectedcriteria.Twowayshavebeenapplied,whicharequalitativeandquantitative. Fig.13.VisualizationofrelationsforE-commercecategory. Fig.14.VisualizationofrelationsforEducationalcategory.R.Rekiketal. Fig.15.VisualizationofrelationsforE-governmentcategory. Fig.16.VisualizationofrelationsforHealthcategory.R.Rekiketal. Thequalitativeapproachischaracterizedbyobservinguserinteractionsfromgatheringuserexplanationsandopinions.SomestudiessuchasHeradio,Cabrerizo,Fernández-Amorós,Herrera,&Herrera-Viedma,Hsuetal.,2012Vatankhah,Wei,&Letchmunan,2014)considerqualitativecriteriafromtheusersjudgmentsthroughquestionnaireswhilethequantitativeapproachisbasedonmeasuringcriteriabyevaluationtools(DeLima,Lima,&DeOliveira,2009RekikandKallel, Fig.17.VisualizationofrelationsforSocialmediacategory. Fig.18.Determinationoffrequentcriteria.R.Rekiketal. Finally,softcomputingmethodsarelargelyappliedinthelastdecadeinsomestepsoftheassessmentprocessasconcludedinan-sweringRQ6.6.ConclusionByfollowingaSLRprocess,itwasseenasasuitablestrategytonetheobjectiveandthequestionsinthisresearch.Manypurposesforassessingwebsitesareidentiedasprovidingrecommendationsforimprovingthequality,collectingasetofcriteriaandweighingthemtoensuretheassessmentandrankingwebsites.TheSLRalsoenablesonetomethodicallycollectasetofpapersinthescopeofwebsitesqualityassessment.Firstly,initialcriteriato

11 performtheevaluationaredenedandregrouped
performtheevaluationaredenedandregroupedsemantically.Then,textminingisappliedtoextractusefulinformationfrompapersandcreateadatasetofcriteria.Thesamemethodologyofextractingandanalyzinginformationisusedtoclassifystudiesaccordingtotheas-sessedcategory.Reliableassociationrulesareobtainedtostudytheinter-dependenciesbetweencriteria.Theyarepresentedbyanetworkgraphtoshowtheserelationsclearlyandtohighlightthemostimportantones.WendthatsomecategoriesdependonaspecicsetofcriteriaforexampletheE-commercecategoryisrelatedtopurchaseintention,product,satisfaction,service,security,aesthetics,etc.Themostim-portantrelationsbetweenthecategoryandcriteriaareanalyzedandclearlyrepresentedbydependencywheelgraphsforE-commerce,Educational,E-government,HealthandSocialmediacategories.Determiningfrequentcriteriafollowedbycurrentstudiesismade.Itisanimportantphaseintheabsenceofstandardstofollow.WithregardtofutureworkweareinterestedinMultipleCriteriaDecisionMakingmethodsexistingintheliteratureforweighinganddecomposingcriteria(Rekik,Kallel,&Alimi,2015Rekik,Kallel,Casillas,&Alimi,2016)andchoosingthesuitableonestoprioritizecriteriacollectedfromthedevelopedworkforsomecategoriesofwebsitessuchastheE-commercecategory(Rekik,Kallel,Alimi,2016TheauthorsexpressthankstotheErasmusMundusAl-IdrisiIIforfundingtheresearchreportedundertheGrantAgreementnumber2013-2401/001-001-EMA2.TherstauthorwouldalsoliketothankKnimecommunityforusingtheirinstrumentsfordatamonitoring.The Table4Softcomputingmethodsintheeldofthequalityassessmentofwebsites.SoftcomputingtechniqueSoftcomputingmethodConcernedphaseoftheevaluationMotivationReferencesFuzzyFuzzylogicInperformingthefuzzycomputationwhenthemeasuredcriteriaarepresentedasinputsforthefuzzysystemRankawebsite(RekikandKallel,2011FuzzyAnalyticHierarchyRepresentationofusersopinionsPrioritizeandweighcriteriatorectifythelimitationofsubjectivehumanjudgmentLin,2010;Xing,2010FuzzylinguisticEvaluatetheservicesoflibrary2.0(Heradioetal.,2013RecommendationtousersinhealthEstebanetal.,2014Fuzzyc-meansDeningthenumberofclustersClusteringresultsanddeterminingthebestqualitysolutionforwebCobos,Mendoza,Manic,León,&Herrera-Viedma,2013BayesiannetworkBayesianInformationEvaluatingthebestsolutionandthenumberofclustersBayesnetPredictionofthebestmodelfortheevaluationofwebpagequalityPredictthequalityofwebsites(DhimanandAnjali,2014FuzzyneuralAdaptiveneuralfuzzyinferencesystem(ANFIS)Intrainingandtestingdataset(LiuandKrasnoproshin,2014SupportVectorMachineSVMTrainingtweetdatasetFilteringandrankingtweetsbyVoseckyetal.,2012Evolotuionary:GeneticAlgorithm(GA)GAbasedlearningmethodIndeterminingtheconnectionweightsforagivenhierarchicalnetworkbyminimizingtheroot-mean-squareerror

12 IdentifycriticalcriteriafortheHu,2009 Ta
IdentifycriticalcriteriafortheHu,2009 Table5Study(Chiouetal.,2010)ThecurrentstudyReviewofliterature1995Systematicliteraturereview2009Collectionofdatafrom83selectedstudiesCollectionofdatafrom532selectedstudiesCategoryassessed:E-commerceCategoryassessed:Anytype,Socialmedia,E-commerce,Health,Educational,E-government,Service,Institutional,Corporate,etc.ComparisonbetweenframeworksPropositionofastrategicframeworkthatdealswith:Websitestrategy-TextminingappliedforstudiestoconstructadatasetofcriteriaStrategyconsistency-AssociationrulesminingappliedtothedatasetanddeterminationofinterdependenciesbetweencriteriaEvaluationfactorsandcriteriamostfrequentlyused-Providingasetoffrequentcriteriagenerallyassessed.R.Rekiketal. authorsacknowledge,too,thenancialsupportofthisworkbygrantsfromtheTunisianGeneralDirectionofScienticResearch(DGRST)undertheARUBprogram.ThisworkalsowaspartiallysupportedbytheSpanishMinistryofScienceandInnovation(grantno.TIN2014-57251-AppendixA.FundamentalnotionsaboutassociationrulesAsetofassociationrulescanbeextractedfromasetofdata.Anassociationruleisanimplicationoftheform:Y,wherebothXandYareitemsetsandXXcanbedenedas{c}andYasc,thenotationoftheassociationrule{cmeansifXcontainsallofcthenitislikelytocontaincThesupportforanitemsetSisthenumberofitemsetsinthedatasetcontainingallitemsinS.denceofaruleTheproblemofdiscoveringassociationrulesischaracterizedbyIf-thenrulesfromthedataset.Inordertomeasurethereliabilityofarule,wecomputethecondenceexpressedbyEq. ConfidenceXYSupportXYSupportX wherebothareitemsetsandXItisthefractionofstudieswithXthatalsocontainsY.Example:given6itemsetsfrom6studieswithsomecriteriaasfollows:={Usability,Content,Aesthetics,E-services};={Usability,Content};={Content,Aesthetics,E={Usability,Content,Reliability,Accessibility};={Security,Privacy};={Usability,Content,Aesthetics,Search}Apossibleassociationruleis:{Usability,Content}Itscondenceis2/4=50%becausethereare4studiesthatfocusonUsabilityandContentwhichareSandSbutjustSandSSupportofassociationruleThesupportofaruleisanimportantmeasurethatindicatesthefrequencyofoccurringpatternsdenedbyEq..Arulethathasverylowsupportmayoccursimplybychance. SupportXYSupportXYNumberoftotalitemsets wherebothareitemsetsandX:if{chashighsupportandcondence,thenboth{c}and{c}willbefrequent.LiftofanassociationruleTheliftofaruleexpressedbyEq. LiftXYSupportXYSupportXSupportY()·() wherebothareitemsetsandXThemoretheliftvalueishigh,themoretheruleisstrong,andvice-versa.Avaluegreaterthan1indicatesthatXandYappearmorefrequentlytogetherthanexpected;thismeansthattheoccurrenceofXhasapositiveectonthatofY,orthatXispositivelycorrelatedwithYandviceversainthecaseofavaluesmallerthan1.AppendixB.Theconceptofin

13 dexingandsearchinginApacheLuceneTheconce
dexingandsearchinginApacheLuceneTheconceptofindexingistheheartofallsearchenginesinordertofacilitatequicksearchamongalargeamountofdata.So,convertingdatatoasuitableformatisthecoreofindexinganditsoutputiscalledanTocreateanwithLucene,therststepistocreateanobject.Theobjectisusedtocreatetheindexandtoaddnewindexentries(i.e.,Documents)tothisindex.ThecodetocreateanispresentedinFig.19Notethattakestwoparameters,,whichareobjects,respectively.R.Rekiketal. Directory:apathtoadirectorywheretheLuceneindexisstored.esthecongurationoftheusinganforindexingdatawhichisinthiscase.SearchinginLuceneissimpleandrapidasindexing.Searchingischaracterizedbylookingupwordsinanindextondthemostrelevantdocumentswheretheyappear.Tosearchinanindex,therststepistoopentheindexwithanFig.20Thenextstepistorunasearchintheindex(Figs.21and22).Thereiscollaborationwiththeandaiscreatedbyinstantiatingitusing(thesameAnalyzerthatthedocumentsintheindexwerecreatedwith).Notethattheparametercanbeastringrepresentingaeldname(e.g.title,author,contents).Onceaparseriscreated,tostartsearchaqueryisthencreatedbypassingasearchexpressionthrough.ThelistofmatchingdocumentsisnallyretrievedbycallingthemethodoftheLuceneFig.23illustratesthesearchingprocessusingthedierentclasses(Agrawal,R.,&Srikant,R.(1994).FastalgorithmsforminingassociationrulesinlargeProceedingsofthe20thinternationalconferenceonverylargedatabases,(pp.487Alanezi,M.A.,Mahmood,A.K.,&Basri,S.(2012).AproposedmodelforassessingE-governmentservicequality:anE-S-QUALapproach.2012internationalconferenceoncomputerandinformationscience,ICCIS2012aconferenceofworldengineering,scienceandtechnologycongress,ESTCON2012,Aleti,A.,Buhnova,B.,Grunske,L.,Koziolek,A.,&Meedeniya,I.(2013).Softwarear-chitectureoptimizationmethods:Asystematicliteraturereview.IEEETransactionsonSoftwareEngineering,39,658Blondel,V.D.,Guillaume,J.-L.,Lambiotte,R.,&Lefebvre,E.(2008).Fastunfoldingofcommunitiesinlargenetworks.JournalofStatisticalMechanics:TheoryandExperiment,2008,P10008Cha,J.(2014).Usageofvideosharingwebsites:Driversandbarriers.TelematicsandInformatics,31,16Chatzopoulos,K.-C.,&Economides,A.(2009).AholisticevaluationofGreekmunici-ElectronicGovernment,6,193Chen,J.V.,Rungruengsamrit,D.,Rajkumar,T.M.,&Yen,D.C.(2013).SuccessofelectroniccommerceWebsites:Acomparativestudyintwocountries.andManagement,50,344Chen,F.H.,Tzeng,G.-H.,&Chang,C.C.(2015).Evaluatingtheenhancementofcor-poratesocialresponsibilitywebsitesqualitybasedonanewhybridMADMmodel.InternationalJournalofInformationTechnologyandDecisionMaking,14,697Chiou,W.-C.,Lin,C.-C.,&Perng,C.(2010).Astrategicframeworkforwebsiteevaluationbasedonareviewoftheliteraturefrom1995to2006.Information&Management,47Cobos,C.,Mendoza,M

14 .,Manic,M.,León,E.,&Herrera-Viedma,E.(2
.,Manic,M.,León,E.,&Herrera-Viedma,E.(2013).ClusteringofwebsearchresultsbasedonaniterativefuzzyC-meansalgorithmandbayesianin-formationcriterion.2013jointIFSAworldcongressandNAFIPSannualmeeting,IFSA/NAFIPS2013,DeLima,S.T.,Lima,F.,&DeOliveira,K.M.(2009).Evaluatingtheaccessibilityofwebsitestodeneindicatorsinservicelevelagreements.Enterpriseinformationsys-tems,lecturenotesinbusinessinformationprocessing,Demeester,T.,Nguyen,D.,Trieschnigg,D.,Develder,C.,&Hiemstra,D.(2012).Whatsnippetssayaboutpagesinfederatedwebsearch.Informationretrievaltechnology,lecturenotesincomputerscience(Includingsubserieslecturenotesinarticialintelligenceandlecturenotesinbioinformatics),P.,&Anjali(2014).Empiricalvalidationofwebsitequalityusingstatisticalandmachinelearningmethods.Proceedingsofthe5thinternationalconferenceoncon2014:Thenextgenerationinformationtechnologysummit,Eidaroos,A.,&Alkraiji,A.(2015).Evaluatingtheusabilityoflibrarywebsitesusinganheuristicanalysisapproachonsmartmobilephones:PreliminaryndingsofastudyinSaudiuniversities.Newcontributionsininformationsystemsandtechnologies,ad-vancesinintelligentsystemsandcomputing,Esteban,B.,Tejeda-LorenteÁ,Porcel,C.,Moral-Muñoz,J.A.,&Herrera-Viedma,E.(2014).AidinginthetreatmentoflowbackpainbyafuzzylinguisticWebsystem.Roughsetsandcurrenttrendsincomputing,lecturenotesincomputerscience(Includingsubserieslecturenotesinarticialintelligenceandlecturenotesinbioinformatics),Esteves,J.,&Lopez,V.W.B.(2010).ComparingthequalityofLatinAmericane-HealthNationalWebsites.16thAmericasconferenceoninformationsystems2010,AMCIS Fig.23.Thesearchingprocess. Fig.19.Thecreationofan Fig.20.Theopeningofanindexwith Fig.21.Thecreationofa Fig.22.Thecreationofaquery.R.Rekiketal. Fogli,D.,&Guida,G.(2013).Assessingcorporatewebsites:Qualitymodelandmetho-Currenttrendsinwebengineering,lecturenotesincomputerscience(Includingsubserieslecturenotesinarticialintelligenceandlecturenotesinbioinformatics),Fuertes-Callén,Y.,Cuellar-Fernández,B.,&Pelayo-Velázquez,M.(2014).DeterminantsofonlinecorporatereportinginthreeLatinAmericanmarkets:Theroleofwebpresencedevelopment.OnlineInformationReview,38,806Heradio,R.,Cabrerizo,F.J.,Fernández-Amorós,D.,Herrera,M.,&Herrera-Viedma,E.(2013).AfuzzylinguisticmodeltoevaluatethequalityofLibrary2.0functionalities.InternationalJournalofInformationManagement,33,642Hernández,B.,Jiménez,J.,&Martín,M.J.(2009).Keywebsitefactorsine-businessInternationalJournalofInformationManagement,29,362Hsu,T.-H.,Hung,L.-C.,&Tang,J.-W.(2012).Themultiplecriteriaandsub-criteriaforelectronicservicequalityevaluation:Aninterdependenceperspective.InformationReview,36,241Hu,Y.-C.(2009).Fuzzymultiple-criteriadecisionmakinginthedeterm

15 inationofcriticalcriteriaforassessingser
inationofcriticalcriteriaforassessingservicequalityoftravelwebsites.ExpertSystemswithApplications,36,6439Kitchenham,B.,&Charters,S.(2007).GuidelinesforperformingSystematicLiteratureReviewsinSoftwareEngineering.Kitchenham,B.,Pretorius,R.,Budgen,D.,PearlBrereton,O.,Turner,M.,Niazi,M.,etal.(2010).Systematicliteraturereviewsinsoftwareengineeringatertiarystudy.InformationandSoftwareTechnology,52,792Kotenko,I.,Chechulin,A.,Shorov,A.,&Komashinsky,D.(2014).Analysisandevaluationofwebpagesclassicationtechniquesforinappropriatecontentblocking.Advancesindatamining.applicationsandtheoreticalaspects,lecturenotesincomputerscience(Includingsubserieslecturenotesinarticialintelligenceandlecturenotesinbioinfor-Leite,P.,Gonçalves,J.,Teixeira,P.,&Rocha,A.(2014).Towardsamodelforthemea-surementofdataqualityinwebsites.NewReviewofHypermediaandMultimedia,20Lin,H.-F.(2010).AnapplicationoffuzzyAHPforevaluatingcoursewebsitequality.ComputersandEducation,54,877Lin,C.L.(2011).TheimprovementstrategyofonlineshoppingservicebasedonSIA-NRMapproach.In:SmartInnovation,SystemsandTechnologies.pp.295Liu,H.,&Krasnoproshin,V.V.(2014).QualityevaluationofE-commercesitesbasedonadaptiveneuralfuzzyinferencesystem.Neuralnetworksandarticialintelligence,communicationsincomputerandinformationscience,G.,&Wang,Y.(2013).Trust-orientedserviceproviderselectionincomplexonlinesocialnetworks.In:AdvancedWebServices.pp.363Luo,J.,Ba,S.,&Zhang,H.(2012).Theeectivenessofonlineshoppingcharacteristicsandwell-designedwebsitesonsatisfaction.MISQuarterly:ManagementInformationSystems,36,11311144A9Mavlanova,T.,Benbunan-Fich,R.,&Koufaris,M.(2012).Signalingtheoryandin-formationasymmetryinonlinecommerce.InformationandManagement,49McCandless,M.,Hatcher,E.,&Gospodnetic,O.(2010).Luceneinaction,secondedition:Coversapachelucene3.0(2nded.).Greenwich,Conn.u.a:ManningPublicationsMuñoz-Leiva,F.,Luque-Martínez,T.,&Sánchez-Fernández,J.(2010).Howtoimprovetrusttowardelectronicbanking.OnlineInformationReview,34,907Murakata,K.,&Matsuo,T.(2011).Informationdisclosureincentivesintourismin-formationsystems.20113rdinternationalconferenceonawarenessscienceandtechnology,ICAST2011,Orriols-Puig,A.,&Casillas,J.(2010).Evolutionofinterestingassociationrulesonlinewithlearningclassiersystems.InJ.Bacardit,W.Browne,J.Drugowitsch,E.Bernadó-Mansilla,&M.V.Butz(Eds.).Learningclassiersystems,lecturenotesincomputerscience(pp.2137).Berlin,Heidelberg:SpringerOzmen-Ertekin,D.,&Ozbay,K.(2012).Dynamicdatamaintenanceforqualitydata,qualityresearch.InternationalJournalofInformationManagement,32,282Pengnate,S.,&Antonenko,P.(2013).Amultimethodevaluationofonlinetrustanditsinteractionwithmetacognitiveawareness:Anemotionaldesignperspectiv

16 e.InternationalJournalofHuman-ComputerIn
e.InternationalJournalofHuman-ComputerInteraction,29,582Rafe,V.,&Monfaredzadeh,M.(2012).Aqualitativeframeworktoassesshospital/Medicalwebsites.JournalofMedicalSystems,36,2927Rekik,R.,&Kallel,I.(2011).Fuzzyreducedmethodforevaluatingthequalityofin-stitutionalwebsites.20117thinternationalconferenceonnextgenerationwebservicespractices(NWeSP),Rekik,R.,&Kallel,I.(2013).Fuzz-Web:Amethodologybasedonfuzzylogicforassessingwebsites.InternationalJournalofComputerInformationSystemsandIndustrialManagementApplications,5,126Rekik,R.,Kallel,I.,&Alimi,A.M.(2014).Extractionofassociationrulesusedforas-sessingwebsitesqualityfromasetofcriteria.201414thinternationalconferenceonintelligentsystems(HIS),Rekik,R.,Kallel,I.,&Alimi,A.M.(2015).Qualityevaluationofwebsites:AcomparativestudyofsomeMultipleCriteriaDecisionMakingmethods.201515thinternationalconferenceonintelligentsystemsdesignandapplications(ISDA),Rekik,R.,Kallel,I.,&Alimi,A.M.(2016).RankingcriteriabasedonfuzzyANPforassessingE-commercewebsites.2016IEEEinternationalconferenceonsystems,man,andcybernetics(SMC),Rekik,R.,Kallel,I.,Casillas,J.,&Alimi,A.M.(2016).Usingmultiplecriteriadecisionmakingapproachestoassessthequalityofwebsites.InternationalJournalofComputerScienceandInformationSecurity,14,747Schäfer,K.,Kummer,T.-F.,&Günther,O.(2011).Measuringwebsiteinformationandservicequality?anextendedmulti-attributeattitudemodel.17thamericasconferenceoninformationsystems2011,AMCIS2011,Silambannan,R.,&Srinath,M.V.(2013).Aconvictiveframeworkforqualitybaseconstructionandevaluationofe-learningwebsite.JournalofTheoreticalandAppliedInformationTechnology,58,147Vatankhah,N.,Wei,K.T.,&Letchmunan,S.(2014).UsabilitymeasurementofMalaysianonlinetourismwebsites.InternationalJournalofSoftwareEngineeringandItsApplications,8Violante,M.G.,&Vezzetti,E.(2015).VirtualinteractiveE-learningapplication:Anevaluationofthestudentsatisfaction.ComputerApplicationsinEngineeringEducation,,72Vosecky,J.,Leung,K.W.-T.,&Ng,W.(2012).Searchingforqualitymicroblogposts:Filteringandrankingbasedoncontentanalysisandimplicitlinks.Databasesystemsforadvancedapplications,lecturenotesincomputerscience(Includingsubserieslecturenotesinarticialintelligenceandlecturenotesinbioinformatics),Xing,H.-H.(2010).Studyofevaluatingweb-basedcoursesbasedonFAHP.Fuzzyin-formationandengineering2010,advancesinintelligentandsoftcomputing,Yagüe,A.,Garbajosa,J.,Pérez,J.,&Díaz,J.(2014).AnalyzingSoftwareProductInnovationAssessmentbyUsingaSystematicLiteratureReview.pp.5049Yang,H.,Guo,X.,Wu,T.,&Ju,X.(2015).Exploringtheeectsofpatient-generatedandsystem-generatedinformationonpatientsonlinesearch,evaluationanddecision.ElectronicCommerceResearchandApplications,14,192R.Reki