Detecting anomaly collections using extreme feature ranks-DMKD

DataMinKnowlDiscDOI10.1007/s10618-014-0360-3DetectinganomalycollectionsusingextremefeatureranksHanboDai·FeidaZhu·Ee-PengLim·HweeHwaPangReceived:24February2013/Accepted:4June2014©TheAuthor(s)2014AbstractDetectinganomalycollectionsisanimportanttaskwithmanyapplications,omalycollection,entitieult,,membersofananomalycollectionareeoduceanovelanomalydefioseanewmeasureofanomaloecanbealargenumberofERACsofvarioussizes,forsimplicity,wefirstinvestigatetheERACdetectionproblemoffindingtop-KERACsofapredefitacklethefollow-upERACexpansionproblemofuncoveringthesupersetthmsareproposedforbothERACdetectionandexpansionproblems,fically,insyntheticdatasets,bothERACdetsibleeditor:(B)TheSchoolofComputerScienceandInformationEngineering,HubeiUniversity,Wuhan,Chinae-mail:daihanbo@··eSchoolofInformationSystems,SingaporeManagementUniversity,Singapore,Singaporee-mail:fdzhu@-mail:eplim@-mail:hhpang@123

spamdataset,bothERACdetectionandexpansionalgorithDBdataset,bothERACdetectionandexpansionalgorithmsidentifyunusualactorcollectionsthatarenoteasilyidentifineseonlineforumdataset,ourERACdetectionalgorithmidentifiessuspicious“waterarmy”pansionalgorithmsuccessfullyrdsAnomalycollection·Extremefeaturerank·Anomalycluster·Outliergroup·Spamdetection·Spamcluster1IntroductionAccordingtoBarnettandLewis(1994),ananomalyoroutlierisadatainstanceorsubsetofdatainstaral,ananomalycanbeclassifianomalyusuallyliesinasparseregionorisfarawayfromnormalones,whereasananomalycollectionareformedbysimilarentities,tice,thisinconsistencyoftenimpliesdifferentagendaspaper,wedetectanomalycollectionsbytheirextremefeatureranks,basedontheobservationthatmembersinananomalycollrtedbyFetterlyetal.(2004),Castilloetal.(2007)andGyöngyietal.(2004),mple,theystuffthepagsogeneratepagesfromsimilartemplatesontheflyinordertoperform“linkspam”.Asaresult,whenmeasuredbythosecharacteristicsorfeatures,spammerhostsconsistentlydemonstrateveryextremetraitsandformanidentifiableanomalouscollection,strate,Fig.1shows30webhosts{e0,...,e29}withthreehostfeatures{f0,f1,f2},reflectingtheaforementionedspammingstrategies:f0representstheaveragenumberofpopularkeywords,f1isthevarianceofthewordcount,hfeature,henidentify{e5,e7,e12}6e29e20e21Fig.1AnexampleofERAC.30entities{e0,...,e29}arerankedaccordingtoeach3features{f0,f1,f2}.Inthisexample,{e5,e7,e12}isanERAC123

Detectinganomalycollectionsf0andf2,tthate5,e7ande12collectivelydisplayextremetraitsacrossthherexample,groupsoffraudulentulentuserwouldcreatesufficientlow-pricetransactionswithotheraccompliceaccountsinashorttimeinordertogaincredibility,beforeperformingfraudtransactionsinvolvinglargesumsofmoneyaccordingtoChuaandWareham(2004)andPanditetal.(2007).Consequently,theyarelikelytorankatextremepositionswithrespeerstudythiskindofanomalycollections,weproposeanoveldefinition,isanentitysubsetclusteredtowardthetoporbottomranks,annotbeeasilydetectedbyexistinganomalydetectionapproaches,becausetheyeitherfocusonsinglepointanomalies,ortatasetofsinglepointanomaliesdoesnotalwaysformanERAC,mple,inFig.1,e12isnotveryextremebyitselfalthoughitispartofanERAC{e5,e7,e12}.Incontrast,e8isveryanomalousasasingleentity,sinceitappearsatextremepositionsonallthreefeatures,hereforecannotbediscoverctERACs,Daietal.(2012)malousnessofanERACisquantifihelargenumberofERACsofvarioussizes,Daietal.(2012)tackletheERACdetectionproblemofdiscoveringtop-KERACswithapredefinedsizelimit,whichissettosmallvaluesforeffiheless,afterbeingofferedwiththetopERACsofapredefinedsize,usersmaywanttomple,inthewebspamcase,usersmayfindthedetectedERAC{e5,e7,e12}ofinterestastheyhavethecom-monspammingstrategyofusinglotsofpopularkeywords,withverylittlevarianceonthewordcount,turaltoask,canwedetectthesupersetofthisERACthatareevenmoreanomalouswithsimilarsetsofspammingstrategies?Therefore,inthispaper,wenotonlyexploretheERACdetectionproblem,butalsoproposetheERACexpansionproblemtouncoverthesupersettheERACdetectionproblem,ERACexpansionisdonewithoutpredefiarizeourcontributionsasfollows:–Wearethefiuretheanomalousnessofacollectionbyhowextremelyrankeditiswithrespecttoanyfeatureset.–Wedevelopbothexactandheuristicalgorithmstofindthetop-Kanomalouscol-lectionsofapredefinedsizelimitondifferentpruningstrategiesunderthefeature123

l.––––providealgorithosethefollow-upproblemdesignefficientgreedyalgoritoseanexploratoryschemeforsearchingERACs,makinguseoyourERACdetectionapproachonsyntheticdatasetswithinjectedERACsandonthreerealdatasetsincludingawebhostgraph,heticdatasets,ourproposedheuristicalgorithmscaleswellwiwebspamdatasetwithlabeledtruespammers,ourapproachdiscoversspammercollectionsthataremoreanomalouswhileachievinghigherprecisions,comoviedataset,wedetectunusuaaluationshowsourapproachsuccessfullyfiultsdemon-stratethatinthesyntheticdatasets,theinjectedERACsareretrievedwithhighsuccessrate;inwebhostdataset,thealgorithmachieveshigherprecisionthanexistingmethods;inthemoviedataset,theexpansionrevealslargeranomalousactorcollectionsthatcannotbediscoveredbytheclustering-basedapproach;intheChineseonlineforumdataset,theexpantroducingtheproblemformulationinSects.3and4presentsourERy,Sect.8concludesthepaperanddiscusseslimitationsandfeaturework.2RelatedworkAccordingtoasurveybyChandolaetal.(2009),ulofapproachesareproposedsuchasaclassification-basedonebyCastilloetal.(2007),adistance-basedonebyKnorrandNg(1998),adensity-basedonebyBreunigetal.(2000)andclusteringbasedonesbyEsteretal.(1996)andGuhaetal.(1999).Sincetheseapproachesassumethatanomaliesappearinsparseregionsorarefarawayfromthenormalentities,anomalouscollectal.(2009),Heetal.(2003)andLoureiroetal.(2004)useclusteringbasedapproachforanomalouscollectiondetection,assumingthatnormalentitiesbelongtolargeanddenseclusters,ingtoDuanetal.(2009)andHeetal.(2003),anomalousclustersarethesmalleronesthat123

Detectinganomalycollectionstogetherconstitutelessthan10%roetal.(2004)r,theassumptionthrmore,l.(2010)kassumesthatafterdatapointsareprojectedtosomehyperplane,theanomalouspointsfollowdistributr,theseassumptionsdonotalwayshold,-Castroetal.(2011)useastatisticalmodeltodetectinagraphananomalycollectioninwhicrktakestheassumptionthatonlyoneanomalousclusterexistsinthewholegraph,inganomalouscollectionsisalsorelatedtothetaskofsubgroupdiscoveryproposedbyKlösgen(1996)andWrobel(1997).AsurveydonebyHerreraetal.(2011)summarizesthetaskastodiscoverthesubgroupsoftheentitypopulationthatarestatistically“mostinteresting”groupsareinducedbyrules,r,theanobecause,(i)notallmembersofananomalycollectionsatisfyrules;(ii)er,theanomalousnessweusetomeasureananomalycollectiondoesnotinvolvetheclasslabels,whereastheinterestingnessmeasureusedforsubgraphdiscoveryusuallydoes.3ExtremerankanomalouscollectionDaietal.(2012)haveshownthattheanomalousnessofacollectionisbettermeasureddirectlyatcollectionlevel,insteadofmeasuringindivadoptthatdefinotetheuniversalentityset,(e).1,rank(e8)=1andrank(e7)=emityindexrreferstoanextremeregion,andSf(r)ef,Table1NotationsNotationESRankf(e)pf(S,r)rf(S)presentativerofSonfNotationFrSf(r)

本文标签： 表格 作者 入门