Database co op eration: classication and middlew are to ols

(1)

Database co op eration: classication and middlew are to ols

PaoloAtzeniUniversita'diRomaTre,Italyincooperationwith

Luca Cabibb o

and

Gianni Mecca

[atzeni,cabibbo,mecca]@inf.uniroma3.ithttp://poincare.inf.uniroma3.it:8080

PaoloAtzeni|CAiSE*971

Outline

introduction

classicationcriteria

classication

architecturaltechniques

architecturesandmiddlewaretools

conclusionanddiscussion

introduction

classication

Focus

databasecooperation

bridgethegapbetweenmethodologiesandtools

classicationofapplications

implementationarchitectures

(2)

Sp ecic framew ork

contractwithAIPA(NationalAuthorityinItalyforComputinginthePublicAdministration)

AIPAispromotingaNationalnetwork:\theInternetoftheItaliangovernmentandadministrationoces"

AIPAwantedtogiveguidelinestothevariousbranchesonhowtoexploitthenetworkandpromotecooperation

General framew ork

theneedforintegratingapplicationsariseseverywhere,stimulatedbythedevelopmentofnetworks

{

integration(orinteraction)ofcomponentsdevelopedseparately,fortechnical,organizational,ortemporalreasons

{

cooperationofpreviouslyseparateprocesses(processreengineering)

{

cooperationofindependentcompanies(ormergesthereof)

Levels of interaction

connectivit y

:systemsandnetworksexchangepacketsofinformation(inInternet,withTCP/IP)

interop erabilit y

:systemsandnetworksinteractbymeansofstandardservices(inInternet,standardprotocolsaboveTCP/IP)

co op eration

:applicationsoverdierentsystemsinteractwithoneanother;attheextremelevel,distributedapplicationscoordinateexistinglocalapplications

Interop erabilit yservices

letransfer(ftp)

virtualterminal(telnet)

electronicmail(X.400orESMTP/MIME)

directoryservice(X.500)

WWW(http)

(3)

Co op eration of Info rmation Systems

requiresthatthecooperatingsystemsoerapplicationservices

ithappenswhensystemsmakeuseofapplicationservicesoeredbyothersystems

cooperatingsystems\sharegoals"

amajorfeature:\continuouschange"

PaoloAtzeni|CAiSE*979 introduction

classication

First level classication

twoformsofcooperation(noclearcut,indeed)

data-o riented

:datainasystemisvisible/accessibletoothersystems

pro cess-o rie nted

:systemsoerservices,exchangemessages,(ordata,documents),triggeractivities

Pro cess-o riented co op eration

exchangeofmessagesanddocuments;simplecooperation;satisedwithinteroperabilityservices

structuredexchangeofmessagesanddocuments;morecomplexcooperation;satisedwithEDIservicesandsophisticatedmailers

cooperatingprocesses:composedofactivitiesofindependentsubjectsthatcooperate;complexcooperation;solution(?):work owmanagementsystems

(4)

Co op eration: requirements

actualcooperationinvolvessystemsthatare

distributed

heterogeneous

autonomous

Distribution

cooperationmeansthatwehavemultiplesystems;datacooperationmeansthatwehavemultipledatabasesthathandledata

asopposedtousualdistributeddatabases,heredistributionisnotadesigndecision,butafact,duetothepreexistenceofthecooperatingdatabases

distributionmayrangefromdierentdatabasesonthesamemachinetodatabasesspreadoverageographicnetwork

Heterogeneit y

manyaspects

dierencesinthecomputingenvironment(hardware,operatingsystem,networksoftware)

dierencesinthedatabasemanagementsystem:

{

dierentdatamodel(relational,hierararchical,OO,les,...)

{

detailsinthesamedatamodel(versionsoftherelationalmodel:types,constraints,...)

{

dierentlanguages(SQLandQBE,versionsofSQL,...)

semanticheterogeneity:dierencesinthemeaningofdata

Autonomy

Absenceofacommon(orcoordinated)controloverthevarioussystems:\distributionofcontrol"Technicalaspects:

designautonomy:thevarioussystemsarebuiltindependently,withdierentchoicesformanyaspects(thusinducingheterogeneity)

serviceautonomy:decisiononifandhowcooperationisestablished(whatservicesareoered)

executionautonomy:cooperationdoesnotinterferewith\private"operations;cooperatingoperationsareexecutedunderlocalcontrolPaoloAtzeni|CAiSE*9716

(5)

Distribution, Heterogeneit y,Autonomy

Classicationcriteria?No,sinceinthecooperationweshouldbereadytotacklehighlydistributed,heterogeneous,andautonomoussystemsConstraints?Probablyyes,atleastintheworstcaseDecienciestoovercome?Possibly,ifthereissomecoordinatingauthority:cooperationcanstimulatereengineering(reducingthedegreesofheterogeneity,autonomy,anddistribution)

The \ideal goal"

completedatabasefeaturesinanintegratedframework(inordertodevelopecient/eectivecooperativeapplications):

{

\integrateddatabase"

{

queryandtransactioncapabilities

{

absenceofredundancy(andinconsitency)

ho wever

currenttechnologydoesnotreallysupportthis(atbestrequireseortandmoney)

New classication criteria for data-o riented co op eration

degreeoftransparencyofcomponentdata

complexityofoperations

levelofliveliness(orup-to-dateness,asopposedtoobsolescenceorlatency)ofdata

Transpa rency

measures the need for hiding distribution and heterogeneit y of comp onent systems in adata-o riented co op eration

+integrationofcomponentdatabases:thecooperativeapplicationneedstoseeone(virtual)database,oeringanintegratedschema(orsetoffunctions)

-itisacceptablethateachcomponentdatabaseoersasetofservices:eachcooperativeapplicationisresponsibleforaccessing,integrating,transformingthevariouspiecesofdata

(6)

Complexit yof op erations

measures the need for co ordination in the execution of op erations (queries and transactions)

+complexoperations(queriesandupdates):joinoflargerelations(fromdierentdatabases)ortransactionswithmultipleupdatesindierentdatabases;requirenontrivialmanagement

-simpleoperations(forexampleread-only,orlocal);donotrequirespecicsupport

Liveliness of data

measures the need for actual availabilit yof current data

+on-lineaccesstotheprimarycopyofdata:\accessofactualdatawhereitis"|theoriginalgoalofintegrateddatabases?

-accesstocopies,withacontrolleddegreeofobsolescenceNote:thiscriterioncanbeappliedindependentlytothevariouscomponents

introduction

classication

A classication for data-o riented co op eration

thereisnoneedtoconsiderallcombinations:

{

thecriteriaarenotindependent:ahighdegreeofcomplexityrequiresanintegrationinfrastructure,thatis,itrequiresahighdegreeoftransparency;

{

somecasesaremarginalorsubsumed

majorclasses

{ multidatabases

:transparency,complexity,up-to-dateness

{ data warehouses

:transparency,complexity

{ local info rmation systems with external data

:varyingdegreeofup-to-datenessPaoloAtzeni|CAiSE*9724

(7)

Multidatabases

intheidealcase,thereisahighdegreeoftransparency,complexity,up-to-dateness

thereisa

global

systemthatintegratesservices:livedataisdirectlyaccessedinatransparentandecientway

methodologiesandtoolsforintegration(ofschemes,data,languages)areneeded

Multidatabase

DBDBDB LocalMgrLocalMgrLocalMgr PPPPPPPP client

clientExporterExporterExporter , , , , , , , ,

@ @ @ @ @ @ @ @ GlobalMgr HHHHHHH

clientclient

Five-level architecture for federated database systems

DB rrrDB LocalSchemaLocalSchema HHHHHH

clientclientComponentSchComponentSch @ @@

, ,, ExportSchemaExportSchemaExportSchema

XXXXXXXXXXXXXXXXX

HHHHHHHHH FederatedSchemaFederatedSchema @ @ @ @@

HHHHHHHHH

, , , ,, ExternalSchExternalSchExternalSch client

@ @ @@ client

, , ,, client

Three-level architecture for traditional database systems

DB InternalSchema LogicalSchema HHHHHHHH

ExternalSchemaExternalSchemaExternalSchema @ @@ , ,, clientclientclientclient

(8)

Data W arehouse

dataareintegratedo-lineandstoredinanewdatabase(the

data warehouse

)

typicalapplications:decisionsupport(formarketing,sales,nancialanalysis),investigation,summarization

greatinterestinthemarketplace(OLAP,datacube,multidimensionaldatabases,extractors,replicators)

Data W arehousing app roach

DBDBDB LocalMgrLocalMgrLocalMgr PPPPPPPP client

clientExtractorExtractorExtractor

XXXXXXXX Integrator DataWarehouse DWMgr HHHHH

clientclient

OPERATIONALDBDATAWAREHOUSEapplicationorientedsubjectorientedservestheoperationlevelservesthemanagersleveldetaileddataalsosummarized,withtimeseriesaccessedaunitatatimeaccessedasetatatimenonredundantdataredundantdatacurrentvaluesovertimeand/ortimeseriesvarieswithtimeisnotupdatedusedstandardlyusedcasuallyrequirementsknownapriorinotknownaprioriperformancesensitiveperformancerelaxedtransactiondrivenanalysisdrivenconcurrencycontrolneedednoneedhighavailabilityrelaxedavailabilitystaticstructure exiblestructuresmallamountofdataperoplargeamountofdata

Advantages of W arehouses wrt Multidatabases

on-lineaccessinqueryprocessingcanbeslow:itcompeteswithoperationalactivities

primarysourcesmaybeunavailable

complexrestructuringandaggregationmaybeneeded(andprimarysourcesmaybeheterogeneous)

(9)

Advantages of Multidatabases wrt W arehouses

sometimesaccesstocurrent(primary)dataisessential

primarydatamaychangerapidly

cansupportunpredictedqueries

A prop osed architecture for DW

operationaldata

enterprisewarehousedata(granular,subjectoriented,historical)

\DataMarts:"departmentaldata(aggregated,themeoriented)

individual(temporary,adhoc,problem-based)Littleredundancyamongthelevels.

(anaside)

Data Mining

Twogoalsoverlarge(derived)setsofdata:

ndrelationships

makeforecasts

PaoloAtzeni|CAiSE*9735 Dataminingisusefulforprovidinginformationofvarioustypes:

classes:groupobjectsonthebasisofproperties

clusters:\unpredicted"categories

associations:correlationsbetweendierentevents

sequences:specicpatterns,forexampleintime-series

forecasts

similarsequences

(10)

No clea rcut

Inacomplexsystemtheremaybe

datawhoseup-to-datenessisessential

datawhoseprimarycopyisexpensivetoaccess(wrttotheactualneedforup-to-dateness)

datathatarealwaysaggregatedinthesamewayTherefore,wecouldneedanintegrationofreplicatedandprimarydata

An intermediate solution

DBDB LocalMgrLocalMgr PPPPPPP clientExporterExporter

DBDB LocalMgrLocalMgr clientExtractorExtractor

HHHH Integrator DW DWMgr

PPPPPPPPP GlobalMgr HHHHHH

clientclient

Lo cal Info rmation Systems with external data

usefulifasystemhastoaccessexporteddatafromanothersystem

theapplicationhastoincludethemanagementofintegration,translation,accesscontrol

meaningfulonlywithsimpleoperations

datacanbeprimaryorreplicated

The application do es the integration

DB LocalMgr PPPPPPP clientExporterDB LocalMgr

DBDB LocalMgrLocalMgr clientExtractorExtractor

HHHH Integrator DW DWMgr

, , , , , , , , , , , , , , , , ,,

PPPPPPPPP client

(11)

introduction

classication

Architectures for data-o riented co op eration

multi-levelclient-serverarchitectures

withmiddlewaretoolsfor

{

complexintegrationor

{

basiccooperationinterfaces

Tw o-level Client/Server architecture

client

:presentation(interface,graphics,somelocalprocessing)

server

:applicationlogicanddataaccess

Three-level Client/Server architecture

(basicidea)

client

intermediate server

:applicationlogic

back-end server

:datamanagement

(12)

Benets of three-level Client/Server architecture

theapplicationprogram,thespecicanddelicatepartofeachapplication,isseparatedfrombothUIandDB

theDBisnotexposed

scalabilityand exibilityofcomponents:nothin/fatclientdoubt

modularity:reasonableencapsulationoflegacyapplications

Multi-level Client/Server architecture

client

intermediate server

:encapsulation(onanopensystem)ofheterogeneous(possiblylegacy)applications

back-end server

:theencapsulatedapplication(withthedataserver)Therecanbemultiple,intermediateservers(andeventheback-endservermaybeaC/Ssystem)

Basic techniques for Client/Server data co op eration

on-linedatatransfer

o-linedatatransfer

on-linemessageexchange

o-linemessageexchange

on-linedataaccess

On-line data transfer

DBDB ApplicationApplication

Gateway @

@

@ @ @ @ @ @ @ @

(13)

On-line data transfer: Gatew ays

allowapplicationsforonedatabasetoaccessdataoveranotherdatabase

therearedierentlevelsoftransparency

typically,theclientmakesuseofSQL

availableintherelationalworldandtoaccesslegacyDBsfromrelationalapplications

exible(ifauthorized,allowaccesstothewholeDB),althoughsometoolsallowread-onlyaccess

ratherinecient(theserverhastoexecutecasualqueries)PaoloAtzeni|CAiSE*9749

O-line data transfer

Replicator

, , , , , , ,,

@ @ @ @ @ @ @@

O-line data transfer

dataareextractedfromoneDB,transformed,andstoredinanother

ad-hocsolutionshavebeenusedfordecades

recentinterestinreplicationandwarehousingtools

toolsfor

{

extraction(incremental,withchangedetection)

{

translation,integration,cleaning,aggregation

{

OLAPprocessing

On-line message exchange

Interface

(14)

On-line message exchange

function-orientedinterface:remoteprocedurecall(RPC)

theclientinvokestheexecutionofaprogramontheserverandgetstheresults

veryroughextreme:screen-scrapers

storedproceduresinDBMSoropensystemsAPIsindistributedenvironments

modernevolution:object-orientedservices

widelyusedintraditionalTPmonitorsandinmoremoderndistributedobjecttechnologiesPaoloAtzeni|CAiSE*9753

O-line message exchange

DBDB ApplicationApplication Queuemanager

O-line message exchange

function-orientedagain(typically),butasynchronous

atoolhandlesqueuesofmessages(message-orientedmiddleware)

toleratesunavailabilityofserverconnection

On-line data access

DB Application client(browser) WebserverandCGI

(15)

On-line data access database access through WWW

verycommon(anduseful)instructuredWWWservers

InternetvsIntranet

allowsthewidedisseminationofinformation

PaoloAtzeni|CAiSE*9757 introduction

classication

Architectures for data co op eration

Basedon

elementarytools

DatabaseGatewayServersorDistributedDBMSswithgateways

DataWarehousingtools

DistributedTransactionMonitors

ObjectRequestBrokers

integratedtools

Architectures based on Database Gatew ay Server or Distributed DBMS with gatew ays

DBDB ApplicationApplicationIntegrator

,

, @

@

@ Application

(16)

Architectures based on Data W arehousing to ols

,

, @

@

@ Integrator DW Application

Architectures based on Distributed TP Monito rs or Object Request Brok ers

DBDB ApplicationApplicationIntegrator Application

Transaction Pro cessing Monito rs

traditionalTPMonitors:ecient(queuemgmt)andreliable(correcttransactions)accessfromremoteterminals

distributedTPMonitors:ecientandreliableaccesstoremoteservicesinadistributedenvironment

currently,thisisnotacompletedistributedcomputationenvironment

Object Request Brok ers

general-purposedistributedcomputingarchitecture

object-oriented:objectinterfacesareusedbyclientsthatdon'tseeimplementations

donotallowcompletedatabasetransparency

couldbeintegratedwithtoolsthatsupporttransactionmanagement

(17)

Integrated Tools

provideasuiteoffeatures

oftenobject-based

oftenoerbothdevelopmentandexecutionsupport

Architectures based on Integrated Tools

DBDB ApplicationApplicationIntegrator

DW ,

,

, @

@

@ Application

introduction

classication

Conclusions

cooperationcaninvolveverydierentneedsandsomanyalternativesexist

carefulevaluationofcostsandbenetsofthearchitecture:somesoultionsarecomplexandexpensive

cooperationdoesnotrequiremigrationnorreengineering

cooperationcanstimulateandencouragemigrationandreengineering:itallowsanincremental,low-riskapproachtomigration

(18)

General References

P.Atzeni,L.Cabibbo,G.Mecca.Databasecooperation:classicationandmiddlewaretools.Draft.Soonavalaibleathttp://poincare.inf.uniroma3.it:8080

G.DeMichelisetal.Cooperativeinformationsystems:Amanifesto.ToappearinCooperativeInformationSystems:Trends&Directions,M.PapazogluandG.Schlageter(eds),AcademicPress,1997.

M.L.BrodieandM.Stonebraker.MigratingLegacySystems:Gateways,Interfaces&theIncrementalApproach.MorganKauman,LosAltos,1995.

W.Kim,editor.ModernDatabaseSystems:theObjectModel,Interoperability,andBeyond.ACMPressandAddisonWesley,1995.

J.A.Larson.DatabaseDirections.PrenticeHall,1995.

P.A.Bernstein.Middleware:Amodelfordistributedsystemservices.CACM39,2,Feb.1996,86{98.

W.H.Inmon.BuildingtheDataWarehouse,SecondEdition.JohnWiley,1996

OMG.OMAExecutiveOverview.http://www.omg.org/omaov.htm

OMG.SuggestedReadings.http://www.omg.org/suggrdgs.htm

A.P.ShethandJ.A.Larson.Federateddatabasesystemsformanagingdistributed,heterogeneous,andautonomousdatabases.ACMComputingSurveys,22(3):183{236,September1990.

UCSMessagingTeamUCSTechKnowShare:APrimeronMiddleware,IndianaUniversity,http://msgwww.ucs.indiana.edu/messaging/infoshare/middleware.htmlPaoloAtzeni|CAiSE*9769

References to to ols and pro ducts

M.L.BrodieandM.Stonebraker.MigratingLegacySystems:Gateways,Interfaces&theIncrementalApproach.MorganKauman,LosAltos,1995,Chapter10.

Websitesofallproductvendors

DBMS1996Buyer'sGuideandClient/ServerSourcebook.Middleware,Connectivity,andInternetTools.http://www.dbmsmag.com/pcmidcon.html

UCSMessagingTeamUCSTechKnowShare:APrimeronMiddleware,IndianaUniversity,http://msgwww.ucs.indiana.edu/messaging/infoshare/middleware.html

Database co op eration: classication and middlew are to ols