LEZIONE
LEZIONE N. 9
• WMS (Workload Management Service)
• JDL (Job Description Language)
• Matchmaking
• Job submission
Introduzione alle griglie
Introduzione alle griglie computazionali computazionali
Università degli Studi di Napoli Federico II
Corso di Laurea in Informatica – III Anno
Workload Management Service
Workload Management Service (1/2) (1/2)
Lo scopo del WMS e’ lo scheduling distribuito dei job in ambiente Grid
funzionalita’ offerte
• job submission
• job execution
• job status monitoring
• output retrieve
Workload Management Service
Workload Management Service (2/2) (2/2)
componenti
• User interface (UI)
punto di accesso a Grid per gli utenti
• Resource Broker (RB)
servizio di scheduling distribuito
• Job Submission Service (JSS)
servizio di job submission
• Logging and Bookkeeping Service (LB)
servizio di job monitoring
Job Description Language
Job Description Language (1/8) (1/8)
informazioni da specificare per l’esecuzione di un Grid-job
• requirement fisici del job (spazio disco, memoria, ...)
• requirement logici del job (environment, ...)
• data requirement (logical file name, ...)
e’ possibile specificare questi parametri utilizzando il JDL
(Job Description Language)
Job Description Language
Job Description Language (2/8) (2/8)
caratteristiche
• basato su Condor ClassAd
(CLASSified ADvertisement language)
• un ClassAd e’ una sequenza di coppie (attributo, valore)
[
attr1 = value1 attr2 = value2 ...
attrn = valuen
]
Job Description Language
Job Description Language (3/8) (3/8)
categorie di attributi
• job attribute
definiscono le caratteristiche del job
• resource attribute
computing resource attribute
specificano i requirement in termini di risorse di computing utilizzano il prefisso “other.”
storage resource attribute
specificano i requirement in termini di risorse di storage
(protocollo di accesso, logical file name, ...)
Job Description Language
Job Description Language (4/8) (4/8)
job attributes
• Executable
nome del file che contiene il codice eseguibile
• Arguments
argomenti da fornire all’eseguibile
• StdInput, StdOutput, StdErr
nomi dei file che conterranno standard input / output / error del job
• Environment
lista di attributi relativi all’environment
Job Description Language
Job Description Language (5/8) (5/8)
job attributes
• InputSandbox
lista di file locali (rispetto alla UI) necessari all’esecuzione del job
• OutputSandbox
lista di file (generati dal job) da recuperare dopo l’esecuzione del job
Job Description Language
Job Description Language (6/8) (6/8)
computing resource attributes
• Requirements
- requirement del job rispetto alle risorse di computing
- vengono specificati utilizzando i nomi degli attributi definiti nell’Information Service
• Rank
- specifica una preferenza rispetto ad un insieme di risorse che soddisfano i requirement richiesti
- viene specificato utilizzando i nomi degli attributi definiti
nell’Information Service
Job Description Language
Job Description Language (7/8) (7/8)
storage resource attributes
• InputData
PFN (Physical File Name) o LFN (Logical File Name) da utilizzare come input del job
• ReplicaCatalog
nome del Replica Catalog da utilizzare per la risoluzione PFN → LFN
• DataAccessProtocol
protocollo da utilizzare per accedere al PFN
• OutputSE
nome dello Storage Element da utilizzare per la copia
dei file di output del job
Job Description Language
Job Description Language (8/8) (8/8)
esempio di JDL file
Executable = “gridTest”;
StdError = “stderr.log”;
StdOutput = “stdout.log”;
InputSandbox = {“home/joda/test/gridTest”};
OutputSandbox = {“stderr.log”, “stdout.log”};
InputData = “LF:testbed0-00019”;
ReplicaCatalog = “ldap://sunlab2g.cnaf.infn.it:2010/
lc=test, rc=WP2 INFN Test, dc=infn, dc=it”;
DataAccessProtocol = “gridftp”;
Requirements = other.Architecture == “INTEL” &&
other.OpSys == “LINUX” &&
other.FreeCpus >= 4;
Rank = “other.MaxCpuTime”;
Comandi della User Interface Comandi della User Interface
• edg-job-list-match
lista delle risorse che soddisfano i requirement specificati nel file JDL (il RB effettua il matchmaking senza sottomettere il job)
• edg-job-cancel
cancellazione di un job
• edg-job-status
visualizza lo stato di un job
• edg-job-get-output
effettua lo stage-out dei file specificati tramite l’attributo OutputSandbox
• edg-job-get-logging-info
visualizza la sequenza completa delle transizioni di stato di un job
(utilizzato per debugging)
Matchmaking
Matchmaking (1/3) (1/3)
Resource Broker
• il suo compito e’ quello di individuare la risorsa di computing
“migliore” su cui sottomettere il job dell’utente
• interagisce con Information Service e Data Management Service
• il Computing Element selezionato deve soddisfare
i requirement specificati nella descrizione del job (JDL file)
• se piu’ Computing Element soddisfano i requirement allora
Matchmaking
Matchmaking (2/3) (2/3)
scenari possibili
1. direct job submission
- l’utente specifica il Computing Element su cui deve essere sottomesso il job - il Resource Broker non effettua il matchmaking
2. job submission senza requirement su storage resource
- il Resource Broker effettua il matchmaking
interroga l’Information Service per ottenere l’elenco delle risorse che soddisfano i requirement e le credenziali dell’utente
se piu’ Computing Element soddisfano i requirement allora viene
scelto il Computing Element con Rank piu’ alto
Matchmaking
Matchmaking (3/3) (3/3)
3. job submission con requirement su storage resource
- il Resource Broker effettua il matchmaking
interroga il Data Management Service per ottenere l’elenco degli Storage Element che contengono i file richiesti
interroga l’Information Service per ottenere l’elenco delle risorse che soddisfano i requirement e le credenziali dell’utente
seleziona (dalla lista di Computing Element ottenuta) i Computing Element “vicini” agli Storage Element ottenuti al passo 1
se il risultato della selezione consiste in piu’ Computing Element
Job submission
Job submission (1/10) (1/10)
UI JDL
Logging &
Bookkeeping (LB)
Resource Broker (RB)
Job Submission Service (JSS)
Storage Element (SE) Information
Service (IS) Replica
Catalogue
(RC)
Job submission
Job submission (2/10) (2/10)
UI JDL
Logging &
Bookkeeping (LB)
Resource Broker (RB)
Job Submission Service (JSS)
Storage Element (SE) Information
Service (IS) Replica
Catalogue (RC)
Job Submit Event
Input Sandbox
submitted
Job Status
Job submission
Job submission (3/10) (3/10)
UI JDL
Logging &
Bookkeeping (LB)
Resource Broker (RB)
Job Submission Service (JSS)
Information Service (IS) Replica
Catalogue (RC)
submitted
waiting
Storage Element (SE)
Job Status
Job submission
Job submission (4/10) (4/10)
UI JDL
Logging &
Bookkeeping (LB)
Resource Broker (RB)
Job Submission Service (JSS)
Information Service (IS) Replica
Catalogue (RC)
submitted
waiting
ready
Storage Element (SE)
Job Status
Job submission
Job submission (5/10) (5/10)
UI JDL
Logging &
Bookkeeping (LB)
Resource Broker (RB)
Job Submission Service
(JSS)
Information Service (IS) Replica
Catalogue (RC)
submitted
waiting
ready
BrokerInfo
scheduled
Storage Element (SE)
Job Status
Job submission
Job submission (6/10) (6/10)
UI JDL
Logging &
Bookkeeping (LB)
Resource Broker (RB)
Job Submission Service (JSS)
Information Service (IS) Replica
Catalogue (RC)
submitted
waiting
ready
scheduled Input Sandbox
running
Storage Element (SE)
Job Status
Job submission
Job submission (7/10) (7/10)
UI JDL
Logging &
Bookkeeping (LB)
Resource Broker (RB)
Job Submission Service (JSS)
Information Service (IS) Replica
Catalogue (RC)
submitted
waiting
ready
scheduled
running
Storage Element (SE)
Job Status
Job submission
Job submission (8/10) (8/10)
UI JDL
Logging &
Bookkeeping
Resource Broker
Job Submission Service
Information Service
Replica Catalogue (RC)
submitted
waiting
ready
scheduled
running
done
Job Status
Job submission
Job submission (9/10) (9/10)
UI JDL
Logging &
Bookkeeping
Resource Broker
Job Submission Service
Information Service
Replica Catalogue (RC)
submitted
waiting
ready
scheduled
running
done
outputready
Job Status
UI JDL
Logging &
Bookkeeping (LB)
Resource Broker (RB)
Job Submission Service (JS)
Storage Element (SE) Information Service (IS) Replica
Catalogue (RC)
Output Sandbox
submitted
waiting
ready
scheduled
running
done
outputready
Job submission
Job submission (10/10) (10/10) Job Status
SUBMITTED
WAITING
READY
SCHEDULED
RUNNING
DONE(ok) DONE(failed)
OUTPUTREADY
ABORTED DONE(cancelled)
Job status
Job status
Job submission
Job submission (1/10) (1/10)
Job submission
Job submission (2/10) (2/10)
dg-job-submit myjob.jdl
Myjob.jdl
Executable = "$(CMS)/exe/sum.exe";
InputData = "LF:testbed0-00019";
ReplicaCatalog = "ldap://sunlab2g.cnaf.infn.it:2010/rc=WP2 INFN Test Replica Catalog,dc=sunlab2g, dc=cnaf, dc=infn, dc=it";
DataAccessProtocol = "gridftp";
InputSandbox = {"/home/user/WP1testC","/home/file*”,
"/home/user/DATA/*"};
OutputSandbox = {“sim.err”, “test.out”, “sim.log"};
Requirements = other.Architecture == "INTEL"
&& other.OpSys== "LINUX Red Hat 6.2";
Rank = other.FreeCPUs;
