COOL Performance Tests - PowerPoint Presentation

Slide1

COOL Performance Tests ATLAS Conditions Database example

Romain

Basset, IT-DM

October 2008

OutlineDescriptionATLAS conditions databaseDifferent performance testsDatabase population

Results

Conclusion

DescriptionATLAS COOL reference workloads

Source :

https://twiki.cern.ch/twiki/bin/view/Atlas/CoolRefWork

, R.

Hawkings

.

Folder(s)

N-Folders

N-channels

Data

#per run

kB

/run

ID/POOL/

POOLDIR_n

2

32

string(160 char)

1

10

CALO/POOL/

POOLDIR_n

17

32

string(160 char)

1

87

MUON/MDT/RT

1

1174

CLOB 4.5k

0.1

500

MUON/MDT/T0

1

1174

CLOB 3k

0.1

350

MUON/POOL/POOLDIR

1

32

string(160 char)

1

5

GLOBAL/DETSTATUS/TISUMM

1

50

3 floats

6

3.6

TDAQ/DCS/TESTDCS_n

10

200

25 floats

12

2400

TDAQ/DCS/TESTDCS_n

5

1000

25 floats

12

6000

Performance testsTwo main cases :Insertion from online or from calibration jobsData readback

from reconstruction jobs

Two types of data in the DB schema :

Simple version folders (DCS)

Multi-version folders (tags, user tags)

The tests should cover both cases with a mix of both types of data.

Performance tests2 other questions with performance implications:Is the Oracle partitioning a possible solution to ease data management in few years ?Is index compression a good solution to decrease the size of the DB ?

To answer these questions, we will also compare the performance of partitioned tables and compressed index to the normal schema.

Database populationA test DB had to be generated from scratch.The AthenaDBTestRec package provided in the ATLAS software was used to insert and retrieve the data.

Several clients were running in

parallel, but it

was not possible to split the work for the DCS data.

The

duplicates for partitioned tables and index compression were created

using Data Pump to accelerate the process.

Test datasetsList of the different datasets :

Data type

Amount (year)

Size (GB)

Rows

ATLAS COOL DB

1

154

-

DCS

1

24

60,000,000

DCS

5

120

300,000,000

DCS

10

230

600,000,000

DCS partitioned w/ global index

10

211

600,000,000

DCS partitioned w/ local index

10

260

600,000,000

DCS w/ comp. index

10

200

600,000,000

MV w/ CLOB

1

0.5

1,100,000

MV w/ CLOB

5

2.5

5,500,000

MV w/ CLOB

10

5

11,000,000

MV CLOB w/ comp. index

10

3.7

11,000,000

10-year dataset : SV folder

The equivalent of a 10 years old SV folder was populated in about

5

days.

Inserting in a SV folderAs seen in previous slide, insertion in a SV folder remains stable over time.Only 1% increase after 10 years.

10-year dataset : MV folder

The equivalent of a 10 years old MV folder was populated in about

2

days.

Inserting in a MV folderAs seen in previous slide, insertion in a MV folder remains stable over time.The increase is less than 2% after 10 years.

Data retrieval in SV folder

Data retrieval in MV folder

Table partitioningTo ease the data management on the long term, we proposed to use Oracle partitioning (eg: to put old data offline).The partitioning should be transparent to the software to reduce the implementation cost.

Several problems were found with this idea

(IOV overlap…)

as well as performance issues.

Partitioning problemReadback test on a partitioned table

ConclusionThe tests show no performance problem when scaling up to a 10-year dataset.The parallel readback tests are the next in line. The preliminary tests by R.

Hawkings

are good so we are confident.

Partitioning and index compression are still being tested.

Questions ? / Discussion ?