DB2 - e-Learning

Transcription

DB2 - e-Learning

IBM Software Group
Scelte tecnologiche di gestione su DBMS centralizzati e distribuiti; servizi automatizzati e ruolo del DBA
DB2 for Linux, Unix & Windows
Preparato per Uni Bicocca - Corso "Architetture Dati"
Dicembre 2010
Michele Benedetti & Francesco Airoldi
Software Group - IBM Italia
© 2010 IBM Corporation
DB2 Information Management
Agenda
Quick review of some DB2 basics
Part 1: how DB2 implements data access, integrity and security
• Transactions: Concurrency, Locking
• Transactions: Recovery
• Backup & Restore
• Security features
• Optimizer
Part 2: aspects of DB2 database administration
• Automated tasks and autonomic capabilities
• High-value DBA tasks
2
3
High level map
End user
Applications
Other
develop
tools
Developer
Other
system
tools
SQL, XQuery
3 Backup & Restore
DBMS engine
Develop
tools
Optimizer
1
5
DBA
tools
&
utilities
Transactions:
concurrency, locking
7
High-value
tasks
2
Transactions:
recovery
DBA
6
Automated tasks &
autonomic capabilities
4
Security
features
Databases
(relational & XML)
4
DB2 architectures
Appl
DB2 has three different proposition
DB2
• Single SMP Server
disk
• Intensive write/update workload
• Scale up
• Shared Nothing Clusters
• Intensive Read workload (BI)
• I/O always in parallel
Appl
DB2
DB2
DB2
DB2
disk
disk
disk
disk
• Scale up & out
• Shared Disk Cluster
• Typical OLTP workload.
•
80% read, 20% write
•
Thousands of users
•
Highest level on SQL execution concurrency
•
Very simple SQL
•
Continuous availability and scalability as main goal
•
Scale up & out
Appl
DB2
DB2
disk
5
DB2 Server Architecture
Instances and databases
• A DB2 instance can
• Access more than one
database at a time
• Most production databases
however are managed by one
instance
• An instance is also called "Data
Base Manager" (DBM)
DB2 Instance: db2inst1
Catalog
Database
PROD_DB3
Log
Catalog
Database
PROD_DB5
Log
DBconfig
Buffer
Pools
DBconfig
Buffer
Pools
• In a DB2 instance
• Many threads are shared by all
databases
• Instance configuration
parameter (dbm cfg) affects all
databases in that instance
• Database configuration (db cfg)
parameters exist for each
database
DBMconfig
DB2 Instance: db2inst2
Catalog
Log
Catalog
SET db2instance = db2inst1
Database
TEST_DB3
Database
TEST_DB5
Log
DBconfig
Buffer
Pools
DBconfig
Buffer
Pools
db2start
DBMconfig
6
DB2 Database Storage Model
DB2 Storage Structure Hierarchy
Logical
Physical
Database
Table space
Container
Object
(Tables – Indexes)
Extent
DB2 Page
OS Page
7
DB2 Administration Tools
DB2 Control
Center
DB2 Health
Center
DB2 Command
Window (CLP)
DB2 Task
Center
8
New: Infosphere Optim Data Studio Portfolio
Menu bar
Tool bar
Perspective
(a collection of
views and
editors)
Data
Project
Explorer
view
Editor view
Drag &
drop views
anywhere
Data
Source
Explorer
view
SQL Results
view
Resize button
9
Part 1
Part 1
How DB2 implements
data access, integrity and security
10
Transactions: concurrency, locking
Transactions: Concurrency & Locking
11
ACID
1 Transactions: concurrency, locking
ACID properties are fundamental for every database management system
In a perfect transaction world, a transaction must contain a series of properties known as ACID. These properties are:
Atomicity
A transaction is an atomic unit of work or collection of separate operations. So, a transaction succeeds and is committed to
the database only when all the separate operations succeed. On the other hand, if any single operations fail during the
transaction, everything will be considered as failed and must be rolled back if it is already taken place. Thus, Atomicity helps
to avoid data inconsistencies in database by eliminating the chance of processing a part of operations only.
Consistency
A transaction must leave the database into a consistent state whether or not it is completed successfully. The data modified
by the transaction must comply with all the constraints in order to maintain integrity.
Isolation
Every transaction has a well defined boundary. One transaction will never affect another transaction running at the same
time. Data modifications made by one transaction must be isolated from the data modification made by all other
transactions. A transaction sees data in the state as it was before the second transaction modification takes place or in the
state as the second transaction completed, but under any circumstance a transaction can not be in any intermediate state.
Durability
If a transaction succeeds, the updates are stored in permanent media even if the database crashes immediately after the
application performs a commit operation. Transaction logs are maintained so that the database can be restored to its
original position before failure takes place.
A number of features in DB2 (concurrency management, locking, logging, recovery...) are designed
and implemented to ensure full compliance with these ACID properties in every possible situation
12
Basic concepts
• Database Transactions
• Concurrency
• Concurrency Issues
• Concurrency Control
• Isolation Levels
• Locking
• Specifying Isolation Levels
13
Database Transactions
• Transaction
• sequence of one or more SQL operations, grouped together
as a single unit
• also known as a unit of work (uow) or logical unit of work (luw)
• Committed Data
• using the COMMIT statement commits any changes made
during the transaction to the database
• Uncommitted Data
• changes during the transaction before the COMMIT statement
is executed
14
Concurrency
• Concurrency
• Sharing of resources by multiple interactive users or application
programs at the same time
• Having multiple interactive users can lead to:
•
•
•
•
Lost Update
Uncommitted Read
Non-repeatable Read
Phantom Read
• Need to be able to control the degree of concurrency:
• With proper amount of data stability
• Without loss of performance
15
Concurrency Issues
• Lost Update
• Occurs when two transactions read and then attempt to update the
same data, the second update will overwrite the first update before
it is committed
1) Two applications, A and B, both read the same row and
calculate new values for one of the columns based on the
data that these applications read
2) A updates the row
3) Then B also updates the row
4) A's update is lost
16
Concurrency Issues...
• Uncommitted Read or Dirty Read
• Occurs when uncommitted data is read during a transaction
• Also known as a Dirty Read
1) Application A updates a value
2) Application B reads that value before it is committed
3) A backs out of that update
4) Calculations performed by B are based on the uncommitted data
17
• Non-repeatable Read
• Occurs when a transaction reads the same row of data twice and
returns different data values with each read
1) Application A reads a row before processing other requests
2) Application B modifies or deletes the row and commits the change
3) A attempts to read the original row again
4) A sees the modified row or discovers that the original row has
been deleted
18
• Phantom Read
• Occurs when a search based on some criterion returns
additional rows after consecutive searches during a transaction
1) Application A executes a query that reads a set of rows based
on some search criterion
2) Application B inserts new data that would satisfy application A's
query
3) Application A executes its query again, within the same unit of
work, and some additional phantom values are returned
19
Concurrency Control
■
Isolation Levels
– determine how data is locked or isolated from other concurrently executing
processes while the data is being accessed
– are in effect while the transaction is in progress
■
There are four (or five?) levels of isolation in DB2:
– Repeatable Read (RR)
– Read Stability (RS)
– Cursor stability (CS)*
• Currently Committed)**
– Uncommitted read
–
–
–
Å ANSI Serializable
Å ANSI Repeatable Read
Å ANSI Read Committed
Å Equivalent to Oracle “Statement Level Snapshot” IL***
Å ANSI Read Uncommitted
* default Isolation Level value up to DB2 9.5
** new in DB2 9.7 and new default Isolation Level
*** Not available in ANSI Standards
20
Locking
• Isolation levels are enforced by locks
• locks limit or prevent data access by concurrent users or
application processes
• Locking Attributes
• resource being locked is called object
• objects which can be explicitly locked are databases, tables and
table spaces
• objects which can be implicitly locked are rows, index keys, and
tables
• implicit locks are acquired by DB2 according to isolation level and
processing situations
• object being locked represents granularity of lock
• length of time a lock is held is called duration and is affected by
isolation level
21
Types of Locks
(simplified overview)
• Share (S)
•
concurrent transactions are limited to read-only operations
• Update (U)
•
•
concurrent transactions are limited to read-only operations
if the transactions have not declared that they might update a row, the database
manager assumes that transaction currently looking at a row might update it
• Exclusive (X)
•
•
concurrent transactions are prevented from accessing the data in any way
does not apply to transactions with an isolation level of UR
• Database manager places exclusive locks on every
row that is inserted, updated, or deleted
22
Types of Locks...
http://publib.boulder.ibm.com/infocenter/db2luw/v9r7/index.jsp
(detailed view)
then search for "Lock management"
23
Types of Locks...
lock compatibility
24
Deadlock
Deadlock Detector
• discovers deadlock cycles
• randomly selects one of
the transactions involved
to roll back and terminate
• transaction chosen is then
sent an SQL error code,
and every lock it had
acquired is released
What is a deadlock?
A deadlock occurs when two separate processes compete
for resources held by one another.
For example, consider the following processing sequence
for two concurrently executing application programs:
Job A
Update Table A/Page 1
Lock established
Intermediate processing
Update Table B/Page 1
Lock (wait)
Job B
Update Table B/Page 1
Lock established
Intermediate processing
Update Table A/Page 1
DEADLOCK Lock (wait)
A deadlock occurs when Job A requests a lock for a data
page held by Job B, and Job B requests a lock for a data
page held by Job A. Both jobs are waiting on the other to
finish before they can proceed.
A deadlock must be resolved before either program can
perform subsequent processing.
DB2's solution is to target one of the two programs as the
victim of the deadlock and deny that program's lock
request by setting the SQLCODE to -911.
25
Repeatable Read
• Highest level of isolation
• No dirty reads, non-repeatable reads or phantom reads
• Locks the entire table or view being scanned
for a query
• Provides minimum concurrency
• When to use Repeatable Read:
• Changes to the result set are unacceptable
• Data stability is more important than performance
26
Read Stability
• Similar to Repeatable Read but not as strict
• No dirty reads or non-repeatable reads
• Phantom reads can occur
• Locks only the retrieved or modified rows in a table
or view
• When to use Read Stability:
• Application needs to operate in a concurrent environment
• Qualifying rows must remain stable for the duration of the unit
of work
• Only issue unique queries during a unit of work
• If the same query is issued more than once during a unit of
work, the same result set should not be required
27
Cursor Stability
• Default isolation level up to DB2 9.5
• No dirty reads
• Non-repeatable reads and phantom reads can occur
• Locks only the row currently referenced by the cursor
• When to use Cursor Stability:
• Want maximum concurrency while seeing only committed data
28
Currently Committed: new default for
DB2 9.7 and higher
• Currently Committed is a variation on Cursor Stability
• Increase TX concurrency (requires specific application logic)
• Log based:
• No management overhead
Cursor Stability
Currently Committed
Situation
Result
Situation
Result
Reader blocks Reader
No
Reader blocks Reader
No
Reader blocks Writer
Maybe
Reader blocks Writer
No
Writer blocks Reader
Yes
Writer blocks Reader
No
Writer blocks Writer
Yes
Writer blocks Writer
Yes
ANSI standard
Implemented in DB2 from V1
Not an ANSI standard
Implemented in Oracle since its birth
Now implemented also in DB2
29
Currently Committed
• Up to DB2 9.5
• Cursor Stability is the default isolation level
• Now in DB2 9.7
• Currently Committed is the default for NEW databases
• Currently Committed is disabled for upgraded databases, i.e.,
Cursor Stability semantics are used
• Applications that depend on the old behavior (writers
blocking readers) will need to update their logic or
disable the Currently Committed semantics
30
Currently Committed Semantics
An example
Transaction A
Transaction B
update T1 set col1 = ? where col2 = 2
update T2 set col1 = ? where col2 = ?
select * from T2 where col2 >= ?
select * from T1 where col5 = ? and col2 = ?
commit
commit
No locking
Reads last committed version
of the data
No locking
Reads last committed version
of the data
No
Nodeadlocks,
deadlocks,no
notimeouts
timeoutsin
inthis
thisscenario!
scenario!
31
Currently Committed – How to use it?
• cur_commit – DB config parameter
• ON: default for new DBs created in DB2 9.7 – CC semantics in place
• DISABLED: default value for existing DBs – old CS semantics in place
• PRECOMPILE/BIND
• CONCURRENTACCESSRESOLUTION: Specifies the concurrent
access resolution to use for statements in the package.
• USE CURRENTLY COMMITTED
• WAIT FOR OUTCOME
32
Uncommitted Read
• Lowest level of isolation
• Dirty reads, non-repeatable reads and phantom reads can occur
• Locks only rows being modified in a transaction
involving DROP or ALTER TABLE
• Provides maximum concurrency
• When to use Uncommitted Read:
• Querying read-only tables
• Using only SELECT statements
• Retrieving uncommitted data is acceptable
• Uncommitted Read with Read-Write tables
• UR behaves like CS with updateable cursors
33
Isolation Levels
Summary
Isolation Level
Dirty Read
Non-repeatable
Read
Phantom
Read
Repeatable Read (RR)
-
-
-
Read Stability (RS)
-
-
Possible
Cursor Stability (CS) and CS
with Currenlty Committed
-
Possible
Possible
Uncommitted read (UR)
Possible
Possible
Possible
DEFAULT
Application Type
High data stability
required
High data stability not
required
Read-write transactions
Read Stability (RS)
Cursor Stability (CS)
Read-only transactions
Repeatable Read (RR) or
Read Stability (RS)
Uncommited Read (UR)
34
Specifying Isolation Levels
• Precompile / Bind
• ISOLATION option of PREP or BIND command
• Can determine isolation level of a package by executing the
following query
SELECT ISOLATION FROM syscat.packages
WHERE pkgname = 'pkgname'
AND pkgschema = 'pkgschema'
• Statement Level
• Use the WITH {RR, RS, CS, UR} clause
• The WITH UR option applies only to read-only operations
• ensure that a result table is read-only by specifying FOR READ ONLY
in the SQL statement
• Overrides the isolation level specified for the package
SELECT * FROM tb1 WITH RR
35
Specifying Isolation Levels
• Dynamic SQL within the current session
• SET CURRENT ISOLATION
• For all subsequent dynamic SQL statements within the same
session
• JDBC or SQLJ at run time
• SQLJ profile customizer (db2sqljcustomize command)
• CLI or ODBC at run time
• CHANGE ISOLATION LEVEL command specified during the
program preparation process
CHANGE ISOLATION LEVEL TO RR
36
Transactions: Recovery
Transactions: Recovery
37
Concetti Generali di Recovery
2 Transactions: recovery
• Chiarire che cosa si intende per “Recovery” di un
database
• Differenze tra Recovery e Disaster Recovery
• L’unità minima di recovery
• Recovery di un database vs recovery di un server
• Implementazione di una strategia di Recovery
• Asset necessari: log transazionali, salvataggi periodici, dischi/nastri, etc.
• Autorizzazioni necessarie
• Capire
• che cosa devo predisporre per poter ripristinare un database che
presenta dei problemi e che non è più accessibile
• quali informazioni troverò nel database dopo il ripristino
38
DB2 Logging
Il "logging" (registrazione continua delle attività sul db)
è il meccanismo fondamentale per il recovery, e anche
per il backup / restore
• Come DB2 gestisce gli aggiornamenti dei dati perché possano
essere “recoverabili”
• Log Buffer e Log Files
• Log Circolare vs Log Archiviabile
• Che cosa succede quando si esegue un Roll Back
• Che cosa significa Crash Recovery
• Automatismi di gestione delle sequenze di DB2 V8 e V9
39
DB2 Logging – Overview
Log Buffer
y
MINCOMMIT
y
SOFTMAX
Online Active Log
Files
asynchronous
write
db2agent
db2pclnr
on COMMIT/ ROLLBACK
db2loggr
synchronous
write
Buffer Pool
when triggered
(chngpgs_threshold)
Database Files
40
ARIES – Write-ahead Logging
• DB2 uses ARIES (Algorithm for Recovery and Isolation Exploiting
Semantics) as the transaction recovery method supporting finegranularity locking and partial rollbacks using write-ahead logging
• ARIES was invented in the IBM Almaden Research Center by Dr. C. Mohan
• Write-ahead logging:
• Must force the log record for an update before the corresponding data page gets
to disk: Guarantees Atomicity (all actions in a transaction happen, or none
happen)
• Must write all log records for a transaction before commit: Guarantees Durability
(if a transactions commits, its effects persist)
• How is it done:
• Each log record has an unique Log Sequence Number (LSN)
• Each data page contains a pageLSN
• System keeps track of flushed LSN: before a page is written, ensure
pageLSN <= flushed LSN
41
DB2 Logging – Circular Logging
Database Configuration:
USEREXIT = OFF
LOGRETAIN = OFF
1
"n"
PRIMARY
2
1
"n"
SECONDARY
3
42
DB2 Logging – Archival Logging
Database Configuration:
USEREXIT = ON
LOGRETAIN = RECOVERY +
Others for variuos
methods – Use wizard
DB2 calls
db2uext2
12
ACTIVE
Contains information
for non-committed trx
13
14
OFFLINE ARCHIVE
Archive moved from
ACTIVE log subdirectory.
(may also be on other media)
ONLINE ARCHIVE
Contains information
for committed trx.
Stored in the ACTIVE
log subdirectory.
15
16
43
Infinite Logging: nuovo da DB2 8.2
• Va configurato con l’apposito Wizard del DB2 CC
• Logica di funzionamento
• Contrariamente al vecchio LOG Archiving utilizza i log secondari
• Log secondari allocati sino ad esaurimento disco
• Se disco esaurito:
• Rollback delle transazioni non completate
• Sospensione degli updates per aggiungere spazio disco
• Possibilità di controllare l’uso dello spazio log da parte di una singola
transazione
• Vantaggioso in caso di esecuzione di workload molto diversi
(es: batch notturni vs online di tipo OLTP)
44
Transaction Log Space Usage
• MAX_LOG (Maximum Log Space Per Transaction)
DB CFG Parameter
• Maximum active log space consumed by one transaction as a percent of primary log
space
• Values: Minimum=0 (ignore), maximum=100
• NUM_LOG_SPAN - Number of Logs Spanned – DB CFG Parameter
• Number of active log files one active unit of work is allowed to span (Also useful for
eliminating idle applications that span multiple logs)
• Values:
Minimum=0 (ignore), maximum=65535
• DB2_FORCE_APP_ON_MAX_LOG – db2set Variable
• Determines error handling when MAX_LOG is reached
• TRUE (default):
Any application exceeding the MAX_LOG will be forced off and rolled back
• FALSE:
The application will receive a SQL1224N error message (Statement that caused
the problem will fail but the application can still issue a ROLLBACK or COMMIT)
45
Block Transactions On Log Disk Full
• DB2 will not report any error because of disk full in the active log
path
• Will attempt to create the log file again after 5 minutes
• Will continue this behavior until log is successfully created
• Record written to the db2diag.Log whenever a log create fails
• Applications that are updating data will not complete their
operation until the log has been created
• Read-only queries are normally not affected
• Read-only queries may block if reading query is dependant on rows being locked by an
update transactions
• On V7 set via DB2 registry variable:
DB2_BLOCK_ON_LOG_DISK_FULL=[ON|OFF]
• On V8, V9 database configuration parameter:
BLK_LOG_DSK_FUL
46
Roll Back & Crash Recovery
• Definizione: Una transazione si dice Recuperabile/Ripristinabile
(ricoverabile) quando tutti gli updates generati sono stati scritti nei Log
transazionali. In tal caso la transazione si dice anche “esternalizzata”.
Una transazione è “completata” quando è stato eseguito un “Commit”
per confermare gli aggiornamenti o un "Roll Back" per rimuoverli.
• Durante il Roll Back, DB2 rilegge tutti i log files che contengono
informazioni sulla transazione alla ricerca dello stato del dato precedente
ciascun aggiornamento.
• Se l’istanza DB2 viene chiusa forzatamente, alla successiva riattivazione
del Database DB2 esegue un processo di Crash Recovery che
comporta la rilettura del LOG per poter eventualmente effettuare un Roll
Back di transazioni non completate per avere certezza della consistenza
del DB stesso. Durante tale fase il DB non è accessibile.
47
Log File Naming
0000000
Sxxxxxxx.LOG
9999999
S
~0.LOG
S ~1.LOG
S
~2.LOG
S ~3.LOG
S ~4.LOG
BACKUP
y When roll-forward is enabled
y RESTORE DATABASE
without rolling forward
y When S9999999.LOG is filled
y ROLLFORWARD DATABASE
y When roll-forward is disabled
To some point in S ~ 2.LOG
RESTART NAMING SEQUENCE
48
Closing Active Log File On Demand
• Command allows users to close and archive the active
log at any time
• Allows users to acquire a complete set of log files up to the point
in time at which the command is executed
• Only useable on recoverable databases
• Command:
db2 archive log for database <db-name>
• Usage notes
• If user exit is enabled, an archive request is issued after the logs
are closed and truncated
• Performance may be impacted during execution of the command
49
Dual Logging
• High availability feature to prevent log failures in providing a
second copy of log files
• If an error is encountered on either log path, that path will no longer be
used until the database attempts to access the next log file
• On DB2 V7 enabled through the DB2 registry variable
DB2_NEWLOGPATH2=ON
• Secondary path is current LOGPATH concatenated with the character '2‘
For example: if LOGPATH is "/db2/U9C/log_dir", then the secondary
path for logging will be "/db2/U9C/log_dir2“
• On DB2 V8,V9 set via the MIRRORLOGPATH database
configuration parameter
• Can be set to any valid file system/ directory name
50
DB2 Roll Forward Versus Restart Recovery
Two types of DB2 recovery:
• Restart recovery (aka crash recovery)
• DB2 reruns all transactions that were performed previously but
have not been written to disk
• The DB2 log control file header contains the log sequence
number, which must be used as the starting point of the
recovery
• Performed automatically during database restart
(if DB parameter AUTORESTART is set to ON)
• Restore & roll forward recovery
• Manually (using DB2 restore and DB2 roll forward commands)
51
DB2 Restore / Roll Forward Workflow
1. Use db2 list history backup command
to look for a backup image to use
2. Issue db2 restore database command to
restore DB2
3. Retrieve the log files you need using db2uext2
(for online backups at the least those
requested in the history file)
4. Roll forward the database using the db2
rollforward command
5. Complete the roll forward and restart the
database
52
Roll Forward Pending
• Roll forward pending is set as a result of:
• Restore of OFFLINE DATABASE backup omitting the command option
"WITHOUT ROLLING FORWARD"
• Restore of an ONLINE DATABASE backup
• Restore of ANY TABLE SPACE level backup
• DB2 detects media failure isolated at a TABLE SPACE
• Scope of pending state managed by DB2:
• DATABASE in pending state will not permit any activity
• TABLE SPACE(s) in pending state will permit access to other table spaces
53
Roll Forward – How Far?
• End of Logs
• "End" means the end of the current log path
• Other logs may need to be moved in the path
• Point in time
• Specified in Coordinated Universal Time (UTC) or Local Time (Version 8)
• Format: yyyy-mm-dd-hh.mm.ss.nnnnnn
• ONLINE backup
• Requires roll forward past end of backup
• Recovery history file is useful
• Table Space Point in time
• Minimum roll forward time maintained for each table space
• Backup required after roll forward
• Minimum roll forward time determined by DB2
• Can roll forward a subset of table spaces to a specified point in time
• Min PIT is the last DDL change or end of backup, whichever is later
54
Nuovo comando RECOVER
• Introdotto con DB2 8.1
• Unisce in un unico comando le operazioni di:
• Restore
• Roll Forward
• Scelta automatica dell’immagine di backup compatibile con il tipo di
forward recovery richiesto
55
Backup & Restore
Backup & Restore
56
Architectural Components of Backup/Restore
(and Recovery) in DB2
Data Files
(Data Pages)
3 Backup & Restore
Recovery
History File
Backup /
Restore
Recovery
(Transaction)
Logs
57
Concepts
3 Backup & Restore
• What kind of backup strategies can I perform?
• Cold backup vs Hot backup
• Full Backup vs Partial Backup
• For each kind of backup strategy, which are the
prerequisite?
• Circular vs Archive Log methods
• With each backup strategy, which kind of recovery
can I perform?
• Simple offline restore
• Full vs partial Restore
• What about the level of updates recovered?
• Forward recoveries
58
Backup and Recovery: How to
3 Backup & Restore
Logging Options & Components
Defined at database level
Circular
- Full "crash" recovery,
- No roll-forward recovery
Archive Log (aka Log Retain)
- Fully recoverable, DB, TS
- Dual logging & mirrored logging
- 3 recovery log states
1 - Active
2 - Online archived
3 - Off-line archived
59
Archive Logging
3 Backup & Restore
DB2 Archive Log
Data Base Level
DB Shared Memory
Package
Cache, DB
Heap, etc.
Log
Buffer
Buffer
Pool
db2logger
db2pgclnr
logarch
method1
[softmax]
Data Files
Log
Control
File
Recovery
Log
Files
Archive
Log Files
Active Log
Non-Committed or Non-Externalized Transactions
Off-line Archive Log
Online Archive Log
Committed & Externalized Transactions
Files moved from active log
subdirectory to another
media
60
Backup and Recovery: How to...
3 Backup & Restore
DB2 CC Logging-Backup-Recover Configuration Wizards
● Excellent learning tools
● Do all of your backups, restores
and even roll-forward recovery
this way
● Perform and generate most of
the command lines shown in
these slides with these GUI
tools
Launch all four
from the Control
Center
61
Backup and Recovery: How to...
3 Backup & Restore
DB2 Control Center Logging Configuration Wizard
● Excellent learning tool
● Perform and generate all of
the logging config command
lines shown in these slides
with this GUI tool
● No need to learn config.
parameters for DB2 logging
up front
Screens walk
you through
logging setup
62
DB2 Logging Parameters
Function
Parameter
3 Backup & Restore
Usage
archretrydelay Wait in second between archive log failure attempts. 0 means no retry.
blk_log_dsk_ful Prevent disk full errors from being generated.
failarchpath
Alternate directory for the archive log files if the log archive method specified
fails.
Log Archive Methods 1 & 2 logarchmeth1
These parameters cause the database manager to archive log files to
logarchmeth2
locations that are not the active log path.
Values are OFF, RETAIN, USEREXIT, DISK, TSM, VENDOR
Log Archive Options 1 & 2 logarchopt1
Works with logarchmeth1 & 2 for vendor options if using TSM or VENDOR
logarchopt2
values.
Log Buffer
logbufsz
Amount of memory to use as a buffer for log records in 4-KB pages.
Log File Size
logfilsiz
The size of each configured log, in number of 4-KB pages.
Max Logs per Transaction max_log
Percentage of primary log space that can be consumed by one transaction.
0 means no limit.
Mirror Log Path
mirrorlogpath
Location where an identical copy of the logs will be kept.
New Log Path
newlogpath
Location to change for log path away from default SQLOGDIR.
Number of Commits to
mincommits
Allows you to delay the writing of log records to disk until a minimum number
Group
of commits have been performed.
Number of Archive Retries numarchretry
Number of attempts that will be made to archive log files using the specified
log archive method before they are archived to the path specified by the
failarchpath.
Number of Log Span
num_log_span Number of active log files that an active transaction can span. 0 means no
limit.
Overflow Log Path
overflowlogpath Multi-function parameter used for rollforward and archived log retrieval
operations.
Primary Logs
logprimary
Number of primary logs. Also determines number of logs in active log path.
Secondary Logs
logsecond
Number of secondary logs allocated if needed. -1 is unlimited active logs.
Archive Retry Delay
Block on Log Disk Full
Failover Archive Path
Default
20
NO
NULL
OFF
NULL
8
1000
0
NULL
NULL
1
5
0
NULL
3
2
63
Offline Backup (aka Cold Backup)
3 Backup & Restore
The simplest backup
• Circular logging - usually (except to make a base
backup for an archive logged database)
• Disconnect all users from database (or quiesce or stop)
• Use Backup command
• At database or table space level
• SYSADM, SYSCTRL or SYSMAINT authority needed
BACKUP
BACKUP DATABASE
DATABASE SAMPLE
SAMPLE TO
TO
c:\db2backup;
c:\db2backup;
• Some consider a manual copy of data files to be a “cold
backup” and you can do this with DB2 also, but this really
is not a “DB2 backup”.
64
Online Backup (aka Hot Backup)
3 Backup & Restore
Archive Logging Configuration Required
• Set LOGARCHMETH1 to enable archive logging
• Set required logging parameters
•
•
•
•
LOGFILSZ
LOGBUFSZ
LOGPRIMARY
LOGSECOND
• Set optional logging parameters
•
•
•
•
NEWLOGPATH
MIRRORLOGPATH
BLK_LOG_DSK_FULL
OVERFLOWLOGPATH
• Take a complete offline database backup
• quiesce database…
• backup database…
• unquiesce database…
• Don’t forget the logging wizard can help you do all of this!
65
Online Backup
3 Backup & Restore
Operation & Scope
• BACKUP command enabled
• Database or table space driven
• Logs hold changed pages for committed transactions
• Very first full backup done offline
• Contains
•
•
•
•
•
•
Changed data pages
Unchanged data pages
System files
History files
Control files
Can include logs
• Can be throttled
• Still, minimum activity usually desired during online backup
• Parallelization is automatic but can be overridden if desired
• Requires sysadmin, sysctrl or sysmaint
66
Online Backup: Full vs Partial
3 Backup & Restore
Examples
•
Online Database backup
BACKUP
BACKUPDATABASE
DATABASESAMPLE
SAMPLEONLINE
ONLINETO
TO
c:\db2backup;
c:\db2backup;
•
Online table space backup including logs
BACKUP
BACKUPDATABASE
DATABASESAMPLE
SAMPLETABLESPACE
TABLESPACEuserspace1
userspace1
ONLINE
ONLINETO
TOc:\db2backup
c:\db2backupINCLUDE
INCLUDELOGS;
LOGS;
•
Online database backup compressed & throttled
BACKUP
BACKUPDATABASE
DATABASESAMPLE
SAMPLEONLINE
ONLINETO
TOc:\db2backup
c:\db2backup
COMPRESS
COMPRESSUTIL
UTILIMPACT
IMPACTPRIORITY
PRIORITY50;
50;
67
Restore
3 Backup & Restore
• Full Database
• From offline backup
• From online backups: need considerations
• Partial Database
• Tablespace level
• From offline or online backup: need considerations
• Forward recovery
• Optional step when restoring full offline backups and Archive Log
is enabled
• Mandatory step when restoring full online backups:
• Mandatory step when restoring partial/tablespace backups
68
Restore...
3 Backup & Restore
• Stop or quiesce database
• Use TIMESTAMP from history file of backup desired
(easy to use)
• Use Restore command:
(no roll-forward necessary)
RESTORE
RESTORE DATABASE
DATABASE SAMPLE
SAMPLE
FROM
FROM "c:\db2backup"
"c:\db2backup" TAKE
TAKE AT
AT 20040901144022
20040901144022
WITHOUT
WITHOUT PROMPTING;
PROMPTING;
• Can relocate containers during restore (“redirected restore”)
• DB2 can generate a script for you to do this
• Can restore to a new database in one easy command
69
Restore / Recover
3 Backup & Restore
Database Considerations
• Database restore/recover done offline from online backup
• Full restore/recover brings up & applies all crucial files
automatically
• DB2 RESTORE: used with roll-forward for more control
• DB2 RECOVER: for ease of use
RECOVER
RECOVER DB
DB SAMPLE
SAMPLE TO
TO END
END OF
OF LOGS;
LOGS;
RESTORE
RESTORE DATABASE
DATABASE SAMPLE
SAMPLE FROM
FROM c:\db2backup
c:\db2backup TAKEN
TAKEN
AT
AT 20041201194626
20041201194626 WITHOUT
WITHOUT PROMPTING;
PROMPTING;
ROLLFORWARD
ROLLFORWARD DATABASE
DATABASE SAMPLE
SAMPLE TO
TO END
END OF
OF LOGS
LOGS AND
AND
COMPLETE;
COMPLETE;
• RECOVER & ROLLFORWARD have two types:
• End of logs
• Point in time
70
Online Restore
3 Backup & Restore
Table Space Considerations
• Table space restore can be performed online like this:
RESTORE
RESTORE DATABASE
DATABASE SAMPLE
SAMPLE TABLESPACE(userspace1)
TABLESPACE(userspace1) ONLINE
ONLINE
FROM
FROM c:\db2backup
c:\db2backup
TAKEN
TAKEN AT
AT 20040901201428
20040901201428
WITHOUT
WITHOUT PROMPTING;
PROMPTING;
ROLLFORWARD
ROLLFORWARD DATABASE
DATABASE SAMPLE
SAMPLE TO
TO END
END OF
OF LOGS
LOGS
AND
AND COMPLETE
COMPLETE TABLESPACE(userspace1)
TABLESPACE(userspace1)
ONLINE;
ONLINE;
• Roll-forward has two types:
• End of logs
• Point in time
• You can also rebuild an entire database from table space backups!
71
Incremental Backups
3 Backup & Restore
DB2 Incremental & Delta Backups
Backup all pages changed since the last full backup
Backup all pages changed since the most recent full, incremental or delta back up
72
Incremental Backups
Using
3 Backup & Restore
Pls. Note: immediately after having set the Trackmode to yes a Full backup is required in order
to get synchronized with the changed pages that will be tracked from now on
• Enable database tracking by setting database
parameter TRACKMOD to YES
UPDATE
UPDATE DB
DB CFG
CFG FOR
FOR SAMPLE
SAMPLE USING
USING TRACKMOD
TRACKMOD YES;
YES;
• Full, then incremental database backups:
BACKUP
BACKUP DATABASE
DATABASE SAMPLE
SAMPLE TO
TO c:\db2backup;
c:\db2backup;
BACKUP
BACKUP DATABASE
DATABASE SAMPLE
SAMPLE ONLINE
ONLINE INCREMENTAL
INCREMENTAL TO
TO
c:\db2backup;
c:\db2backup;
BACKUP
BACKUP DATABASE
DATABASE SAMPLE
SAMPLE ONLINE
ONLINE INCREMENTAL
INCREMENTAL
DELTA
DELTA TO
TO c:\db2backup;
c:\db2backup;
73
Incremental Restore
3 Backup & Restore
Using
• To restore without loss of data
• Apply full backup
• Use AUTOMATIC keyword to automatically restore from the full backup and apply
incremental backup images, then Roll-forward
RESTORE
RESTOREDATABASE
DATABASESAMPLE
SAMPLEINCREMENTAL
INCREMENTALAUTOMATIC
AUTOMATIC
FROM
FROMc:\DB2Backup
c:\DB2Backup
TAKEN
TAKENAT
AT20040901230402
20040901230402WITHOUT
WITHOUTPROMPTING;
PROMPTING;
For automatic restore, the timestamp is of
The most recent incremental or delta backup
ROLLFORWARD
ROLLFORWARDDATABASE
DATABASESAMPLE
SAMPLETO
TOEND
ENDOF
OFLOGS
LOGS
AND
ANDCOMPLETE;
COMPLETE;
• Don’t forget the Restore Wizard can help you!
74
Recover Dropped Table
3 Backup & Restore
• Dropped table history is maintained
(if DROPPED TABLE RECOVERY is on)
• Restore table space from previous backup
• Unload dropped Table data
ROLLFORWARD
ROLLFORWARD DB
DB SAMPLE
SAMPLE TO
TO END
END OF
OF LOGS
LOGS AND
AND COMPLETE
COMPLETE
RECOVER
RECOVER DROPPED
DROPPED TABLE
TABLE 00000000000002010002000d
00000000000002010002000d TO
TO
c:\db2backup;
c:\db2backup;
• Recreate table using DDL from recovery history file
• Load the data
LOAD
LOAD FROM
FROM c:\db2backup\node0000\data
c:\db2backup\node0000\data OF
OF DEL
DEL METHOD
METHOD p(1)
p(1)
INSERT
INSERT INTO
INTO recoverme;
recoverme;
75
Backup & Restore
3 Backup & Restore
Supporting Utilities
• db2ckbkp (check backup)
• Checks for completeness of backup images (which also may contain logs)
• db2ckrst (check incremental restore image)
• Helps manage log chains to make sure they are complete for any given incremental restore
• db2adutl (Tivoli Storage Manager object management)
• Allows users to query, extract, verify, and delete backup images, logs, and load copy images
saved using Tivoli Storage Manager
• db2tapemgr
• Stores archived log files to tape
• db2pd (problem determination)
• -LOGS option gives current information on all logs
Self tuning features extended to
BACKUP utility
• Dramatic performance benefits
• In worst case scenario, 4 times
performance gain
• Automatic setting of:
• Buffer size
• Number of buffers
• Parallelism
76
Backup and Recovery
3 Backup & Restore
Recovery History Information
Recovery History File updated during
•
•
•
•
•
•
Database backup, restore, roll-forward or recovery
Table space backup, restore or roll-forward
Table space is created, dropped, altered, renamed or quiesced
Table is loaded, dropped or reorganized
Log file is archived
New log file is written to
list
list history
history backup
backup all
all for
for sample;
sample;
select
select ** from
from table(sysproc.admin_list_hist())
table(sysproc.admin_list_hist())
as
as history;
history;
77
Backup and Recovery
3 Backup & Restore
Automated Management of Recovery Artifacts
• To automatically prune recovery history information for a specific
timeframe, set DB CFG:
• REC_HIS_RETENTN
• To manually prune your recovery history file, use this command:
prune
prune history
history 200412
200412 and
and delete;
delete;
Format YYYYMM
• To automate the cleanup of backup images, log files & load copy
images to coincide with the recovery history file, set DB CFG:
• AUTO_DEL_REC_OBJ
• To keep a specific number of backups, set DB CFG:
• NUM_DB_BACKUPS
78
DB2 Recovery Expert
3 Backup & Restore
Additional DB2 Tool Goes Beyond Core Backup/Restore Functionality
• Row level recovery
• Undo or redo any row or set of rows
• Detailed log activity reports
• Recovery of other SQL
object types
•
•
•
•
•
•
•
•
Stored procedures,
UDFs
Triggers
UDTs
Grants
Monitors
Indexes
…and more
• Grouper utility to recover related objects
•
•
•
•
Database RI
Application logic RI
Trigger relationships
User defined relationships
• Undo for LOB and long fields
• Flash copy backup support
• API interfaces
• Faster recovery options
• Mini logs can be used without requiring a restore & roll forward
• e.g. recover a single table with indexes, triggers & grants without a restore
• Autonomic “multiple recovery path”
• Has estimated recovery “costs”
• You choose path & point-in-time and Recovery Expert does the rest
79
Security Features
Security Features
80
4 Security features
IBM Security Framework
Visibilità
Controllo
Automazione
IBM Security Solutions
Security Compliance
Dimostrazione dell’applicazione delle
policies, in conformità a standard,
normative, leggi, accordi (PCI, FISMA,
ecc…)
Identity and Access
Collaborazione sicura di utenti interni ed
esterni; accesso controllato e sicuro agli
asset aziendali: processi, applicazioni,
informazioni…
IBM Security Framework
Data and Information Security
Protezione e messa in sicurezza dei dati
e degli asset informativi; controllo degli
accessi; monitoring, auditing, compliance
Application Security
Gestione continua, monitoraggio e audit
della sicurezza delle applicazioni in tutto il
loro ciclo di vita
Infrastructure Security
Gestione completa delle minacce e delle
vulnerabilità sulla rete, sui server e sui
sistemi utente
81
4 Security features
Data and Information Security
Protezione del patrimonio informativo aziendale
e conformità a policies, leggi e regolamentazioni
¾ Protezione completa per dati ed informazioni
¾ Dati strutturati (dbms)
¾ Dati semi-strutturati (p.e. xml)
¾ Dati non struttutari (documenti, contenuti…)
Define
Controls
¾ Mascheramento di dati sensibili nella creazione (ed eventuale
esternalizzazione) di ambienti di sviluppo e test
¾ Scoperta, identificazione, classificazione, difesa e controllo di
proprietà intellettuale critica e di informazioni aziendali sensibili
Monitor ,
Audit,
Report
Information
i
¾ Monitoraggio in tempo reale dell’infrastruttura dei dbms aziendali:
sicurezza, integrità, compliance
Retention
Tivoli Access Manager
Discovery
&
Classification
Enforce
Controls
IBM Data Server Security
IBM Optim Data Privacy Solutions
Guardium Solutions
FileNet Compliance Solutions
FileNet Records Manager
FileNet Records Crawler with IBM Classification Module
82
4 Security features
IBM Data Server Security
Data threats
Come utilizzare al meglio le feature di security
dei dbms IBM e di specifici tools
• DB2 z/OS e DB2 for Linux, Unix, Win
• basic security
• encryption, LBAC, identity assertion,
db roles, audit facility…
• Informix IDS
• Audit Management Expert
• Database Encryption Expert
• Optim
• …
Countermeasures
Products
recommended
Config. threats
Audit threats
Exec. threats
83
Security
4 Security features
Terms used by DB2 (and other RDBMS)
Authentication vs. Authorization
Authentication: Determines who the user is by identifying with a password
DB2 use external as default
Authorization: Grants privileges to that user once authenticated
Explicit vs. Implicit Access
Explicit: Granted directly to user, role or group
Implicit: Granted when a database or database object is created
Indirect: Inherited through execution of packaged code
Data Control Language
DCL: Subset of SQL that performs security definitions in the database
Security Views: Where security definitions are kept in the database
Catalog (DB2): SYSCAT.%AUTH
Schemas: Where database objects are logically grouped together
True (DB2) Creates schemas even if a user does not exist
84
DB2 Security Model Overview
AUTHENTICATION
DB2 Client
DB2 Client
4 Security features
DB2 Client
DB2 Client
Authenticate
Here!
Data
Encryption
Data
Encryption
Data
Encryption
Authenticate
Here!
Authenticate
Here!
Instance 1
SERVER
AUTHENTICATION
AUTHORIZATION
Data
Encryption
Plug-ins
Instance 2
Instance 3
CLIENT
AUTHENTICATION
GSSPLUGIN
AUTHENTICATION
System (instance) Authorities
• SYSADM
• SYSCTRL
• SYSMAINT
• SYSMON
Kerberos
Server
Instance 4
KERBEROS
AUTHENTICATION
Authenticate
Here!
Database Authorities
• SECADM
• DBADM
• CONNECT
• CREATETAB
• LOAD
• IMPLICIT_SCHEMA
• QUIESCE_CONNECT
• CREATE_EXTERNAL_ROUTINE
• CREATE_NOT_FENCED_ROUTINE
• BINDADD
• ACCESSCTRL
• DATAACCESS
• SQLADM
• WLMADM
• EXPLAIN
Object Privileges
Table
Index
Package
Workload
Sequence
Security
Label
Nickname
Server
View
Role
XSR
Schema
Routine
Table
Space
Global
Variable
Module
Setsessionuser
85
85
Security Basics
Level: Security in DB2 starts at the instance
User definition: defined outside the database
Authentication: is plug-in controlled
● Security plug-ins (DLLs or exits) for client and server
authentication as well as group management
● Default plug-ins use OS authentication DLLs
● Authentication is by one of these methods
►
►
►
►
Client side
Server side
GSS Plug-in
Kerberos server
Groupings:
● Groups: Supports groups as defined in the OS
● Authorities: Instance or database level groupings of privileges
● Roles: Supports roles similar to Oracle, but can grant to users,
groups or other roles
● Trusted Contexts: “Trusted Connections” established through
Trusted Contexts, can be tied to a role
4 Security features
A role is a database object that
groups together one or more
privileges and can be assigned
to users, groups, PUBLIC, or
other roles by using a GRANT
statement, or can be assigned
to a trusted context by using a
CREATE TRUSTED CONTEXT
or ALTER TRUSTED
CONTEXT statement.
Roles provide several
advantages that make it easier
to manage privileges in a
database system:
• Security administrators
can control access to their
databases in a way that
mirrors the structure of
their organizations (they
can create roles in the
database that map directly
to the job functions in their
organizations).
• ……….
86
DB2 Authorities
Database level
4 Security features
Instance level
87
Grant / Revoke
4 Security features
The GRANT statement allows an authorized user to
grant privileges. A privilege can be granted to one or
more authorization names in one statement; or to
PUBLIC, which makes the privileges available to all
users. Note that an authorization name can be either
an individual user or a group.
The following example grants SELECT privileges on the EMPLOYEE table to the user HERON:
GRANT SELECT ON EMPLOYEE TO USER HERON
The following example grants SELECT privileges on the EMPLOYEE table to the group HERON:
GRANT SELECT ON EMPLOYEE TO GROUP HERON
The REVOKE statement allows authorized users to revoke privileges previously granted to other users.
88
DB2 Label Based Access Control
4 Security features
(Also called “DB2 Advanced Access Control Feature *”)
• Label Based Access Control (LBAC)
• Also called “DB2 Advanced Access Control Feature”
• A “label” is associated with both user sessions and data
rows or columns
SELECT
SELECT* *FROM
FROMEMP
EMP
WHERE
SALARY
>=
WHERE SALARY >=50000
50000; ;
• Rules for comparing users & data labels allow access
controls to be applied at the row level
• Labels may consist of multiple
components
No LBAC
SEC=254
• 1. Hierarchical (tree)
2. Group (array)
3. Set types
• Row labels appear as a single
additional column in a protected table
• Regardless of the number of label
components
• User labels are granted by a
security administrator
• SECADM authority granted by
SYSADM
SEC=100
SEC=50
ID
SALARY
255
60000
100
50000
50
70000
50
45000
60
30000
250
56000
102
82000
100
54000
75
33000
253
46000
90
83000
200
78000
= row returned
* This feature available for Enterprise Server Edition only
89
Trusted Context
Basics
4 Security features
In a typical three tiers scenario (Web + Appl. Server + DB2) the mid tier authorization
is used for all the web users. It is therefore convenient to set up a "trusted context"
for security across the application server and the database server environments.
Trusted context can be used:
•
•
•
•
Setup
To avoid/prevent from loss of user identity
To avoid lacking of accountability/auditing
To avoid over granting of privileges to the middle tier’s authid
To enforce security
CREATE TRUSTED CONTEXT CTX1
BASED UPON CONNECTION USING SYSTEM AUTHID USER1
ATTRIBUTES (ADDRESS '192.0.2.1')
WITH USE FOR USER2 WITH AUTHENTICATION, USER3 WITHOUT
AUTHENTICATION
ENABLE
Usage
Open/establish the
trusted connection
Switch to a user (with or
without authentication)
90
Trusted Context...
4 Security features
Trusted Contexts:
• Are database objects that provide greater control when you use
restricted and/or sensitive privileges
•
userA
userB
userD
Especially useful when many application servers use a single user id
• Allow middle-tier servers or applications to assert the identity of
the end-user to the database
•
userC
Middle Tier
Application Server
Can then acquire additional capabilities not allowed outside the trusted context
•
Implicitly: Application uses trusted context as-is
•
Explicitly: Application can switch users without re-authenticating
• Create a “trusted connection” when a connection attribute
matches trusted context attributes:
•
System authorization ID: Represents the user who establishes a database
connection
•
IP address (or domain name): Represents the host from which a database
connection was established
•
Data stream encryption: Represents the encryption setting (if any) for the data
communication between the database server and the database client
Trusted Contexts defined having:
Trusted Contexts defined having:
[1] System authid – used to establish the connection
[1] System authid – used to establish the connection
[2] Trust attributes – used to establish the connection
[2] Trust attributes – used to establish the connection
[3] Default role (optional)
[3] Default role (optional)
[4] A list of authids allowed to switch (optional)
[4] A list of authids allowed to switch (optional)
Trusted Connection
wasUserB, context2
Trusted Connection
wasUserA, context1
DB2 Database
Context2:
Context1:
wasUserB allow
userC; userD
wasUserA allow
userA; userB
CREATE TRUSTED CONTEXT appsrvercx
CREATE TRUSTED CONTEXT appsrvercx
BASED UPON CONNECTION USING
BASED UPON CONNECTION USING
[1] SYSTEM AUTHID appsrvid
[1] SYSTEM AUTHID appsrvid
ENABLE
ENABLE
[2] ATTRIBUTES (ADDRESS ‘9.13.116.224’ ENCRYPTION HIGH)
[2] ATTRIBUTES (ADDRESS ‘9.13.116.224’ ENCRYPTION HIGH)
[3] DEFAULT ROLE appsrvrl
[3] DEFAULT ROLE appsrvrl
[4] WITH USE FOR usermgr WITH AUTHENTICATION ROLE mgrrole;
[4] WITH USE FOR usermgr WITH AUTHENTICATION ROLE mgrrole;
91
DB2 Secure Sockets Layer (SSL) and
Transport Layer Security (TLS)
4 Security features
• DB2 can use both SSL & TLS
• Provides Federal Information Processing Standard (FIPS) compliance
• Enable a client to authenticate a server
• By exchanging digital certificates
In cryptography,
the Advanced
Encryption
Standard (AES) is
an encryption
standard adopted
by the U.S.
government.
• Provide private communication between client and server
• By using AES encryption
• Easy setup through configuration parameters
• There is an SSL handshake that does the following:
•
•
•
•
•
•
Client requests an SSL connection listing its SSL version and supported cipher suites
Server responds with a selected cipher suite
Server sends its digital certificate to the client
Client verifies the validity of the server's certificate (server authentication)
Client and server securely negotiate a session key
Client and server securely exchange information using the key selected above
Signer
Certificate
Database
iKeyman tool
JCC Client
SSL (JSSE)
TCP/IP
DB2 Server
Encrypted
Communication
SSL (GSKit)
TCP/IP
Digital
Certificate
Database
iKeyman tool
92
DB2 Audit
•
Managed by SYSADM and SECADM
•
•
•
•
4 Security features
At Instance level by SYSADM by using db2audit tool
At database level by SECADM by using AUDIT POLICY and AUDIT commands
Audit is granular to the level of table, user, group, role, SYSADM, DBADM, SECADM
Audit Features
•
•
•
•
•
•
•
Audit logs are separate for each database
DBAs should ask
Audit logs location can be changed
application developers to
start using CURRENT
Audit logs can be archived
CLIENT_* registers in
Audit policies can be created to control audit configuration within a database
applications for better
Individual databases and objects can have their own audit configurations
audits
Auditing individual SQL statements is completely configurable
CURRENT CLIENT special registers allows client information to be put in audit logs
SYSADM (Instance Level - db2audit)
db2audit configure datapath "C:\db2audit"
db2audit configure datapath "C:\db2audit"
archivepath "C:\db2auditArchive"
archivepath "C:\db2auditArchive"
db2audit configure scope all status both
db2audit configure scope all status both
db2audit start
db2audit start
[…. SQL statements that trigger auditing]
[…. SQL statements that trigger auditing]
db2audit flush
db2audit flush
db2audit stop
db2audit stop
db2audit archive
db2audit archive
db2audit extract delasc to c:\db2audit from files
db2audit extract delasc to c:\db2audit from files
c:\db2auditarchive\db2audit.instance.
c:\db2auditarchive\db2audit.instance.
log.0.20071215103050
log.0.20071215103050
SECADM (Database Level)
db2 CREATE AUDIT POLICY SENSITIVE_DATA_POLICY
db2 CREATE AUDIT POLICY SENSITIVE_DATA_POLICY
CATEGORIES EXECUTE WITH DATA STATUS BOTH
CATEGORIES EXECUTE WITH DATA STATUS BOTH
ERROR TYPE AUDIT
ERROR TYPE AUDIT
db2 AUDIT TABLE EMPLOYEE USING POLICY
db2 AUDIT TABLE EMPLOYEE USING POLICY
SENSITIVE_DATA_POLICY
SENSITIVE_DATA_POLICY
db2 CALL SYSPROC.AUDIT_ARCHIVE('C:\db2auditArchive', -2 )
db2 CALL SYSPROC.AUDIT_ARCHIVE('C:\db2auditArchive', -2 )
db2 CALL SYSPROC.AUDIT_DELIM_EXTRACT
db2 CALL SYSPROC.AUDIT_DELIM_EXTRACT
(NULL,'C:\db2auditarchive‘,null, ‘%20071215%',null)
(NULL,'C:\db2auditarchive‘,null, ‘%20071215%',null)
93
Optimizer
Optimizer
94
DB2 Optimizer
5 Optimizer
The optimizer is the heart and soul of DB2. It analyzes SQL statements and determines the most
efficient access path available for satisfying each statement.
DB2 accomplishes this by parsing the SQL statement to determine which tables and columns must be
accessed. The DB2 optimizer then queries system information and statistics stored in the DB2 system
catalog to determine the best method of accomplishing the tasks necessary to satisfy the SQL request.
http://www.globalguideline.com/articles/analysis.php?k=Optimization_of_DB2
95
DB2 Optimizer...
5 Optimizer
Optimizing data access in DB2
The notion of optimizing data access in the DBMS is one of the most powerful capabilities of DB2. Remember, you access DB2 data
by telling DB2 what to retrieve, not how to retrieve it. Regardless of how the data is physically stored and manipulated, DB2 and SQL
can still access that data. This separation of access criteria from physical storage characteristics is called physical data
independence.
DB2's optimizer is the component that accomplishes this physical data independence.
If you remove the indexes, DB2 can still access the data (although less efficiently). If you add a column to the table being accessed,
DB2 can still manipulate the data without changing the program code. This is all possible because the physical access paths to DB2
data are not coded by programmers in application programs, but are generated by DB2.
Compare this with non-DBMS systems in which the programmer must know the physical structure of the data. If there is an index, the
programmer must write appropriate code to use the index. If someone removes the index, the program will not work unless the
programmer makes changes. Not so with DB2 and SQL. All this flexibility is attributable to DB2's capability to optimize data
manipulation requests automatically.
The optimizer performs complex calculations based on a host of information.
To visualize how the optimizer works, picture the optimizer as performing a four-step process:
1 . Receive and verify the syntax of the SQL statement.
2 . Analyze the environment and optimize the method of satisfying the SQL statement.
3. Create machine-readable instructions to execute the optimized SQL.
4. Execute the instructions or store them for future execution.
The second step of this process is the most intriguing. How does the optimizer decide how to execute the vast array of SQL
statements that you can send its way?
The optimizer has many types of strategies for optimizing SQL. How does it choose which of these strategies to use in the optimized
access paths? IBM does not publish the actual, in-depth details of how the optimizer determines the best access path, but the
optimizer is a cost-based optimizer. This means the optimizer will always attempt to formulate an access path for each query that
reduces overall cost. To accomplish this, the DB2 optimizer applies query cost formulas that evaluate and weigh four factors for each
potential access path: the CPU cost, the I/O cost, statistical information in the DB2 system catalog, and the actual SQL statement.
96
DB2 Access Plan
An access plan specifies the order of
operations for accessing data
necessary to resolve a SQL or XQuery
statement
5 Optimizer
Different from
Oracle, DB2
doesn't use
“hints”
Static
statements
Dynamic
statements
Legend
tables
Provide Information
on these objects
and parameters
DB2 automatically detects
outdated statistics and
update them on-the-fly!
Indexes
Table
Functions
operators
97
DB2 SQL Compiler Process
5 Optimizer
Parse Query
Validate Syntax. Query created
and stored in Query Graph
Model.
Check Semantics
Ensure no inconsistencies, verify
data types, etc.
Rewrite Query
Alter entire query into a form
more easily optimized.
Pushdown Analysis Federated database usage only,
remote
or local evaluation
determined.
Optimize Plan
Determine best plan based on
cost.
Remote SQL Gen.
Federated database usage only,
makes possible some part of the
final plan to be remote.
Gen. Exec. Code
Takes plan and Query Graph
Model to build executable access
plan.
98
Have to do with the Optimizer
5 Optimizer
Reorg
Runstats
Learning Optimizer (LEO) --> autonomics
Static & Dynamic SQL
Explain facilities
99
Explain Facilities
5 Optimizer
DB2 Explain Tools Overview
Visual Explain
●
●
●
●
Run from the Control Center or Command Editor or IBM Data Studio
Entirely GUI driven
Provides a visual graph of the plan
Allows drill-down on objects and nodes for more detail
db2exfmt
● Formatter for previously stored explain data from optimizer information only
● Line command driven
● Text output with text based graphs
db2expln
● To see the access plan information available for packages of static SQL statements (static
mode)
● Shows the actual implementation of access plan, not just optimizer information
● Shows on the fly execute and explain or explain from cache (dynamic mode)
● Line command driven
● Text output with text based graphs
100
Explain Facilities
5 Optimizer
DB2 Tool Characteristics
Characteristics Desired
Explain
Tables
GUI Interface
Visual
Explain
db2exfmt db2expln
Yes
Text Output
Yes
Yes
Static SQL supported
Yes
Yes
Yes
Yes
Dynamic SQL supported
Yes
Yes
Yes
Yes
CLI Applications supported
Yes
Yes
Yes
Detailed Optimizer information supplied
Yes
Yes
Yes
Suited for multiple statement analysis
Yes
Information accessible from within
application
Yes
Available to DRDA* application requests
Yes
“Quick & Dirty” static SQL analysis
Yes
Yes
Yes
* Distributed Database Relational Architecture
101
Explain Facilities
5 Optimizer
DB2 Explain Tables
Explain Table
Function
EXPLAIN_INSTANCE
Main control table that has RI to all others.
A DELETE from this table will cascade to all the others.
EXPLAIN_STATEMENT
Contains the text of the SQL statement as it exists for the different levels
of Explain information.
EXPLAIN_ARGUMENT
Represents the unique characteristics for each individual operator, if there
are any.
EXPLAIN_OBJECT
Identifies those data objects required by the access plan generated to
satisfy the SQL statement.
EXPLAIN_OPERATOR
Contains all the operators needed to satisfy the SQL statement by the
SQL compiler.
EXPLAIN_PREDICATE
Identifies which predicates are applied by a specific operator.
EXPLAIN_STREAM
Represents the input and output data streams between individual
operators and data objects.
“Advise” Tables: tables named ADVISE_* used for explain mode processing shown in following slide
102
Explain Facilities
5 Optimizer
DB2 Dynamic SQL Explain Modes
DB2 Dynamic SQL Explain Control:
● Set special register with “SET CURRENT EXPLAIN MODE = [value]”
● Set interactively or embed in an application; it can be dynamically prepared and executed.
Explain Mode
Function
NO (default)
• Disables explain. No explain info. captured.
• SQL statements executed and real-time statistics are used if available.
YES
• Enables explain. Causes explain info. to be captured into explain tables.
• SQL statements executed and real-time statistics are used if available..
EXPLAIN
• Enables explain. Causes explain info. to be captured into explain tables.
• SQL statements not executed.
REOPT
• Enables explain. Causes explain info. to be captured into explain tables but only at
re-optimization time when host variables are resolved.
• SQL statements executed and real-time statistics are used if available.
RECOMMEND INDEXES
• Enables explain. Causes explain info. to be captured into explain tables & populates
ADVISE_INDEX table.
EVALUATE INDEXES
• Enables explain. Uses indexes in ADVISE_INDEX table.
RECOMMEND PARTITIONINGS
• Enables explain. Causes explain info. to be captured into explain tables & populates
ADVISE_PARTITION table.
EVALUATE PARTITIONINGS
• Enables explain. Uses indexes in ADVISE_PARTITION table.
103
5 Optimizer
Explain Facilities
DB2 Static SQL Explain Bind Values
DB2 Static SQL explain control:
● External programs: Explained at bind (or pre-compile) time
In bind (or pre-compile) command, use “EXPLAIN [value]”
● SQL PL routines (stored procedures, UDFs and methods): Explained when created
Global level - use registry variable setting “DB2_SQLROUTINE_PREPOPTS=EXPLAIN [value]”
Session level - use stored procedure setting “SET_ROUTINE_OPTS(EXPLAIN [value])”
Explain Bind Values
Function
NO (default)
• Disables explain. No explain info. captured.
• SQL statements executed.
YES
• Enables explain. Causes explain info. to be captured into explain tables for:
Static SQL at prep / bind time
Reoptimizable incremental statements at run time
SQL statements executed.
REOPT
Dynamic SQL at run time overriding CURRENT EXPLAIN MODE.
ALL
Static SQL at prep / bind time
Dynamic SQL at run time overriding CURRENT EXPLAIN MODE.
104
Explain Facilities
5 Optimizer
Explain Terminology
Timerons
An invented relative unit of measure. Timerons are determined by the optimizer based on internal values such as
statistics that change as the database is used.
Node
Numbered section of each explain plan. To read a plan, start at bottom right and work right to left, bottom to top. There
are two kinds of nodes: Operator (the action that is performed) & Operand (the object the action is performed against)
OPERATOR TYPE
FUNCTION
INSERT, UPDATE, DELETE
Perform the operations named.
FETCH
Retrieves data from a table, given a specific pointer to the record.
TBSCAN
A table scan is examination of an entire table row by row.
Index Scanning –
IXSCAN, EISCAN, RIDSCN
IXSCAN points to rows, sorted for fast access.
EISCAN operates with multiple ranges of values.
RIDSCN retrieves the row pointers (Row IDs) from a set of index records.
XSCAN, XISCAN, XANDOR
XML scan & navigation, XML index access, XML ANDing
FILTER
Reduces a set of row to only those required.
UNIQUE
Removes duplicates
Joins – NLJOIN, MSJOIN, UNION,
IXAND, HSJOIN
Nested Loop, Merge Scan, Union (concatenation), Index ANDing (dynamic bit map), Hash
GRPBY
Group by rows, and performs column function calculations on the groups.
SORT
Sorts rows in the order of specified columns, and optionally eliminates duplicate entries.
TQUEUE
Transfers table data between agents
GENROW
Generates a table of rows
RETURN
Return data from a query to a user
TEMP
Stores a set of results for re-use elsewhere in the query.
105
Explain Facilities
5 Optimizer
DB2 Visual Explain in IBM Data Studio
Operator Type
SHAPE KEY
Tables:
Indexes:
Operators:
Functions:
Tqueues:
Rectangles
Diamonds
Octagons
Hexagons
Parallelograms
Timerons
Operator #
Nodes
106
Explain Facilities
5 Optimizer
DB2 Visual Explain Control Center
Operator Type
SHAPE KEY
Timerons
Tables:
Indexes:
Operators:
Functions:
Tqueues:
Rectangles
Diamonds
Octagons
Hexagons
Parallelograms
Operator #
Nodes
107
Explain Facilities
5 Optimizer
db2exfmt Output
Operator Type
Timerons (can be changed to show other stats)
Operator #
Nodes
Rows
Total Cost
I/O Cost
108
Extra: beyond the "local" optimization
5 Optimizer
"Global Optimizer" in federated environments
Federated
database
server
Federated
database
server
local data
database virtuale
DB2
• memorizzazione locale di risultati
di query federate
• a supporto di prestazioni migliori
• trasparenza applicativa
SQL Srv
non relational
relational
Accesso trasparente a dati eterogenei
Aree di applicabilità
Benefici
• integrazione senza rimpiazzo delle basi dati esistenti
• semplificazione e standardizzazione delle interfacce di
accesso ai dati
• visione unificata di dati su sistemi diversi
standardizzazione nomenclature)
• apertura a dati non strutturati
(p.e.
•
•
•
•
•
•
•
architetture applicative
ambienti analitici
acquisizioni e fusioni aziendali
ambienti multi-sito
portali
processi IT
….
109
IBM InfoSphere Federation Server
5 Optimizer
Federation Server usa il dbms "engine" del DB2, esteso con:
Global optimization
• database statistics
• ulteriori fattori di costo
• network bandwidth
• data source capabilities
• data volumes
• predicate pushdown
Supporto dei
“nicknames”
Caching eterogeneo
• trasparenza applicativa
• migliori prestazioni per le query federate
• estensione delle MQT ad ambienti federati
• i nicknames sono usati
come normali tabelle o
viste
•
•
•
•
memorizzazione permanente di risultati
usate dall’optimizer
enable / disable caching
refresh manuale o per replica
applicazione
EMP - owner J15USER3
EMPNO
100
200
300
400
500
EMPNAME
Smith
Jones
Adams
Miller
Bennett
write
SELECT X_EMP.EMPNAME,
O_OFFICE.OFFICENO
FROM X_EMP, O_OFFICE
WHERE X_EMP.EMPNO=
OFFICE - owner J15USER1 O_OFFICE.EMPNO
DB2 for z/OS
Oracle
EMPNO
OFFICENO
100
200
300
400
500
C200
C202
C204
C206
C208
EMPNAME
OFFICENO
Smith
Jones
Adams
Miller
Bennett
C200
C202
C204
C206
C208
read
read
A B C
IBM Federation Server
A
CREATE NICKNAME O_OFFICE FOR
ORACLE.J15USER1.OFFICE
CREATE NICKNAME X_EMP FOR
DB2OS390.J15USER3.EMP
B
DB2
WS II
C
Microsoft
110
IBM InfoSphere Federation Server...
5 Optimizer
Prestazioni delle query federate: un esperimento interessante
Native Query Interface
SQL-H
workload
IBM Federated Server
Tecnologia di
“global optimizing”
Non-DB2
db
(4GB)
2.0
Ratio of query elapsed time
Federated vs. Native interface
• Le prestazioni dipendono dalla
configurazione e dal tipo di query
1.5
• Le query federate sono in un
range 20% di quelle native nella
maggioranza dei casi
1.0
• Per poche query:
Prestazioni peggiori: pushdown
vs.network traffic
Prestazioni migliori: più efficiente
ottimizzazione delle query
0.5
14
22
19
16
02
10
18
07
12
08
21
01
11
05
06
03
04
09
15
20
0.0
TPC-H query number
111
5 Optimizer
Federation Terminology
Server2
Wrapper A
Nickname
Nickname
Server1
Nickname
Nickname
Server: represents a specific
data source
Federated views
Table
Wrapper: a library allowing
access to a particular class of
data sources or protocols
(Net8, DRDA, CTLIB...).
Gathers information about
data source characteristics
Server3
WrapperB
Nickname: a local alias to data
on a remote server (mapped
to rows and columns)
(Optional
Local Data)
(remote data
source 1)
(remote data
source 2)
(remote data
source 3)
112
5 Optimizer
How Federation Server Works
Federated views
Server2
Wrapper A
Nickname
Nickname
Server1
Nickname
Table
2. Federation Server parses the
request and creates query
fragments to send to the sources
Nickname
1. User’s client sends a request
which goes to the Federation
Server interface
Server3
WrapperB
3. Each source system receives its
query fragment, processes it and
returns the result
4. Federation Server assembles the
final result (which may mean
additional processing) and the
result is sent back to the client
(Optional (remote data
Local Data) source 1)
(remote data
source 2)
(remote data
source 3)
113
5 Optimizer
Data Federation Query Processing
• Decomposes, rewrites and
distributes queries
• Invokes functions at
remote sources as needed
Local + Remote
Execution Plans
Rel. Wrapper
NR. Wrapper
Client library
Client library
Table
• Compensates for missing
function in data source
Cost-based optimizer
Nickname
• Query execution engine
drives wrappers, combines
results
Federated Server
Nickname
• Cost-based optimizer
chooses query plan with
pushdown as appropriate
Non-SQL
114
5 Optimizer
Federated Cost-Based Query Optimization
Physical Properties:
Federated system configuration
Query Properties:
Optimization class, data distribution,
operators used, query type,
cost models, FIRST N ROWS ?
Statistics:
•Table Statistics
•Column statistics
•Index statistics
SERVER
•Characteristics
•Cpu/io ratio,
•Commrate
•Capabilities
•Type/version
Non-Relational Wrapper
Wrapper Plans
Cost Models
115
5 Optimizer
Local Caching
Materialized Query Tables (MQTs) Over Nicknames
• MQT: local table defined by
the result of a query
•
Can include joins, aggregations
over multiple nicknames
•
Can be indexed, replicated in
partitioned environment
•
Optimizer “routes to” them
transparently as appropriate
•
Can include both local DB2 tables
and nicknames
•
Can include nicknames to
nonrelational objects
• Use to replace remote access
with local access
Remote
data
Nickname
Fed server
MQT
Local Data
Remote
data
Remote
data
Local Data
116
5 Optimizer
Multi-site Update
V8.x
Disallowed:
transaction
atomicity
Cannot be
guaranteed
V9.1
UPDATE S1
UPDATE S2
Federated
Server
UPDATE S1
UPDATE S2
Federated
Server
UPDATE S1
S1
S2
UPDATE S1
S1
UPDATE S2
S2
• location transparency: no need to track # update sites
• transaction atomicity: all succeed or all fail
117
Part 2
Part 2
Aspects of DB2 database administration
118
DBA day-by-day tasks
Automatic Storage
• Storage management
• Backup / Restore
Automatic Backup
• Performance & Tuning
Automatic Maintenance
• System and dbm configuration
• Memory mgmt
• ...
• Workload mgmt
• Disegno fisico dei dati
• Indici, partizioni, MQT, MDC...
STMM
WorkLoad Manager
Advisors
• Data maintenance
• Reorg, runstats, rebind...
• Monitoring
• a livello dbms
• di sistema
Tools
Health Monitor
Tivoli sw
119
DBA high value tasks
• Security management
• accessi, compliance, encryption...
• Supporto alla ottimizzazione delle applicazioni
• xml data management
• Web Applications
• ....
• Aspetti architetturali
• High Availability / Disaster Recovery
• Data volume explosion (Storage)
• Integrazione del "DB2 subsystem" con altri ambienti
•
•
•
•
•
DB2 Data WareHouse con ETL e front-end tools
Data replication, Data federation, Data events...
Service Oriented Architectures (--> Info Services)
Master Data Management
....
• Data Governance
120
Automated Tasks & Autonomic Capabilities
Automated Tasks & Autonomic Capabilities
121
Autonomic Computing
6 Automated tasks & Autonomic capabilities
Definition
Autonomic computing systems have the ability to manage themselves and dynamically adapt to change
in accordance with business policies and objectives.
This enables computers to identify and correct problems often before they are noticed by IT personnel.
They can also “learn”, adapt and protect themselves for future situations.
• Autonomic computing architecture
provides a foundation on which selfmanaging information technology
systems can be built
• Self-managing autonomic systems
exhibit the characteristics of CHOP:
•
•
•
•
Self-Configuring
Self-Healing
Self-Optimizing
Self-Protecting
• Originally, “autonomics” described
the human body’s ability to regulate
itself, e.g.
•
•
Increased heart and breathing rate
for more blood and oxygen flow
during a crisis
Sweating to cool down the body
when it gets hot
Autonomic Computing Attributes
Self-managing systems that deliver
Increased
Responsiveness
Business resiliency
Adapt to dynamic- Selfally changing
Configuring
environments
Self-
Operational
Optimizing
efficiency
Tune resource &
balance workloads to
maximize use of IT resource
SelfHealing
SelfProtecting
Discover, diagnose
& act to prevent
disruptions
Secure information
and resources
Anticipate, detect, identify
and protect against attacks
As described by: Brent Miller Senior Technical Staff Member,
Autonomic Computing Architecture, IBM Corporation
122
DB2 Self-Tuning Memory Manager
Perf. & Tuning --> System and dbm cfg --> Memory mgmt
Autonomics For Managing DB2 Memory Usage
QUESTION:
Do I need to learn the DB2 cached
based memory model in order to tune it?
ANSWER:
Not at all. DB2 Self-Tuning Memory
Manager handles this for you!
DB2 Self-Tuning Memory Manager uses “out of the box” autonomics which:
• Constantly re-evaluated cached based memory usage
•
•
Optimizes current workload up to 60 times an hour
Stops tuning if it reaches optimal configuration
• Is on by default, but can be turned off
•
Use db parameter SELF_TUNING_MEM OFF
• Can share OS memory with database memory automatically
•
Use db parameter DATABASE_MEMORY AUTOMATIC
• Can share various cached based database memory heaps with each other
•
Set these to AUTOMATIC
: BUFFER POOLS
: PCKCACHESZ
: LOCKLIST
: SHEAPTHRES_SHR
: SORTHEAP
NOTE:
DB2 Self-Tuning Memory Manager manages cache based
memory in DB2. Functional based memory used by application
programs are controlled by the parameter
APPLICATION_MEMORY, or if this is set to AUTOMATIC then
INSTANCE_MEMORY with control this.
123
Basic Memory Usage in DB2
DB2 9.5 introduces Thread Based Engine
• One Process per Instance base.
• Any other “agent” servicing applications or internally
working for the Instance is a “thread” of the main instance
process
Memory is allocated by the Instance process
• For itself
• On behalf of each thread
Memory consumption is
based on three concepts
• Instance Memory
• Database Memory
• Application Memory
General Rule
Instance Memory = Sum(all DB memories + all Application Memories) + overhead (10%)
When more than 1 instance is installed
Each must address the Max memory addressable by DB2 Edition installed
The SUM of all the Instance Memory should not be more than real addressable memory (to
avoid OS paging activities)
Better run at least WSE on 64 bits servers
Instance Memory
• Is the total amount of memory
addressable by DB2 Instance
• 2 GB for Express-C, 4 GB for
Express, 16 GB for Workgroup
• Managed by specific Parameter (can be
automatic and managed by STMM)
Database Memory
• Is the total amount of memory used
internally by each DB and include:
• All the buffers
• All the caches
• All the sort areas
• Locks
Application Memory
• Is the total amount of memory used by
agents (instance threads) servicing local
and remote applications when connected
to the DB
Avoid relying on 32 bit OS algorythm to go beyond 4GB (like Windows one)
Not efficient
DB2 can think to have more real memory to address leading to less optimization and pure
performance
124
DB2 “Automatic” Memory Usage
• Setting a few database parameters & the buffer pools to AUTOMATIC
• Makes DB2 share the entire system memory, when needed, as shown below
•
•
•
The database borrows memory from the operating system, or gives it back
The buffer pools share memory with each other, even converting the various page sizes
The other database memory heaps share with each other as well as the buffer pools
OPERATING SYSTEM MEMORY
DATABASE_MEMORY
BUFFER
POOL 1
BUFFER
POOL 2
BUFFER
POOL 3
SHEAPTHRES
_SHR
BUFFER POOL 4
PCKCACHESZ (Package Cache)
SORTHEAP
(Sorts &
Hash Joins)
LOCKLIST (Lock List)
* Registry variable db2_mem_tuning_range limits upper and lower ranges of OS memory usage
125
Parameter Setting Details
• DB2 Self-Tuning Memory Manager itself is on by default for newly created databases
• To turn it off, use the Control Center GUI to change SELF_TUNING_MEM to OFF
• Or use the following CLP command:
UPDATE
UPDATEDB
DBCFG
CFGFOR
FORDATABASE
DATABASE[db-name]
[db-name]USING
USINGSELF_TUNING_MEM
SELF_TUNING_MEMOFF;
OFF;
• DATABASE_MEMORY is AUTOMATIC by default for newly created databases
• To turn it off, use the Control Center GUI to change DATABASE_MEMORY to COMPUTED
UPDATE
UPDATEDB
DBCFG
CFGFOR
FORDATABASE
DATABASE[db-name]
[db-name]USING
USINGDATABASE_MEMORY
DATABASE_MEMORY
COMPUTED;
COMPUTED;
• Buffer pools are set to SIZE AUTOMATIC by default if size is not specified
• To freeze this, use the Control Center GUI to alter the buffer pool size to a set page amount
ALTER
ALTERBUFFERPOOL
BUFFERPOOL[bufferpool-name]
[bufferpool-name]size
size[pages];
[pages];
126
Parameter Setting Details (Continued)
• Sorting and hash joins are controlled by three different parameters
• By default the DBM parameter SHEAPTHRES is set to 0 to put sorts in database shared
memory; it must be set to 0 for STMM to manage sort memory
• By default DB parameters SHEAPTHRES_SHR and SORTHEAP are set to AUTOMATIC
• Use the Control Center GUI to alter these or the following CLP commands to change them:
UPDATE
UPDATEDBM
DBMCFG
CFGUSING
USINGSHEAPTHRES
SHEAPTHRES[pages];
[pages];
UPDATE
UPDATEDB
DBCFG
CFGFOR
FORDATABASE
DATABASE[db-name]
[db-name]USING
USINGSHEAPTHRES_SHR
SHEAPTHRES_SHR[pages];
[pages];
UPDATE
UPDATEDB
DBCFG
CFGFOR
FORDATABASE
DATABASE[db-name]
[db-name]USING
USINGSORTHEAP
SORTHEAP[pages];
[pages];
• Package Cache and Lock List set to AUTOMATIC by default
• To freeze their sizes, use Control Center GUI to alter their sizes to a fixed page amount
• Or use the following CLP commands:
UPDATE
UPDATEDB
DBCFG
CFGFOR
FORDATABASE
DATABASE[db-name]
[db-name]USING
USINGPCKCACHE_SZ
PCKCACHE_SZ[pages];
[pages];
UPDATE
UPDATEDB
DBCFG
CFGFOR
FORDATABASE
DATABASE[db-name]
[db-name]USING
USINGLOCKLIST
LOCKLIST[pages];
[pages];
127
Table Space Design in DB2
Storage management
DB2 V8.x
• For tables in all table spaces
(regular, temporary, DMS, SMS)
• Large Tablespaces ONLY for LOB
DB2 V9.x
New Large and Temporary Table Spaces
• For tables in LARGE table spaces
(DMS only)
• Also all SYSTEM and USER
temporary table spaces
For Large and Temporary Tablespaces:
128
Automatic Storage
Storage management
DB2 Automatic Storage – How does it work?
1.
Create a DB2 database with the following to enable it:
CREATE
CREATEDATABASE
DATABASE…
…AUTOMATIC
AUTOMATICSTORAGE
STORAGEYES
YESON
ON[path/drive]…
[path/drive]…
… or alter an existing DB2 database with the following to enable it:
ALTER
ALTERDATABASE
DATABASE…
…ADD
ADDSTORAGE
STORAGEON
ON[path/drive]…
[path/drive]…
2.
Create table space with the following:
CREATE
CREATETABLESPACE
TABLESPACE…
…MANAGED
MANAGEDBY
BYAUTOMATIC
AUTOMATICSTORAGE
STORAGE…
…
• That’s it! Under the covers, DB2 now creates, names and sizes the data files
needed to support the table space without any DBA intervention.
• Path or drive can be any number of subdirectories, mount points or drives
• Path or drive list can be added to later with an ALTER DATABASE command
• DB2 manages all automatic storage containers within this path
• Smart enough to make temporary table spaces SMS, all others DMS
• Automatically stripes containers for you by your number of paths or drives
• Create other DMS or SMS table spaces outside automatic storage if desired
• Health Monitor indicators signal if paths or drives about to become full
129
Automatic Storage
Some common operations
Alter Table Space to enable Automatic Storage
Convert table spaces to automatic storage
ALTER TABLESPACE <tbspname> MANAGED BY
AUTOMATIC STORAGE
or
RESTORE DB <dbname> REDIRECT SET
TABLESPACE CONTAINERS FOR
<tablespaceID> USING
AUTOMATIC STORAGE
Affects:
Converts DMS tablespace to a automatic
storage managed tablespace
No container operations allowed after conversion
Dropping Storage Paths
ALTER DATABASE DROP STORAGE PATH ON ‘path1’,…
If storage path is “Not In Use” then it is immediately removed
from database
If storage path is “In Use” then:
Storage path state is changed to “Drop Pending”
Affected table spaces are marked as “An Underlying
Storage Path(s) has been Dropped”
A “Drop Pending” storage path is removed when no
automatic storage table space is using it: Requires
REBALANCE
Storage management
Alter Table Space: Reduce
ALTER TABLESPACE <tbspname> REDUCE
[New Options]
New Options:
MAX – (Only Automatic Storage
table spaces) moves water mark to
lowest possible page and removes
all data pages above water mark
STOP – (Only
Automatic Storage
table spaces) Stops
extent movement
operation
<value> [K|M|G] (Only Automatic
Storage table spaces)
Moves Extents
<value> PERCENT (Only Automatic
Alter Table Space: Rebalance
Storage table spaces)
ALTER TABLESPACE
<tbspname>
Moves Extents
Affects:REBALANCE
Limitations:
Will try to move water mark if
possible Only Automatic Storage
table spaces
Will remove
all unused pages
Affects:
above watermark
Rebalance data across
storage paths
Moves data from previous
nonautomatic storage
containers to automatic
storage paths
130
D
t ff t d t
Automatic Storage Rebalance:
REBALANCE
Storage management
ALTER TABLESPACE myts REBALANCE
High Water Mark
Two storage paths
and a table space has
a container on each
p1
p2
C0 C1
New storage paths
not used by the
table space
immediately
p1
p2
C0 C1
ALTER DATABASE …
ADD p3, p4
p3
REBALANCE causes DB2
to create equal-sized
containers in new paths, and
redistribute extents to them
p4
p1
p2
p3 p4
C0 C1 C2 C3
ALTER TABLESPACE ...
REBALANCE
If table space is not growing
rapidly, consider REDUCing
it to make space available
for other table spaces
p1
p2
p3 p4
C0 C1 C2 C3
ALTER TABLESPACE ...
REDUCE
(optional)
High Water Mark
131
Automatic Storage
Storage management
adding or dropping storage
Alter Table Space: Add or Drop storage
ALTER DATABASE <dbname> (options) STORAGE ON <path>
New Options:
ADD – Enables database for automatic storage if not previously enabled and/or adds
new storage path
DROP – Removes storage path
Affects:
Enables database for automatic storage if not previously enabled
Adds new storage paths
Drop removes storage path once all extents have been moved
Dropping Storage Paths : Example
p1
p2
p3
p1
p2
p3
C1 C2
C3
C1
C2
C3
ALTER DATABASE …
DROP p2
p1
p3
C1
C3
ALTER TABLESPACE ..
REBALANCE
Drop Pending
132
Automatic Storage: monitoring
Storage management
Some examples
New database snapshot field
Automatic storage path
Node number
State
File system ID
Storage path free space (bytes)
File system used space (bytes)
File system total space (bytes)
=
=
=
=
=
=
=
/storagePath1
0
In Use | Not In Use | Drop Pending
64768
66725142528
50100731904
317068410880
New tablespace snapshot field
Tablespace name = USERSPACE1
Tablespace ID = 2
...
Rebalancer Mode = No Rebalancing
Underlying storage path has been dropped = No | Yes
Minimum Recovery Time =
...
SQL to find tablespaces using dropped paths
SELECT DISTINCT(A.TBSP_NAME)
FROM SYSIBMADM.SNAPTBSP A, SYSIBMADM.SNAPTBSP_PART B
WHERE A.TBSP_ID = B.TBSP_ID AND
A.TBSP_CONTENT_TYPE IN ('ANY','LARGE') AND
B.PATH_DROPPED = 1
133
• DB2 Automatic Maintenance:
• Only performs activities that need to be done
• Uses “set it and forget it” autonomics
• Determines when (and whether) to run
• Optimizes resources by
• Phase I - Determining best time to run any given activity in:
•
•
Online window
Offline window
UP
CK ATS
BA NST
RU
• BACKUPs
• RUNSTATs
• REORGs
You’ll like me.
I’ll make your job
easier!
RG
Autonomics Approach Overview
Perf. & Tuning --> Data maintenance --> Reorg & Runstats
Backup / Restore
RE
O
DB2 Automatic Maintenance
• Phase II - Running only required activities
• Is easy to manage
• GUI wizard walks you through the entire process
• Saves your settings in a profile & can be run over and over again
• Has a “learning” capability
• Internal scheduling continually estimates times to completion
• If window not large enough
•
•
Will not run those activities it cannot complete
Will notify you through the Health Monitor
134
Backup / Restore
Setup GUI Wizard vs. Line Command
GUI
LINE COMMAND CONFIGURATION
There are stored procedures you
can use to manually update
automatic maintenance:
AUTOMAINT_GET_POLICY
AUTOMAINT_GET_POLICYFILE
AUTOMAINT_SET_POLICY
AUTOMAINT_SET_POLICYFILE
135
Backup / Restore
Concepts Used
All options use these concepts:
• Maintenance Windows (or Timing)
•
Defined as:
• Periods of time for hours of the day, days of the week or month
• For online and offline time periods
• Inside or outside defined time periods
• DB2 uses this window to consider running selected utilities
• This window should be a relatively unused time period for your database
• Notification
•
List of contacts can be built and used to notify results of auto maintenance actions
• Throttling
•
Auto maintenance uses “adaptive utility throttling” which it controls all by itself
• The defined maintenance window is used as the overriding priority
• Uses a fixed 7% limit on resources
136
Backup / Restore
Auto_% DB Configuration Parameters
auto_maint
db cfg
~~~~~ ~~
~~~~~~~
~~~~~ ~~~
~~ ~~~~ ~
auto_db_backup
auto_tbl_maint
auto_reorg
auto_runstats
auto_stats_prof
auto_stmt_stats
auto_prof_upd
AUTO_MAINT
Parent to all other auto_% parameters. Sets auto maintenance at a global level.
Enabled: All recorded values for child parameters take effect.
AUTO_DB_BACKUP
Enabled: Allows for automatic BACKUP operations.
AUTO_TBL_MAINT
Parent parameter to all other (non backup) auto_% parameters.
Enabled: All recorded values for child parameters take effect.
AUTO_REORG
Enabled: Allows for automatic table and index REORG operations.
AUTO_RUNSTATS
Enabled: Allows for automatic asynchronous RUNSTATS operations.
AUTO_STMT_STATS
Enabled: Allows real-time, synchronous statistics gathering.
AUTO_STATS_PROF
Turns on statistical profile generation, designed to improve applications whose workloads include
complex queries, many predicates, joins, and grouping operations over several tables.
Enabled: Recommendations are stored in the opt_feedback_ranking table.
AUTO_PROF_UPD
Enabled: Runstats profile is to be updated with recommendations from opt_feedback_ranking table.
137
Backup / Restore
Backup Settings & Workings
BACKUP option uses these settings:
• Criteria
• Time elapsed since last full backup since specified by number of hours
• Also, if backup has never run before
• Log space used between backups more than specified number of 4K pages
• Location
• File System (Disk)
• Automatically deletes old copy in chosen directory
• Tape
• Tivoli Storage Manager
• X/Open Backup Services API (XBSA)
• Vendor DLL
• Mode
• Online - Archive log (retain) mode supported
• Default maintenance mode for this setting is 24 x 7
• Offline – Archive log and circular log both supported
• Uses QUIESCE DATABASE in DEFER mode to allow transactions to complete
138
REORG Settings & Workings
Auto REORG option uses these settings:
• All tables
• Can include or exclude system catalog tables
• Selected tables
• Can use a simple or custom filter to select range of tables to run against
• Custom filter can point to a “control table” which contains only the names of tables you desire
an automatic REORG to be performed on
• Has other options for REORG like:
•
•
•
•
Size of table limit (helps avoid REORG on very large tables)
Offline storage location (temporary storage location)
Dictionary rebuild/no rebuild (for compressed tables)
Index reorg mode (read or write options)
Auto REORG works by
• Using reorgchk_tb_stats and reorgchk_ix_stats stored procedures
• Determines which tables and indexes to perform REORG on
• Performs “classic” table REORG on table data
• Only during a specified offline table maintenance window
• with no write access during reorganization
• with no access at all during shadow copy change-over
• Performs index REORGs during online or offline table maintenance windows
• Keeps working data in SYSTOOLS schema tables
139
RUNSTATS Settings & Workings
Auto RUNSTATS – asynchronous collection
• DBA selects range of tables for consideration (using GUI)
• Utility is non contentious – uses new special kernel locking protocols
• Utility uses a collection algorithm that learns over time how fast the
statistics change
• Compares histograms on a per table basis
• Determines the minimum set of statistics that give optimal performance
improvement
• Asynchronously runs background RUNSTATS for tables determined to need it
• Keeps working data in SYSTOOLS schema tables
Auto profile generation customizes your statistics gathering
• DBA sets AUTO_STATS_PROF db cfg parameter – gathers
automatically
• DBA sets AUTO_PROF_UPD db cfg parameter – updates automatically
• Stores data in a “feedback warehouse” in SYSTOOLS, tables
OPT_FEEDBACK_*:
• By default, statistics collected for synchronous and asynchronous
operations are basic table statistics with distribution information and
detailed index statistics using sampling. This feature customizes this.
140
RUNSTATS Real-Time Statistics Collection
Real-time statistics - synchronous collection
• DBA sets AUTO_STMT_STATS db cfg parameter
• Utility collecting statistics when needed by that query at run time
• Ignores the maintenance windows; runs anytime queries run
• Uses many different techniques to maximize synchronous collection
• Like placing statistics in statistics cache and writing to the catalog later
• Limiting the time spent to collect statistics (5 seconds by default)
• Submitting asynchronous collection if necessary
• Fabricating statistics through extrapolation
ASAP!
141
DBA high value tasks
DBA High value tasks
142
7 DBA high-value tasks
DB2 architectures
High Availability
& Disaster Recovery
Appl
DB2 has three different proposition
DB2
• Single SMP Server
disk
• Intensive write/update workload
• Scale up
• Shared Nothing Clusters
• Intensive Read workload (BI)
• I/O always in parallel
Appl
DB2
DB2
DB2
DB2
disk
disk
disk
disk
• Scale up & out
• Shared Disk Cluster
• Typical OLTP workload.
•
80% read, 20% write
•
Thousands of users
•
Highest level on SQL execution concurrency
•
Very simple SQL
•
Continuous availability and scalability as main goal
•
Scale up & out
Appl
DB2
DB2
disk
143
How DB2 address Availability Issues
High Availability
& Disaster Recovery
Three main scenarios:
• High Availability
• Disaster Recovery
• Continuous Availability
Various possibilities to address each of those
• By use of integrated DB2 functions/implementation
• By use of external Software
144
High Availability
High Availability
& Disaster Recovery
Two servers nearby
• Can use native DB2 HADR function
• Double writings on two syncronous copies of the same DB
• Active-Standby Concept
• Can rely on OS SW
• HACMP on IBM AIX, MS CS on Windows etc
• Need an external shared disk
• Can be implemented via Scripts
• c
145
High Availability Disaster Recovery Basics
High Availability
& Disaster Recovery
HADR replication takes place at the database level
Database A
Server 1
Server 2
Primary
Standby
TCP/IP
HADR
HADR
Database A
Database C
Database B
Database D
Standby
Database E
Primary
TCP/IP
HADR
HADR
Database E
146
DB2 HADR Synchronization Modes
Primary
Standby
DB2
DB2
DB2 instance A
TCP/IP port
DB2 database
from instance A
memory
nsync
async
Transaction
DB2 database
from instance B
async
TCP
TCP
IP
IP
MAC
MAC
PHY
PHY
Log writer
1
disk2
DB2 instance B
TCP/IP port
1
32
disk
sync
High Availability
& Disaster Recovery
3
disk
2
memory
nsync
Log writer
1
Transaction
disk
3
Transaction
Log
sync
Transaction
Log
HADR Replication
147
Disaster Recovery
High Availability
& Disaster Recovery
Two or more Servers located in a WAN
• Can be addressed by HADR
• Can be addressed by Mirroring/Replicating Disks
on a WAN
• Can be address on subset of a Database by use
of Q-Replication SW
148
Continuous Availability
High Availability
& Disaster Recovery
Reduce Single Point of Failure
• More than two servers in a cluster
Provide Transparent Recoveries
• No interrupt on applications
Can handle more than one failure at the same time
Suited for OLTP applications
149
DB2 pureScale Architecture
High Availability
& Disaster Recovery
Automatic workload balancing
Cluster of DB2 nodes
running on Power servers
Leverages the global lock
and memory manager
technology from z/OS
Integrated Cluster
Manager
InfiniBand network & DB2
Cluster Services
Shared Data
150
DB2 pureScale main goals
High Availability
& Disaster Recovery
Address Continuous Availability
• By use of numbers of DB2 members
• Relying on a centralized lock manager (Global Lock Mgr) and on a centralized
Cache (Global Buffer Pool) provided by Cluster Facilities Members
• Specific implementation that tolerates multiple Member/CFs failure
• Transparent to applications
Address Scalability
• Can add nodes whenever needed without stopping applications
• Can scale up (on a single member) or out (by adding members)
Address Workload Balancing
• Implicit in DB2 pureScale server architecture and standard DB2 drivers
(CLI/ODBC, .NET and Java)
• Transparent to applications
151
Scalability for OLTP Applications
High Availability
& Disaster Recovery
112 Members
81% Scalability
88 Members
87% Scalability
2, 4 and 8
Members
Over 95%
Scalability
64 Members
91% Scalability
32 Members
Over 95%
Scalability
16 Members
Over 95%
Scalability
Number of Members in the Cluster
152
Single Failures Toleration
High Availability
& Disaster Recovery
153
Simultaneous Failures Toleration
High Availability
& Disaster Recovery
154
Replication: another way to integration
Data Replication
Needed when integration allow/requires multiple
data copies kept in sync
Two technologies so far
• SQL Replication
• Q – Replication
data
Can work on both:
• Homogeneous RDBMS (eg: DB2)
• Etherogeneous RDBMS (eg: Source Oracle, target DB2,
SQL Server etc)
Various products in IBM Portfolio
155
SQL replication (DB2 – DB2)
Data Replication
Filtering is possible
Vertical (selected colums)
Horizontal (selected rows=
Applied at source (capture) and/or at target
(apply)
156
Scenarios
Data Replication
157
Q - Replication
Data Replication
158
Scenarios
Data Replication
159
Esplosione dei volumi di dati
Data volume growth
IBM Optim Enterprise Data Mgmt
Scenario / Esigenze
• Gestione del ciclo di vita dei dati
in funzione delle applicazioni
• crescita dei volumi: archiviazione di dati
raramente acceduti (“svecchiamento”)
• creazione di ambienti di test, anche con
protezione dei dati sensibili
• dismissione di applicazioni
Data Growth - Data Privacy - Test Data Mgmt - Appl Upgrades - Appl Retirement
Optim
Extract - Store - Port - Protect
Estrazione e
Archiviazione
degli “oggetti”
applicativi
Soluzione: IBM Optim EDM sw
• Optim Data Growth
• Optim Test Data Mgmt
• Optim Data Privacy
extract
files
extract
Data Governance
Database
di
produzione
Storici
mask
archive
Recuperati
Correnti
retrieve
masked
files
Database
di sviluppo
e di test
load
Dati di
riferimento
Archivi
Dati di
reporting
Dati
storici
Accesso universale ai dati applicativi
Applicazione
Applicazione
XML
ODBC/JDBC
160
DWH architectural framework
Data Warehousing
Example: real customer case
SAP R/3 environment
IDMS
DB2
IDMS
Ora
Ora
Ora
Ora
15
2
Data modelling
Administration
1 Primary ETL
5
Ora
R/3 conn
BW conn
17
3
BW
PSA
ODS
Infocubes
Aggreg.
Transf
Other envir
Mainframe environment
ETL
4
metadata
ODS DWH
EDW
Enablers
ETL int
14
DMarts
12 Mdim ETL
Mdim
DMarts
6
7
DW
8
DMarts
Query governor
9
front-end
13
virtualization layer
11 End user tools
Appl
16
embedded
analytical
BEX
18
10
161
IOD e SOA: l’informazione come servizio
Service Oriented Arch
L’uso flessibile delle informazioni è la base di qualsiasi architettura SOA
"You will waste your investment in SOA unless you
have enterprise information that SOA can exploit."
Gartner, March 2005
L’informazione corretta, univoca,
fornita come servizio “su richiesta”
Information Services
Conformi alla SOA
Riutilizzabili
Semanticamente
consistenti
Esempi:
• verfiche di
correttezza
• trasformazioni di
formato
• arricchimento,
aggregazione
• sincronizzazione
• query federate
• ……
Info 2.0
Information Services
162
SOA
Info
Services
IOD
Analityc Services
• BI services (Cognos 8 BI)
• On-line analytics, DMining (IWH)
• …
Service Oriented Arch
Master Data Mgmt Services
• All the 800+ services offered by
InfoSphere MDM Server
• Entity Analytics services (EAS)
Information Integration Services
• Reat-time ETL (IS DataStage)
• Data cleansing (IS QualityStage)
• Federated queries (IS Federation Srv)
• Data event capture (IS Data Event Publ)
• …..
Metadata Services (and repository)
• Import /export, Data lineage, Impact
analysis (IIS)
• Business Glossary (IIS)
Content Services
• Content mgmt, Record mgmt, eForms
services (FileNet P8)
• Content federation (WS II CE)
• ….
Supporting Tools
(service generation)
• IS Service Director
• IBM DataStudio
• Other IBM tools
(e.g. WS Integration Developer)
• …
Service choreography
• Content-centric business
processes & workflows
(FileNet P8 BPM)
• Other IBM sw
(e.g. WS Process Server)
Data Services
• Relational data access (DB2, IDS)
• Stored Procedures (DB2)
• XML data access (DB2)
• Data access (IMS)
• …
163
Per maggiori dettagli e altre informazioni
164
DB2 for Linux, Unix & Windows 9.7
Home page (ESE)
http://www-01.ibm.com/software/data/db2/linux-unix-windows/edition-enterprise.html
165
Infocenter
http://publib.boulder.ibm.com/infocenter/db2luw/v9r7/index.jsp
166
Tech Guides
http://www-01.ibm.com/support/docview.wss?rs=71&uid=swg27009474
167
http://www-03.ibm.com/certify/certs/dm_index.shtml#db
http://www.ibm.com/developerworks/offers/lp/db2cert/db2-cert730.html?S_TACT=105AGX19&S_CMP=db2certlp
168
http://www.ibm.com/developerworks/offers/lp/db2cert/db2-cert731.html?S_TACT=105AGX19&S_CMP=ZHP
169
Questions ?
170

DB2 - e-Learning

Transcription

Similar documents

DECEMBER BOOK.indb

What can I do to Protect DB2 Against Warehousing

zIM News in 2012

(C) Engine - Automotriz En Video

Eloxite Corporation

Optimize Access to DB2 for i5OS from Java and WebSphere_S

Dianne Fodell Charlotte Informatics Talent Panel

IBM Systems Tecdhnical University

Services for better business – and for better world.

IBM DB2 Analytics Accelerator