Performing an Oracle DBA 1.0 to DBA 2.0 Upgrade

Harald van Breederode and Joel Goodman
Insights into the Expansion of the DBA Skill Set
Performing an Oracle
DBA 1.0 to DBA 2.0 Upgrade

---

Introduction
Historically, the Oracle DBA skill set was database-centric, usually limited to software installa-
tion, database creation, day-to-day maintenance, performance monitoring, tuning and most
of all, backup and recovery. Since the arrival of Oracle 10g, the technology within the DBA
arena has changed, due to the increase in automated monitoring, tuning and manageability
features within the Oracle kernel, and also expanded into areas that were formerly the re-
sponsibility of OS, storage, and network administrators. Due to these changes, the knowledge
required by the typical Oracle DBA has increased, requiring additional skills and in some cases
job responsibilities. This shift from the traditional DBA, which we call DBA 1.0, toward the mod-
ern, post Oracle 9i DBA, which we call DBA 2.0, has occurred gradually over the past two major
releases of the Oracle Database Server.
We recently discussed this shift in DBA skills as part of our regular collaboration as the found-
ing members of the OU EMEA SME team, as well as in leading the Oracle DBA Certification
development projects by mentoring the exam developers on the changing skill set that we
describe here.
ii

---

Performing an
Oracle DBA 1.0 to
DBA 2.0 Upgrade
Insights into the Expansion of the DBA Skill Set

---

I remember seeing “the writing on the wall” when Oracle 10g Release 1 was in Beta. I looked at the
JOEL
traditional DBA job role and the required skills and thought, “I had better start teaching myself
certain concepts and technologies or I will not be able to deliver in the classroom nor lead the
certification development team”. How would you characterise the typical DBA that you taught in
the days of Oracle 9i and earlier.
Prior to Oracle 10g, most DBA skills were centred purely on the Oracle database server. This
HARALD
included installation, database and Oracle network administration, performance monitoring,
database tuning, occasional SQL tuning and first and foremost backup and recovery.
Is that all? I taught RAC, Data Guard, data warehouse database administration, security and vari-
JOEL
ous other advanced topics during releases 8.0, 8i and 9i. Surely the DBA skill set was larger than
what you claim.
The skill set I refer to is the “core” set of skills that most DBAs required for any job, according
HARALD
to a typical technical job description. The advanced courses you and I both delivered were re-
quired for those DBAs who used those specific options, but they were not part of every DBA’s
skill set. When I taught these advanced courses, I struggled with delegates who lacked the
necessary skills in the OS, Storage and Network areas precisely because DBAs did not have,
nor were they expected to have those skills. I also recall helping you at the start in learning
these concepts so that you could teach these skills to course delegates.
Yes I too recall those early days. I also remember looking at both Oracle 10g release 1 Clusterware
JOEL
and ASM, and thinking “this is easy if you are an OS, Storage or Network administrator but how
am I going to learn this well enough to teach it?” When did you first realise that DBA 2.0 was an
emerging requirement?
I realised this even before Oracle 10g. Some of Oracle 9i technology was moving in this direc-
HARALD
tion with features such as Oracle Cluster Manager and Oracle Cluster File System, both for
Windows and Linux, and RAC Guard for Windows. Furthermore, from the start of 8.0, RMAN
could interface to media managers using the SBT API and the use of “dbstart” and “dbshut” on
UNIX systems required some OS knowledge as well. But by 10g it was no longer possible to
remain a DBA 1.0. You said so yourself, that you saw things changing.
Yes I did indeed. But what still causes confusion for some people is the discrepancy between the
JOEL
DBA 2.0 skill set that we saw emerging at that time, and the DBA job role that I saw and still see in
the industry. We are saying that DBA 2.0 requires OS and Network skills especially in the area of the
Oracle Clusterware and Storage administrator skills for ASM and OCFS2. But in the UK, especially in
large organisations, the traditional organisational divisions between the DBA team, the OS admin-
istrators and the Storage administrators remain. So the job roles and responsibilities are the same
perhaps, but the skill set is different.
In The Netherlands, I have seen many cases where companies converting to Linux have asked
HARALD
the DBAs to administer the OS as well as the database. In such cases it is obvious that DBA
2.0 has arrived. But even in large companies with separate departments, DBA 2.0 is needed to
allow the DBA to discuss requirements and implementation details with the OS, Storage and
Network administrators, so that the optimal Oracle technology stack configuration may be
implemented. It is also true that in some companies the application server is part of the DBA
job role further expanding the required skills.
2

---

H
E
Regarding large companies, I taught an ASM seminar recently, for a customer who uses Oracle
JOEL
N
on Solaris, with RAC and ASM where the ASM disks are built on LUNs served by a Fibre Chan-
D
nel SAN. In this company, the DBAs do Oracle database administration but the Clusterware is
I
T
installed and maintained by the OS administrators although DBAs have use of many of the clus-

l
terware utilities. The ASM environment is installed by the DBAs but the storage is managed by
A
the Storage administrators, including the creation of ASM disks. In such cases, it is crucial that the
M
DBAs know about the ASM storage choices, as they impact on database performance. Likewise,
E
T
DBAs must understand how the Clusterware maintains high availability of Nodeapps, ASM, da-

V
tabase instances and database services because adhering to SLAs for availability of Oracle is part
E
of the DBA job requirement.
I agree that in both cases DBA 2.0 skills are required, but we have not yet discussed migrat-
HARALD
ing to DBA 2.0. How did you begin to upgrade yourself aside from our joint sessions?
The first skill I acquired was Storage administration. ASM disks may be created on one of three
JOEL
possible storage types: SAN (Storage Area Network) volumes, local volumes also called JBOD
(Just a Bunch Of Disks) or using certified NAS appliances (Network Attached Storage.) But when
I began learning ASM, I could understand only the highest-level concepts based on my previous
OS and storage skills from my Mainframe days and not this level of detail. But you may recall
that you recommended some good books to read on storage technology and I supplemented
this by creating my own SAN server and NAS Filer using Openfiler in a virtual machine based on
your suggestion. By researching these, and playing around with NAS and SAN using iSCSI, I have
learned much about storage administration. Of course, creating my own virtual machines and
then installing, configuring and maintaining the Operating Systems, has helped me to learn a
great deal about OS administration skills as well.
Of course there is more to storage admin than just creating LUNs or shared file systems.
HARALD
Multi-pathing is a skill that is important for high availability and for performance. In addi-
tion, Oracle 11g has a feature called Direct NFS, which a DBA 2.0 may configure, permitting
the Oracle instance to act directly as an NFS client. This avoids making file I/O system calls
to the underlying OS, thereby improving performance. The recent announcement of Exa-
data extends the skill set of the DBA into the world of storage even further by offloading
SQL filters and certain other operations to the storage servers allowing a vast reduction in
the volume of data that gets returned over the storage network to the database.
I suppose that with Exadata, we could argue that DBA 2.1 may be just around the corner. But
JOEL
there is one more skill that we have not mentioned in the storage area, which is specific to Linux
at the moment. ASMLIB, which is optional, was created to address the lack of persistent device
naming on Linux prior to Linux Kernel 2.6 when UDEV did not exist. ASMLIB is installed as a
set of three Linux packages, and would normally be the responsibility of the OS administrator.
But whoever administers the ASM environment, must specify the size of the disk groups and
communicate this to the OS administrator. If ASMLIB is used with SAN served volumes, then
this requires DBAs, OS administrators and Storage administrators to work together to provide a
solution.
And a similar issue faces the DBA 2.0 when dealing with Clusterware. The OS administra-
HARALD
tors normally install and maintain the HA software on a cluster. But Oracle Clusterware
requires them to use the Oracle universal installer rather than a package manager to install
the clusterware. The OS administrator therefore needs to know how to install and use this
Oracle product and a DBA 2.0 must know about many areas of technology used for and
by the Clusterware as well. In addition, the day-to-day monitoring and maintenance of the
Clusterware involves both the OS and Database administrators. For example, maintenance
of the Clusterware may require shutdown of instances, ASM, Nodeapps or all of them. The
DBA may need to schedule the outage, but the OS administrator may perform the mainte-
nance. It is clear that combining both skills in one person has many advantages.
3

---

JOEL
I know it may be easy for Unix administrators but could you elaborate a bit more on these Cluster-
ware skills for us?
For Clusterware, DBAs must understand I/O fencing and how and why nodes get evicted
HARALD
to prevent problems when a split brain is detected. This is why there is a voting disk for ex-
ample. And the Clusterware is activated automatically at boot time running under “init”. This
requires the DBA to learn how Unix systems initialise, and how to control some of the system
configuration related to initialisation by modifying the correct parameters in the correct files.
Furthermore, network skills are required to change IP addresses and subnet masks for main-
tenance purposes such as network topology changes.
JOEL
I also remember learning how to configure NTP with you for my virtual cluster to make certain that
the cluster nodes did not get their clocks out of sync by enough time to cause node evictions. And
you also began providing 100hz Kernels for our Linux virtual machines to prevent the guest OS
time from falling behind the Host OS time because timekeeping itself is very important in a cluster.
Yes that is true, but setting up access to the NTP server is only one of several network-related
HARALD
skills that a DBA 2.0 requires.
When I began teaching myself Oracle Clusterware, all these skills were needed. OS Bootstrap, NTP,
JOEL
IO fencing and Network technology generally. Most DBA 1.0s have exposure to IP addresses from
administering the database listener and occasionally CMAN or a name server. But Clusterware
takes this further, using several protocols in the TCP/IP stack and for redundant interconnects,
some sort of NIC bonding or link aggregation is required.
And remember the problems we had with the cluster verification utility complaining about
HARALD
the non-routable IP addresses used for the OU RAC courses. There was a workaround, but a
O
DBA 1.0 would not necessarily have known about routable and non-routable IP addresses
M
and why there was a problem. A DBA 2.0 must understand this.
M
Y

N
At the time, you were so concerned about this problem that you wrote a paper called “Cluvfy’s
JOEL
O
Complaint” explaining these concepts to the DBA trainer and student community.
N
S
E

And of course there is much, much more to networks than the minimum skills required. Some
C
HARALD
O
companies use proprietary protocols when using vendor Clusterware together with Oracle
R
Clusterware. And there is Infiniband as well. And let’s not forget that firewalls are used exten-
E
sively in the industry and knowledge of this is a must if the firewalls are not to interfere with
proper network communication using VIPs and public addresses. Of course the use of virtual
machines allows DBAs to practice all of these network and Clusterware skills quite easily.
JOEL
We have discussed many skill areas and technologies, and we have both mentioned previously the
use of virtual machines as a learning tool, but they are also excellent for demos, or proof of concept
activities. I have virtual machines for Openfiler, Oracle Enterprise Linux 4, and Oracle Enterprise
Linux 5 and for SUSE Linux Enterprise Server supporting a virtual standalone server, a virtual two-
node cluster, a virtual SAN server and a virtual NAS filer. It has been great for learning and also for
having complete demo environments for seminars, courses and user group presentations. What
else would you like to add on the subject of virtualisation?
Virtualisation comes in two different flavours. There is virtualisation based on hardware em-
HARALD
ulation, and true virtualisation based on a hypervisor controlling real hardware. Emulation
based virtualisation products such as Vmware Server or Vmware Workstation, are more flex-
ible in their deployment as they may be installed on an existing “Host” system. The virtual ma-
4

---

chines then run as “guests” on the “host”. Hypervisor based virtualisation products, such
as Oracle VM, offer performance advantages but are less flexible in their deployment, as
they must be installed directly on the bare metal, thereby replacing the existing OS.
JOEL
Do you think that the portability advantages of Emulation based virtualisation make it the best
environment for DBAs to use? What disadvantages are there for example in using hypervisor
based virtual machines?
Hypervisor based solutions such as Oracle VM provide lower virtualisation overheads,
HARALD
but the guest operating systems must have modifications to work with a hypervisor.
Many operating systems are not modifiable in a way that supports hypervisor-based vir-
tualisation, so this is not the most portable solution. Since a DBA 2.0 would normally use
these environments for learning and demos, it is probably better to use a more portable
approach because emulation based virtualisation supports almost any “guest” OS. But
for Production environments Oracle VM is the only Oracle supported solution.
JOEL
I found emulation-based virtualisation easy, because at the beginning of my research into OS
administration, I was able to “borrow” a VMware image from a colleague so that I could start
teaching myself OS administration quickly. As this was fully virtualised, the image created on
another machine worked first time on mine, after changing the eth0 network address in the vir-
tual machine to match the subnet used by VMware on my laptop. This portability is very useful
for sharing. It was only a bit later that I began creating my own virtual machines based on your
suggestion, and installing the Operating Systems myself.
It is very important for a DBA 2.0 to learn OS administration and it is better to create
HARALD
the machines oneself, but sharing information is also important and I must agree that
the portability is a key benefit of emulation-based virtualisation. Since we began using
virtual machines in EMEA, I note that many UK trainers are now using then for many ac-
tivities. It seems to me that in the UK at least, the DBA delivery team is well on its way to
the Oracle DBA 2.0 upgrade.
We are in agreement about DBA 2.0, having begun from different starting points. But it is very
JOEL
challenging for people to be faced with this learning curve if they are just starting now.
DBA 2.0 is a reality for some and a goal for others. You have got there for example and
HARALD
other UK DBA trainers are on their way. But the journey will take some time. For example
how much time have you spent doing this?
JOEL
I don’t know how many hours or days I have spent but I acquired DBA 2.0 skills over a period of
two to three years by reading, and playing around with many systems. You also provided excel-
lent and challenging mentoring and advice to myself and other UK trainers as required. However
long it may take someone on their journey of expanding their skills to DBA 2.0, the trick I think is
to enjoy the journey and not just the destination. I have had fun learning new things and meet-
ing the challenges.
For me it is all about the interest and the fun. Let’s hope that others will follow you on
HARALD
the journey.
JOEL
You have inspired others already. I’m sure we can get more people along for the ride.
5

---

Conclusion
We strongly recommend that Oracle DBAs make the effort to enhance their skill set to include
OS, Storage and Network administration. The technology has moved on, but the creation of
many new automatic management and maintenance capabilities in the Oracle server means
that more time can be devoted to the acquisition of other skills.
We, as trainers and Oracle Subject Matter Experts have the responsibility to advise other train-
ers, others within Oracle, and customers alike on the best approach to keeping current with
trends in the industry. We hope that readers will benefit from our discussion and use it as a
roadmap to attaining DBA 2.0 qualification for themselves.
About the Authors:
Joel Goodman of Oracle University UK and EMEA, has taught Oracle DBA technology for 11
years after 2 years in support and 20 years in Mainframe OS technology. He has presented
courses and seminars for Oracle University as well as speaking at the UK Oracle User Group
Annual Conference, the UKOUG and other Technical SIGs, and the Oracle RAC SIG in the US. He
is also the Global Team leader of DBA Certification Exam development team. When Oracle 10g
release 1 was in Beta, Joel saw the direction in which the Oracle DBA skill set was moving and
decided he had better move with it.
O
M
Harald van Breederode of Oracle University NL and EMEA, has taught Oracle DBA tech-
nology for 10 years and presented courses and seminars for Oracle University as well as
M
speaking at the UK Oracle User Group Annual Conference, the OKOUG RAC SIG and The Dutch
Y
Planboard DBA Symposium. He is also the Global Team Leader of the Linux Certification Exam

N
development team. Harald began his career at Oracle as a Unix system administrator and
O
worked in the Dutch Oracle Data centre before moving to Oracle University. He has 30 years
N
Unix experience and clearly saw the need to help the “traditional” DBAs move their skills in a
S
new direction.
E

C
O
R
E
6

---

Copyright © 2008 Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only
and the contents hereof are subject to change without notice. This document is not warranted to be error-free nor is it
subject to any other warranties or conditions whether expressed orally or implied in law including implied warranties
and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to
this document and no contractual obligations are formed either directly or indirectly by this document. This document
may not be reproduced or transmitted in any form or by any means electronic or mechanical for any purpose without
our prior written permission.
7

---

O R A C L E U N I V E R S I T Y
w w w. o r a c l e . c o m / e d u c a t i o n

---