Friday, January 1, 2010

Veritas Volume Manager FAQ 2

Note that the first 28 questions have answers related to using the command line. Sun do not normally support this method of bottom up creation of volumes. An explanation of why can be seen in Q97. The questions are designed to give you an idea of what goes on in the background. However, if at all possible, use the GUI.

29. How can I tell if a disk has been hot spared ?

30. How do I redirect mail to be sent to another user instead of root in the event of a failure & hot spare replacement ?

31. How do I force a disk back into it's original configuration after it has been hot-spared ?

32. How do I make Volume Manager see newly added disks ?

33. How to determine unused space in a diskgroup config ?

34. How do I list all the subdisks in my volume ?

35. How can I find out which physical disks are in a disk group ?

36. How can I change the name of a VM disk ?

37. How do I move all the subdisks from one physical disk to another disk ?

38. Is it safe to use vxevac on a live system ?

39. Can I use a hot-spare across multiple disk groups ?

40. How does Dirty Region Logging (DRL) work ?

41. Early Notifier Alert RAID-5

42. Hot reloction VS hot sparing

43. How can I manually failover a diskgroup in an SSA ?

44. How can I grow the private region of a disk group ?

45. Where can I get the latest SSA Matrix - (Patches & Compatibility) ?

46. Where can I get Demo License for VxVM (2.6) ?

47. How can I create a small rootdg without encapsulating a disk ?

48. What should I consider when configuring swap ?

49. How can I stop volume creation ?

50. How can I recover a failing disk - FAILING ?

51. How do I move disks from one disk group to another ?

52. How do I return to underlying devices ?

53. How could I backup and restore private regions ?

54. How can I add more swap from another disk under Veritas ?

55. How can I setup an alternate dump device (rcS.d/S35newdumpdevice) ?

56. Official SSA/A5000 Software/Firmware Configuration Matrix

57. Veritas On-Line Storage Manager (www.veritas.com)

58. Download Veritas Software

59. Veritas Utilities and Scripts (Unsupported)

60. What can the vxprivutil command be used for ?

61. How can I clear the message 'volume ? is locked by another utility' ?

62. How can I clear subdisks which are in IOFAIL state ?

63. How do I upgrade to Solaris 2.5 & VM 2.1.1 ?

64. How does a hotspare know when to kick in ?

65. How do I encapsulate the boot disk ?

66. How can I clone a Sparc Storage Array which is under Veritas control ?

67. How to import rootdg from another host ?

68. How do I increase the size of a striped mirrored volume ?

69. How do I boot from a SSA ?

70. How can I set the ownership of a volume ?


29. How can I tell if a disk has been hot spared ?

A mail message will have been sent to root (by default) indicating
that a hot relocation has taken place.

The message subjects should be :

Subject: Volume Manager failures on host xxx

Subject: Attempting VxVM relocation on host xxx

30. How do I redirect mail to be sent to another user instead of root in the event of a failure & hot spare replacement ?

Edit /etc/rc2.d/S95vxvmrecover and enter 'vxsparecheck user-name &'
where user-name is the user you wish to have mailed.

Note this is hot sparing and not hot relocation.

31. How do I force a disk back into it's original configuration after it has been hot-spared ?

/usr/sbin/vxassist -g move !disk05 disk02

where disk05 is the 'relocated to' disk and disk02 is the 'relocated from' disk

Of course, before moving any relocated subdisks back, ensure the disk thatexperienced the failure has been fixed.

An alternative would be to use vxevac. See option 37 below.

32. How do I make Volume Manager see newly added disks ?

/usr/sbin/vxdctl enable

33. How to determine unused space in a diskgroup config ?

/usr/sbin/vxassist -g maxsize layout=nolog

34. How do I list all the subdisks in my volume ?

/usr/sbin/vxedit -g list

35. How can I find out which physical disks are in a disk group ?

/usr/sbin/vxdisk (-s) list

36. How can I change the name of a VM disk ?

/usr/sbin/vxedit -g rename

An alternative would be to use vxassist. See option 31 above.

37. How do I move all the subdisks from one physical disk to another disk ?

/usr/sbin/vxevac -g disk03 disk02

38. Is it safe to use vxevac on a live system ?

Yes. Volume manger will build a mirror and do a full sync of the original disk to the new disk.

39. Can I use a hot-spare across multiple disk groups ?

No. Hot-spares only act as replacements for other disk in the same disk group.


This rest of this section is designed to give you an idea of various disk management tips and hints for Volume Manager.

----40. How does Dirty Region Logging (DRL) work ?
What is DRL ?
Dirty Region Logging is a bit map mechanism used to facilitate mirror resynchronisation after a system failureWhere DRL's can be used
A DRL is ONLY used when a volume has 2 plexes or more (ie, a mirror) and the system panics (or someone yanks out the power cord).
NB. One mistake people make is assuming that the DRL is the same thing as the log used on a Raid5 volume. They function completely different.
The DRL is not added automatically, because it is an "option" and not always necessary.
If one side of a mirror goes down and is marked as bad and needs to be restarted, a full resync will be done and the DRLs have no value. The main use for DRLs is in system failure. In most other scenarios they are not used.

What Veritas DRL's will NOT do
In SDS you can do a metaoffline and the SDS DRL keeps track of which regions are changed then when doing a metaonlinesome time later SDS only resync's the changed regions. This 'feature' or functionality is not part of the design of the Veritas software.
In the case where a plex is disassociated, this causes the entire plex to go stale, so when you reattach the plex to the mirror, it requires a full resync.
They are not used for on-line reconfigurations operations.
Again, the DRL is ONLY used if the system crashes and is subsequently rebooted.

How DRL's work When the system is rebooted after a 'non-graceful' shutdown, volume manager needs to syncronize the plexes to guarantee that both mirrors are identical. It does this by reading each and every block and copying the data to the "other" plex. Obviously, this will take a long time. If there's a DRL attached to the volume, volume manager will only have to 'sync' the regions which are dirty. Since it works on a "region" basis, the resync times are greatly reduced, in most cases, but not necessarily all cases.

Performance
The performance difference you may notice during normal operation may be due to the updates in the log. There is some amount of overhead involved, though normally it is minimal.

Fast writes with DRL's
Fast_write has nothing to do with DRL.... Veritas has no knowledge of fast_write... The SSA fast_write feature is very safe with DRL's in with all configurations. There were some problems in the past but that is ancient history.

41. EARLYNOTIFIER_ALERT_RAID-5
The EARLYNOTIFIER_ALERT_RAID-5 says:
For non-typical RAID-5 configurations which have the following characteristics:
- a RAID-5 column which is split into multiple subdisks where the subdisks do NOT end and begin on a stripe-unit aligned boundary
- and a RAID-5 reconstruction operation was performed.Data corruption can be present, in the region of the RAID-5 column where the split subdisks align.
So for a preventive action I want to verify the customers RAID-5 configurations. ( using explorer results )
This can be done as follows.
Here's an example:A. Good case

Case-1:
# vxprint -g kuro -hqvt


vvol01raid5ENABLEDACTIVE640000RAID-

plvol01-01vol01ENABLEDACTIVE640000RAID3/32RW

sdkuro01-01vol01-01kuro0102240000/0c2t1d0ENA

sdkuro02-01vol01-01kuro0202240001/0c2t1d1ENA

sdkuro03-01vol01-01kuro0302240002/0c2t1d2ENA

plvol01-02vol01ENABLEDLOG1008CONCAT
RW

sdkuro04-01vol01-02kuro04010080c2t1d3ENA

Subdisk size is multiple of stripe width. So you don't worry about this.

Case-2:
# vxprint -g kuro -hqvt

vvol01raid5ENABLEDACTIVE640000RAID-
plvol01-01vol01ENABLEDACTIVE640000RAID3/32RW
sdkuro01-01vol01-01kuro0102240000/0c2t1d0ENA
sdkuro05-01vol01-01kuro050960000/224000c2t4d0ENA
sdkuro02-01vol01-01kuro0202240001/0c2t1d1ENA
sdkuro06-01vol01-01kuro060960001/224000c2t4d1ENA
sdkuro03-01vol01-01kuro0302240002/0c2t1d2ENA
sdkuro07-01vol01-01kuro070960002/224000c2t4d2ENA
plvol01-02vol01ENABLEDLOG1008CONCAT-RW
sdkuro04-01vol01-02kuro04010080c2t1d3ENA

Subdisk is concatanated, but still they're matched with stripe size boundary. Also you don't worry about this.

B. Bad case --- You need to care about those configuration.

Case-1:
# vxprint -g kuro -hqvt

vvol01raid5ENABLEDACTIVE614400RAID-
plvol01-01vol01ENABLEDACTIVE617728RAID3/32RW
sdkuro01-01vol01-01kuro0102052000/0c2t1d0ENA
sdkuro05-01vol01-01kuro0501036800/205200c2t4d0ENA
sdkuro02-01vol01-01kuro0202052001/0c2t1d1ENA
sdkuro06-01vol01-01kuro0601028161/205200c2t4d1ENA
sdkuro03-01vol01-01kuro0302052002/0c2t1d2ENA
sdkuro07-01vol01-01kuro0701028162/205200c2t4d2ENA
plvol01-02vol01ENABLEDLOG1008CONCAT-RW
sdkuro04-01vol01-02kuro04010080c2t1d3ENA

This means the last stripe unit of "kuro01-01" contains only 16 blks. This means this bottom section isn't aligned stripe size. As the result, the first 16 blks in "kuro05-01" is used for this stripe. You need to pay attention to this configuration. The Early Notifier tells us this.

Case-2:
# vxprint -g kuro -hqvt

vvol01raid5ENABLEDACTIVE614400RAID-
plvol01-01vol01ENABLEDACTIVE617728RAID3/32RW
sdkuro01-01vol01-01kuro0102052000/0c2t1d0ENA
sdkuro02-01vol01-01kuro0202052001/0c2t1d1ENA
sdkuro03-01vol01-01kuro0302052002/0c2t1d2ENA
plvol01-02vol01ENABLEDLOG1008CONCAT-RW
sdkuro04-01vol01-02kuro04010080c2t1d3ENA

Currently, there're no concatinated subdisk in this raid5 volume. But each subdisk size is not aligned with stripe width. The last stripe has only 16 blks. At this moment, VxVM doesn't use this last stripe part. But think again, when you resize this volume with new 3 drives with "vxva" or "vxassist growto", this configuration will changed to "Case-1" above. So you need to pay attention to this configuration too.42. Hot reloction VS hot sparing
Hot reloction VS hot sparing.....
Presently if there is a media (read) failure of redundant data, Volume Manager
will read from the other plex and then write the data to the plex that
encountered the error data. The data is then re-read from the newly
written data to ensure it is good. Should this fail again, Volume Manager then
performs a diagnostic in its private region for that disc. This
diagnostic (for the lack of a better descriptive word) consist basically
of writing and reading in the private region. Should it fail then Volume Manager
will replace the disc with another disc (if the Hot Sparing option is
selected and a disc is available), and then copy good data from the good
plex to the newly replaced disc. As you notice this is a disc for disc
replacement for redundant data.In the new Hot Relocation scheme, pretty much the same is carried on up
to the point of the diagnostic in the private region. Instead of
performing the diagnostic in the private region and then replacing the
disc with another. Now Volume Manager will replace the errored sub-disc with a
matching sized area out of the disc(s) allocated for the Hot Relocation
pool and copies data from the good plexs sub-disc to the newly acquired
sub-disc. As you will notice this is a sub-disc for sub-disc
replacement of redundant data, without the requirement that the complete
disc fail.

In both the above cases non-redundant data is not moved to the newly
aquired disc space, be it a complete disc or sub-disc failure. This
operation becomes the function of the systems operator...

Obviously there is much more to the operation, this is meant to be
nothing more than just a thumb nail sketch of the differences..

A dead disc type failure is pretty much the same, with all the
operations failing on the bad disc.

43. How can I manually failover a diskgroup in an SSA
If SSA linked to two systems and one system fails
To list groups out there
# vxdiskadm
8 Enable access to (import) a disk group
list

To force the import of a disk group
# vxdg -fC import disk_group

To start the volumes
# vxvol -g disk_group -f start vol01 vol02 vol03 etc..

To mount volumes
# mount /dev/vx/dsk/disk_group/vol01 /data01

To bring in rootdg disks on SSA into different group
# vxdg -tC -n "newgroup" import "diskgroup"

44. How can I grow the private region of a disk group ?
The following information is from "Ultra Enterprise PDB 1.2 Release Notes - December 1996"Large System Configurations
Systems that have a very large disk group configurations can eventually exhaust the private region size in the disk group, which results in not being able to add more configuration objects. At this time, the configuration either has to be split into multiple disk groups, or the private regions have to be enlarged. This task involves re-initializing each disk in the disk group. There is no method to dynamically enlarge this private region.The private region of a disk is specified by using the privlen option to the vxdisksetup cammand. For example,
# vxdisksetup -i c1t0d0 privlen=2048

Reasons why this is not a good idea
Increasing the size of the private region is tricky because you have to increase it on ALL the disks in the dg before the database itself will "expand". Therefore, running vxdissetup on only one disk is not going to work. Unfortunately, it's not that easy.If you increase the size of the private area, that means less space for data on the disk, so you'r PROBABLY looking at a complete backup, tear down of the dg, remaking the dg with the newly formatted/initialized disks, and then remaking the volumes and restoring data. Obviously, this is very time consuming.

Once enlarged, if ANY failed disk is replaced by use of normal means the dfault region size will be on the new device, thus forcing the entire group to default back to that size.

Lastly, if the customer ever in the future runs 'vxdiskadm' to initialize a new disk, it'll put the default (small) private area on the disk. This won't work, cause all the disks in the dg HAVE to have the larger private area. So for the rest of this guys life he needs to run vxdisksetup with the appropriate parameters to give the disk the larger private area.

Options
They should look into just putting any new volumes and disks into a new diskgroup.Split the disk group that has the problem up, move some of the volumes out to a different disk group, so that they do not run into problems under live running conditions. As the Veritas volume manager requires space in the private region for say when it has to use hot sparing/relocation etc... Only where necessary use enlarged private regions.

45. Where can I get the latest SSA Matrix - (Patches & Compatibility) ?
SSA Matrix - (Patches & Compatibility)

46. Where can I get Demo License for VxVM (2.6) ?
Demo License for VxVM (2.6)

47. How can I create a small rootdg without encapsulating a disk ?

Create Small Rootdg Group
There MUST be a rootdg group for each node when using the SPARCcluster.
It is the default disk group and cannot be omitted.
It is more difficult to upgrade to new version or recover from certain
errors when the root is encapsulated.Prerequsists:-
Install Solaris 2.x plus patches.
Install Volume Manager and patches.
Reboot.Create a 2 cylinder raw partition on a disk.NB DO NOT RUN VXINSTALL.

Set the initial operating mode to disable. This disables
transactions and creates the rendevous file install-db,
for utilities to perform diagnostic and initialization
operations.


# vxconfigd -m disable
# ps -ef | grep vxconfigd

root 608 236 6 16:20:20 pts/0 0:00 grep vxconfigd
root 605 1 18 16:17:44 ? 0:00 vxconfigd -m disable

NB. If the above command fails its probably because vxconfigd is already running.
Just kill off vxconfigd and re-run the above command again or alternatively try vxconfigd -k which kills the daemon and restarts it.

Initialise database
# vxdctl init Make new rootdg group
# vxdg init rootdg Add a simple slice
# vxdctl add disk c0t3d0s7 type=simple
vxvm:vxdctl: WARNING: Device c0t3d0s7: Not currently in the configuration Add disk record
# vxdisk -f init c0t3d0s7 type=simple Add disk name to rootdg disk group
# vxdg adddisk c0t3d0s7 Enable transactions
# vxdctl enable Remove file
# rm /etc/vx/reconfig.d/state.d/install-db

48. What should I consider when configuring swap ?


Dump device
Primary swap needs to be a phyiscal partition. This is not to say it cannot be under vertias control.
This is because in the event of a panic VM doesn't support volumes as dump devices.
It gets it from a physical slice initially which is then is changed to a vertias volume.
During the startup from /etc/rcS.d/S35vxvm-startup1 it executes the following.

"vxvm: NOTE: Setting partition $dumpdev as the dump device."
swap -a $dumpdev
swap -d $dumpdev
breakdoneMore than one swap

One place for the primary swap.
You can have other swap areas which could be raw partitions or a swap file.

How much swap must be non striped

We do not support a striped primary swap device. This could be made smaller than memory.

How big must his swap be

We advise this to be at least the same size as his systems memory, so that crash dumps can be taken. - not really relevant to Enterprise class systems

Maximum limit

Up to a max of 2GB.
All core dumps will be truncated to 2GB, and in fact a partition over 2GB is a problem for 'savecore'.
The core dump is copied to the _end_ of the primary swap partition,
and savecore can't read it because it can't seek over 2GB.

Direct mapping between devices

Veritas finds the 'real' partition that the swapvol maps to and uses this for the 'dumpdevice' ( for core dumps ).
If there is not direct mapping, then this will break and the system will have not dumpdevice.

How swap works

There is no point in striping swap.
Swap is allocated on a round robin basis in 1MB pieces from all the partitions or files that are assigned as swap space.
Just take all the pieces of disk that you were planning on striping, and assign them as swap space in their own right.
"Striping" will essentially occur automatically.

Creating primary swap on different disk

The swap partition should still be on slice 1 but must not start at cylinder 0 (start at cylinder 1) and it's tag must be swap.
This is because it stops the veritas volume manager startup script S35vxvm-config from mapping the underlying partition
as a dumpdevice after the disk has been encapsulated.

Although this was done on a raid5 volume, the same could be done for any type of volume.

49. How can I stop volume creation ?


We created a raid5 volume that was imediately seen to be the wrong size.
By running the following commands ment we can have another
go at creating it the correct size, without having to wait
for it to finish the raid5 setup.// Find the id of the process creating the raid5 volume.
# ps -ef

// Kill off the process.
# kill

// Clear the tutil and putil fields (we want dashes in the fields).
// These fields can be seen by running vxprint.
// May only have to do it for the plex.
# vxmend clear tutil0 vol01-01
# vxmend clear tutil0 vol01

// Remove the volume.
// You will not be able to do this from the GUI.
# vxedit -fr rm vol01

50. How can I recover a failing disk - FAILING ?
In VXVM 2.3 when a subdisk gets an I/O error, hot relocation kicks in and the
VMdisk that contained the failed VMsubdisk is set to "failing=on".This prevents vxassist from using the disk to create new subdisks unless explicitly directed to do so by the user.

You can use the following command to turn the flag off if you think it got turned
on due to a transient error. VXVM 2.3 GUI also has a checkbox for this functionality.
# vxedit set failing=off disk05 51. How do I move disks from one disk group to another ?
NB.
If these disks are going into an existing disk group then
you do not need to initialise the disk group and the disk
names must be different,
eg If you have c1t0d0s2=disk01 then the disk group you
are moving the disk into must not have a disk01.

Moving Populated Volume Manager Disks between Disk Groups

i) Assume I intend to move volumes vol02 and vol04 from the disk
group olddg to a new group, newdg.

ii) Get a list of disks in the disk group you intend to split.

# vxdisk list | grep olddg
c1t0d0s2 sliced olddg01 olddg online
c1t0d1s2 sliced olddg06 olddg online
c1t1d0s2 sliced olddg04 olddg online
c1t1d1s2 sliced olddg09 olddg online
c1t2d0s2 sliced olddg02 olddg online
c1t2d1s2 sliced olddg07 olddg online
c1t4d0s2 sliced olddg03 olddg online
c1t4d1s2 sliced olddg08 olddg online
c1t5d0s2 sliced olddg05 olddg online
c1t5d1s2 sliced olddg10 olddg online

iii) Get the configuration.

iv) Determine which disks contain the volumes to be moved. Insure that all
volume allocations are self-contained in the set of disks to be moved.
In this case, my volumes are contained on disks olddg04 through olddg09,
with no unassociated plexes or subdisks, and no allocations which cross
out of this set of disks.

v) Save the configuration, in a format that can be plugged back into
the vxmake utility. Specify all volumes on the disks in question (plus
any unassociated plexes and their child subdisks, plus any unassociated
subdisks).

# vxprint -hmQqvps -g olddg vol02 vol04 > movers

vi) Unmount the appropriate file systems, and/or stop the processes
which hold the volumes open.

vii) Stop the volumes.

# vxvol -g olddg stop vol02 vol04

viii) Remove from the configuration database the definitions of the
structures (volumes, plexes, subdisks) to be moved. (NOTE that this
does not affect your data.)

# vxedit -g olddg -r rm vol02 vol04

ix) Remove the disks from the original diskgroup.

# vxdg -g olddg rmdisk olddg04 olddg05 olddg06 olddg07 olddg08 olddg09

x) Initialize the new diskgroup using one of your disks. DO NOT
reinitialize the disk itself. (vxdisk init). (If you are moving the disks
to a disk group that already exists, skip this step.) It is simplest to
keep their old names until a later step.

# vxdg init newdg olddg04=c1t1d0s2

xi) Add the rest of the moving disks to the new disk group.

# vxdg -g newdg adddisk olddg05=c1t5d0s2
# vxdg -g newdg adddisk olddg06=c1t0d1s2
# vxdg -g newdg adddisk olddg07=c1t2d1s2 olddg08=c1t4d1s2 olddg09=c1t1d1s2

xii) See the disks in the new disk group.

# vxdisk list | grep newdg
c1t0d1s2 sliced olddg06 newdg online
c1t1d0s2 sliced olddg04 newdg online
c1t1d1s2 sliced olddg09 newdg online
c1t2d1s2 sliced olddg07 newdg online
c1t4d1s2 sliced olddg08 newdg online
c1t5d0s2 sliced olddg05 newdg online

xiii) Reload the object configuration into the new disk group.

# vxmake -g newdg -d movers

xiv) Bring the volumes back on-line.

# vxvol -g newdg init active vol02
# vxvol -g newdg init active vol04

xv) Observe the configuration of the new disk group.

# vxprint -ht -g newdg

xvi) Test the data. Remember that the device names have changed to refer
to newdg instead of olddg; you'll need to modify /etc/vfstab and/or your
database configurations to reflect this. Then you'd mount your file
systems, start your database engines, etc.


xvii) Note that the original database is intact, though the disk naming is
a bit odd. You *can* rename your disks and their subdisks to reflect the
change. This is optional.

# vxprint -ht -g olddg

# vxedit rename olddg10 olddg04

# vxprint -g olddg -s -e "name~/olddg10/"

# vxedit rename olddg10-01 olddg04-01

# vxprint -g olddg -ht

xviii) Do the same for the disks in newdg and their subdisks.

# vxprint -g newdg -s

# vxprint -g newdg -e "name~/olddg/"

# vxedit rename olddg04 newdg01
# vxedit rename olddg05 newdg02
# vxedit rename olddg06 newdg03
# vxedit rename olddg07 newdg04
# vxedit rename olddg08 newdg05
# vxedit rename olddg09 newdg06
# vxedit rename olddg04-01 newdg01-01
# vxedit rename olddg05-01 newdg02-01
# vxedit rename olddg06-01 newdg03-01
# vxedit rename olddg07-01 newdg04-01
# vxedit rename olddg08-01 newdg05-01
# vxedit rename olddg09-01 newdg06-01
# vxedit rename olddg08-02 newdg05-02
# vxedit rename olddg09-02 newdg06-02

# vxprint -g newdg -ht 52. How do I return to underlying devices ?
Notes:- Use the name of your system disk rather than c0t3d0, which is
used in this guide only as an example.

- When rebooting, you might have to specify a disk if the disk is not
the default boot disk, instead of init 6.


Sequence:
Boot from CD-ROM to convert back to underlying devices and stop Volume
Manager.
Boot from cdrom
>ok boot cdrom -sw

# fsck /dev/rdsk/c0t3d0s0
FILE SYSTEM STATE IN SUPERBLOCK IS WRONG; FIX? y

# mount /dev/dsk/c0t3d0s0 /a

Take a copy of the vfstab and system files
# cp /a/etc/vfstab /a/etc/vfstab.vx
# cp /a/etc/system /a/etc/system.vx

Edit vfstab file to replace any of the system partition entries with
entries for the underlying partitions.
Also remove/convert/hash out, any other entries to vx devices.

# vi /a/etc/vfstab

/proc - /proc proc - no -
fd - /dev/fd fd - no -
swap - /tmp tmpfs - yes -
/dev/dsk/c0t3d0s0 /dev/rdsk/c0t3d0s0 / ufs 1 no -
/dev/dsk/c0t3d0s6 /dev/rdsk/c0t3d0s6 /usr ufs 1 no -
/dev/dsk/c0t3d0s3 /dev/rdsk/c0t3d0s3 /var ufs 1 no -
/dev/dsk/c0t3d0s5 /dev/rdsk/c0t3d0s5 /opt ufs 2 yes -
/dev/dsk/c0t3d0s1 - - swap - no -

Check the rest of the system filesystems.
# fsck /dev/rdsk/c0t3d0s6
# fsck /dev/rdsk/c0t3d0s3
# fsck /dev/rdsk/c0t3d0s5
Edit system file to remove the following entries for Volume Manager.
# vi /a/etc/system
rootdev:/pseudo/vxio@0:0
set vxio:vol_rootdev_is_volume=1

Stop Volume Manager.
# touch /a/etc/vx/reconfig.d/state.d/install-db
install-db should be the only file in this directory, remove any other
files like root-done.

# cd /
# umount /a
# fsck /dev/rdsk/c0t3d0s0

Reboot on the underlying disk
# init 6 53. How could I backup and restore private regions ?
The following proceedure is not a substitute for backing up your valuable
data. Please ensure you have reliable backups of your data.
Take a snapshot of the current configuration

When you have volume manager up and running, you need to run the following
command for each disk group.
# vxprint -hmQqsvp -g dg > /directory-path/filenameWe also need a list of the names which equate to the physical disks,
this can be obtained be keeping the output from
# vxdisk list

Create the new disk group
NB. Disk names MUST correspond to the same disks exactly as they were
originally (see you backup copy vxdisk list output).

Initalise the diskgroup, using one of the disks from the lost disk
group.
# vxdg init dg access name=medianameNB. Substitute the disk group, disk access name and media name
from the saved vxdisk list output. e.g.
# vxdg init datadg disk01=c2t0d0s2Add in the rest of the disks.
# vxdg -g new_data_dg adddisk =[other disks]Recreate the volume(s) configuration from the configuration file.
# vxmake -g dg -d /directory-path/filename

NB. If this fails saying input file too large, then split the
file up and run the above for each one.
In most cases it works, its just for very large configurations
and then we have only split it into two pieces.You can use the above command with the -V option which will
go through the motions but not actually do anything.
You may have to bring the volume back online (each vol must be done
one at a time).

# vxvol -g init active 54. How can I add more swap from another disk under Veritas ?

Create more swap in a different disk group
NB Name no longer than swapvol ie swapvl2 or swap02 is ok.To add extra swap manually:-
# swap -l
# swap -a /dev/vx/rdsk/swapdg/swap02
# swap -l

To pick up the extra swap at boot time:-
# vi /etc/vfstab
/dev/vx/rdsk/swapdg/swap02 - - swap - no -

# init 6 55. How can I setup an alternate dump device (rcS.d/S35newdumpdevice) ?

#!/bin/sh
#
# /etc/rcS.d/S35newdumpdevice
#
# Author: Bede Seymour (bede.seymour@Eng.Sun.COM)
#
# Created: Wed Apr 30 14:41:09 EST 1997
#
# This script *MUST* come before /etc/rcS.d/S40standardmounts.sh !!!
#
# This only works because the swapadd/swap -a command only edits the
# dumpvp vnode ptr in the kernel if not already set.
#
# Set DUMPDEV to the device/slice on which you would like to dump.
#
# set defaults
PATH=/sbin:/etc:/bin:/usr/sbin:/usr/bin:/usr/ucb ; export PATH
PROG=`basename $0` ; export PROG
DUMPDEV="/dev/dsk/c0t0d0s1" ; export DUMPDEV

echo "$

Unknown macro: {PROG}

: configuring $

Unknown macro: {DUMPDEV}

as dump device ..."
swap -a $

swap -d $

Unknown macro: {DUMPDEV}

echo "$

: done." 56. Official SSA/A5000 Software/Firmware Configuration Matrix


Official SSA/A5000 Software/Firmware Configuration Matrix 57. Veritas On-Line Storage Manager (www.veritas.com)


Veritas On-Line Storage Manager (www.veritas.com) 58. Download Veritas Software


Download Veritas Software 59. Veritas Utilities and Scripts (Unsupported)


Veritas Utilities and Scripts (Unsupported) - Australian 'Spider' server 60. What can the vxprivutil command be used for ?


Check the consistancy of the information in the private region try the following:
# /etc/vx/diag.d/vxprivutil dumpconfig /dev/rdsk/c?t?d?s2 > /tmp/vx.output
# cat /tmp/vx.output | vxprint -Ath -D -or
# cat /tmp/vx.output | vxprint -vpshm -D - > /tmp/vxprint.vpshm
# vxmake -g diskgroup -d /tmp/vxprint.vpshm

You should see a normal vxprint output. 61. How can I clear the message 'volume ? is locked by another utility' ?


If you get the following error message :
vxvm:vxplex: ERROR: Plex vol01-02 in volume vol01 is locked by another utility

This can be fixed by running the following commands:
$ vxmend clear tutil plex
$ vxmend clear tutil vol
$ vxplex -g diskgroup det plex
$ vxrecover -s

If you look at the TUTIL0 in a vxprint you will see that there are
some flags set call ATT, all you need to do is to clear them. 62. How can I clear subdisks which are in IOFAIL state ?


How to get rid of the subdisk in a IOFAIL state, if volume is mirror'd
This is a known bug (Bug ID: 4050048), the workaround is as follows

NOTE:-
Before you start make sure you have an output of
$ vxprint -Ath

In this example it is the rootdisk which has had this problem, hence

sdrootdiskPriv-ENABLED2015---PRIVATE
vrootvolrootENABLED1919232-ACTIVE--
plrootvol-03rootvolENABLED1919232-ACTIVE--
sddisk01-01rootvol-03ENABLED19192320---
plrootvol-01rootvolENABLED1919232-ACTIVE--
sdrootdisk-B0rootvol-01ENABLED10--Block0
sdrootdisk-02rootvol-01ENABLED19192311IOFAIL--Both sides of the plex are working but the sd rootdisk-02 is marked as
IOFAIL, the only way to clear this state is to remove the subdisk and
recreate it will the same subdisk offsets and length. this can be done
as follows

# vxplex -g rootdg dis rootvol-01
# vxsd -g rootdg dis rootdisk-02
# vxedit -g rootdg rm rootdisk-02
# vxmake -g rootdg sd rootdisk-02 rootdisk,0,1919231

where DISK is rootdisk, DISKOFFS is 0 and LENGTH is 1919231The DISK, DISKOFFS and LENGTH can be found from the vxprint -th output

VNAMEUSETYPEKSTATESTATELENGTHREADPOLPREFPLEX
PLNAMEVOLUMEKSTATESTATELENGTHLAYOUTNCOL/WIDMODE
SDNAMEPLEXDISKDISKOFFSLENGTH[COL/]OFFDEVICEMODE
vrootvolrootENABLEDACTIVE1919232ROUND-
plrootvol-03rootvolENABLEDACTIVE1919232CONCAT-RW
sddisk01-01rootvol-03disk01019192320c1t0d0ENA
plrootvol-01rootvolENABLEDACTIVE 1919232CONCAT-RW
sdrootdisk-B0rootvol-01rootdisk191923110c0t3d0ENA
sdrootdisk-02rootvol-01rootdisk019192311c0t3d0ENANOTE:
depending where the subdisk is used, ie with in a stripe you may need
to use the -l option to specifiy where the subdisk is located in the stripe.
# vxsd -g rootdg assoc [-l n] rootvol-01 rootdisk-02

where n is the position in the stripe, this again can be determined from the vxprint -th output.
for subdisk datadg08-01 the -l number would be 2 (STRIPE 2/0)
for subdisk datadg06-01 the -l number would be 0 (STRIPE 0/0)| V | NAME | USETYPE | KSTATE | STATE | LENGTH | READPOL | PREFPLEX |

PLNAMEVOLUMEKSTATESTATELENGTHLAYOUTNCOL/WIDMODE
SDNAMEPLEXDISKDISKOFFSLENGTH[COL/]OFFDEVICEMODE
plvol01-02-DISABLED-104672STRIPE4/128RW
sddatadg06-01vol01-02datadg060262080/0c1t1d0ENA
sddatadg07-01vol01-02datadg070262081/0c1t3d0ENA
sddatadg08-01vol01-02datadg080257602/0c1t3d1ENA
sddatadg09-01vol01-02datadg090257603/0c1t5d1ENA


# vxplex -g rootdg att rootvol rootvol-01

Once the re-sunc has finished then everything should be back to normal.


This section deals more specifically with issues related to Veritas Volume Manager and the Sparc Storage Array


63. How do I upgrade to Solaris 2.5 & VM 2.1.1 ?

We had a customer who installed Solaris 2.5 on top of Vm 2.1.... This is not a supported config, because it does not work. This left the customer in a position of not being able to get his system back up. After jumping thru some hoops the customer was back up on line. This to point out a few things to try and save others from the same or similar problems....

o The Vm 2.1.1 binaries for Solaris 2.3 and 2.4 are NOT the same as the binaries for Solaris 2.5.

o This a departure from the past, due to some other problems (to be solved in the next release).

o This also means that the Vm 2.1 binaries will NOT work with Solaris 2.5...

o If you are going to use Solaris 2.5 and Vm, it MUST be Vm 2.1.1(release)....

o If a customer upgrades Vm to 2.1.1 on either Solaris 2.3 or 2.4 and then decides to go to Solaris 2.5, caution should be taken. As mentioned above the Vm 2.1.1 binaries for Solaris 2.3/2.4 and 2.5 are not the same, so both Vm and Solaris need to be upgraded in this case.

o The upgrade procedures for Vm 2.1.1 provided with the CD, do not cover this situation, a procedure needs to be developed and tested to handle this.

64. How does a hotspare know when to kick in ?

This is how a hotspare works with Veritas Volume Manager....

Specifically, here's the procedure Vol Mgr goes through to determine
whether a disk has failed:

If read error on non-protected (non-mirrored and non-RAID5) volume:
do nothing, other than report the error
(this is the same thing Solaris does)


If read error on protected volume:
read the data from alternate drives
(from alternate mirror, or by reconstructing the
data from the remaining disks in RAID5)
return data to app

write data to drive which originally reported read error
(Reason: if the original read error was due to
a bad spot on the disk, then this write may
cause the SCSI drive to re-vector the block to an
alternate track)


If write error:
Try to read & write the private region on the disk.


If private region accessible:
Must be just a bad spot on the disk.
Do not consider disk failed, and do not hot spare.
Detach plex(es) associated with this disk and email root

Else:
The whole disk is almost surely bad.
Consider disk failed, and initiate hot spare.

Note that it takes a little while to go through all this, doing extra
I/O's to make sure the disk has really failed, waiting for commands to
time out, etc. This is why you don't see hot spares kick in instantaneously.
Volume Manager is trying to make sure that the disk really is bad. This
is Good. You don't want to start swapping in hot spares unless the disk
is really bad.

So, if Vol Mgr decides to kick in a hot spare, it proceeds to build the
appropriate subdisk(s), plex(es), and volume(s) that were on the failed
disk onto the hot spare. It then populates the subdisks with the data
that was on the failed disk.

If all you had on the failed disk was 100MB of volume stuff, then that's
all that gets written to the hot spare. It doesn't write the whole disk
unless it has to.

Look at this example Failures have been detected by the VERITAS Volume Manager:

failed plexes:
ukdb00-01
ukdb02-01
ukdb03-01
ukdb04-01
No data appears to have been lost.

The line "No data appears to have been lost." inidcates this, and inform's root
that he needs to decide what to do

1. try and re-attach disk and see if he has a geniune h/w Error or
2. replace faulty disk, and rebuild the mirror

In a nutshell a hotspare will only be initated by Volume Manager if/when Volume Manager cannot read/write
to the private region of the disk. In all other circumstances Volume Manager will email root (to configure
Volume Manager to email more users edit /etc/rc2.d/S95vxvm-recover and modify the line that says
"vxsparecheck root &" to vxsparecheck root other-users &") and inform him/her that it has detached a plex

Unknown macro: {mirror}

as it encountered problems, and needs attention (this is better than ODS which just sits there until,
you issue a metastat command too see whats happening)

65. How do I encapsulate the boot disk ?
SRDB ID: 10923
SYNOPSIS: How to encapsulate root disk for mirroring

DETAIL DESCRIPTION:

Bring the internal bootable disk under the control of the
Veritas Volume Manager software so it can be mirrored.

SOLUTION SUMMARY:

The disk must have the following file systems:
/ /usr /tmp /var

There must be some swap space allocated on the bootable
disk, or the Veritas software will not encapsulate the disk.

Set up for encapsulation as follows:

* Select two unused slices on the disk and verify that these are
unassigned, 0 (zero) length slices. If there aren't any unused
slices, then they must be made; i.e., if all seven allocated slices
of the disk have been used, the disk cannot be encapsulated.
These two unused slices will become the Volume Manager's private
and public regions.

* Next, try to "steal" two cylinders (at least 2MB are needed) from
the last used slice. It is best to use the two cylinders (or
the amount needed) at the end of the disk. These are not set up
as a slice, they are just not part of the last allocated slice.
This means if slice "g" is the last used slice and extends all the way
to the last cylinder, for example, cylinder number 1254, make
this slice stop at cylinder 1252 instead. This area will be used
by the Veritas software to hold its private region data.

If this space is not allocated for the software to use, it will
take it from the swap area of the disk. So, if swap is at a premium
on this system, allocate the space.

* Encapsulate the disk by running the following from the command line:

# vxdiskadm

* Select "encapsulate disk" from the menu to complete the
encapsulation operation.

Once the disk has been encapsulated, it is readyto be mirrored.

PRODUCT AREA: SunOS Unbundled
PRODUCT: Install
SUNOS RELEASE: Solaris 2.3
HARDWARE: any
66. How can I clone a Sparc Storage Array which is under Veritas control ?
These scripts might prove useful to some of you, to use as-is or
to modify as needed.This is used for cloning purposes.
Here is a script to replicate a SSA plex/subdisk configuration
from a command line instead of each machine doing the GUI work.

This may be of interest to you to take a look at. I thought it
might be useful for say disaster recover purposes.


To make an identical disk storage array set up to the one you have at
another site do the following:

1) at the original site do the command:

# vxprint -t > /tmp/vxprint.out

2) copy the vxprint.out file and the vxawk script to the new site.

3) install the SSA according to the instructions including running the vxinstall script.

4) make an install script with the commands:

# awk -f vxawk vxprint.out > vxscript
# chmod 755 vxscript

5) run the vxscript:

# ./vxscript

6) start the vxva program:

# vxva &

7) select the Volumes button.

8) use the select followed by mutiple adjust mouse buttons to select all the volumes.

9) under the Advanced-Ops/Volume/Initialize Volumes menu chose Active.

10) if needed under Basic-Ops/File-System-Operations chose Make File System.

11) The system should now be ready for operations.


# VXAWK SCRIPT TO CREATE A DUPLICATE DISK STORAGE ARRAY SET-UP.
# THIS CAN BE USED AS A PART OF A RESTORATION PROCEDURE OF
# A DISASTER RECOVERY PLAN, WHERE THE ORIGINAL STORAGE ARRAY
# MUST BE REPLACED IN TOTAL. TO USE IN THIS MANNER A vxprint -t
# OUTPUT FILE MUST BE SAVED IN OFFLINE MEDIA TO ASSIST IN THE
# RECONSTRUCTION OF THE OLD STORAGE ARRAY. SEE ACCOMPANYING
# HOWTO FILE FOR MORE INFORMATION.
#
# AUTHOR: N. Derek Arnold (derek.arnold@cbis.com)
# COMPANY: Cincinnati Bell Information Systems.
# COPYRIGHT: 1995
# EDIT HISTORY:
# 05-28-95 nda (creation)
#
# THIS PROGRAM CONTAINS UNPUBLISHED PROPRIETARY MATERIAL OF CBIS
# (CINCINNATI BELL INFORMATION SYSTEMS INC.). IT MAY NOT BE
# REPRODUCED OR INCLUDED IN ANY OTHER PACKAGES WITHOUT WRITTEN
# CONSENT FROM CBIS. THE INCLUSION OF THIS COPYRIGHT DOES NOT
# IMPLY INTENT TO PUBLISH. ALL RIGHTS, BOTH FOREIGN AND DOMESTIC,
# ARE RESERVED.
#
BEGIN {
print "PATH=$

Unknown macro: {PATH}

:/usr/sbin:/usr/lib/vxvm/bin"
print "export PATH"
print "set -x"
print ""
print "vxconfigd -k"
print ""
{color:#3366ff}}

# DO NOTHING FOR DGs - this should be done by the vxinstall script
# IF disk groups other than root, this could be a problem in that
# we assume that the dg is always root.
# $1 == "dg" {
# print "vxdg init " $2 " nconfig=" $3 " nlog=" $4 " minor=" $5
# print ""
# }

# SAME COMMENT HERE AS FOR DGs
#$1 == "dm" {
# access = substr( $3, 1, 6 )
# print "vxdisksetup " access " privlen=" $5 " publen=" $6
# print "vxdisk init " $3 " type=" $4
# print "vxdg adddisk " $2 "=" $3
# print ""
#}

# NEED TO DO REMAINING.
$1 == "sd" {
split( $7, col, "/" )
print "vxmake sd " $2 " disk=" $4 " offset=" $5 " len=" $6 " column=" col[1]
print ""
if( plex[$3] == "" )
plex[$3] = $2
else
plex[$3] = plex[$3] "," $2
{color:#3366ff}}

$1 == "pl" {
if( $7 == "STRIPE" )
{
split( $8, f, "/" )
tag = " st_width=" f[2] " sd_num=" f[1]
}
else
{
tag=""
}
output = "vxmake plex " $2 " kstate=" $4 " state=" $5
output = output " len=" $6 " iomode=" $9 tag " sd=" plex[$2]
print output
print ""
if( vol[$3] == "" )
vol[$3] = $2
else
vol[$3] = vol[$3] "," $2
{color:#3366ff}}

$1 == "v" {
output = "vxmake vol " $2 " use_type=" $3 " kstate=" $4
output = output " state=" $5 " len=" $6 " read_pol=" $7
output = output " plex=" vol[$2]
print output
print ""
{color:#3366ff}}

{ }
67. How to import rootdg from another host ?
Note: you *can* import the rootdg from system 1 on system 2 with a click
or two from the vxvm GUI after you move the array, however, you will have
to do some serious handwaving to move the array back to its original
configuration.From the Veritas System Administrators Guide, Release 2.0/Jan 95,section 3.7.4.1 Renaming Disk Groups

The following set of steps can be used to temporarily move the rootdg disk group from one host to another
(for repair work on the root volume, forinstance) and then move it back:

1) On the original host, identify the disk group ID of the rootdg disk group to be import to the other host:

# vxdisk -s list

This command results in output that includes disk group information similar to the following.

dgname: rootdg
dgid: 774226267.1025.tweety

2) On the importing host, import and rename the rootdg disk group as
follows:

# vxdg -tC -n "newdg_name" import "diskgroup"

(quotes not needed)

where -t indicates a temporary import name; -C clears import locks; -n specifies a temporary name for the rootdg to be imported (so that it does not conflict with the existing rootdg); and "diskgroup" is the disk group ID of the disk group being imported (774226267.1025.tweety,for example).

If a reboot or crash occurs at this point, the temporarily-imported disk group will become unimported and will require a reimport.

3) After the necessary work has been done on the imported rootdg, deport it back to its original host as follows:

# vxdg -h newhost_id deport diskgroup

where newhost_id is the hostid of the system whose rootdg is being returned (this is the system's namd, which can be confirmed with the command uname -n). This command removes the imported rootdg from the importing host and returns locks to its original host. The originalhost will then autoimport its rootdg on its next reboot.

68. How do I increase the size of a striped mirrored volume ?
This is how I did it, but note that I used the GUI. I found that the autoresizing of the volume did NOT occur when I used the command line interface.At this point I have my two volumes:


vol02 is my larger volume who's plex is going to hold the data eventually: 1) stop vol02

# vxvol -g Daviddg stop vol02

2) Disassociate plex from volume

vxplex -g Daviddg dis vol02-01 (this also sets the volume size to 0 by itself)
vxvol -g Daviddg set len=0 vol02 ( not for command line you need to do this manually)

3) I then remove that volume as I no longer need it

# vxedit -g Daviddg rm vol02

4) time to attach the larger plex to my original volume:

# vxplex -g Daviddg att vol01 vol02-01

Note there is an error at this point when it tries to do a vxpvol set len
command. (I don't get the error when I use the command line interface though)

5) Wait for the sync up, remove the smaller plex :

# vxplex -g Daviddg dis vol01-01
( and a vxvol command comes up to and changes the size of the volume to the larger plex.)
I had to do this by hand using the command line interface.

# vxvol -g Daviddg set len=205632 vol01

vxprint shows the volumes new size:

6) I remove the old plex/subdisks This is what the GUI ran :

# vxedit -g Daviddg rm vol01-01 (plex)
# vxedit -g Daviddg rm David13-01 David14-01 (subdisks)

On the command line I did this which deleted the subdisks too.

# vxedit -g Daviddg -rf rm vol01-01

7) Increase the size of the filesystem with mkfs

Before:

/dev/vx/dsk/Daviddg/vol01 47855 9 43066 0% /foobar

# /usr/lib/fs/ufs/mkfs -F ufs -M /foobar /dev/vx/rdsk/Daviddg/vol01 205632
Warning: 192 sector(s) in last cylinder unallocated
/dev/vx/rdsk/Daviddg/vol02: 205632 sectors in 402 cylinders of 16 tracks, 32 sectors
100.4MB in 26 cyl groups (16 c/g, 4.00MB/g, 1920 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
32, 8256, 16480, 24704, 32928, 41152, 49376, 57600, 65824,
74048, 82272, 90496, 98720, 106944, 115168, 123392, 131104, 139328,
147552, 155776, 164000, 172224, 180448, 188672, 196896, 205120,

# df -k /foobar
Filesystem kbytes used avail capacity Mounted on
/dev/vx/dsk/Daviddg/vol01 96143 9 86524 0% /foobar

Another plex can be added to the vlume to provide protection.

I have done this with files on a volume, and fsck'd the disk afterwards and it all seems fine. YOU MUST take
backups in case things go wrong, and althoughthis works in the lab I am sure you understand I cannot
say this will work for you, although someone from Veritas has looked over this.
69. How do I boot from a SSA ?
you need the following: Fcode 1.33 on the sbus card.
Firmware 1.9 in the array.
Optical Modules rev -03.
2.4 Hardware 3/95.

These are all minimum requirements because:

lower rev fcode: scrambles the world address of the array in
the boot prom, never gets to open the disk device.

lower rev firmware: loadsabugs. not recommended anyway.

lower rev optical modules: give SOC-OFFLINE messages then times
out at the boot prom.

lower rev OS: not got the soc or pln drivers bundled so can't
find the array at boot.

The new firmware is supplied with both SSA 2.0 and 2.1 cd's, the fcode
and the program to check / update this is only with the 2.1 cd.

To update / check the fcode, use the fc_update program as follows:

fc_update [return] will check for SOC cards and bring them
all to the current fcode revision, asking for confirmation
on each one.

fc_update -v [return] will go through the system looking for
SOC cards and reporting the fcode revision levels it finds.
No changes are made to the system.

In order to set up a system to boot from an array without any other
disks during the install, you will need either two cd drives to hold
the OS and SSA cd's, or a copy of the fc_update directory on tape that
you can use.

During the install you will be offered the chance to put the OS onto
any of the array disks, along with any other disks on the system
under the "c0t0d0" naming scheme.

To get the boot device, do not reboot after the install has finished,
instead cd into /dev/dsk, and use ls -l on the boot device. This will
show the full symlink to the /devices entry for the array disk. Remove
the initial "/devices" and the tailing ":a" (for slice 0) and this is
your boot device.

To store the boot device into the eeprom, shutdown the system and use
nvedit. eg for an example ss1000, booting from an array connected to
the SOC in sbus slot 3 on the first system board, with the world address
"a0000000,740d10"

nvedit [return]
0 devalias ssa /io-unit@f,e0200000/sbi@0,0/SUNW,soc@3,0/
SUNW,pln@a0000000,740d10/SUNW,ssd@0,0 [return]
1 [ctrl-c]
nvstore [return]
reset

then you can "boot ssa" to start the system.
70. How can I set the ownership of a volume ?

Say for example you had a volume you wanted to use with sybase
then you'd issue a command something like this.... # vxedit set user=sybase group=sybase mode=600 /dev/vx/dg/volumename

volume manager will set this everytime upon bootup.

No comments:

Post a Comment