Here I will cover the shrink of ASM diskgroup in Exadata environment running VM’s. The process here is the opposite of what I wrote in the previous post, but have a tricky part that demands attention to avoid errors. The same points that you checked for extending are valid now: number the cells, disks per cell, ASM mirroring, and the VM that you want to change continue to be important, but we have more now. Besides that, the post shows how to verify if you have something in the ASM internal extent map that can block the shrink and “fix” this.

Here I will cover the shrink of ASM diskgroup in Exadata environment running VM’s. The process here is the opposite of what I wrote in the previous post, but have a tricky part that demands attention to avoid errors. The same points that you checked for extending are valid now: number the cells, disks per cell, ASM mirroring, and the VM that you want to change continue to be important, but we have more now. Besides that, the post shows how to verify if you have something in the ASM internal extent map that can block the shrink and “fix” this.

Here, in this scenario, I will reduce the size for grid disks linked with diskgroup DATAC8 (that run in a cluster for VM #08).

And before continuing, be aware of Exadata disk division:

So, my actual usage space for DATAC8 is:

ASMCMD> lsdg

State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB   Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name

MOUNTED  NORMAL  N         512             512   4096  4194304  29921280  29790140           498688        14645726              0             Y  DATAC8/

MOUNTED  NORMAL  N         512             512   4096  4194304   6144000   6142020           102400         3019810              0             N  RECOC8/

ASMCMD>

SQL> select name, total_mb, free_mb, total_mb - free_mb used_mb, round(100*free_mb/total_mb,2) pct_free

  2  from v$asm_diskgroup

  3  order by 1;

NAME                             TOTAL_MB    FREE_MB    USED_MB   PCT_FREE

------------------------------ ---------- ---------- ---------- ----------

DATAC8                           29921280   29790140     131140      99.56

RECOC8                            6144000    6142020       1980      99.97

SQL>

As you can see above, the RAW size is around 29.2 TB (14.6 after the ASM NORMAL mirror) and the actual free space is around 14.3TB. So, plenty of space for the reduction in this case. Unfortunately, even if I have 99% of free space don’t mean that I can reduce instantly to 1TB.

This occurs because, probably, I have a fragmented diskgroup and something can be allocated at the end of some disk and will deny me do reduce the disk. I will not dig in this topic but will show you how to check this. If you want to check with more details you can read these two articles: http://asmsupportguy.blogspot.com/2011/06/asm-file-extent-map.html and http://asmsupportguy.blogspot.com/2012/10/where-is-my-data.html.

The idea is to use the ASM extend map from X$KFFXP to discover the object that has the MAX AU_KFFXP into some disk (whatever which one). So, I made:

SQL> select GROUP_NUMBER, name from v$asm_diskgroup;

GROUP_NUMBER NAME

------------ ------------------------------

           1 DATAC8

           2 RECOC8

SQL>

SQL> select VALUE from V$ASM_ATTRIBUTE where NAME='au_size' and GROUP_NUMBER=1;

VALUE

--------------------------------------------------------------------------------

4194304

SQL>

SQL> select max(AU_KFFXP) from X$KFFXP where GROUP_KFFXP=1;

MAX(AU_KFFXP)

-------------

       114195

SQL>

SQL> select NUMBER_KFFXP from X$KFFXP where AU_KFFXP = 114195;

NUMBER_KFFXP

------------

         262

SQL>

SQL> select name from v$asm_alias where FILE_NUMBER = 262 and GROUP_NUMBER = 1;

NAME

----------------------------------------------------------------------

group_1.262.983716383

SQL>

Above I discovered that the file 252 (that it is a redo – will show after), is allocated in the allocation unit 114195 from diskgroup 1 (that have – and default fore Exadata – allocation unit of 4MB). Doing a little count (114195*4), this means that this file resides in somewhere around 446GB into some disk. And if I try to reduce the disk below this value, I will receive an error.

To solve this we need to move files, in this case, the file is related to MGMTDB:

ASMCMD> ls -l +DATAC8/_MGMTDB/ONLINELOG/

Type       Redund  Striped  Time             Sys  Name

ONLINELOG  MIRROR  COARSE   MAY 03 10:00:00  Y    group_1.262.983716383

ONLINELOG  MIRROR  COARSE   MAY 03 10:00:00  Y    group_2.264.983716383

ONLINELOG  MIRROR  COARSE   MAY 03 10:00:00  Y    group_3.263.983716383

ASMCMD>

ASMCMD> ls -l +DATAC8

Type      Redund  Striped  Time         Sys  Name

                                        Y    ASM/

                                        Y    _MGMTDB/

                                        Y    exa-cl8/

PASSWORD  HIGH    COARSE   JAN 20 2017  N    orapwasm => +DATAC8/ASM/PASSWORD/pwdasm.256.933784119

PASSWORD  HIGH    COARSE   AUG 09 2018  N    orapwasm_backup => +DATAC8/ASM/PASSWORD/pwdasm.1471.983713235

ASMCMD> ls -l +RECOC8

ASMCMD>

To move MGMTDB we can follow the steps from note 2065175.1 and use the script mdbutil.pl. In this case, I moved to RECOC8:

[grid@exa01vm08 +ASM1]$ /tmp/MGMTDB/mdbutil.pl --mvmgmtdb --target=+RECOC8 -debug

mdbutil.pl version : 1.98

2019-06-21 11:43:33: D Executing: /u01/app/18.0.0/grid/bin/srvctl status diskgroup -g RECOC8

2019-06-21 11:43:34: D Exit code: 0

2019-06-21 11:43:34: D Output of last command execution:

Disk Group RECOC8 is running on exa01vm08

2019-06-21 11:43:34: D Executing: /u01/app/18.0.0/grid/bin/srvctl status mgmtdb

2019-06-21 11:43:35: D Exit code: 0

2019-06-21 11:43:35: D Output of last command execution:

…

…

2019-06-21 11:53:20: D Executing: /u01/app/18.0.0/grid/bin/crsctl query crs activeversion

2019-06-21 11:53:20: D Exit code: 0

2019-06-21 11:53:20: D Output of last command execution:

Oracle Clusterware active version on the cluster is [18.0.0.0.0]

2019-06-21 11:53:20: I Starting the Cluster Health Analysis Resource

2019-06-21 11:53:20: D Executing: /u01/app/18.0.0/grid/bin/srvctl start cha

2019-06-21 11:53:22: D Exit code: 0

2019-06-21 11:53:22: D Output of last command execution:

2019-06-21 11:53:22: I MGMTDB Successfully moved to +RECOC8!

[grid@exa01vm08 +ASM1]$

The output above was cropped and you can see in raw here. And after that, doing the same query as above, I checked again the extent map:

SQL> select max(AU_KFFXP) from X$KFFXP where GROUP_KFFXP=1;

MAX(AU_KFFXP)

-------------

       114132

SQL> select NUMBER_KFFXP from X$KFFXP where AU_KFFXP = 114132;

NUMBER_KFFXP

------------

         255

SQL> select name from v$asm_alias where FILE_NUMBER = 255 and GROUP_NUMBER = 1;

NAME

----------------------------------------------------------------------

REGISTRY.255.933784121

SQL>

So, more one file to move and not it is related to OCR. To move I made:

[root@exa01vm08 ~]# export ORACLE_HOME=/u01/app/18.0.0/grid

[root@exa01vm08 ~]# export PATH=$ORACLE_HOME/bin:$PATH

[root@exa01vm08 ~]#

[root@exa01vm08 ~]# ocrconfig -add +RECOC8

[root@exa01vm08 ~]# ocrcheck

Status of Oracle Cluster Registry is as follows :

         Version                  :          4

         Total space (kbytes)     :     491684

         Used space (kbytes)      :      88624

         Available space (kbytes) :     403060

         ID                       :  354072626

         Device/File Name         :    +DATAC8

                                    Device/File integrity check succeeded

         Device/File Name         :    +RECOC8

                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

[root@exa01vm08 ~]#

[root@exa01vm08 ~]# ocrconfig -delete +DATAC8

[root@exa01vm08 ~]#

[root@exa01vm08 ~]# ocrcheck

Status of Oracle Cluster Registry is as follows :

         Version                  :          4

         Total space (kbytes)     :     491684

         Used space (kbytes)      :      88624

         Available space (kbytes) :     403060

         ID                       :  354072626

         Device/File Name         :    +RECOC8

                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

[root@exa01vm08 ~]#

Above I added a new place to OCR store the OCR files(RECOC8) and deleted the other (DATAC8). So, we still have more to move (votedisks, older OCR backups and ASM password file). To move votedisks I made:

[root@exa01vm08 ~]# crsctl query css votedisk

##  STATE    File Universal Id                File Name Disk group

--  -----    -----------------                --------- ---------
  1. ONLINE 077367b255804f0abf804a3a3ca8045d (o/200.200.10.11;200.200.10.12/DATAC8_CD_02_exaceladm04) [DATAC8]
  2. ONLINE 59ef748a7d2a4f43bffb54a90fe1b1a9 (o/200.200.10.13;200.200.10.14/DATAC8_CD_02_exaceladm05) [DATAC8]
  3. ONLINE cbaad30809f94fdcbfe5d60f6529ba63 (o/200.200.10.7;200.200.10.8/DATAC8_CD_02_exaceladm02) [DATAC8]
Located 3 voting disk(s).

[root@exa01vm08 ~]#

[root@exa01vm08 ~]# crsctl replace votedisk +RECOC8

Successful addition of voting disk 081e9e767bc44ff2bff6067229378db5.

Successful addition of voting disk 82649e3309d34fa0bf4fd3c89c93e42f.

Successful addition of voting disk 57f3ec44b44b4fcdbf35e716e13011e9.

Successful deletion of voting disk 077367b255804f0abf804a3a3ca8045d.

Successful deletion of voting disk 59ef748a7d2a4f43bffb54a90fe1b1a9.

Successful deletion of voting disk cbaad30809f94fdcbfe5d60f6529ba63.

Successfully replaced voting disk group with +RECOC8.

CRS-4266: Voting file(s) successfully replaced

[root@exa01vm08 ~]#

[root@exa01vm08 ~]# crsctl query css votedisk

##  STATE    File Universal Id                File Name Disk group

--  -----    -----------------                --------- ---------
  1. ONLINE 081e9e767bc44ff2bff6067229378db5 (o/200.200.10.11;200.200.10.12/RECOC8_CD_02_exaceladm04) [RECOC8]
  2. ONLINE 82649e3309d34fa0bf4fd3c89c93e42f (o/200.200.10.9;200.200.10.10/RECOC8_CD_02_exaceladm03) [RECOC8]
  3. ONLINE 57f3ec44b44b4fcdbf35e716e13011e9 (o/200.200.10.13;200.200.10.14/RECOC8_CD_03_exaceladm05) [RECOC8]
Located 3 voting disk(s).

[root@exa01vm08 ~]#

And to backups for OCR:

[root@exa01vm08 ~]# ocrconfig -showbackuploc

The Oracle Cluster Registry backup location is [+DATAC8].

[root@exa01vm08 ~]# ocrconfig -backuploc +RECOC8

[root@exa01vm08 ~]#

[root@exa01vm08 ~]#

[root@exa01vm08 ~]# ocrconfig -showbackuploc

The Oracle Cluster Registry backup location is [+RECOC8].

[root@exa01vm08 ~]#

[root@exa01vm08 ~]# ocrconfig -manualbackup

exa01vm08     2019/06/21 12:08:05     +RECOC8:/exa-cl8/OCRBACKUP/backup_20190621_120805.ocr.4722.1011528485     671056737

exa01vm08     2018/08/16 12:38:11     +DATAC8:/exa-cl8/OCRBACKUP/backup_20180816_123811.ocr.1511.984314291     671056737

exa01vm08     2018/08/09 15:56:08     +DATAC8:/exa-cl8/OCRBACKUP/backup_20180809_155608.ocr.1508.983721369     2960767134

[root@exa01vm08 ~]#

After one more backup, I deleted manually the old OCR backups. And to move ASM password:

[grid@exa01vm08 +ASM1]$ asmcmd pwget --asm

+DATAC8/orapwASM

[grid@exa01vm08 +ASM1]$ srvctl config asm -detail

ASM home: <CRS home>

Password file: +DATAC8/orapwASM

Backup of Password file: +DATAC8/orapwASM_backup

ASM listener: LISTENER

ASM is enabled.

ASM is individually enabled on nodes:

ASM is individually disabled on nodes:

ASM instance count: ALL

Cluster ASM listener: ASMNET1LSNR_ASM

[grid@exa01vm08 +ASM1]$

[grid@exa01vm08 +ASM1]$

[grid@exa01vm08 +ASM1]$ asmcmd pwmove --asm +DATAC8/orapwASM +RECOC8/orapwASM -f

moving +DATAC8/orapwASM -> +RECOC8/orapwASM

[grid@exa01vm08 +ASM1]$

[grid@exa01vm08 +ASM1]$ srvctl config asm -detail

ASM home: <CRS home>

Password file: +RECOC8/orapwASM

Backup of Password file: +DATAC8/orapwASM_backup

ASM listener: LISTENER

ASM is enabled.

ASM is individually enabled on nodes:

ASM is individually disabled on nodes:

ASM instance count: ALL

Cluster ASM listener: ASMNET1LSNR_ASM

[grid@exa01vm08 +ASM1]$

[grid@exa01vm08 +ASM1]$ asmcmd pwmove --asm +DATAC8/orapwASM_backup +RECOC8/orapwASM_backup -f

moving +DATAC8/orapwASM_backup -> +RECOC8/orapwASM_backup

[grid@exa01vm08 +ASM1]$

[grid@exa01vm08 +ASM1]$

Maybe you don’t need to move everything, I recommend that after every move you check the extent map for ASM and verify if the value less the minimum size per disk that you want.

Since I changed a lot of things I made a restart for the cluster in both nodes:

[root@exa01vm08 ~]# crsctl stop cluster -all

CRS-2673: Attempting to stop 'ora.crsd' on 'exa01vm08'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'exa01vm08'

…

CRS-2673: Attempting to stop 'ora.diskmon' on 'exa01vm08'

CRS-2677: Stop of 'ora.diskmon' on 'exa01vm08' succeeded

[root@exa01vm08 ~]#

[root@exa01vm08 ~]# crsctl start cluster -all

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'exa01vm08'

…

…

CRS-2672: Attempting to start 'ora.crsd' on 'exa01vm08'

CRS-2676: Start of 'ora.crsd' on 'exa01vm08' succeeded

[root@exa01vm08 ~]#

And after all of these moves I got the extent map with:

SQL> select max(AU_KFFXP) from X$KFFXP where GROUP_KFFXP=1;

MAX(AU_KFFXP)

-------------

        15149

SQL>

SQL> select NUMBER_KFFXP from X$KFFXP where AU_KFFXP = 15149;

NUMBER_KFFXP

------------

           1

SQL>

Basically, this means that something is around 60Gb of some disk. But the link is for a file #1. So, to check what is this you can query inside X$KFFXP:

SQL> select NUMBER_KFFXP "ASM file number",

  2  DECODE (NUMBER_KFFXP,

  3          1, 'File directory'

  4          , 2, 'Disk directory'

  5          , 3, 'Active change directory'

  6          , 4, 'Continuing operations directory'

  7          , 5, 'Template directory'

  8          , 6, 'Alias directory'

  9          , 7, 'ADVM file directory'

 10          , 8, 'Disk free space directory'

 11          , 9, 'Attributes directory'

 12          , 10, 'ASM User directory'

 13          , 11, 'ASM user group directory'

 14          , 12, 'Staleness directory'

 15          , 253, 'spfile for ASM instance'

 16          , 254, 'Stale bit map space registry '

 17          , 255, 'Oracle Cluster Repository registry') "ASM metadata file name",

 18  count(AU_KFFXP) "Allocation units"

 19  from X$KFFXP

 20  where GROUP_KFFXP=1

 21  group by NUMBER_KFFXP

 22  order by 1;

ASM file number ASM metadata file name             Allocation units

--------------- ---------------------------------- ----------------

              1 File directory                                   15

              2 Disk directory                                    3

              3 Active change directory                          69

              4 Continuing operations directory                   6

              5 Template directory                                3

              6 Alias directory                                   3

              8 Disk free space directory                         3

              9 Attributes directory                              3

             12 Staleness directory                               3

             13                                                   3

             16                                                   3

ASM file number ASM metadata file name             Allocation units

--------------- ---------------------------------- ----------------

            120                                                   3

            121                                                 180

            253 spfile for ASM instance                           2

            254 Stale bit map space registry                     18

            255 Oracle Cluster Repository registry               83

            262                                                  99

            321                                                 185

            428                                                 221

            492                                                 305

            551                                                 129

            584                                                 705

ASM file number ASM metadata file name             Allocation units

--------------- ---------------------------------- ----------------

            740                                                1865

           1003                                                 105

           1417                                                   2

           4708                                                   6

           4710                                                   6

           4713                                                   6

           4716                                                   6

           4719                                                   6

           4722                                                   6

           4731                                                  52

           4733                                                  52

ASM file number ASM metadata file name             Allocation units

--------------- ---------------------------------- ----------------

           4736                                                  52

           4739                                                 379

           4742                                                 379

           4744                                                  52

           4747                                                 379

           4750                                                  24

           4753                                                 129

           4756                                                5125

           4759                                                7685

           4762                                                1029

           4765                                                 517

ASM file number ASM metadata file name             Allocation units

--------------- ---------------------------------- ----------------

           4768                                                  52

           4771                                                   4

           4773                                               11819

47 rows selected.

SQL>

In this case, it is a file directory ASM, but I will not try to move because it is even impossible in my case because does not appears in ASMCMD command. Basically, can be internal file/directory that we can’t see. Since I know that the diskgroup is empty and the allocation is related to internal ASM, I continued.

Remember at the end of the procedure, after shrinking in grid disk (that I will show later) to move again these files to original diskgroup. I will not cover this new move here in the post but the steps are the same that I showed before.

The shrinking part start in ASM it is simple and you just need to define the value that you want per disk and reduce them. The most critical part here is to define one size for the disk that it is aligned with 16 MB of the grid disk. Different from when you are increasing the space, here, for shrink, you start to define the size in ASM side and after go to grid disk. And since the grid disk in storage cell round down the value to the nearest 16MB value, you can misalign and have ASM disk with a different value than grid disk (and this will be REALLY BAD). A trick secret is to think directly in GB to avoid the error.

For 16 Mb explanation you can check in the Exadata docs https://docs.oracle.com/en/engineered-systems/exadata-database-machine/sagug/exadata-administering-asm.html#GUID-42DA2512-667D-443C-93C5-6E5110DFAE21:

Find the closest 16 MB boundary for the new grid disk size. If you do not perform this check, then the cell will round down the grid disk size to the nearest 16 MB boundary automatically, and you could end up with a mismatch in size between the Oracle ASM disks and the grid disks.

Since I defined to reduce the DATAC8 to around 1.8TB I defined the disks with 30GB:

SQL> select ((30*12)*5) as sizeGB FROM dual;

    SIZEGB

----------

      1800

SQL>

Remember in the beginning that you need to know your environment? Here I have 12 disks per cell and 5 cells:

SQL> set linesize 250

SQL> select dg.name, d.failgroup, d.state, d.header_status, d.mount_status, d.mode_status, count(1) num_disks

  2  from v$asm_disk d, v$asm_diskgroup dg

  3  where d.group_number = dg.group_number

  4  and dg.name IN ('DATAC8')

  5  group by dg.name, d.failgroup, d.state, d.header_status, d.mount_status, d.mode_status

  6  order by 1,2,3;

NAME   FAILGROUP                      STATE    HEADER_STATU MOUNT_S MODE_ST  NUM_DISKS

------ ------------------------------ -------- ------------ ------- ------- ----------

DATAC8 EXACELADM01                    NORMAL   MEMBER       CACHED  ONLINE          12

DATAC8 EXACELADM02                    NORMAL   MEMBER       CACHED  ONLINE          12

DATAC8 EXACELADM03                    NORMAL   MEMBER       CACHED  ONLINE          12

DATAC8 EXACELADM04                    NORMAL   MEMBER       CACHED  ONLINE          12

DATAC8 EXACELADM05                    NORMAL   MEMBER       CACHED  ONLINE          12

SQL>

After choosing the size that you want, you need to check if all the grid disk as online. It is not recommended to do the shrink with missing disks or even faulty failgroups. To check both at the same time you can do:

SQL> col path format a100

SQL> select dg.name, d.failgroup, d.path

  2  from v$asm_disk d, v$asm_diskgroup dg

  3  where d.group_number = dg.group_number

  4  and dg.name IN ('DATAC8') and state = 'NORMAL'

  5  order by 1,2,3;

NAME                           FAILGROUP                      PATH

------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_00_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_01_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_02_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_03_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_04_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_05_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_06_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_07_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_08_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_09_exaceladm01

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_10_exaceladm01

NAME                           FAILGROUP                      PATH

------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------

DATAC8                         EXACELADM01                    o/200.200.10.5;200.200.10.6/DATAC8_CD_11_exaceladm01

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_00_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_01_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_02_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_03_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_04_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_05_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_06_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_07_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_08_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_09_exaceladm02

NAME                           FAILGROUP                      PATH

------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_10_exaceladm02

DATAC8                         EXACELADM02                    o/200.200.10.7;200.200.10.8/DATAC8_CD_11_exaceladm02

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_00_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_01_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_02_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_03_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_04_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_05_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_06_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_07_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_08_exaceladm03

NAME                           FAILGROUP                      PATH

------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_09_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_10_exaceladm03

DATAC8                         EXACELADM03                    o/200.200.10.9;200.200.10.10/DATAC8_CD_11_exaceladm03

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_00_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_01_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_02_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_03_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_04_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_05_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_06_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_07_exaceladm04

NAME                           FAILGROUP                      PATH

------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_08_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_09_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_10_exaceladm04

DATAC8                         EXACELADM04                    o/200.200.10.11;200.200.10.12/DATAC8_CD_11_exaceladm04

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_00_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_01_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_02_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_03_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_04_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_05_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_06_exaceladm05

NAME                           FAILGROUP                      PATH

------------------------------ ------------------------------ ----------------------------------------------------------------------------------------------------

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_07_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_08_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_09_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_10_exaceladm05

DATAC8                         EXACELADM05                    o/200.200.10.13;200.200.10.14/DATAC8_CD_11_exaceladm05

60 rows selected.

SQL>

After all the checks 16MB align, missing disks, and online failgroups you can do the resize in ASM:

SQL> alter diskgroup DATAC8 resize all size 30720M rebalance power 1024;

Diskgroup altered.

SQL>

As you can see above, I specified in MB and I made this to show you where the error can occurs. If I defined (as example) disk size as 30700M it will not be aligned to 16MB (30700/16 = 1918,75) and the griddisk will be 30688M and if ASM allocate something in the end of the disk you will corrupt something.

After executing the change in ASM you just continue after having nothing in v$asm_operation. And for ASM now you see the size for your diskgroup:

SQL> select name, total_mb, free_mb, total_mb - free_mb used_mb, round(100*free_mb/total_mb,2) pct_free

  2  from v$asm_diskgroup

  3  order by 1;

NAME                             TOTAL_MB    FREE_MB    USED_MB   PCT_FREE

------------------------------ ---------- ---------- ---------- ----------

DATAC8                            1843200    1837896       5304      99.71

RECOC8                            6144000    6016048     127952      97.92

SQL>

After shrinking in ASM side, you need to reduce in storage side the grid disk to release the space to celldisk. The procedure is the same for the increase and you use ALTER GRIDDISK to specify the new value.

Just to show, that before the shrink in storage cell I have disk defined for grid disk (487GB) and celldisk (346GB free):

CellCLI> list griddisk where name = 'DATAC8_CD_00_exaceladm01' detail;

         name:                   DATAC8_CD_00_exaceladm01

         asmDiskGroupName:       DATAC8

         asmDiskName:            DATAC8_CD_00_EXACELADM01

         asmFailGroupName:       EXACELADM01

         availableTo:

         cachedBy:               FD_00_exaceladm01

         cachingPolicy:          default

         cellDisk:               CD_00_exaceladm01

         comment:                "Cluster exa-cl8 diskgroup DATAC8"

         creationTime:           2017-01-20T16:19:29+01:00

         diskType:               HardDisk

         errorCount:             0

         id:                     0ddfb7c0-1351-4df3-b5d6-82d3bbffa6e2

         size:                   487G

         status:                 active

CellCLI>

CellCLI> list celldisk where name = 'CD_00_exaceladm01' detail;

         name:                   CD_00_exaceladm01

         comment:

         creationTime:           2016-11-29T10:44:55+01:00

         deviceName:             /dev/sda

         devicePartition:        /dev/sda3

         diskType:               HardDisk

         errorCount:             0

         freeSpace:              346.0625G

         id:                     f73cfdb7-aa40-47d9-99e0-e39e456b0b55

         physicalDisk:           PUUK3V

         size:                   7.1192474365234375T

         status:                 normal

CellCLI>

As before, you have two option: execute manually disk by disk or using the script with dcli. Below you see that I created and called the script (I cropped the output but you can see the RAW execution here):

[DOM0 - root@exadbadm01 tmp]$  vi Change_Disk_Size_Of_DATAC8_Cluster_To_30G.sh

[DOM0 - root@exadbadm01 tmp]$

[DOM0 - root@exadbadm01 tmp]$  chmod +x Change_Disk_Size_Of_DATAC8_Cluster_To_30G.sh

[DOM0 - root@exadbadm01 tmp]$

[DOM0 - root@exadbadm01 tmp]$  cat Change_Disk_Size_Of_DATAC8_Cluster_To_30G.sh

dcli -l root -c exaceladm01 cellcli -e ALTER GRIDDISK DATAC8_CD_00_EXACELADM01 size=30720M;

dcli -l root -c exaceladm02 cellcli -e ALTER GRIDDISK DATAC8_CD_00_EXACELADM02 size=30720M;

…

…

dcli -l root -c exaceladm02 cellcli -e ALTER GRIDDISK DATAC8_CD_11_EXACELADM02 size=30720M;

dcli -l root -c exaceladm03 cellcli -e ALTER GRIDDISK DATAC8_CD_11_EXACELADM03 size=30720M;

dcli -l root -c exaceladm04 cellcli -e ALTER GRIDDISK DATAC8_CD_11_EXACELADM04 size=30720M;

dcli -l root -c exaceladm05 cellcli -e ALTER GRIDDISK DATAC8_CD_11_EXACELADM05 size=30720M;

[DOM0 - root@exadbadm01 tmp]$

[DOM0 - root@exadbadm01 tmp]$

[DOM0 - root@exadbadm01 tmp]$  ./Change_Disk_Size_Of_DATAC8_Cluster_To_30G.sh

exaceladm01: GridDisk DATAC8_CD_00_exaceladm01 successfully altered

exaceladm02: GridDisk DATAC8_CD_00_exaceladm02 successfully altered

…

…

exaceladm02: GridDisk DATAC8_CD_11_exaceladm02 successfully altered

exaceladm03: GridDisk DATAC8_CD_11_exaceladm03 successfully altered

exaceladm04: GridDisk DATAC8_CD_11_exaceladm04 successfully altered

exaceladm05: GridDisk DATAC8_CD_11_exaceladm05 successfully altered

[DOM0 - root@exadbadm01 tmp]$

Check again that I used the value defined in MB, 30720MB in this case. Again, be careful with 16MB align. After the change I have in storage cell:

CellCLI> list griddisk where name = 'DATAC8_CD_00_exaceladm01' detail;

         name:                   DATAC8_CD_00_exaceladm01

         asmDiskGroupName:       DATAC8

         asmDiskName:            DATAC8_CD_00_EXACELADM01

         asmFailGroupName:       EXACELADM01

         availableTo:

         cachedBy:               FD_00_exaceladm01

         cachingPolicy:          default

         cellDisk:               CD_00_exaceladm01

         comment:                "Cluster exa-cl8 diskgroup DATAC8"

         creationTime:           2017-01-20T16:19:29+01:00

         diskType:               HardDisk

         errorCount:             0

         id:                     0ddfb7c0-1351-4df3-b5d6-82d3bbffa6e2

         size:                   30G

         status:                 active

CellCLI> list celldisk where name = 'CD_00_exaceladm01' detail;

         name:                   CD_00_exaceladm01

         comment:

         creationTime:           2016-11-29T10:44:55+01:00

         deviceName:             /dev/sda

         devicePartition:        /dev/sda3

         diskType:               HardDisk

         errorCount:             0

         freeSpace:              803.0625G

         id:                     f73cfdb7-aa40-47d9-99e0-e39e456b0b55

         physicalDisk:           PUUK3V

         size:                   7.1192474365234375T

         status:                 normal

CellCLI>

Execute the shrink for Exadata is something that you don’t realize every day in your daily tasks. I suppose that not in a quarterly task because today is always “add, add, add”. Besides the increase of space, the shrink is trickier, and you really need to take care for more steps. Maybe you need to move some data, restart the cluster, move data again.

But, the most critical part is the 16MB alignment, more than when you add space. This is important because for shrink you define the value in two places: for ASM and for grid disk. If you choose a bad value, the size for grid disk can differ ASM and you will corrupt something. As I told, if you think directly in GB instead of MB for disk sizes, you are safer because GB is always compatible with 16MB.

This article Shrink ASM Diskgroup and Exadata Grid Disks was initially published on Fernando Simon’s Blog