Recently, in March, I made the reimage from an X5-2 HA ODA and saw a strange behavior during the diskgroup creation and couldn’t reproduce (because involve reimaging again). Basically, the FLASH diskgroup was not created.
But in last May I reimaged another ODA using the same patch/imageversion (18.3.0.0 – Patch 27604623) and was possible to verify again. In both cases, I created the appliance using the CLI “odacli create-appliance” using JSON file because the network uses VLAN (what it is impossible to create using the web interface), and both appliances are identical (X5-2 HA with SSD for RECO and FLASH).
To reimage, I followed the steps in the docs for this version and used the ISO to do the baremetal procedure. If you look in the docs about the options for storage (check here) you can see that there is no single reference to use FLASH diskgroup (or that you need to do that). Checking in the readme/reference JSON files that exist in the folder “/opt/oracle/dcs/sample” under file “sample-oda-ha-json-readme.txt”:

Recently, in March, I made the reimage from an X5-2 HA ODA and saw a strange behavior during the diskgroup creation and couldn’t reproduce (because involve reimaging again). Basically, the FLASH diskgroup was not created.

But in last May I reimaged another ODA using the same patch/imageversion (18.3.0.0 – Patch 27604623) and was possible to verify again. In both cases, I created the appliance using the CLI “odacli create-appliance” using JSON file because the network uses VLAN (what it is impossible to create using the web interface), and both appliances are identical (X5-2 HA with SSD for RECO and FLASH).

To reimage, I followed the steps in the docs for this version and used the ISO to do the baremetal procedure. If you look in the docs about the options for storage (check here) you can see that there is no single reference to use FLASH diskgroup (or that you need to do that). Checking in the readme/reference JSON files that exist in the folder “/opt/oracle/dcs/sample” under file “sample-oda-ha-json-readme.txt”:

grid:

    diskGroup: (ODA HA contains DATA, RECO and REDO Diskgroups)

        diskgroupName: DATA|RECO|REDO

        redundancy: Normal|High

            If the system has less than 5 NVMe storage devices, redundancy is Normal.

            If the system has 5 or more NVMes, then Normal or High are supported.

        diskPercentage: Percentage of NVMe drive capacity is used for this particular diskgroup.

And the example that come with image (sample-oda-ha.json):

  "grid" : {

    "diskGroup" : [ {

      "diskGroupName" : "DATA",

      "redundancy" : "",

      "diskPercentage" :80

    }, {

      "diskGroupName" : "RECO",

      "redundancy" : "",

      "diskPercentage" :20

    }, {

      "diskGroupName": "REDO",

      "diskPercentage": 100,

      "redundancy": "HIGH"

    } ],

As you can see there is no reference to FLASH, or even if it is possible to use that (because the options are diskgroupName: DATA|RECO|REDO). Based on that (and supposing that FLASH will be created automatically – since X5-2 have these SSD disk dedicated to that) I made JSON file with:

  "grid" : {

    "diskGroup" : [ {

      "diskGroupName" : "DATA",

      "redundancy" : "NORMAL",

      "disks" : null,

      "diskPercentage" : 90

    }, {

      "diskGroupName" : "RECO",

      "redundancy" : "NORMAL",

      "disks" : null,

      "diskPercentage" : 10

    }, {

      "diskGroupName" : "REDO",

      "redundancy" : "HIGH",

      "disks" : null,

      "diskPercentage" : null

    } ],

And after the create-appliance command I got just these filegroups in ASM:

ASMCMD> lsdg

State    Type    Rebal  Sector  Logical_Sector  Block       AU   Total_MB    Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name

MOUNTED  NORMAL  N         512             512   4096  4194304  108019712  108000740          6751232        50624754              0             Y  DATA/

MOUNTED  NORMAL  N         512             512   4096  4194304   11997184   11995456           749824         5622816              0             N  RECO/

MOUNTED  HIGH    N         512             512   4096  4194304     762880     749908           190720          186396              0             N  REDO/

ASMCMD>

And the describe-system reports:

[root@ODA1 ~]# /opt/oracle/dcs/bin/odacli describe-system

Appliance Information

----------------------------------------------------------------

                     ID: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX

               Platform: X5-2-HA

        Data Disk Count: 24

         CPU Core Count: 6

                Created: March 26, 2019 11:56:24 AM CET

System Information

----------------------------------------------------------------

                   Name: ODA

            Domain Name: XYZ.XYZ

              Time Zone: Europe/Luxembourg

             DB Edition: EE

            DNS Servers: 1.1.1.1 2.2.2.2

            NTP Servers: 3.3.3.3

Disk Group Information

----------------------------------------------------------------

DG Name                   Redundancy                Percentage

------------------------- ------------------------- ------------

Data                      Normal                    90

Reco                      Normal                    10

Redo                      High                      100

[root@ODA1 ~]#

After that, the only option was creating the FLASH manually:

SQL> create diskgroup FLASH normal redundancy

  2     DISK 'AFD:SSD_E0_S16_1320409344P1' NAME SSD_E0_S16_1320409344P1

  3     ,  'AFD:SSD_E0_S17_1320408816P1' NAME SSD_E0_S17_1320408816P1

  4     ,  'AFD:SSD_E0_S18_1320404784P1' NAME SSD_E0_S18_1320404784P1

  5     ,  'AFD:SSD_E0_S19_1320406740P1' NAME SSD_E0_S19_1320406740P1

  6     attribute  'COMPATIBLE.ASM' =   '18.0.0.0.0'

  7     , 'COMPATIBLE.rdbms' = '12.1.0.2'

  8     , 'compatible.advm' = '18.0.0.0'

  9     , 'au_size'='4M'

  10  ;

Diskgroup created.

SQL> select NAME, SECTOR_SIZE, ALLOCATION_UNIT_SIZE, COMPATIBILITY, DATABASE_COMPATIBILITY from v$asm_diskgroup;

NAME                           SECTOR_SIZE ALLOCATION_UNIT_SIZE COMPATIBILITY                                                DATABASE_COMPATIBILITY

------------------------------ ----------- -------------------- ------------------------------------------------------------ ------------------------------------------------------------

DATA                                   512              4194304 18.0.0.0.0                                                   12.1.0.2.0

RECO                                   512              4194304 18.0.0.0.0                                                   12.1.0.2.0

REDO                                   512              4194304 18.0.0.0.0                                                   12.1.0.2.0

FLASH                                  512              4194304 18.0.0.0.0                                                   12.1.0.2.0

SQL>

Reimage again, or cleanup everything, sometimes it is not an option. Change the JSON file and modify the parameter “dbOnFlashStorage” it is not correct since will fail again as you can see in the release notes for version 12.2.1.3.0 that says in workaround: “Provide information for DATA, REDO, RECO, and FLASH disk groups at the time of provisioning. If you have provisioned your environment without creating the FLASH disk group, then create the FLASH disk group manually. This issue is tracked with Oracle bug 27721310.

For the second reimage I specified my JSON file as:

"grid" : {

    "diskGroup" : [ {

      "diskGroupName" : "DATA",

      "redundancy" : "NORMAL",

      "disks" : null,

      "diskPercentage" : 90

    }, {

      "diskGroupName" : "RECO",

      "redundancy" : "NORMAL",

      "disks" : null,

      "diskPercentage" : 10

    }, {

      "diskGroupName" : "REDO",

      "redundancy" : "HIGH",

      "disks" : null,

      "diskPercentage" : null

    }, {

      "diskGroupName" : "FLASH",

      "redundancy" : "NORMAL",

      "disks" : null,

      "diskPercentage" : null

    } ],

And the describe for system now says:

[root@OAK1 ~]# /opt/oracle/dcs/bin/odacli describe-system

Appliance Information

----------------------------------------------------------------

                     ID: XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX

               Platform: X5-2-HA

        Data Disk Count: 24

         CPU Core Count: 36

                Created: May 22, 2019 11:03:36 AM CEST

System Information

----------------------------------------------------------------

                   Name: OAK

            Domain Name: XYZ.XYZ

              Time Zone: Europe/Luxembourg

             DB Edition: EE

            DNS Servers: 1.1.1.1 2.2.2.2

            NTP Servers: 3.3.3.3

Disk Group Information

----------------------------------------------------------------

DG Name                   Redundancy                Percentage

------------------------- ------------------------- ------------

Data                      Normal                    90

Reco                      Normal                    10

Redo                      High                      100

Flash                     Normal                    100

[root@OAK1 ~]#

And I can confirm everything as expected in ASM:

ASMCMD> lsdg

State    Type    Rebal  Sector  Logical_Sector  Block       AU   Total_MB    Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name

MOUNTED  NORMAL  N         512             512   4096  4194304  108019712  108007188          6751232        50627978              0             Y  DATA/

MOUNTED  NORMAL  N         512             512   4096  4194304    1525760    1518672           381440          568616              0             N  FLASH/

MOUNTED  NORMAL  N         512             512   4096  4194304   11997184   11995736           749824         5622956              0             N  RECO/

MOUNTED  HIGH    N         512             512   4096  4194304     762880     749908           190720          186396              0             N  REDO/

ASMCMD>

If you compare the job report for the new create-appliance execution:

Post cluster OAKD configuration          May 22, 2019 11:53:45 AM CEST       May 22, 2019 11:59:11 AM CEST       Success

Disk group 'RECO'creation                May 22, 2019 11:59:20 AM CEST       May 22, 2019 11:59:38 AM CEST       Success

Disk group 'REDO'creation                May 22, 2019 11:59:38 AM CEST       May 22, 2019 11:59:47 AM CEST       Success

Disk group 'FLASH'creation               May 22, 2019 11:59:47 AM CEST       May 22, 2019 11:59:58 AM CEST       Success

Volume 'commonstore'creation             May 22, 2019 11:59:58 AM CEST       May 22, 2019 12:00:48 PM CEST       Success

It is different from the previous:

Post cluster OAKD configuration          March 26, 2019 1:47:06 PM CET       March 26, 2019 1:52:32 PM CET       Success

Disk group 'RECO'creation                March 26, 2019 1:52:41 PM CET       March 26, 2019 1:52:57 PM CET       Success

Disk group 'REDO'creation                March 26, 2019 1:52:57 PM CET       March 26, 2019 1:53:07 PM CET       Success

Volume 'commonstore'creation             March 26, 2019 1:53:07 PM CET       March 26, 2019 1:53:53 PM CET       Success

So, be careful during your ODA deployment, even if the doc says something, check again to avoid errors. It is important to know your environment and be aware of what is expected as a result. Here in this case, even if the docs says nothing about specifying the FLASH diskgroup, it is needed and if you forgot, not FLASH for you.

The article ODA, JSON and FLASH was initially published on Fernando Simon’s Blog