Recently, in March, I made the reimage from an X5-2 HA ODA and saw a strange behavior during the diskgroup creation and couldn’t reproduce (because involve reimaging again). Basically, the FLASH diskgroup was not created.
But in last May I reimaged another ODA using the same patch/imageversion (18.3.0.0 – Patch 27604623) and was possible to verify again. In both cases, I created the appliance using the CLI “odacli create-appliance” using JSON file because the network uses VLAN (what it is impossible to create using the web interface), and both appliances are identical (X5-2 HA with SSD for RECO and FLASH).
To reimage, I followed the steps in the docs for this version and used the ISO to do the baremetal procedure. If you look in the docs about the options for storage (check here) you can see that there is no single reference to use FLASH diskgroup (or that you need to do that). Checking in the readme/reference JSON files that exist in the folder “/opt/oracle/dcs/sample” under file “sample-oda-ha-json-readme.txt”:
Recently, in March, I made the reimage from an X5-2 HA ODA and saw a strange behavior during the diskgroup creation and couldn’t reproduce (because involve reimaging again). Basically, the FLASH diskgroup was not created.
But in last May I reimaged another ODA using the same patch/imageversion (18.3.0.0 – Patch 27604623) and was possible to verify again. In both cases, I created the appliance using the CLI “odacli create-appliance” using JSON file because the network uses VLAN (what it is impossible to create using the web interface), and both appliances are identical (X5-2 HA with SSD for RECO and FLASH).
To reimage, I followed the steps in the docs for this version and used the ISO to do the baremetal procedure. If you look in the docs about the options for storage (check here) you can see that there is no single reference to use FLASH diskgroup (or that you need to do that). Checking in the readme/reference JSON files that exist in the folder “/opt/oracle/dcs/sample” under file “sample-oda-ha-json-readme.txt”:
grid: diskGroup: (ODA HA contains DATA, RECO and REDO Diskgroups) diskgroupName: DATA|RECO|REDO redundancy: Normal|High If the system has less than 5 NVMe storage devices, redundancy is Normal. If the system has 5 or more NVMes, then Normal or High are supported. diskPercentage: Percentage of NVMe drive capacity is used for this particular diskgroup.
And the example that come with image (sample-oda-ha.json):
"grid" : {
"diskGroup" : [ {
"diskGroupName" : "DATA",
"redundancy" : "",
"diskPercentage" :80
}, {
"diskGroupName" : "RECO",
"redundancy" : "",
"diskPercentage" :20
}, {
"diskGroupName": "REDO",
"diskPercentage": 100,
"redundancy": "HIGH"
} ],
As you can see there is no reference to FLASH, or even if it is possible to use that (because the options are diskgroupName: DATA|RECO|REDO). Based on that (and supposing that FLASH will be created automatically – since X5-2 have these SSD disk dedicated to that) I made JSON file with:
"grid" : {
"diskGroup" : [ {
"diskGroupName" : "DATA",
"redundancy" : "NORMAL",
"disks" : null,
"diskPercentage" : 90
}, {
"diskGroupName" : "RECO",
"redundancy" : "NORMAL",
"disks" : null,
"diskPercentage" : 10
}, {
"diskGroupName" : "REDO",
"redundancy" : "HIGH",
"disks" : null,
"diskPercentage" : null
} ],
And after the create-appliance command I got just these filegroups in ASM:
ASMCMD> lsdg State Type Rebal Sector Logical_Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name MOUNTED NORMAL N 512 512 4096 4194304 108019712 108000740 6751232 50624754 0 Y DATA/ MOUNTED NORMAL N 512 512 4096 4194304 11997184 11995456 749824 5622816 0 N RECO/ MOUNTED HIGH N 512 512 4096 4194304 762880 749908 190720 186396 0 N REDO/ ASMCMD> And the describe-system reports: [root@ODA1 ~]# /opt/oracle/dcs/bin/odacli describe-system Appliance Information ---------------------------------------------------------------- ID: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX Platform: X5-2-HA Data Disk Count: 24 CPU Core Count: 6 Created: March 26, 2019 11:56:24 AM CET System Information ---------------------------------------------------------------- Name: ODA Domain Name: XYZ.XYZ Time Zone: Europe/Luxembourg DB Edition: EE DNS Servers: 1.1.1.1 2.2.2.2 NTP Servers: 3.3.3.3 Disk Group Information ---------------------------------------------------------------- DG Name Redundancy Percentage ------------------------- ------------------------- ------------ Data Normal 90 Reco Normal 10 Redo High 100 [root@ODA1 ~]#
After that, the only option was creating the FLASH manually:
SQL> create diskgroup FLASH normal redundancy 2 DISK 'AFD:SSD_E0_S16_1320409344P1' NAME SSD_E0_S16_1320409344P1 3 , 'AFD:SSD_E0_S17_1320408816P1' NAME SSD_E0_S17_1320408816P1 4 , 'AFD:SSD_E0_S18_1320404784P1' NAME SSD_E0_S18_1320404784P1 5 , 'AFD:SSD_E0_S19_1320406740P1' NAME SSD_E0_S19_1320406740P1 6 attribute 'COMPATIBLE.ASM' = '18.0.0.0.0' 7 , 'COMPATIBLE.rdbms' = '12.1.0.2' 8 , 'compatible.advm' = '18.0.0.0' 9 , 'au_size'='4M' 10 ; Diskgroup created. SQL> select NAME, SECTOR_SIZE, ALLOCATION_UNIT_SIZE, COMPATIBILITY, DATABASE_COMPATIBILITY from v$asm_diskgroup; NAME SECTOR_SIZE ALLOCATION_UNIT_SIZE COMPATIBILITY DATABASE_COMPATIBILITY ------------------------------ ----------- -------------------- ------------------------------------------------------------ ------------------------------------------------------------ DATA 512 4194304 18.0.0.0.0 12.1.0.2.0 RECO 512 4194304 18.0.0.0.0 12.1.0.2.0 REDO 512 4194304 18.0.0.0.0 12.1.0.2.0 FLASH 512 4194304 18.0.0.0.0 12.1.0.2.0 SQL>
Reimage again, or cleanup everything, sometimes it is not an option. Change the JSON file and modify the parameter “dbOnFlashStorage” it is not correct since will fail again as you can see in the release notes for version 12.2.1.3.0 that says in workaround: “Provide information for DATA, REDO, RECO, and FLASH disk groups at the time of provisioning. If you have provisioned your environment without creating the FLASH disk group, then create the FLASH disk group manually. This issue is tracked with Oracle bug 27721310.”
For the second reimage I specified my JSON file as:
"grid" : {
"diskGroup" : [ {
"diskGroupName" : "DATA",
"redundancy" : "NORMAL",
"disks" : null,
"diskPercentage" : 90
}, {
"diskGroupName" : "RECO",
"redundancy" : "NORMAL",
"disks" : null,
"diskPercentage" : 10
}, {
"diskGroupName" : "REDO",
"redundancy" : "HIGH",
"disks" : null,
"diskPercentage" : null
}, {
"diskGroupName" : "FLASH",
"redundancy" : "NORMAL",
"disks" : null,
"diskPercentage" : null
} ],
And the describe for system now says:
[root@OAK1 ~]# /opt/oracle/dcs/bin/odacli describe-system Appliance Information ---------------------------------------------------------------- ID: XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX Platform: X5-2-HA Data Disk Count: 24 CPU Core Count: 36 Created: May 22, 2019 11:03:36 AM CEST System Information ---------------------------------------------------------------- Name: OAK Domain Name: XYZ.XYZ Time Zone: Europe/Luxembourg DB Edition: EE DNS Servers: 1.1.1.1 2.2.2.2 NTP Servers: 3.3.3.3 Disk Group Information ---------------------------------------------------------------- DG Name Redundancy Percentage ------------------------- ------------------------- ------------ Data Normal 90 Reco Normal 10 Redo High 100 Flash Normal 100 [root@OAK1 ~]#
And I can confirm everything as expected in ASM:
ASMCMD> lsdg State Type Rebal Sector Logical_Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name MOUNTED NORMAL N 512 512 4096 4194304 108019712 108007188 6751232 50627978 0 Y DATA/ MOUNTED NORMAL N 512 512 4096 4194304 1525760 1518672 381440 568616 0 N FLASH/ MOUNTED NORMAL N 512 512 4096 4194304 11997184 11995736 749824 5622956 0 N RECO/ MOUNTED HIGH N 512 512 4096 4194304 762880 749908 190720 186396 0 N REDO/ ASMCMD>
If you compare the job report for the new create-appliance execution:
Post cluster OAKD configuration May 22, 2019 11:53:45 AM CEST May 22, 2019 11:59:11 AM CEST Success Disk group 'RECO'creation May 22, 2019 11:59:20 AM CEST May 22, 2019 11:59:38 AM CEST Success Disk group 'REDO'creation May 22, 2019 11:59:38 AM CEST May 22, 2019 11:59:47 AM CEST Success Disk group 'FLASH'creation May 22, 2019 11:59:47 AM CEST May 22, 2019 11:59:58 AM CEST Success Volume 'commonstore'creation May 22, 2019 11:59:58 AM CEST May 22, 2019 12:00:48 PM CEST Success
It is different from the previous:
Post cluster OAKD configuration March 26, 2019 1:47:06 PM CET March 26, 2019 1:52:32 PM CET Success Disk group 'RECO'creation March 26, 2019 1:52:41 PM CET March 26, 2019 1:52:57 PM CET Success Disk group 'REDO'creation March 26, 2019 1:52:57 PM CET March 26, 2019 1:53:07 PM CET Success Volume 'commonstore'creation March 26, 2019 1:53:07 PM CET March 26, 2019 1:53:53 PM CET Success
So, be careful during your ODA deployment, even if the doc says something, check again to avoid errors. It is important to know your environment and be aware of what is expected as a result. Here in this case, even if the docs says nothing about specifying the FLASH diskgroup, it is needed and if you forgot, not FLASH for you.
The article ODA, JSON and FLASH was initially published on Fernando Simon’s Blog
