Enterprise Manager 24ai & Fleet Patching

Enterprise Manager is usually installed once, by hand, by someone who then leaves. This build treats it as infrastructure code: silent installs, Ansible for OS preparation, emcli for configuration, and a written record of every step taken from real command output rather than reconstructed afterwards.

The certification work came first and settled two constraints before anything was installed. Oracle Linux 10 is not certified for the management server, so the OMS runs OL9. And because the repository runs 23ai, capabilities that normally need a separate target database can run inside the repository itself.

Fleet Patching and Provisioning

FPP is the piece that makes patching an estate tractable — gold images built once and deployed to many homes, rather than an opatch run per server. Standing it up meant a single-node Grid Infrastructure cluster with Flex ASM, three iSCSI LUNs presented as ASM disk groups, an ACFS checkpoint filesystem, and the RHP server registered as a target in Enterprise Manager.

Four gold images are published, covering both 19c and 23ai lines.

Three faults worth documenting

19c Grid Infrastructure will not run on UEK7. root.sh failed at the OCR write with ORA-15081 and PROC-26 Insufficient quorum. The obvious reading is a storage problem — but fio submitted I/O to the same devices successfully while ASM’s own io_submit failed. The fault was the kernel, not the disks. Diagnosing that saved a pointless storage migration.

Recurring CRSD fencing. The cluster daemon died roughly every two-and-a-half to four hours with OCR disk fencing and no underlying SCSI error, on two different kernel builds. Patching Grid Infrastructure from 19.25 to 19.32 fixed it — and, importantly, the fix was proven rather than assumed: a watcher recorded 149 continuous online heartbeats well past the previous failure window, with zero recurrences. Claiming a fix for an intermittent fault without passing the failure window is not a fix, it is a hope.

An Enterprise Manager product defect. Every call from the EM Fleet Maintenance UI to the FPP server failed with a JSON parse error. The cause: EM sends an Accept-Encoding: gzip header but never inflates the compressed response it gets back. The RHP side was searched exhaustively — server configuration, properties files, admin JARs, deployed web applications, the CLI — and no supported server-side fix exists.

The decision was to stop looking and drive FPP through rhpctl, which is fully functional, with a support request as the proper route to a fix. Recognising an unwinnable fight early and routing around it is worth more than another day of searching.

Local AI instead of cloud

The build connects Enterprise Manager’s AI assistant to a self-hosted, OpenAI-compatible LLM gateway rather than a cloud generative-AI service. The release update ships a generic LLM provider mechanism — a credential type with a free-text provider URL and a management-server-level allow-list — that Oracle has not publicly documented. Proving it worked on the box was a prerequisite to using it, and it means database telemetry and DBA questions never leave the network.

Operational honesty

The FPP host is documented as non-standard, in writing, on the box itself. Its reboot persistence depends on a specific driver-loading order that a well-meaning administrator would “fix” by restoring the default configuration — and thereby break the boot. A README on the host and a warning in the repository say exactly that, because the most dangerous state for infrastructure is a workaround nobody recorded.