Package namespace registration and MyDomain fails when namespace or MyDomain contains any reserved keyword like "Apex".

1) When you change or setup a new My Domain, following the below document:-

Change Your My Domain Details

https://help.salesforce.com/s/articleView?id=xcloud.domain_name_rename.htm&type=5

For example : You entered any one of the below as your new my domain'

"apexabc", "abcapex", "abapexc" (The following strings contain 'apex' as a substring)

An error will popup The domain name "abapexc" is taken or cannot be used. Try another value

2) Similarly, if you create and register Your Namespace following the below document

Create and Register Your Namespace

https://developer.salesforce.com/docs/atlas.en-us.sfdx_dev.meta/sfdx_dev/sfdx_dev_unlocked_pkg_create_namespace.htm

For example : You entered any one of the below as your new namespace

"apexabc", "abcapex", "abapexc" (The following strings contain 'apex' as a substring)

An error will popup Invalid namespace.