Enterprise integration • API security • Kubernetes
A 401 Unauthorized response usually sends teams straight to API Manager, client credentials, or application code. In this incident, all three were healthy. The failure was introduced earlier in the request path, where an NGINX ingress controller silently discarded the credential headers before MuleSoft could evaluate them.
Why This Case Matters
The incident shows why enterprise API troubleshooting must follow the complete request path, not only the API runtime. A valid Client ID Enforcement policy can produce the correct 401 response even when the actual defect sits in ingress configuration.
The Environment
The same MuleSoft API was deployed to two production Runtime Fabric environments:
- Shore production: MuleSoft Runtime Fabric deployed on Alibaba Cloud Container Service for Kubernetes (ACK).
- Ship production: MuleSoft Runtime Fabric deployed on a self-hosted, on-premises Kubernetes cluster.
Both deployments referenced the same managed API instance through the same API Manager api.id. The same client application credentials worked in the shore environment. On the ship endpoint, however, an equivalent request returned 401 Unauthorized.
curl --location 'https://<host>/<api>/v1/integrationTasks' \
--header 'client_id: <client-id>' \
--header 'client_secret: <client-secret>' \
--header 'Content-Type: application/json' \
--data '{}'The First Investigation Path
The initial checks focused on the most common MuleSoft causes:
- incorrect or environment-specific API Manager
api.id - missing, rejected, or revoked client application contract
- failed API autodiscovery or control-plane pairing
- Client ID Enforcement policy not deployed
- old, malformed, or incorrectly injected client credentials
The runtime logs eliminated a major branch of that investigation:
Applied policy client-id-enforcement... to API... in application...This confirmed that the managed API was discovered and that the Client ID Enforcement policy had been attached to the application. A nearby policy shut down normally entry was part of the normal stop/start and policy-redeployment sequence, not evidence of a policy crash.
Diagnostic principle: do not interpret an isolated shutdown line without its surrounding lifecycle events. Correlate policy stop, policy apply, application startup, pod restart, and request timestamps before concluding that a policy is unstable.
The Root Cause: Headers Were Lost at Ingress
The key difference between the two environments was the ingress path. The ship Runtime Fabric environment used the Kubernetes community ingress-nginx controller. The Client ID Enforcement policy was configured with MuleSoft's default credential headers:
client_id: <client-id>
client_secret: <client-secret>Those names contain underscores. NGINX treats underscore-containing request headers according to the underscores_in_headers directive, which is off by default. In the affected environment, the ingress controller did not forward the two credential headers upstream. The Mule application therefore received a request without the values required by the policy.
The resulting 401 was technically correct: MuleSoft rejected the request because the credentials were absent at the point of policy evaluation. The misleading part was that the original client request did contain them.
The Configuration Change
The ingress-nginx controller ConfigMap was updated at controller scope, not on the individual application Ingress resource:
apiVersion: v1
kind: ConfigMap
metadata:
name: ingress-nginx-controller
namespace: ingress-nginx
data:
enable-underscores-in-headers: "true"
ignore-invalid-headers: "false"After the ConfigMap was applied and the ingress controller was reloaded or restarted, the same request reached Runtime Fabric with both headers intact and the Client ID Enforcement policy validated the client application successfully.
Security note: use the narrowest configuration change that solves the problem. Enabling underscore headers is the setting directly related to client_id and client_secret. Disabling invalid-header filtering is broader and should be retained only when testing proves it is required, after security review. An alternative is to configure the policy and consumers to use hyphenated custom headers.
A Faster Diagnostic Sequence
This sequence separates policy, credential, routing, and ingress problems with fewer assumptions:
| Check | What it proves | Next action |
|---|---|---|
| Confirm policy Applied log | API autodiscovery and policy attachment are operating. | Move from deployment checks to request-path validation. |
| Run the same credentials against both endpoints | Separates credential validity from environment-specific behavior. | Compare ingress, DNS, routes, and upstream services. |
| Test the internal service or pod path | Bypasses the external ingress layer. | If internal works, inspect ingress, load balancer, or proxy configuration. |
| Inspect generated NGINX configuration | Confirms whether underscore headers are accepted. | Validate controller ConfigMap and reload status. |
| Verify headers at the closest safe upstream point | Shows whether values survive every hop. | Never log complete secrets; log presence, length, or a one-way hash only. |
Architecture Lessons Beyond This Incident
Treat ingress as part of API security
Authentication behavior depends on every hop that carries identity data. API governance must include load balancers, WAFs, ingress controllers, service meshes, and reverse proxies, not only the gateway policy.
Compare the whole path, not only the deployment artifact
Two Runtime Fabric environments can run the same Mule application and API policy but behave differently because their Kubernetes, ingress, TLS termination, DNS, or network controls differ.
Prefer explicit, portable header conventions
Hyphenated HTTP header names are generally less likely to encounter intermediary compatibility issues. MuleSoft's policy supports custom header names, allowing teams to standardize a safer contract where change control permits.
Separate proof from assumption
A 401 proves that an authentication control rejected the request. It does not, by itself, prove that the caller supplied bad credentials. Validate whether the credentials reached the enforcement point.
How Rafiqy Frames Problems Like This
Rafiqy is not simply a collection of technical posts. It is a practical knowledge and delivery platform for teams working across enterprise integration, APIs, automation, and AI-enabled engineering. Cases like this are documented to turn isolated production incidents into reusable design guidance, diagnostic playbooks, and stronger platform standards.
For integration and platform teams, the lasting value is not only the ConfigMap change. It is the improved operating model: trace the full request path, distinguish policy behavior from transport behavior, capture evidence at each layer, and feed the lesson back into architecture and deployment standards.
Rafiqy takeaway: reliable enterprise integration comes from understanding how application logic, API governance, Kubernetes infrastructure, and security controls behave as one system, not as separate teams and tools.
Production Checklist
- Confirm which component terminates TLS and which component first receives the HTTP headers.
- Compare API Manager
api.id, contract status, and policy configuration across environments. - Validate the ingress class and the exact controller ConfigMap used by that class.
- Check whether client credentials are expected in default headers, custom headers, query parameters, or HTTP Basic Authentication.
- Test through both the external ingress path and an internal service path.
- Avoid logging client secrets; rotate any credential exposed during troubleshooting.
- Manage the final controller setting through Helm, GitOps, or the platform's source of truth so that upgrades do not overwrite it.