Loading

Applications in RTF are Failing to Deploy due to Connection to Resource-cache Refused

发布日期: Jul 18, 2025
解决方案

SYMPTOM


All Applications in Test/QA RTF are failing to deploy with the below error - 
1 or more replicas in unexpected state: [Kubernetes] Container "init" - CrashLoopBackOff - /opt/mule/conf/testextsys-hydronet-forecast.yml: dial tcp 10.100.110.114:80: connect: connection refused - Exit code: 1, Restart count: 1
Check the init container logs of the app  
$ sudo kubectl logs -n <namespace> <app pod name> -c init 

...
time="2022-01-14T03:47:29Z" level=info msg="Reading annotations..."
time="2022-01-14T03:47:29Z" level=info msg="Parsing resources annotation..."
time="2022-01-14T03:47:29Z" level=info msg="Application resources:"
time="2022-01-14T03:47:29Z" level=info msg=" 1. /opt/mule/apps/testextsys-osipi-hist-loggerdata.jar (https://configuration-resolver.prod.cloudhub.io/api/v1/deployments/e9af161e-c413-4921-b832-01234567890a/assets/eyJ0eXAiOiJKV1QiLCJhb...
...
time="2022-01-14T03:47:29Z" level=warning msg="/opt/mule/apps/testextsys-osipi-hist-loggerdata.jar: dial tcp 10.100.110.114:80: connect: connection refused, retrying in 1s"
time="2022-01-14T03:47:29Z" level=warning msg="/opt/mule/conf/testextsys-osipi-hist-loggerdata.yml: dial tcp 10.100.110.114:80: connect: connection refused, retrying in 1s"
time="2022-01-14T03:47:30Z" level=warning msg="/opt/mule/conf/testextsys-osipi-hist-loggerdata.yml: dial tcp 10.100.110.114:80: connect: connection refused, retrying in 1.664560053s"
time="2022-01-14T03:47:30Z" level=warning msg="/opt/mule/apps/testextsys-osipi-hist-loggerdata.jar: dial tcp 10.100.110.114:80: connect: connection refused, retrying in 1.437714187s"
time="2022-01-14T03:47:31Z" level=warning msg="/opt/mule/apps/testextsys-osipi-hist-loggerdata.jar: dial tcp 10.100.110.114:80: connect: connection refused, retrying in 2.273912491s"
time="2022-01-14T03:47:31Z" level=warning msg="/opt/mule/conf/testextsys-osipi-hist-loggerdata.yml: dial tcp 10.100.110.114:80: connect: connection refused, retrying in 3.060469218s"
time="2022-01-14T03:47:33Z" level=warning msg="/opt/mule/apps/testextsys-osipi-hist-loggerdata.jar: dial tcp 10.100.110.114:80: connect: connection refused, retrying in 1.459459134s"
time="2022-01-14T03:47:35Z" level=warning msg="/opt/mule/conf/testextsys-osipi-hist-loggerdata.yml: dial tcp 10.100.110.114:80: connect: connection refused, retrying in 2.095634783s"
time="2022-01-14T03:47:35Z" level=error error="/opt/mule/apps/testextsys-osipi-hist-loggerdata.jar: dial tcp 10.100.110.114:80: connect: connection refused" res=/opt/mule/apps/testextsys-osipi-hist-loggerdata.jar
time="2022-01-14T03:47:35Z" level=fatal msg="/opt/mule/apps/testextsys-osipi-hist-loggerdata.jar: dial tcp 10.100.110.114:80: connect: connection refused"
...

Note 10.100.110.114 is the service endpoint of the resource-cache 
$ sudo kubectl get svc -n rtf
NAME                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
...
resource-cache           ClusterIP   10.100.110.114   <none>        80/TCP           394d
...

CAUSE

Test the request to the service endpoint on a worker node 
$ curl -v http://10.100.110.114
Rebuilt URL to: http://10.100.110.114/
*   Trying 10.100.110.114...
* TCP_NODELAY set
* connect to 10.100.110.114 port 80 failed: Connection refused
* Failed to connect to 10.100.110.114 port 80: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 10.100.110.114 port 80: Connection refused
Test the request to the resource-cache directly and bypass the service endpoint. 10.244.85.48 is the pod IP 
$ sudo kubectl get po -n rtf -o wide
NAME                                      READY   STATUS        RESTARTS   AGE     IP              NODE           NOMINATED NODE   READINESS GATES
...
resource-cache-6f668b86f7-mbj62           2/2     Running       0          4d22h   10.244.85.48    172.19.0.122   <none>           <none>
...
Test with curl. Note the pod listens on port 8010 
$  curl -v http://10.244.85.48:8010
* Rebuilt URL to: http://10.244.85.48:8010/
*   Trying 10.244.85.48...
* TCP_NODELAY set
* Connected to 10.244.85.48 (10.244.85.48) port 8010 (#0)
> GET / HTTP/1.1
> Host: 10.244.85.48:8010
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 502 Bad Gateway
< Server: nginx/1.17.8
< Date: Wed, 19 Jan 2022 03:38:00 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 56
< Connection: keep-alive
< X-Upstream: rtf-resource-fetcher
<
* Connection #0 to host 10.244.85.48 left intact
{"error":"WARN: value of X-Original-Host is invalid ()"}
We are seeing a 502 response code rather than connection refused. Check the service endpoint 
$ sudo kubectl describe -n rtf svc resource-cache
Name:              resource-cache
Namespace:         rtf
Labels:            app=resource-cache
                   app.kubernetes.io/instance=runtime-fabric
Annotations:       kubectl.kubernetes.io/last-applied-configuration:
                     {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"resource-cache","app.kubernetes.io/instance":"runtime-fa...
Selector:          app=resource-cache
Type:              ClusterIP
IP:                10.100.110.114
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Endpoints:         10.244.85.48:80
Session Affinity:  None
Events:            <non

We can see it forwards the request to port 80 which is wrong. This is due to a solution suggested for an old version  
kubectl delete service -n rtf resource-cache
kubectl create service -n rtf clusterip resource-cache --tcp=80
The above port should be updated to 8010 as the resource-cache listening port was changed. 
 

SOLUTION

Edit the service endpoint and update the targetPort to 8010 
$ sudo kubectl edit service -n rtf resource-cache
...
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8010
...
知识文章编号

001119138

 
正在加载
Salesforce Help | Article