Error: No valid responses from any peers. Errors: peer=undefined, status=grpc, message=Endorsement has failed - kubernetes

Question

I m working on hyperledger blockchain and Kubernetes(minikube) on Ubuntu 18-04. In my network there two Organisations with one peer each and the orderer type is Solo. This whole network I deployed on minikube. Chaincode install and instantiation did successfully on pods. After that am trying to invoke by using SDK.
I am using the below code as invoke.js
'use strict';
const { Gateway, Wallets } = require('fabric-network');
const fs = require('fs');
const path = require('path');
async function main() {
try {
// load the network configuration
const ccpPath = path.resolve(__dirname, '..', '..', 'first-network', 'connection1-org1.json');
let ccp = JSON.parse(fs.readFileSync(ccpPath, 'utf8'));
// Create a new file system based wallet for managing identities.
const walletPath = path.join(process.cwd(), 'wallet');
const wallet = await Wallets.newFileSystemWallet(walletPath);
console.log(`Wallet path: ${walletPath}`);
// Check to see if we've already enrolled the user.
const identity = await wallet.get('user1');
if (!identity) {
console.log('An identity for the user "user1" does not exist in the wallet');
console.log('Run the registerUser.js application before retrying');
return;
}
// Create a new gateway for connecting to our peer node.
const gateway = new Gateway();
await gateway.connect(ccp, { wallet, identity: 'user1',discovery: { enabled: true, asLocalhost: true } });
// Get the network (channel) our contract is deployed to.
const network = await gateway.getNetwork(channelname);
// Get the contract from the network.
const contract = network.getContract(contractname);
// Submit the specified transaction.
// createCar transaction - requires 5 argument, ex: ('createCar', 'CAR12', 'Honda', 'Accord', 'Black', 'Tom')
// changeCarOwner transaction - requires 2 args , ex: ('changeCarOwner', 'CAR10', 'Dave')
await contract.submitTransaction('arumnet','argument');
console.log('Transaction has been submitted');
// Disconnect from the gateway.
await gateway.disconnect();
} catch (error) {
console.error(`Failed to submit transaction: ${error}`);
process.exit(1);
}
}
main();
configTx.yaml
Organizations:
- &Orderer
Name: Orderer
ID: OrdererMSP
MSPDir: ./crypto-config/ordererOrganizations/acme.com/msp
# Policies are mandatory starting 2.x
Policies: &OrdererPolicies
Readers:
Type: Signature
Rule: "OR('OrdererMSP.member')"
Writers:
Type: Signature
Rule: "OR('OrdererMSP.member')"
Admins:
Type: Signature
# ONLY Admin Role can carry out administration activities
Rule: "OR('OrdererMSP.admin')"
Endorsement:
Type: Signature
Rule: "OR('OrdererMSP.member')"
- &Acme
Name: Acme
ID: AcmeMSP
MSPDir: ./crypto-config/peerOrganizations/acme.com/msp
Policies: &AcmePolicies
Readers:
Type: Signature
# Any member can READ e.g., query
Rule: "OR('AcmeMSP.member')"
Writers:
Type: Signature
# Any member can WRITE e.g., submit transaction
Rule: "OR('AcmeMSP.member')"
Admins:
Type: Signature
# Either Acme admin OR Orderer Admin can carry out admin activities
Rule: "OR('AcmeMSP.admin')"
Endorsement:
Type: Signature
# Any member can act as an endorser
Rule: "OR('AcmeMSP.member')"
AnchorPeers:
- Host: acme-peer-clusterip
Port: 30751
- &Budget
Name: Budget
ID: BudgetMSP
MSPDir: ./crypto-config/peerOrganizations/budget.com/msp
Policies: &BudgetPolicies
Readers:
Type: Signature
# Any member
Rule: "OR('BudgetMSP.member')"
Writers:
Type: Signature
# Any member
Rule: "OR('BudgetMSP.member')"
Admins:
Type: Signature
# BOTH Budget Admin AND Orderer Admin needed for admin activities
Rule: "OR('BudgetMSP.member')"
Endorsement:
Type: Signature
Rule: "OR('BudgetMSP.member')"
AnchorPeers:
- Host: budget-peer-clusterip
Port: 30851
Connection.json
{
"name": "first-network-acme",
"version": "1.0.0",
"client": {
"organization": "AcmeMSP",
"connection": {
"timeout": {
"peer": {
"endorser": "300"
}
}
}
},
"organizations": {
"AcmeMSP": {
"mspid": "AcmeMSP",
"peers": [
"peer1.acme.com"
],
"certificateAuthorities": [
]
}
},
"channel":{
"airlinechannel":{
"orderers": [
"orderer.acme.com"
],
"peers": {
"peer1.acme.com": {}
}
}
},
"peers": {
"peer1.acme.com": {
"url": "grpc://10.109.214.71:3005",
"tlsCACerts": {
"pem": "/crypto-config/peerOrganizations/acme.com/tlsca/tlsca.acme.com-cert.pem"
},
"grpcOptions": {
"ssl-target-name-override": "peer1.acme.com",
"hostnameOverride": "peer1.acme.com"
}
}
},
"certificateAuthorities": {
}
}
Logs after running invoke.js
2020-11-26T05:31:09.252Z | connectivity_state | dns:localhost:30751 CONNECTING -> CONNECTING
2020-11-26T05:31:09.252Z | dns_resolver | Resolved addresses for target dns:localhost:30751: [127.0.0.1:30751]
2020-11-26T05:31:09.252Z | pick_first | IDLE -> IDLE
2020-11-26T05:31:09.252Z | resolving_load_balancer | dns:localhost:30751 CONNECTING -> IDLE
2020-11-26T05:31:09.253Z | connectivity_state | dns:localhost:30751 CONNECTING -> IDLE
2020-11-26T05:31:09.253Z | pick_first | Connect to address list 127.0.0.1:30751
2020-11-26T05:31:09.253Z | subchannel | 127.0.0.1:30751 refcount 3 -> 4
2020-11-26T05:31:09.253Z | pick_first | IDLE -> TRANSIENT_FAILURE
2020-11-26T05:31:09.253Z | resolving_load_balancer | dns:localhost:30751 IDLE -> TRANSIENT_FAILURE
2020-11-26T05:31:09.253Z | connectivity_state | dns:localhost:30751 IDLE -> TRANSIENT_FAILURE
2020-11-26T05:31:12.254Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Endorser- name: acme-peer-clusterip:30751, url:grpc://localhost:30751, connected:false, connectAttempted:true
2020-11-26T05:31:12.254Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server acme-peer-clusterip:30751 url:grpc://localhost:30751 timeout:3000
2020-11-26T05:31:12.254Z - error: [DiscoveryService]: _buildPeer[dsg-test] - Unable to connect to the discovered peer acme-peer-clusterip:30751 due to Error: Failed to connect before the deadline on Endorser- name: acme-peer-clusterip:30751, url:grpc://localhost:30751, connected:false, connectAttempted:true
2020-11-26T05:31:12.261Z - error: [DiscoveryHandler]: _build_endorse_group_member >> G1:0 - returning an error endorsement, no endorsement made
2020-11-26T05:31:12.261Z - error: [Transaction]: Error: No valid responses from any peers. Errors:
peer=undefined, status=grpc, message=Endorsement has failed
Failed to submit transaction: Error: No valid responses from any peers. Errors:
peer=undefined, status=grpc, message=Endorsement has failed
Below kubernetes pods setup
kubectl get all
NAME READY STATUS RESTARTS AGE
pod/acme-orderer-0 1/1 Running 0 107m
pod/acme-peer-0 2/2 Running 0 107m
pod/budget-peer-0 2/2 Running 0 107m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/acme-orderer-clusterip ClusterIP 10.108.218.191 <none> 30750/TCP 107m
service/acme-orderer-nodeport NodePort 10.111.186.82 <none> 30750:30750/TCP 107m
service/acme-peer-clusterip ClusterIP 10.98.236.210 <none> 30751/TCP,30752/TCP 107m
service/acme-peer-nodeport NodePort 10.101.38.254 <none> 30751:30751/TCP,30752:30752/TCP 107m
service/budget-peer-clusterip ClusterIP 10.108.194.45 <none> 30851/TCP 107m
service/budget-peer-nodeport NodePort 10.100.136.250 <none> 30851:30851/TCP,30852:30852/TCP 107m
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 112m
service/svc-acme-orderer LoadBalancer 10.105.155.207 10.105.155.207 6005:30696/TCP 27m
service/svc-acme-peer LoadBalancer 10.98.44.14 10.109.214.71 3005:30594/TCP 56m
NAME READY AGE
statefulset.apps/acme-orderer 1/1 107m
statefulset.apps/acme-peer 1/1 107m
statefulset.apps/budget-peer 1/1 10

Answer 1

Some things to check:
connection1-org1.json - you've supplied connection1-org1.json in the code for invoke.js but connection.json config as a candidate you presumably expect to be used to load/access the network. Pls confirm if this is correct.
connection.json - specifies a tlsCACerts but the url scheme used is grpc:// rather than grpcs://
connection.json - does not specify the orderer url. This may not be necessary if it is discoverable via other means
Is TLS enabled on the peers/orderer?
If you can confirm/amend as necessary it will provide a better chance for someone to help.