ipfs pubsub not working across two remote peers - publish-subscribe

If I run this code on my pc, I see my message, but another person on a different pc running the same code doesn't see mine. Is this a NAT thing perhaps? Or am I using this wrong?
const Room = require("ipfs-pubsub-room");
const ipfs = require("ipfs");
ipfs.create({}).then(async (node) => {
const room = new Room(node, "room-name");
room.on("peer joined", (peer) => {
console.log("Peer joined the room", peer);
});
room.on("peer left", (peer) => {
console.log("Peer left...", peer);
});
// now started to listen to room
room.on("subscribed", () => {
console.log("Now connected!");
});
room.on("message", ({ from, data }) =>
console.log(from, JSON.parse(data.toString()))
);
room.broadcast(JSON.stringify({ bla: "hello" }));
});

Are you using this in Node.js or browser?
The problem should be that your peers are not connected between each other (or with a common set of peers running pubsub) to have the pubsub messages reliably forwarded. The first step for you to diagnose this is to use node.swarm.peers() and see if you have the other peer.
Considering your peer is not connected to the other person node, you need to manually connect it, or configure a discovery mechanism to help you (this is automatic discovery and connectivity is a known problem for the community and we will be working on improving this experience).
The simplest option is to use webrtc-star transport+discovery. You can see examples with it in ipfs browser exchange files and with libp2p.
With this, you will likely see the other peers connected and then be able to exchange Pubsub messages. You can see more about the discovery mechanisms available in Libp2p config.md and libp2p discovery examples.
Let me know if you could get your issue fixed

Related

gRPC Node microservice talking to another microservice in istio mesh

I've got several gRPC microservices deployed via Istio in my k8s pod behind a gateway that handles the routing for web clients. Things work great when I need to send an RPC from client (browser) to any of these services.
I'm now at the point where I want to call service A from service B directly. How do I go about doing that?
Code for how both the servers are instantiated:
const server = new grpc.Server();
server.addService(MyService, new MyServiceImpl());
server.bindAsync(`0.0.0.0:${PORT_A}`, grpc.ServerCredentials.createInsecure(), () => {
server.start();
});
A Service Account is being used with GOOGLE_APPLICATION_CREDENTIALS and a secret in my deployment YAML.
To call service A from service B, I was thinking the code in service B would look something like:
const serviceAClient: MyServiceClient = new MyServiceClient(`0.0.0.0:${PORT_A}`, creds);
const req = new SomeRpcRequest()...;
serviceAClient.someRpc(req, (err: grpc.ServiceError, response: SomeRpcResponse) => {
// yay!
});
Is that naive? One thing I'm not sure about is the creds that I need to pass when instantiating the client. I get complaints that I need to pass ChannelCredentials, but every mechanism I've tried to create those creds has not worked.
Another thing I'm realizing is that 0.0.0.0 can't be correct because each service is in its own container paired with a sidecar proxy... so how do I route RPCs properly and attach proper creds?
I'm trying to construct the creds this way:
let callCreds = grpc.CallCredentials.createFromGoogleCredential(myOauthClient);
let channelCreds = grpc.ChannelCredentials.createSsl().compose(callCreds);
const serviceAClient = new MyServiceClient(`0.0.0.0:${PORT_A}`, channcelCreds);
and I'm mysteriously getting the following error stack:
UnhandledPromiseRejectionWarning: TypeError: Channel credentials must be a ChannelCredentials object
at new ChannelImplementation (/bish/proto/activities/node_modules/#grpc/grpc-js/build/src/channel.js:69:19)
at new Client (/bish/proto/activities/node_modules/#grpc/grpc-js/build/src/client.js:58:36)
at new ServiceClientImpl (/bish/proto/activities/node_modules/#grpc/grpc-js/build/src/make-client.js:58:5)
at PresenceService.<anonymous> (/bish/src/servers/presence/dist/presence.js:348:44)
at step (/bish/src/servers/presence/dist/presence.js:33:23)
at Object.next (/bish/src/servers/presence/dist/presence.js:14:53)
at fulfilled (/bish/src/servers/presence/dist/presence.js:5:58)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
Which is odd because channelCreds is a ComposedChannelCredentialsImpl which, in fact, extends ChannelCredentials
Ok, at least the root cause of the "Channel credentials must be a ChannelCredentials object" error is now known. I'm developing node packages side-by-side as symlinks and each of the dependencies has their own copy of grpc-js in it.
https://github.com/npm/npm/issues/7742#issuecomment-257186653

NestJS Mongoose connection dies on load testing

When multiple devs use my API, multiple concurrent requests are being sent to Mongoose.
When the concurrency is high, the connection just "dies" and refuses to fulfil any new request, no matter how long I wait (hours!).
I just want to state everything is working fine on regular use. Heavy use leads the connection to crash.
My MongooseModule initialization:
MongooseModule.forRoot(DatabasesService.MONGO_FULL_URL, {
useNewUrlParser: true,
useUnifiedTopology: true,
useFindAndModify: false,
autoEncryption: {
keyVaultNamespace: DatabasesService.keyVaultNamespace,
kmsProviders: DatabasesService.kmsProviders,
extraOptions: {
mongocryptdSpawnArgs: ['--pidfilepath', '/tmp/mongocryptd.pid']
}
} as any
})
Module that imports the feature:
#Module({
imports: [MongooseModule.forFeature([{ name: 'modelName', schema: ModelNameSchema }])],
providers: [ModelNameService],
controllers: [...],
exports: [...]
})
Service:
#Injectable()
export class ModelNameService {
constructor(
#InjectModel('modelName') private modelName: Model<IModelName>
) {}
async findAll(): Promise<IModelName[]> {
const result: IModelName[] = await this.modelName.find().exec();
if (!result) throw new BadRequestException(`No result was found.`);
return result;
}
}
I've tried loadtesting using different utils, the easiest was:
ab -c 200 -n 300 -H "Authorization: Bearer $TOKEN" -m GET -b 0 https://example.com/getModelName
Any new request after the connection hangs gets stuck at ModelNameService.findAll() first line (the request to mongo).
On mongodb logs with verbosity of "-vvvvv" I can see few suspicious lines:
User Assertion: Unauthorized: command endSessions requires authentication src/mongo/db/commands.cpp
Cancelling outstanding I/O operations on connection to 127.0.0.1:33134
And I've also found that it doesn't exceed 12 open connections at the same time. It always waits to close one before opening a new one.
Other key points:
Mongoose doesn't return any value or notifies about any error. It just hangs without notifying anything.
Terminus health check able to ping the DB and returns a healthy status.
NestJS API still works - I'm able to send new requests and receive a response. Just requests that are related to the faulty connection hang.
When I inject connection and check its readyState it returns connected.
Restarting the API fixes it immediately.
MongoDB itself keeps working as normal.
Increasing Mongoose poolSize is able to handle more requests at the same time but will still crash on a larger amount of requests.
My main question here is how do I handle this case? Currently, I've added another health check to try and send a query to the problematic connection every half a minute, and k8s restarts the pod if it determines a failure. This works but it is not optimal.

ECONNREFUSED during 'next build'. Works fine with 'next dev' [duplicate]

This question already has an answer here:
Fetch error when building Next.js static website in production
(1 answer)
Closed last year.
I have a very simple NextJS 9.3.5 project.
For now, it has a single pages/users and a single pages/api/users that retrieves all users from a local MongoDB table
It builds fine locally using 'next dev'
But, it fails on 'next build' with ECONNREFUSED error
page/users
import fetch from "node-fetch"
import Link from "next/link"
export async function getStaticProps({ params }) {
const res = await fetch(`http://${process.env.VERCEL_URL}/api/users`)
const users = await res.json()
return { props: { users } }
}
export default function Users({ users }) {
return (
<ul>
{users.map(user => (
<li key={user.id}>
<Link href="/user/[id]" as={`/user/${user._id}`}>
<a>{user.name}</a>
</Link>
</li>
))}
</ul>
);
}
pages/api/users
import mongoMiddleware from "../../lib/api/mongo-middleware";
import apiHandler from "../../lib/api/api-handler";
export default mongoMiddleware(async (req, res, connection, models) => {
const {
method
} = req
apiHandler(res, method, {
GET: (response) => {
models.User.find({}, (error, users) => {
if (error) {
connection.close();
response.status(500).json({ error });
} else {
connection.close();
response.status(200).json(users);
}
})
}
});
})
yarn build
yarn run v1.22.4
$ next build
Browserslist: caniuse-lite is outdated. Please run next command `yarn upgrade`
> Info: Loaded env from .env
Creating an optimized production build
Compiled successfully.
> Info: Loaded env from .env
Automatically optimizing pages ..
Error occurred prerendering page "/users". Read more: https://err.sh/next.js/prerender-error:
FetchError: request to http://localhost:3000/api/users failed, reason: connect ECONNREFUSED 127.0.0.1:3000
Any ideas what is going wrong ? particularly when it works fine with 'next dev' ?
Thank you.
I tried the same few days ago and didn't work... because when we build the app, we don't have localhost available... check this part of the doc - https://nextjs.org/docs/basic-features/data-fetching#write-server-side-code-directly - that said: "You should not fetch an API route from getStaticProps..." -
(Next.js 9.3.6)
Just to be even more explicit on top of what Ricardo Canelas said:
When you do next build, Next goes over all the pages it detects that it can build statically, i.e. all pages that don't define getServerSideProps, but which possibly define getStaticProps and getStaticPaths.
To build those pages, Next calls getStaticPaths to decide which pages you want to build, and then getStaticProps to get the actual data needed to build the page.
Now, if in either of getStaticPaths or getStaticProps you do an API call, e.g. to a JSON backend REST server, then this will get called by next build.
However, if you've integrated both front and backend nicely into a single server, chances are that you have just quit your development server (next dev) and are now trying out a build to see if things still work as sanity check before deployment.
So in that case, the build will try to access your server, and it won't be running, so you get an error like that.
The correct approach is, instead of going through the REST API, you should just do database queries directly from getStaticPaths or getStaticProps. That code never gets run on the client anyways, only server, to it will also be slightly more efficient than doing a useless trip to the API, which then calls the database indirectly. I have a demo that does that here: https://github.com/cirosantilli/node-express-sequelize-nextjs-realworld-example-app/blob/b34c137a9d150466f3e4136b8d1feaa628a71a65/lib/article.ts#L4
export const getStaticPathsArticle: GetStaticPaths = async () => {
return {
fallback: true,
paths: (await sequelize.models.Article.findAll()).map(
article => {
return {
params: {
pid: article.slug,
}
}
}
),
}
}
Note how on that example, both getStaticPaths and getStaticProps (here generalized HoC's for reuse, see also: Module not found: Can't resolve 'fs' in Next.js application ) do direct database queries via sequelize ORM, and don't do any HTTP calls to the external server API.
You should then only do client API calls from the React components on the browser after the initial pages load (i.e. from useEffect et al.), not from getStaticPaths or getStaticProps. BTW, note that as mentioned at: What is the difference between fallback false vs true vs blocking of getStaticPaths with and without revalidate in Next.js SSR/ISR? reducing client calls as much as possible and prerendering on server greatly reduces application complexity.

Logstash TCP Input Crashing

We have a logstash (v2.3) setup with 1 queue server running RabbitMQ, 10 elasticsearch nodes and a web node for kibana. Everything "works" and we have a large number of servers sending logs at the queue server. Most of the logs make it in, but we've noticed many that just never show up.
Looking in the logstash.log file we'll see the following start showing:
{:timestamp=>"2016-07-15T16:21:34.638000+0000", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::Tcp type=>\"syslog\", port=>5544, codec=><LogStash::Codecs::JSONLines charset=>\"UTF-8\", delimiter=>\"\\n\">, add_field=>{\"deleteme\"=>\"\"}, host=>\"0.0.0.0\", data_timeout=>-1, mode=>\"server\", ssl_enable=>false, ssl_verify=>true, ssl_key_passphrase=><password>>\n Error: closed stream", :level=>:error}
This repeats about every second or so. We initially thought maybe max connections limit was being met but a netstat only shows about ~4000 connections, and our limit should be upwards of 65,000.
Why is this TCP plugin crashing so much?
Everything I've read online hints towards this being an older issue that was resolved with newer versions of Logstash, which we've long since installed. Whats confusing is that it is partially working, we're getting a ton of logs but also seem to be missing quite a bit.
Relevant conf file on Queue server:
queue.mydomain.com:
input {
tcp {
type => "syslog"
port => "5544"
}
udp {
type => "syslog"
port => "5543"
}
}
output {
rabbitmq {
key => "thekey"
exchange => "theexchange"
exchange_type => "direct"
user => "username"
password => "password"
host => "127.0.0.1"
port => 5672
durable => true
persistent => true
}
}
We recently added UDP to the above conf to test with it, but logs aren't making it in reliably to it either.
Just in case the Elasticsearch cluster conf is relevant:
We have a 10 node Elasticsearch cluster, setup to pull from the queue server, this works as intended and is on same version of Logstash as Queue server. They pull from the rabbitMQ server with the conf:
input {
rabbitmq {
durable => "true"
host => "***.**.**.**"
key => "thekey"
exchange => "theexchange"
queue => "thequeue"
user => "username"
password => "password"
}
}
Anyone have any ideas for us to try to figure out whats up with tcp-input plugin?
Thanks.

Riak Java HTTPClientAdapter TCP CLOSE_WAIT

TLDR:
Lots of TCP connections in OPEN_WAIT status shutting down server
Setup:
riak_1.2.0-1_amd64.deb installed on Ubuntu12
Spring MVC 3.2.5
riak-client-1.1.0.jar
Tomcat7.0.51 hosted on Windows Server 2008 R2
JRE6_45
Full Description:
How do I ensure that the Java RiakClient is properly cleaning up it's connections to that I'm not left with an abundance of CLOSE_WAIT tcp connections?
I have a Spring MVC application which uses the Riak java client to connect to the remote instance/cluster.
We are seeing a lot of TCP Connections on the server hosting the Spring MVC application, which continue to build up until the server can no longer connect to anything because there are no ports available.
Restarting the Riak cluster does not clean the connections up.
Restarting the webapp does clean up the extra connections.
We are using the HTTPClientAdapter and REST api.
When connecting to a relational database, I would normally clean up connections by either explicitly calling close on the connection, or by registering the datasource with a pool and transaction manager and then Annotating my Services with #Transactional.
But since using the HTTPClientAdapter, I would have expected this to be more like an HttpClient.
With an HttpClient, I would consume the Response entity, with EntityUtils.consume(...), to ensure that the everything is properly cleaned up.
HTTPClientAdapter does have a shutdown method, and I see it being called in the online examples.
When I traced the method call through to the actual RiakClient, the method is empty.
Also, when I dig through the source code, nowhere in it does it ever close the Stream on the HttpResponse or consume any response entity (as with the standard Apache EntityUtils example).
Here is an example of how the calls are being made.
private RawClient getRiakClientFromUrl(String riakUrl) {
return new HTTPClientAdapter(riakUrl);
}
public IRiakObject fetchRiakObject(String bucket, String key, boolean useCache) {
try {
MethodTimer timer = MethodTimer.start("Fetch Riak Object Operation");
//logger.debug("Fetching Riak Object {}/{}", bucket, key);
RiakResponse riakResponse;
riakResponse = riak.fetch(bucket, key);
if(!riakResponse.hasValue()) {
//logger.debug("Object {}/{} not found in riak data store", bucket, key);
return null;
}
IRiakObject[] riakObjects = riakResponse.getRiakObjects();
if(riakObjects.length > 1) {
String error = "Got multiple riak objects for " + bucket + "/" + key;
logger.error(error);
throw new RuntimeException(error);
}
//logger.debug("{}", timer);
return riakObjects[0];
}
catch(Exception e) {
logger.error("Error fetching " + bucket + "/" + key, e);
throw new RuntimeException(e);
}
}
The only option I can think of, is to create the RiakClient separately from the adapter so I can access the HttpClient and then the ConnectionManager.
I am currently working on switching over to the PBClientAdapter to see if that might help, but for the purposes of this question (and because the rest of the team may not like me switching for whatever reason), let's assume that I must continue to connect over HTTP.
So it's been almost a year, so I thought I would go ahead and post how I solved this problem.
The solution was to change the client implementation we were using to the HTTPClientAdapter provided by the java client, passing in the configuration to implement pools and max connections. Here's some code example of how to do it.
First, we are on an older version of RIAK, so here's the amven dependency:
<dependency>
<groupId>com.basho.riak</groupId>
<artifactId>riak-client</artifactId>
<version>1.1.4</version>
</dependency>
And here's the example:
public RawClient riakClient(){
RiakConfig config = new RiakConfig(riakUrl);
//httpConnectionsTimetolive is in seconds, but timeout is in milliseconds
config.setTimeout(30000);
config.setUrl("http://myriakurl/);
config.setMaxConnections(100);//Or whatever value you need
RiakClient client = new RiakClient(riakConfig);
return new HTTPClientAdapter(client);
}
I actually broke that up a bit in my implementation and used Spring to inject values; I just wanted to show a simplified example for it.
By setting the timeout to something less than the standard five minutes, the system will not hang to the connections for too long (so, 5 minutes + whatever you set the timeout to) which causes the connectiosn to enter the close_wait status sooner.
And of course setting the max connections in the pool prevents the application from opening up 10's of thousands of connections.