At a Vert.x verticle I'm implementing I have a Buffer that was previously loaded into memory and now I want to dump it into disk.
As far as I understood we should use a Pump to make sure not to overload the WriteStream.
But I'm not finding a way to get a ReadStream child instance from a Buffer. Shouldn't there be an easy / standard way to do this?
Regards
Generally, vert.x does not warn on any issues writing directly into AsyncFiles. Furthermore, they provide the corresponding example of using AsyncFile.write directly here and state that you can use those to write directly: http://vertx.io/docs/vertx-core/java/#_asynchronous_files
However, if you want the pump with Buffer you need an instance of ReadStream<Buffer> along with an AsyncFile to pump into. You can make use of the implementation by PitchPoint Solutions (Copyright 2016 The Simple File Server Authors):
https://github.com/pitchpoint-solutions/sfs/blob/master/sfs-server/src/main/java/org/sfs/io/BufferReadStream.java
Putting it all together:
CompletableFuture<Void> done = new CompletableFuture<>();
Buffer buffer = Buffer.buffer(new byte[100]);
Vertx.vertx().fileSystem.open("myfile.txt", new OpenOptions(), res -> {
if (res.succeeded()) {
AsyncFile outputFile = res.result();
BufferReadStream reader = new BufferReadStream(buffer)
Pump pump = Pump.pump(reader, outputFile);
pump.start();
reader.endHandler((r) -> {
pump.stop(); // not sure this is required
done.complete(null);
});
} else {
// Something went wrong!
}
});
// wait elsewhere
done.get();
I have a C# library which connects to 59 servers of the same database structure and imports data to my local db to the same table. At this moment I am retrieving data server by server in foreach loop:
foreach (var systemDto in systems)
{
var sourceConnectionString = _systemService.GetConnectionStringAsync(systemDto.Ip).Result;
var dbConnectionFactory = new DbConnectionFactory(sourceConnectionString,
"System.Data.SqlClient");
var dbContext = new DbContext(dbConnectionFactory);
var storageRepository = new StorageRepository(dbContext);
var usedStorage = storageRepository.GetUsedStorageForCurrentMonth();
var dtUsedStorage = new DataTable();
dtUsedStorage.Load(usedStorage);
var dcIp = new DataColumn("IP", typeof(string)) {DefaultValue = systemDto.Ip};
var dcBatchDateTime = new DataColumn("BatchDateTime", typeof(string))
{
DefaultValue = batchDateTime
};
dtUsedStorage.Columns.Add(dcIp);
dtUsedStorage.Columns.Add(dcBatchDateTime);
using (var blkCopy = new SqlBulkCopy(destinationConnectionString))
{
blkCopy.DestinationTableName = "dbo.tbl";
blkCopy.WriteToServer(dtUsedStorage);
}
}
Because there are many systems to retrieve data, I wonder if it is possible to use Pararel.Foreach loop? Will BulkCopy lock the table during WriteToServer and next WriteToServer will wait until previous will complete?
-- EDIT 1
I've changed Foreach to Parallel.Foreach but I face one problem. Inside this loop I have async method: _systemService.GetConnectionStringAsync(systemDto.Ip)
and this line returns error:
System.NotSupportedException: A second operation started on this
context before a previous asynchronous operation completed. Use
'await' to ensure that any asynchronous operations have completed
before calling another method on this context. Any instance members
are not guaranteed to be thread safe.
Any ideas how can I handle this?
In general, it will get blocked and will wait until the previous operation complete.
There are some factors that may affect if SqlBulkCopy can be run in parallel or not.
I remember when adding the Parallel feature to my .NET Bulk Operations, I had hard time to make it work correctly in parallel but that worked well when the table has no index (which is likely never the case)
Even when worked, the performance gain was not a lot faster.
Perhaps you will find more information here: MSDN - Importing Data in Parallel with Table Level Locking
I wrote a piece of code that is going to perform a request to Facebook.
Now i wrapped this code into a infinite loop which is going to send those requests every 10 seconds using timeouts.
Code:
var poll = function(socket, userProvider) {
var lastCallTime = new Date();
var polling = true;
// The stream itself, non blocking
function performPoll() {
var results = feed(function (err, data) {
lastCallTime = new Date();
// PROCESS DATA
// Check new posts
if (polling) {
setTimeout(performPoll, 1000 * 10);
}
});
};
// Start infinite loop
performPoll();
};
The feed(cb) is just going to call a request to Facebook requesting data, this works 100% and does what i want it to do, the only problem that i am having now is that this piece of code is keeping to increase my memory usage. After a few minutes it increased by 50MB already (From 50 -> 100).
Is there anybody that can help me identify the cause of this?
v8 does not collect memory immediately. If it stabilizes at 100mb, then it is to be expected. For more information, checkout nodejs setTimeout memory leak?
If you really, really want to clear the memory, use global.gc(). Read this blog about how to call garbage collector manually.
I am trying to implement Facebook X_FACEBOOK_PLATFORM SASL mechanism so I could integrate Facebook Chat to my application over XMPP.
Here is the code:
var ak = "my app id";
var sk = "access token";
var aps = "my app secret";
using (var client = new TcpClient())
{
client.Connect("chat.facebook.com", 5222);
using (var writer = new StreamWriter(client.GetStream())) using (var reader = new StreamReader(client.GetStream()))
{
// Write for the first time
writer.Write("<stream:stream xmlns=\"jabber:client\" xmlns:stream=\"http://etherx.jabber.org/streams\" version=\"1.0\" to=\"chat.facebook.com\"><auth xmlns=\"urn:ietf:params:xml:ns:xmpp-sasl\" mechanism=\"X-FACEBOOK-PLATFORM\" /></stream:stream>");
writer.Flush();
Thread.Sleep(500);
// I am pretty sure following works or at least it's not what causes the error
var challenge = Encoding.UTF8.GetString(Convert.FromBase64String(XElement.Parse(reader.ReadToEnd()).Elements().Last().Value)).Split('&').Select(s => s.Split('=')).ToDictionary(s => s[0], s => s[1]);
var response = new SortedDictionary<string, string>() { { "api_key", ak }, { "call_id", DateTime.Now.Ticks.ToString() }, { "method", challenge["method"] }, { "nonce", challenge["nonce"] }, { "session_key", sk }, { "v", "1.0" } };
var responseString1 = string.Format("{0}{1}", string.Join(string.Empty, response.Select(p => string.Format("{0}={1}", p.Key, p.Value)).ToArray()), aps);
byte[] hashedResponse1 = null;
using (var prov = new MD5CryptoServiceProvider()) hashedResponse1 = prov.ComputeHash(Encoding.UTF8.GetBytes(responseString1));
var builder = new StringBuilder();
foreach (var item in hashedResponse1) builder.Append(item.ToString("x2"));
var responseString2 = Convert.ToBase64String(Encoding.UTF8.GetBytes(string.Format("{0}&sig={1}", string.Join("&", response.Select(p => string.Format("{0}={1}", p.Key, p.Value)).ToArray()), builder.ToString().ToLower()))); ;
// Write for the second time
writer.Write(string.Format("<response xmlns=\"urn:ietf:params:xml:ns:xmpp-sasl\">{0}</response>", responseString2));
writer.Flush();
Thread.Sleep(500);
MessageBox.Show(reader.ReadToEnd());
}
}
I shortened and shrunk the code as much as possible, because I think my SASL implementation (whether it works or not, I haven't had a chance to test it yet) is not what causes the error.
I get the following exception thrown at my face: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine.
10053
System.Net.Sockets.SocketError.ConnectionAborted
It happens every time I try to read from client's stream for the second time. As you can see i pause a thread here so Facebook server has enough time to answer me, but I used asynchronous approach before and I encountered the exact same thing, so I decided to try it synchronously first. Anyway actual SASL mechanism implementation really shouldn't cause this because if I don't try to authenticate right away, but I send the request to see what mechanisms server uses and select that mechanism in another round of reading and writing, it fails, but when I send mechanism selection XML right away, it works and fails on whatever second I send.
So the conclusion is following: I open the socket connection, write to it, read from it (first read works both sync and async), write to it for the second time and try to read from it for the second time and here it always fails. Clearly then, problem is with socket connection itself. I tried to use new StreamReader for second read but to no avail. This is rather unpleasant since I would really like to implement facade over NetworkStream with "Received" event or something like Send(string data, Action<string> responseProcessor) to get some comfort working with that stream, and I already had the implementation, but it also failed on second read.
Thanks for your suggestions.
Edit: Here is the code of facade over NetworkStream. Same thing happens when using this asynchronous approach, but couple of hours ago it worked, but for second response returned same string as for first. I can't figute out what I changed in a meantime and how.
public void Send(XElement fragment)
{
if (Sent != null) Sent(this, new XmppEventArgs(fragment));
byte[] buffer = new byte[1024];
AsyncCallback callback = null;
callback = (a) =>
{
var available = NetworkStream.EndRead(a);
if (available > 0)
{
StringBuilder.Append(Encoding.UTF8.GetString(buffer, 0, available));
NetworkStream.BeginRead(buffer, 0, buffer.Length, callback, buffer);
}
else
{
var args = new XmppEventArgs(XElement.Parse(StringBuilder.ToString()));
if (Received != null) Received(this, args);
StringBuilder = new StringBuilder();
// NetworkStream.BeginRead(buffer, 0, buffer.Length, callback, buffer);
}
};
NetworkStream.BeginRead(buffer, 0, buffer.Length, callback, buffer);
NetworkStreamWriter.Write(fragment);
NetworkStreamWriter.Flush();
}
The reader.ReadToEnd() call consumes everything until end-of-stream, i.e. until TCP connection is closed.
I have a regular TCP socket connection between a Node.js application (using standard net.Socket), and some other application.
Suppose I send a 1MB buffer:
socket.write(new Buffer(1048576));
While this is done in a non-blocking manner, obviously the data is not transmitted instantaneously.
How can I measure the data rate that the underlying system is sending the internal stream buffer, from my Node.js application?
Ultimately I just need to know the average speed of data being sent to the client on the other end over the last several seconds. Is this possible?
try to use callback in write
var net = require('net');
var s = net.createConnection(8888);
s.on('connect', function() {
function test() {
var len = 512*1048576;
var start = +new Date();
var b = new Buffer(len);
b.fill('x');
s.write(b, function() {
console.log(len + ' bytes written, ' + (1000*len/(+new Date() - start)).toString() + ' bytes/sec');
test();
});
}
test();
});
with nc -l 8888 > /dev/null on the other end I get around 350M bytes/sec