I have blob type generated by webaudio API, but the file that is saved have to high sample rate.
How can I convert it to lower maybe something like https://developer.mozilla.org/en-US/docs/Web/API/OfflineAudioContext can help?
Here is some sample of code:
var xhr = new XMLHttpRequest();
/* HERE IS SOME CONVERTATION TO LOWER RATE */
var fd = new FormData();
fd.append("randomname", bigBlob);
xhr.open("POST",url,false);
xhr.send(fd);
xhr.onload=function(e) {
alert(e.target.responseText);
};
Create an OfflineAudioContext with the rate you want at the end, and the number of frames there will be at the end
Create an AudioBuffer from your raw data buffer
Create an AudioBufferSourceNode, set its buffer attribute to the AudioBuffer you just created, and connect this AudioBufferSourceNode to the destination of the OfflineAudioContext
Start the AudioBufferSourceNode at 0
Start the rendering
I couldn't find a way to control the sample rate but here is a way to re-sample (up/down sampling)
function reSample(audioBuffer, targetSampleRate, onComplete) {
var channel = audioBuffer.numberOfChannels;
var samples = audioBuffer.length * targetSampleRate / audioBuffer.sampleRate;
var offlineContext = new OfflineAudioContext(channel, samples, targetSampleRate);
var bufferSource = offlineContext.createBufferSource();
bufferSource.buffer = audioBuffer;
bufferSource.connect(offlineContext.destination);
bufferSource.start(0);
offlineContext.startRendering().then(function(renderedBuffer){
onComplete(renderedBuffer);
})
}
Extracted from here:
https://github.com/notthetup/resampler
Related
How I can calculate the arithmetic mean of a large vector(series) in distributed computing where I partition the data on multiple nodes. I do not want to use map reduce paradigm. Is there any distributed algorithm to efficiently compute the mean besides the trivial computation of individual sum on each node and then bringing the result at master node and dividing with the size of the vector(series).
distributed average consensus is an alternative.
The problem with the trivial approach of map-reduce with a master is that if you have a vast set of data, in essence to make everything dependent on each other, it could take a very long time to calculate the data, by which time the information is very out of date, and therefore wrong, unless you lock the entire dataset - impractical for a massive set of distributed data. Using distributed average consensus (the same methods work for alternative algorithms to Mean), you get a more up to date, better guess at the current value of the Mean without locking the data, and in real time.
Here is a link to a paper on it, but it's math heavy :
http://web.stanford.edu/~boyd/papers/pdf/lms_consensus.pdf
You can google for many papers on it.
The general concept is like this: say on each node you have a socket listener. You evaluate your local sum and average, then publish it to the other nodes. Each node listens for the other nodes, and receives their sum and averages on a timescale that makes sense. You can then evaluate a good guess at the total average by (sumForAllNodes(storedAverage[node] * storedCount[node]) / (sumForAllNodes(storedCount[node])). If you have a truly large dataset, you could just listen for new values as they are stored in the node, and amend the local count and average, then publish them.
If even this is taking too long, you could average over a random subset of the data in each node.
Here is some c# code that gives you an idea (uses fleck to run on more versions of windows than windows-10-only microsoft websockets implementation). Run this on two nodes, one with
<appSettings>
<add key="thisNodeName" value="UK" />
</appSettings>
in the app.config, and use "EU-North" in the other. Here is some sample code. The two instances exchange means using websockets. You just need to add your back end enumeration of the database.
using Fleck;
namespace WebSocketServer
{
class Program
{
static List<IWebSocketConnection> _allSockets;
static Dictionary<string,decimal> _allMeans;
static Dictionary<string,decimal> _allCounts;
private static decimal _localMean;
private static decimal _localCount;
private static decimal _localAggregate_count;
private static decimal _localAggregate_average;
static void Main(string[] args)
{
_allSockets = new List<IWebSocketConnection>();
_allMeans = new Dictionary<string, decimal>();
_allCounts = new Dictionary<string, decimal>();
var serverAddresses = new Dictionary<string,string>();
//serverAddresses.Add("USA-WestCoast", "ws://127.0.0.1:58951");
//serverAddresses.Add("USA-EastCoast", "ws://127.0.0.1:58952");
serverAddresses.Add("UK", "ws://127.0.0.1:58953");
serverAddresses.Add("EU-North", "ws://127.0.0.1:58954");
//serverAddresses.Add("EU-South", "ws://127.0.0.1:58955");
foreach (var serverAddress in serverAddresses)
{
_allMeans.Add(serverAddress.Key, 0m);
_allCounts.Add(serverAddress.Key, 0m);
}
var thisNodeName = ConfigurationSettings.AppSettings["thisNodeName"]; //for example "UK"
var serverSocketAddress = serverAddresses.First(x=>x.Key==thisNodeName);
serverAddresses.Remove(thisNodeName);
var websocketServer = new Fleck.WebSocketServer(serverSocketAddress.Value);
websocketServer.Start(socket =>
{
socket.OnOpen = () =>
{
Console.WriteLine("Open!");
_allSockets.Add(socket);
};
socket.OnClose = () =>
{
Console.WriteLine("Close!");
_allSockets.Remove(socket);
};
socket.OnMessage = message =>
{
Console.WriteLine(message + " received");
var parameters = message.Split('~');
var remoteHost = parameters[0];
var remoteMean = decimal.Parse(parameters[1]);
var remoteCount = decimal.Parse(parameters[2]);
_allMeans[remoteHost] = remoteMean;
_allCounts[remoteHost] = remoteCount;
};
});
while (true)
{
//evaluate my local average and count
Random rand = new Random(DateTime.Now.Millisecond);
_localMean = 234.00m + (rand.Next(0, 100) - 50)/10.0m;
_localCount = 222m + rand.Next(0, 100);
//evaluate my local aggregate average using means and counts sent from all other nodes
//could publish aggregate averages to other nodes, if you wanted to monitor disagreement between nodes
var total_mean_times_count = 0m;
var total_count = 0m;
foreach (var server in serverAddresses)
{
total_mean_times_count += _allCounts[server.Key]*_allMeans[server.Key];
total_count += _allCounts[server.Key];
}
//add on local mean and count which were removed from the server list earlier, so won't be processed
total_mean_times_count += (_localMean * _localCount);
total_count = total_count + _localCount;
_localAggregate_average = (total_mean_times_count/total_count);
_localAggregate_count = total_count;
Console.WriteLine("local aggregate average = {0}", _localAggregate_average);
System.Threading.Thread.Sleep(10000);
foreach (var serverAddress in serverAddresses)
{
using (var wscli = new ClientWebSocket())
{
var tokSrc = new CancellationTokenSource();
using (var task = wscli.ConnectAsync(new Uri(serverAddress.Value), tokSrc.Token))
{
task.Wait();
}
using (var task = wscli.SendAsync(new ArraySegment<byte>(Encoding.UTF8.GetBytes(thisNodeName+"~"+_localMean + "~"+_localCount)),
WebSocketMessageType.Text,
false,
tokSrc.Token
))
{
task.Wait();
}
}
}
}
}
}
}
Don't forget to add static lock or separate activity by synchronising at given times. (not shown for simplicity)
There are two simple approaches you can use.
One is, as you correctly noted, to calculate the sum on every node and then combine the sums and divide by the total amount of data:
avg = (sum1+sum2+sum3)/(cnt1+cnt2+cnt3)
Another possibility is to calculate the average on every node and then use weighted average:
avg = (avg1*cnt1 + avg2*cnt2 + avg3*cnt3) / (cnt1+cnt2+cnt3)
= avg1*cnt1/(cnt1+cnt2+cnt3) + avg2*cnt2/(cnt1+cnt2+cnt3) + avg3*cnt3/(cnt1+cnt2+cnt3)
I don't see anything wrong with these trivial ways and am wondering why you would want to use a different approach.
I work in theoretical physics and I do lot of computer simulations. An important part of my duty is the analysis of the results. I make simulations and store the numerical results in a file with some simple name. Typically I have lot of data files with very similar name and after a while I do not remember what kind of parameters the file corresponds to. I was thinking that maybe there exists a better way to store numerical results from a simulation e.g. some database (SQL, MongoDB etc.) where I could put some comments about parameters of the program, names, date etc. - a sort of a library with numerical data. I just have everything in a one place well organized. Do you know of anything like this? How do you store you numerical data from computer simulations?
More details
Typical procedure looks like this. Let say we want to simulate time evolution of the three body problem. We have three bodies of different masses interacting with Newton forces. I want to test how these objects move in space depending on: relative mass value, initial position - 6 parameters. I run simulation for one choice of parameters and save it in file: three_body_m1=0p1_m2=0p3_(the rest).dat - all double precision in total 1+3*3 (3d) columns of data in one file. Then I lunch gnuplot, python etc. and visualize them. In principle there is no relation between the data from different simulations, but I can use them to make comparison plot.
Within same nodejs context, you can,
Stream big xyz data file to server using socket.io-stream + fs modules and save filename+parameters to database using mongodb module.(max 1-page of coding but more for complex server talking)
If data fits in ram and if you don't have to save immediately, you can use redis module to send everything to server cache easily(as key-value pairs such as data->xyzData and parameters->simulationParameters and user->name_surname) and read from it high speed. If you need data as file by other processes in server, you can stram to a ramdisk instead and have most of RAM bandwidth as a file cache.(needs more ram ofcourse but fast)
mongodb is slow(even with optimizations) for saving millions of particles xyz data but is most easiest and quickest install for parameter saving and sharing.
Using all could be better.
Saving: stream file to physical disk using socket.io-stream and fs. Send parameters to mongodb.
Loading: check redis if user is registered, check if data is in cache, if yes, get it, if no, stream from physical disk and also save some of it to redis at the same time.
Editing: check if cache exists, if yes then edit it. Another serverside process can update physical disk from that cache, if no then update physical disk directly.
The communication scheme could be:
data server talks to cache server if there is any pending writes/reads/edits, consumes jobs from there.
compute server talks to cache server for producing read/write/edit jobs or consuming compute jobs.
clients can talk to cache server for reading only.
admins can also place their own data or produce compute jobs or read stuff.
compute server, data server and cache server can be on same computer easily or moved to other computers thanks to nodejs's awesomeness and countless modules of it such as redis, socket.io-stream, fs, ejs, express(for clients for example), etc.
a cache server can offload some data to another cache server and have a redirection to it(or some mapping of data to it)
a cache server can communicate N number of data servers and M number of compute servers at the same time as long as RAM holds.
You have slow network? You can use gzip module to compress the data on-the-fly with just 3-5 lines of extra code(at both ends)
You don't have money?
Nodejs works on raspberry pi (as data server maybe?)
Nvidia GTX660 can work with an Intel galileo (compute server?) using nodejs with some extra native modules for opencl(could be hard to implement)(also connecting(and powering) gpu and galileo may not be easy but should be much faster than a cluster of raspberry pi boards for fp32 number crunching)
bypass cache, RAM is expensive for now.
data server cluster
\
\
\ client
\ client /
\ / /
\ / /
mainframe cache and database server ----- compute cluster
| \
| \
support cache server admin
A very simple example to send some files to another computer(or same):
var pipeline_n = 8;
var fs = require("fs");
// server part accepting files
{
var io = require('socket.io').listen(80);
var ss = require('socket.io-stream');
var path = require('path');
var ctr = 0;
var ctr2 = 0;
io.of('/user').on('connection', function (socket) {
var z1 = new Date();
for (var i = 0; i < pipeline_n; i++) {
ss(socket).on('data'+i.toString(), function (stream, data) {
var t1 = new Date();
stream.pipe(fs.createWriteStream("m://bench_server" + ctr + ".txt"));
ctr++;
stream.on("finish", function (p) {
var len = stream._readableState.pipes.bytesWritten;
var t2 = new Date();
ctr2++;
if (ctr2 == pipeline_n) {
var z2 = new Date();
console.log(len * pipeline_n);
console.log((z2 - z1));
console.log("throughput: " + ((len * pipeline_n) / ((z2 - z1)/1000.0))/(1024*1024)+" MB/s");
}
});
});
}
});
}
//client or another server part sending a file
//(you can change it to do parts of same file instead of same file n times),
//just a dummy file sending code to stress other server
for (var i = 0; i < pipeline_n; i++)
{
var io = require('socket.io-client');
var ss = require('socket.io-stream');
var socket = io.connect('http://127.0.0.1/user');
var stream = ss.createStream();
var filename = 'm://bench.txt'; // ramdrive or cluster of hdd raid
ss(socket).emit('data'+i.toString(), stream, { name: filename });
fs.createReadStream(filename).pipe(stream);
}
Here is testing insert vs bulk insert performance of mongodb(this could be a wrong way to benchmark but is simple, just uncomment-in the part you want to benchmark)
var mongodb = require('mongodb');
var client = mongodb.MongoClient;
var url = 'mongodb://localhost:2019/evdb2';
client.connect(url, function (err, db) {
if (err) {
console.log('fail:', err);
} else {
console.log('success:', url);
var collection = db.collection('tablo');
var bulk = collection.initializeUnorderedBulkOp();
db.close();
//benchmark insert
//var t = 0;
//t = new Date();
//var ctr = 0;
//for (var i = 0; i < 1024 * 64; i++)
//{
// collection.insert({ x: i + 1, y: i, z: i * 10 }, function (e, r) {
// ctr++;
// if (ctr == 1024 * 64)
// {
// var t2 = 0;
// db.close();
// t2 = new Date();
// console.log("insert-64k: " + 1000.0 / ((t2.getTime() - t.getTime()) / (1024 * 64)) + " insert/s");
// }
// });
//}
// benchmark bulk insert
//var t3 = new Date();
//for (var i = 0; i < 1024 * 64; i++)
//{
// bulk.insert({ x: i + 1, y: i, z: i * 10 });
//}
//bulk.execute();
//var t4 = new Date();
//console.log("bulk-insert-64k: " + 1000.0/((t4.getTime() - t3.getTime()) / (1024 * 64)) + " insert/s");
// db.close();
}
});
be sure to setup mongodb and or redis servers before this. Also "npm install module_name" necessary modules from nodejs command prompt.
I am uploading files using the following code:
using (var s = File.OpenRead(#"C:\2gbDataTest.zip"))
{
var t = Task.Run<ObjectId>(() =>
{
return fs.UploadFromStreamAsync("2gbDataTest.zip", s);
});
return t.Result;
}
//works for the files below 2gb
var t1 = fs.DownloadAsBytesAsync(id);
Task.WaitAll(t1);
var bytes = t1.Result;
I am getting error
I am new to MongoDb and C#, can any one please show me how to download files greater than 2GB in size?
You are hitting the limit in terms of the size a byte array (kept in memory) download can be, so your only choice is to use a Stream instead like you are doing when you upload, something like (with a valid destination):
IGridFSBucket fs;
ObjectId id;
FileStream destination;
await fs.DownloadToStreamAsync(id, destination);
//Just writing complete code for others, This will work ;
//Thanks to "Adam Comerford"
var fs = new GridFSBucket(database);
using (var newFs = new FileStream(filePathToDownload, FileMode.Create))
{
//id is file objectId
var t1 = fs.DownloadToStreamAsync(id, newFs);
Task.WaitAll(t1);
newFs.Flush();
newFs.Close();
}
I'm facing an issue with Mirthconnect.
I just have a trouble in this process. I like to read the data from mail, is it possible to acheive this in the open source mirthconnect? of version 3.3.1, if so is it possible to read from direct mail?. Apart from the commerical versions like mirth mails.
I made use of JAVA mail library and inserted it in the custom library folder of mirth connect then used the following code in the connector portion of mirth. It works well.
//Fetchmail from Gmail
var props = new Packages.java.util.Properties();
props.setProperty("mail.store.protocol", "imaps");
var session = new Packages.javax.mail.Session.getInstance(props, null);
var store = session.getStore();
store.connect("imap.gmail.com", "xxxxxxxx#gmail.com", "xxxxxxxxx");
var inbox = store.getFolder("INBOX");
inbox.open(Packages.javax.mail.Folder.READ_ONLY);
var msgs = inbox.getMessage(inbox.getMessageCount());
var currentMessage = inbox.getMessage(inbox.getMessageCount());
var mp = currentMessage.getContent();
var bp = mp.getBodyPart(0);
var content = "" + bp.getContent();
content = content.replace(/''/g, "");
globalMap.put('gcon', content);
logger.info("SENT DATE:" + msgs.getSentDate());
logger.info("SUBJECT:" + msgs.getSubject());
logger.info("CONTENT:" + content);
//bp.getContent()
var receiveId = UUIDGenerator.getUUID();
logger.info("incomingMailID : "+receiveId);
//Database Connectivity
var time= msgs.getSentDate();
var con = bp.getContent();
var sub = msgs.getSubject();
//global variable declaration
globalMap.put('glcontent',con);
globalMap.put('glsubject',sub);
globalMap.put('gltime',time);
return sub;
Then you can set the the polling frequency time intervalin the listener which the mirth channel will poll for that specific time interval.
I am hoping to implement audio effects in the Web Audio API which require continuous access to two or more audio streams.
I can define a script processor with 2 input channels and 2 output channels:
var mod = context.createScriptProcessor(4096,2,2);
I can then connect a few sine waves to this processor:
mySine.connect(mod);
mySine2.connect(mod);
Is there a way to connect them to a specific input channel of the audio processor?
Eventually, when I write an onaudioprocess function and listen to each input channel individually, each input channel contains all the sounds connected to the processor. I have no way to access each sine wave individually within the onaudioprocess function. Is this correct? Or is there a way to connect sounds to a single input channel of the scriptprocessor?
You could either create two identical single channel ScriptProcessors or maybe use a channel merger to assign the two sine waves to each channel of the script processor like this:
var context = new AudioContext();
var sineA = context.createOscillator();
sineA.type = 'sine';
sineA.frequency.value = 300;
var sineB = context.createOscillator();
sineB.type = 'sine';
sineB.frequency.value = 100;
var script = context.createScriptProcessor(4096, 2, 2);
// create 2 channel merger node
var merger = context.createChannelMerger(2);
// connect sineA to channel 0
sineA.connect(merger, 0, 0);
// connect sineA to channel 1
sineB.connect(merger, 0, 1);
// connect the script to the merger
merger.connect(script);
// process the audio data of each channel
script.onaudioprocess = function(event) {
var input = event.inputBuffer;
var output = event.outputBuffer;
var inputA = input.getChannelData(0);
var inputB = input.getChannelData(1);
var outputA = output.getChannelData(0);
var outputB = output.getChannelData(1);
for (var i = 0; i < input.length; i++) {
outputA[i] = inputA[i];
outputB[i] = inputB[i];
}
}
script.connect(context.destination);
sineA.start();
sineB.start();