How to include a json-rpc in the pow consensus building on Substrate? - json-rpc

In the pow consensus module of substrate, the miner does not dig ore by means of RPC access, how to access?
I don't have idea.
fn mine(
&self,
parent: &BlockId<B>,
pre_hash: &H256,
difficulty: Difficulty,
round: u32,
) -> Result<Option<RawSeal>, String> {
let mut rng = SmallRng::from_rng(&mut thread_rng())
.map_err(|e| format!("Initialize RNG failed for mining: {:?}", e))?;
let key_hash = key_hash(self.client.as_ref(), parent)?;
for _ in 0..round {
let nonce = H256::random_using(&mut rng);
let compute = Compute {
key_hash,
difficulty,
pre_hash: *pre_hash,
nonce,
};
let seal = compute.compute();
if is_valid_hash(&seal.work, difficulty) {
return Ok(Some(seal.encode()))
}
}
Ok(None)
}

I suggest you follow a pattern similar to Kulupu for creating a PoW Substrate blockchain.
In Kulupu, it seems the Substrate service starts mining if it detects your are an "authority":
/// Builds a new service for a full client.
pub fn new_full<C: Send + Default + 'static>(config: Configuration<C, GenesisConfig>, author: Option<&str>, threads: usize, round: u32)
-> Result<impl AbstractService, ServiceError>
{
let is_authority = config.roles.is_authority();
let (builder, inherent_data_providers) = new_full_start!(config, author);
let service = builder
.with_network_protocol(|_| Ok(NodeProtocol::new()))?
.with_finality_proof_provider(|_client, _backend| {
Ok(Arc::new(()) as _)
})?
.build()?;
if is_authority {
for _ in 0..threads {
let proposer = basic_authorship::ProposerFactory {
client: service.client(),
transaction_pool: service.transaction_pool(),
};
consensus_pow::start_mine(
Box::new(service.client().clone()),
service.client(),
kulupu_pow::RandomXAlgorithm::new(service.client()),
proposer,
None,
round,
service.network(),
std::time::Duration::new(2, 0),
service.select_chain().map(|v| v.clone()),
inherent_data_providers.clone(),
);
}
}
Ok(service)
}
In this case, you just need to start your node with the --validator flag and an --author flag with address:
cargo run --release -- --validator --author 0x7e946b7dd192307b4538d664ead95474062ac3738e04b5f3084998b76bc5122d

Related

the trait `LoadConnection` is not implemented for `&diesel::PgConnection`

I want to create a rest api with rust and can't make it work.
My relevant code so far:
In the main.rs:
#[actix_web::main]
async fn main() -> std::io::Result<()> {
// Loading .env into environment variable.
dotenv::dotenv().ok();
env_logger::init_from_env(env_logger::Env::new().default_filter_or("info"));
// set up database connection pool
let database_url = std::env::var("DATABASE_URL").expect("DATABASE_URL");
let manager = ConnectionManager::<PgConnection>::new(database_url);
let pool: DbPool = r2d2::Pool::builder()
.test_on_check_out(true)
.build(manager)
.expect("Could not build connection pool");
let port = std::env::var("PORT").expect("$PORT is not set.");
HttpServer::new(move || {
App::new()
.app_data(web::Data::new(pool.clone()))
.wrap(middleware::Logger::default())
.route("/", web::get().to(|| async { "Actix REST API" }))
.service(handlers::common::houses::index)
})
.bind(("0.0.0.0", port.parse().unwrap()))?
.run()
.await
}
The schema:
diesel::table! {
houses (id) {
id -> Int4,
user_id -> Varchar,
street -> Varchar,
figure -> Varchar,
floor -> Varchar,
create_date -> Timestamp,
update_date -> Timestamp,
is_deleted -> Bool,
}
}
The model:
#[derive(Debug, Serialize, Deserialize, Queryable)]
pub struct House {
pub id: i32,
pub user_id: String,
pub street: String,
pub figure: String,
pub floor: String,
pub create_date: chrono::NaiveDateTime,
pub update_date: chrono::NaiveDateTime,
pub is_deleted: bool,
}
The handler:
#[get("/houses")]
async fn index(pool: web::Data<DbPool>) -> Result<HttpResponse, Error> {
let houses = web::block(move || {
let conn = &pool.get()?;
find_all(&conn)
})
.await?
.map_err(actix_web::error::ErrorInternalServerError)?;
Ok(HttpResponse::Ok().json(houses))
}
fn find_all(conn: &PgConnection) -> Result<Vec<House>, DbError> {
use crate::schemas::common::houses::houses::dsl::*;
let items =houses.load::<House>(&mut conn)?;
Ok(items)
}
The dependencies are:
[dependencies]
actix-web = "4"
chrono = { version = "0.4.19", features = ["serde"] }
diesel = { version = "2.0.3", features = ["postgres", "r2d2", "chrono"] }
dotenv = "0.15.0"
env_logger = "0.10.0"
serde = { version = "1.0.136", features = ["derive"] }
serde_json = "1.0"`
It keeps giving an error, and I don't understand why.
The error is:
`error[E0277]: the trait bound `&diesel::PgConnection: LoadConnection` is not satisfied src\handlers\common\houses.rs:25:37
| 25 | let items =houses.load::<House>(&mut conn)?;
| ---- -^^^^^^^^
| | | the trait `LoadConnection` is not implemented for `&diesel::PgConnection` help: consider removing the leading `&`-reference required by a bound introduced by this call
| note: required for `table` to implement `LoadQuery<'_, &diesel::PgConnection, House>` note: required by a bound in `diesel::RunQueryDsl::load`
I've seen a similar error with the diesel version 1.4, but I think that this version is different.
Plus I'm starting with rust and I'm a little lost in general at the moment.
I was hopping someone knows what the problem is and how to fix it.
PgConnection implements LoadConnection but &PgConnection does not (note the extra &).
Make conn mutable and pass as a mutable reference:
#[get("/houses")]
async fn index(pool: web::Data<DbPool>) -> Result<HttpResponse, Error> {
let houses = web::block(move || {
let mut conn = pool.get()?; // <------------
find_all(&mut conn) // <------------
})
.await?
.map_err(actix_web::error::ErrorInternalServerError)?;
Ok(HttpResponse::Ok().json(houses))
}
fn find_all(conn: &mut PgConnection) -> Result<Vec<House>, DbError> {
// ^^^ <------------
use crate::schemas::common::houses::houses::dsl::*;
let items = houses.load::<House>(conn)?; // <------------
Ok(items)
}

UdpFramed with Actix Rust. Can't send messages using SinkWrite

I'm trying to write a Udp Client Actor using Actix. I've followed this example, UDP-Echo, but I can't seem to send a message to the server using the UdpFramed tokio struct.
Here's what I have so far, this is the Udp Client Actor implementation
use std::collections::HashMap;
use std::net::{SocketAddr};
use actix_rt::net::UdpSocket;
use actix::{Actor, Addr, AsyncContext, Context, Handler, StreamHandler, Message};
use actix::io::SinkWrite;
use actix_web::web::{Bytes, BytesMut};
use futures_util::stream::{SplitSink};
use futures_util::StreamExt;
use log::info;
use serde_json::Value;
use tokio_util::codec::BytesCodec;
use tokio_util::udp::UdpFramed;
use crate::rosclient::messages::Subscribe;
use std::io::Result;
mod messages;
type SinkItem = (Bytes, SocketAddr);
type UdpSink = SplitSink<UdpFramed<BytesCodec, UdpSocket>, SinkItem>;
pub struct UdpClientActor {
pub address: SocketAddr,
pub sink: SinkWrite<SinkItem, UdpSink>,
}
impl UdpClientActor {
pub fn start(udp: UdpSocket, address: SocketAddr) -> Addr<UdpClientActor> {
let framed = UdpFramed::new(udp, BytesCodec::new());
let (split_sink, split_stream) = framed.split();
UdpClientActor::create(|ctx| {
ctx.add_stream(split_stream.filter_map(
|item: Result<(BytesMut, SocketAddr)>| async {
item.map(|(data, sender)| UdpPacket(data, sender)).ok()
},
));
UdpClientActor {
address,
sink: SinkWrite::new(split_sink, ctx),
}
})
}
}
impl Actor for UdpClientActor {
type Context = Context<Self>;
fn started(&mut self, ctx: &mut Self::Context) {
let mut hashmap = HashMap::new();
hashmap.insert(String::from("topic"), Value::String(String::from("/client_count")));
let subscription = Subscribe {
id: Default::default(),
op: "subscribe".to_string(),
extra: hashmap
};
ctx.notify(subscription);
}
}
#[derive(Message)]
#[rtype(result = "()")]
struct UdpPacket(BytesMut, SocketAddr);
impl StreamHandler<UdpPacket> for
UdpClientActor {
fn handle(&mut self, item: UdpPacket, _ctx: &mut Self::Context) {
println!("Received: ({:?}, {:?})", item.0, item.1);
self.sink.write((item.0.into(), item.1)).unwrap();
}
}
impl actix::io::WriteHandler<std::io::Error> for UdpClientActor {}
impl Handler<Subscribe> for UdpClientActor {
type Result = ();
fn handle(&mut self, msg: Subscribe, _ctx: &mut Self::Context) -> Self::Result {
let js = serde_json::json!(msg).to_string();
let _ = self.sink.write((Bytes::from(msg.to_string()), self.address));
info!("Subscribing to topic {}", js);
}
}
My main function creates the udp socket and spawns the actor.
fn main() {
////////////////////////////////////////////////////////////////////////////
let fut = async {
////////////////////////////////////////////////////////////////////////////
/////////// UDP_ACTOR
let sock = tokio::net::UdpSocket::bind("0.0.0.0:9091").await.unwrap();
let remote_addr = "172.30.89.169:9091".parse::<SocketAddr>().unwrap();
// let message = b"{ \"op\": \"subscribe\", \"topic\": \"/client_count\"}";
let _ = sock.connect(remote_addr).await;
// sock.send(message).await.unwrap();
let _udp_client = UdpClientActor::start(sock, remote_addr);
};
actix_rt::Arbiter::new().spawn(fut);
// system.block_on(fut);
system.run().unwrap();
}
If I remove the comments on
let message = b"{ \"op\": \"subscribe\", \"topic\": \"/client_count\"}";
and
sock.send(message).await.unwrap();
I can at least check that the server can actually receive messages. So I know that the problem must lie in my implementation of the actor. I do have another one which uses the LinesCodec instead of the BytesCodec, which follows the exact same implementation. Only difference is that SinkWrite becomes this:
SinkWrite<(String, SocketAddr), SplitSink<UdpFramed<codec::LinesCodec>,
(String, SocketAddr)>>
Here is my Cargo.toml for reference.
[package]
name = "local_websocket_client"
version = "0.1.0"
edition = "2018"
[dependencies]
actix="0.12"
actix-codec = "0.4"
actix-rt = "2.5"
bytestring = "1.0"
serde = {version="1.0", features=["serde_derive"]}
log = "0.4"
env_logger = "0.9.0"
chrono = "0.4"
dashmap = "4.0"
futures = "0.3"
openssl = "0.10"
tokio = { version = "1", features = ["full"] }
actix-web = "4.0.0-beta.15"
futures-util = "0.3"
tokio-util = { version="0.6", features=["net", "codec"] }
tokio-udp = "0.1.6"
bytes= { version="0.6", features=["serde"] }
[dependencies.awc]
features = ["openssl"]
version = "3.0.0-beta.9"
[dependencies.serde_json]
features = ["default"]
version = "1.0"
[dependencies.uuid]
features = ["v4", "serde", "v5"]
version = "0.8"
There are some extra crates there because I'm running 2 other websocket clients on the same application.
I would really appreciate some help on this matter.
Thank you
Solved it by wrapping the UdpSocket in an Arc and keeping the reference in the actor for later use. Using the socket to write messages works. The split stream used for the streamhandler needs no change, as it works as expected.

Swift Command Line Tool utilizing Process() and multithreading crashes after a certain number of execution rounds (~3148)

I implemented a password generator script in Swift which utilizes Process() to execute Mac OS X command line tasks. The passwords themselves are just random Strings which then are encrypted (bcrypt) by the command line task as follows:
/usr/sbin/htpasswd -bnBC 10 '' this_is_the_password | /usr/bin/tr -d ':\n'
Also multiple threads are used to generate passwords with their hashes in parallel.
Note: Both the multithreading and the command line task (compared to serveral other native Swift libraries I tried) improved performance in terms of execution time drastically.
The Problem
The Programm runs fine for the first ~3148 rounds and always crashes around this number (problably correlated to the number of threads running).
For example, if I configure 2000 passwords the code executes as expected terminates without any errors.
Common Error Messages
Setting a breakpoint in Process+Pipe.swift in the catch block of the execute(...) function at BREAKPOINT_1 results in:
Thread 1: signal SIGCHLD
po error.localizedDescription
"The operation couldn\\U2019t be completed. (NSPOSIXErrorDomain error 9 - Bad file descriptor)"
When uncommenting the four //return self.hash(string, cost: cost) code snippets to ignore the error the following errors finally crash the execution (again in execute(...), but not necessarily in the catch block):
Program stops ...
Thread 32: EXC_BAD_ACCESS (code=2, address=0x700003e6bfd4)
... on manual continue ...
Thread 2: EXC_BAD_ACCESS (code=2, address=0x700007e85fd4)
po process
error: Trying to put the stack in unreadable memory at: 0x700003e6bf40.
Code
The relevant code componets are the Main.swift which initialized and starts (and later stops) the PasswordGenerator and then loops n times to get passwords via nextPassword() from the PasswordGenerator. The PasswordGenerator itself utilized execute(...) from the Process extension to run commandline tasks which generate the hash.
Main.swift
class Main {
private static func generate(...) {
...
PasswordGenerator.start()
for _ in 0..<n {
let nextPassword = PasswordGenerator.nextPassword()
let readablePassword = nextPassword.readable
let password = nextPassword.hash
...
}
PasswordGenerator.stop()
...
}
}
PasswordGenerator.swift
The PasswordGenerator runs multiple Threads in parallel.
nextPassword() tries to get a password (if there is one in the passwords Array) or else waits for 100 seconds.
struct PasswordGenerator {
typealias Password = (readable: String, hash: String)
private static let semaphore = DispatchSemaphore(value: 1)
private static var active = false
private static var passwords: [Password] = []
static func nextPassword() -> Password {
self.semaphore.wait()
if let password = self.passwords.popLast() {
self.semaphore.signal()
return password
} else {
self.semaphore.signal()
sleep(100)
return self.nextPassword()
}
}
static func start(
numberOfWorkers: UInt = 32,
passwordLength: UInt = 10,
cost: UInt = 10
) {
self.active = true
for id in 0..<numberOfWorkers {
self.runWorker(id: id, passwordLength: passwordLength, cost: cost)
}
}
static func stop() {
self.semaphore.wait()
self.active = false
self.semaphore.signal()
}
private static func runWorker(
id: UInt,
passwordLength: UInt = 10,
cost: UInt = 10
) {
DispatchQueue.global().async {
var active = true
repeat {
// Update active.
self.semaphore.wait()
active = self.active
print("numberOfPasswords: \(self.passwords.count)")
self.semaphore.signal()
// Generate Password.
// Important: The bycrypt(cost: ...) step must be done outside the Semaphore!
let readable = String.random(length: Int(passwordLength))
let password = Password(readable: readable, hash: Encryption.hash(readable, cost: cost))
// Add Password.
self.semaphore.wait()
self.passwords.append(password)
self.semaphore.signal()
} while active
}
}
}
Encryption.swift
struct Encryption {
static func hash(_ string: String, cost: UInt = 10) -> String {
// /usr/sbin/htpasswd -bnBC 10 '' this_is_the_password | /usr/bin/tr -d ':\n'
let command = "/usr/sbin/htpasswd"
let arguments: [String] = "-bnBC \(cost) '' \(string)".split(separator: " ").map(String.init)
let result1 = Process.execute(
command: command,//"/usr/sbin/htpasswd",
arguments: arguments//["-bnBC", "\(cost)", "''", string]
)
let errorString1 = String(
data: result1?.error?.fileHandleForReading.readDataToEndOfFile() ?? Data(),
encoding: String.Encoding.utf8
) ?? ""
guard errorString1.isEmpty else {
// return self.hash(string, cost: cost)
fatalError("Error: Command \(command) \(arguments.joined(separator: " ")) failed with error: \(errorString1)")
}
guard let output1 = result1?.output else {
// return self.hash(string, cost: cost)
fatalError("Error: Command \(command) \(arguments.joined(separator: " ")) failed! No output.")
}
let command2 = "/usr/bin/tr"
let arguments2: [String] = "-d ':\n'".split(separator: " ").map(String.init)
let result2 = Process.execute(
command: command2,
arguments: arguments2,
standardInput: output1
)
let errorString2 = String(
data: result2?.error?.fileHandleForReading.readDataToEndOfFile() ?? Data(),
encoding: String.Encoding.utf8
) ?? ""
guard errorString2.isEmpty else {
// return self.hash(string, cost: cost)
fatalError("Error: Command \(command) \(arguments.joined(separator: " ")) failed with error: \(errorString2)")
}
guard let output2 = result2?.output else {
// return self.hash(string, cost: cost)
fatalError("Error: Command \(command) \(arguments.joined(separator: " ")) failed! No output.")
}
guard
let hash = String(
data: output2.fileHandleForReading.readDataToEndOfFile(),
encoding: String.Encoding.utf8
)?.replacingOccurrences(of: "$2y$", with: "$2a$")
else {
fatalError("Hash: String replacement failed!")
}
return hash
}
}
Process+Pipe.swift
extension Process {
static func execute(
command: String,
arguments: [String] = [],
standardInput: Any? = nil
) -> (output: Pipe?, error: Pipe?)? {
let process = Process()
process.executableURL = URL(fileURLWithPath: command)
process.arguments = arguments
let outputPipe = Pipe()
let errorPipe = Pipe()
process.standardOutput = outputPipe
process.standardError = errorPipe
if let standardInput = standardInput {
process.standardInput = standardInput
}
do {
try process.run()
} catch {
print(error.localizedDescription)
// BREAKPOINT_1
return nil
}
process.waitUntilExit()
return (output: outputPipe, error: errorPipe)
}
}
Question(s)
Why does the program crash?
Why does it not crash for also huge numbers like 2000 passwords?
Is the multithreading implemented correct?
Is there a problem in the execute(...) code?
I have found a fix while researching this bug. It seems that, despite what the documentation claims, Pipe will not automatically close its reading filehandle.
So if you add a try outputPipe.fileHandleForReading.close() after reading from it, that will fix the issue.
It seems this is a Swift bug. I did some testing and could reproduce it by just running a lot of Process.run(). Filed an issue against Swift:
https://bugs.swift.org/browse/SR-15522

Swift, macOS, mac with 2 GPUs, matrix operations work on one GPU, not on the other

I'm running macOS on my MacBook Pro (Retina, 15-inch, Mid 2015), which has two GPUs in it, according to "About this Mac" in the Apple menu. One GPU is an AMD Radeon R9 M370X 2 GB, the other is an Intel Iris Pro 1536 MB -- the standard chips, I guess? They're the chips that were in there when I bought it, nothing I added myself.
I'm using the Swift MPS library for matrix computations; it works great on the Intel GPU, but when I select the Radeon, I only ever get back zeros from every operation, with no error reported. I've looked around for documentation on it, but I can't find anything. The only clue I have so far is that the Radeon reports "not integrated" (or at least, I think it does, based on the sample code at Finding GPUs on macOS, which is about as useful as Apple's doc ever is, meaning not very). If I've read that page correctly, this is what my two GPUs are telling me.
Device Intel Iris Pro Graphics; caps: headful, not discrete, integrated, not external
Device AMD Radeon R9 M370X; caps: headful, discrete, not integrated, not external
I can't find any doc that would suggest what I'm doing wrong. I've been all over Apple's MPS documentation, to no avail. And as I say, the code works great on the Intel GPU, so I should think it would run on the Radeon too. I've run some downloadable diagnostic tools to check on the Radeon, but it doesn't show up in the menus of those tools. So I don't even know whether this is something I'm doing wrong in the code, or if the chip itself is broken.
Below is the code, which you can build as a console app by pasting into main.swift. Find the following line:
let device = MTLCopyAllDevices()[1]
I use [0] for the Intel, [1] for the Radeon, and you can see that the output is different, that is, all zeros for the Radeon. I suppose your mileage may vary depending on your machine. I welcome any input, cheers
import MetalPerformanceShaders
typealias MPSNumber = Float32
let MPSNumberSize = MemoryLayout<MPSNumber>.size
let MPSNumberTypeInGPU = MPSDataType.float32
class MPSNet {
let commandBuffer: MTLCommandBuffer
let commandQueue: MTLCommandQueue
let device = MTLCopyAllDevices()[1]
var neuronsInMatrix1: MPSMatrix?
var neuronsInMatrix2: MPSMatrix?
var neuronsOutMatrix: MPSMatrix?
init() {
guard let cq = device.makeCommandQueue() else { fatalError() }
guard let cb = cq.makeCommandBuffer() else { fatalError() }
commandQueue = cq
commandBuffer = cb
let cMatrices = 2
let cRows = 1
let cColumns = 3
let sensoryInputs1: [MPSNumber] = [1, 2, 3]
let sensoryInputs2: [MPSNumber] = [4, 5, 6]
neuronsInMatrix1 = makeMatrix(device, sensoryInputs1)
neuronsInMatrix2 = makeMatrix(device, sensoryInputs2)
let rowStride = MPSMatrixDescriptor.rowBytes(fromColumns: cColumns, dataType: MPSNumberTypeInGPU)
neuronsOutMatrix = makeMatrix(device, cRows, cColumnsOut: cColumns, rowStride: rowStride)
let adder = MPSMatrixSum(
device: device, count: cMatrices, rows: cRows, columns: cColumns, transpose: false
)
adder.encode(
to: commandBuffer,
sourceMatrices: [neuronsInMatrix1!, neuronsInMatrix2!],
resultMatrix: neuronsOutMatrix!, scale: nil, offsetVector: nil,
biasVector: nil, start: 0
)
commandBuffer.addCompletedHandler { _ in
let motorOutputs = self.getComputeOutput(self.neuronsOutMatrix!)
let discrete = !self.device.isLowPower && !self.device.isRemovable
let caps = "\(self.device.isHeadless ? " headless" : " headful")" +
"\(discrete ? ", discrete" : ", not discrete")" +
"\(self.device.isLowPower ? ", integrated" : ", not integrated")" +
"\(self.device.isRemovable ? ", external" : ", not external")"
print("Device \(self.device.name); caps:\(caps); motor outputs \(motorOutputs)")
}
}
func compute() {
commandBuffer.commit()
commandBuffer.waitUntilCompleted()
}
}
extension MPSNet {
func getComputeOutput(_ matrix: MPSMatrix) -> [Double] {
let rc = matrix.data.contents()
return stride(from: 0, to: matrix.columns * MPSNumberSize, by: MPSNumberSize).map {
offset in
let rr = rc.load(fromByteOffset: offset, as: MPSNumber.self)
return Double(rr)
}
}
func loadMatrix(_ data: MTLBuffer, _ rawValues: [MPSNumber]) {
let dContents = data.contents()
zip(stride(from: 0, to: rawValues.count * MPSNumberSize, by: MPSNumberSize), rawValues).forEach { z in
let (byteOffset, rawValue) = (z.0, MPSNumber(z.1))
dContents.storeBytes(of: rawValue, toByteOffset: byteOffset, as: MPSNumber.self)
}
}
func makeMatrix(_ device: MTLDevice, _ rawValues: [MPSNumber]) -> MPSMatrix {
let rowStride = MPSMatrixDescriptor.rowBytes(
fromColumns: rawValues.count, dataType: MPSNumberTypeInGPU
)
let descriptor = MPSMatrixDescriptor(
dimensions: 1, columns: rawValues.count, rowBytes: rowStride,
dataType: MPSNumberTypeInGPU
)
guard let inputBuffer = device.makeBuffer(
length: descriptor.matrixBytes, options: MTLResourceOptions.storageModeManaged
) else { fatalError() }
loadMatrix(inputBuffer, rawValues)
return MPSMatrix(buffer: inputBuffer, descriptor: descriptor)
}
func makeMatrix(_ device: MTLDevice, _ cRowsOut: Int, cColumnsOut: Int, rowStride: Int) -> MPSMatrix {
let matrixDescriptor = MPSMatrixDescriptor(
dimensions: cRowsOut, columns: cColumnsOut,
rowBytes: rowStride, dataType: MPSNumberTypeInGPU
)
return MPSMatrix(device: device, descriptor: matrixDescriptor)
}
}
let net = MPSNet()
net.compute()
It appears you have failed to use -[MPSMatrix synchronizeOnCommandBuffer:]. On discrete devices, some explicit synchronization is required before the data will come back from the GPU.
The problem lies in the storage mode of your matrix buffers. You're using MTLResourceOptions.storageModeManaged, which tells Metal that you want to manage synchronization of the memory being shared between the CPU and GPU. As mentioned in another answer here, you must use MPSMatrix.synchronize(on: MTLCommandBuffer) after a GPU operation before you try to read the data with the CPU. But you must also synchronize in the other direction, ie, after CPU operations before you commit the command to the GPU, using MTLBuffer.didModifyRange(_: Range).
Alternatively, you can use the shared storage mode, MTLResourceOptions.storageModeShared, which takes care of the synchronization for you.
See Synchronizing a Managed Resource in the Apple doc for details.
Below is a working version of your example using the managed storage mode as you have. Notice the differences in function MPSNet.compute(). You can leave that stuff out and just change the storage mode when creating the MTLBuffers for your matrices if your app is ok to use the shared storage mode.
import MetalPerformanceShaders
typealias MPSNumber = Float32
let MPSNumberSize = MemoryLayout<MPSNumber>.size
let MPSNumberTypeInGPU = MPSDataType.float32
class MPSNet {
let commandBuffer: MTLCommandBuffer
let commandQueue: MTLCommandQueue
let device = MTLCopyAllDevices()[1]
var neuronsInMatrix1: MPSMatrix?
var neuronsInMatrix2: MPSMatrix?
var neuronsOutMatrix: MPSMatrix?
init() {
guard let cq = device.makeCommandQueue() else { fatalError() }
guard let cb = cq.makeCommandBuffer() else { fatalError() }
commandQueue = cq
commandBuffer = cb
let cMatrices = 2
let cRows = 1
let cColumns = 3
let sensoryInputs1: [MPSNumber] = [1, 2, 3]
let sensoryInputs2: [MPSNumber] = [4, 5, 6]
neuronsInMatrix1 = makeMatrix(device, sensoryInputs1)
neuronsInMatrix2 = makeMatrix(device, sensoryInputs2)
let rowStride = MPSMatrixDescriptor.rowBytes(fromColumns: cColumns, dataType: MPSNumberTypeInGPU)
neuronsOutMatrix = makeMatrix(device, cRows, cColumnsOut: cColumns, rowStride: rowStride)
let adder = MPSMatrixSum(
device: device, count: cMatrices, rows: cRows, columns: cColumns, transpose: false
)
adder.encode(
to: commandBuffer,
sourceMatrices: [neuronsInMatrix1!, neuronsInMatrix2!],
resultMatrix: neuronsOutMatrix!, scale: nil, offsetVector: nil,
biasVector: nil, start: 0
)
commandBuffer.addCompletedHandler { _ in
let motorOutputs = self.getComputeOutput(self.neuronsOutMatrix!)
let discrete = !self.device.isLowPower && !self.device.isRemovable
let caps = "\(self.device.isHeadless ? " headless" : " headful")" +
"\(discrete ? ", discrete" : ", not discrete")" +
"\(self.device.isLowPower ? ", integrated" : ", not integrated")" +
"\(self.device.isRemovable ? ", external" : ", not external")"
print("Device \(self.device.name); caps:\(caps); motor outputs \(motorOutputs)")
}
}
func compute() {
for matrix in [neuronsInMatrix1!, neuronsInMatrix2!, neuronsOutMatrix!] {
let matrixData = matrix.data
matrixData.didModifyRange(0..<matrixData.length)
matrix.synchronize(on: commandBuffer)
}
commandBuffer.commit()
}
}
extension MPSNet {
func getComputeOutput(_ matrix: MPSMatrix) -> [Double] {
let rc = matrix.data.contents()
return stride(from: 0, to: matrix.columns * MPSNumberSize, by: MPSNumberSize).map {
offset in
let rr = rc.load(fromByteOffset: offset, as: MPSNumber.self)
return Double(rr)
}
}
func loadMatrix(_ data: MTLBuffer, _ rawValues: [MPSNumber]) {
let dContents = data.contents()
zip(stride(from: 0, to: rawValues.count * MPSNumberSize, by: MPSNumberSize), rawValues).forEach { z in
let (byteOffset, rawValue) = (z.0, MPSNumber(z.1))
dContents.storeBytes(of: rawValue, toByteOffset: byteOffset, as: MPSNumber.self)
}
}
func makeMatrix(_ device: MTLDevice, _ rawValues: [MPSNumber]) -> MPSMatrix {
let rowStride = MPSMatrixDescriptor.rowBytes(
fromColumns: rawValues.count, dataType: MPSNumberTypeInGPU
)
let descriptor = MPSMatrixDescriptor(
dimensions: 1, columns: rawValues.count, rowBytes: rowStride,
dataType: MPSNumberTypeInGPU
)
guard let inputBuffer = device.makeBuffer(
length: descriptor.matrixBytes, options: MTLResourceOptions.storageModeManaged
) else { fatalError() }
loadMatrix(inputBuffer, rawValues)
return MPSMatrix(buffer: inputBuffer, descriptor: descriptor)
}
func makeMatrix(_ device: MTLDevice, _ cRowsOut: Int, cColumnsOut: Int, rowStride: Int) -> MPSMatrix {
let matrixDescriptor = MPSMatrixDescriptor(
dimensions: cRowsOut, columns: cColumnsOut,
rowBytes: rowStride, dataType: MPSNumberTypeInGPU
)
return MPSMatrix(device: device, descriptor: matrixDescriptor)
}
}
let net = MPSNet()
net.compute()

How to return a result from the Blake2 crate in Rust?

I am struggling getting the hash of the passed file name using the blake2 crate. From the documentation:
extern crate blake2;
use blake2::{Blake2b, Digest};
use std::env;
use std::fs;
use std::io::{self, Read};
const BUFFER_SIZE: usize = 1024;
fn print_result(sum: &[u8]) {
for byte in sum {
print!("{:02x}", byte);
}
}
fn process<D: Digest + Default, R: Read>(reader: &mut R) {
let mut sh = D::default();
let mut buffer = [0u8; BUFFER_SIZE];
loop {
let n = match reader.read(&mut buffer) {
Ok(n) => n,
Err(_) => return,
};
sh.input(&buffer[..n]);
if n == 0 || n < BUFFER_SIZE {
break;
}
}
print_result(&sh.result());
}
fn main() {
let args = env::args();
if args.len() > 1 {
for path in args.skip(1) {
if let Ok(mut file) = fs::File::open(&path) {
process::<Blake2b, _>(&mut file);
}
}
} else {
process::<Blake2b, _>(&mut io::stdin());
}
}
blake-test $ cargo run hoge.txt
Compiling blake-test v0.1.0 (/Users/hoge/blake-test)
Finished dev [unoptimized + debuginfo] target(s) in 0.61s
Running `target/debug/blake-test hoge.txt`
eefea9ae6b7fb678ed54e6d58d46aed9eae6d003f29419948cdb42a44a7016dee3eb566e7e95c68ac7587d5debd516a3b195eed0db84d72819e387d687fd06a6
It can successfully print the the &[u8] slice.
However, I want to receive/return the results instead of printing them.
When you're returning a newly-created object, you have to return it as an owned value.
Borrowed references, such as &[u8] are temporary and can't exist by themselves, they're merely a views of data that has storage in an owned form elsewhere.
You can for example, call .to_vec() on the slice and return Vec<u8>.