Assuming we have multiple notification sources (channels) and we want to perform some similar tasks on some of these notification events. For example, consider the case where we call doSomething() every 10 minutes and also upon user request. A possible implementation in Go could be something like this:
var ticker = time.NewTicker(10*time.Minute)
nowDoSomething := make(chan time.Time, 1)
for {
select {
case <-ticker.C:
nowDoSomething<-time.Now()
case <-userReq: // some channel activated occasionally
nowDoSomething<-time.Now()
case <-nowDoSomething:
doSomething()
}
}
What would be other, perhaps better, ways of achieving a similar functionality?
Perhaps you could do this using the Select function from the reflect package:
func Select(cases []SelectCase) (chosen int, recv Value, recvOK bool)
Why signaling to yourself using a channel if you can just call doSomething()? I understand do Something might not be exactly a function, but you can always use a local closure.
See a working example:
package main
import (
"fmt"
"time"
)
func someRoutine (userReq chan int) {
doSomething := func () {
fmt.Printf ("OK %v\n", time.Now())
}
var ticker = time.NewTicker(10*time.Second)
for {
select {
case <-ticker.C:
doSomething()
case <-userReq: // some channel activated occasionally
doSomething()
}
}
}
func main() {
userReq := make(chan int, 1)
go someRoutine(userReq);
userReq <- 1 // user requested some action here (just to mark the start)
time.Sleep(15*time.Second)
userReq <- 1 // user requested some action here
time.Sleep(30*time.Second)// Put your own waiting and exiting logic here
}
If you still want to optimize it beyond this simple correction, there are always some alternatives available. But with just two channels to read from, make sure you're not doing premature optimization.
Related
I want to build a microservice that uses jrpc2 and mongodb to manage small IoT devices. These are the libraries in question:
https://godoc.org/bitbucket.org/creachadair/jrpc2
https://godoc.org/github.com/globalsign/mgo
The problem is, being rather new to Golang in general, I'm not sure how to combine these things together. Here's the code I have:
func DeviceAdd(ctx context.Context) (map[string]string, error) {
m := make(map[string]string)
m["token"] = "0xdeadbeef"
return m, nil
}
func DeviceBootstrap(ctx context.Context, params map[string]string) (map[string]string, error) {
m := make(map[string]string)
m["entered_token"] = params["token"]
return m, nil
}
...
func NewServer() *jrpc2.Server {
assigner := jrpc2.MapAssigner{
"device_add": jrpc2.NewHandler(DeviceAdd),
"device_bootstrap": jrpc2.NewHandler(DeviceBootstrap),
"device_update": jrpc2.NewHandler(DeviceUpdate),
"device_get_status": jrpc2.NewHandler(DeviceGetStatus),
}
srv := jrpc2.NewServer(assigner, nil)
return srv
}
Basically, you write some basic functions that accept a context.Context or any other JSON-RPC params, and jrpc2.NewHandler(function) will attach them to the jrpc2.Server.
The problem is, I cannot change the function handler signatures to put a mgo.Session in there. Or should I make DeviceAdd/DeviceBootstrap methods of a struct, which has access to a mgo.Session?
According to another StackOverflow question, it seems I should make DeviceAdd/DeviceBootstrap methods of a struct that has an mgo.Session, but I'm really not sure if I understood it correctly, or if jrpc2.NewHandler will accept the newly converted methods, or if it's the right thing to do. Also, Golang's context.Context seems like the right place to put such a thing. Help!
A request can be handled by any value that implements the jrpc2.Handler interface. Perhaps the simplest way to adapt an mgo.Session to this interface would be to wrap it in a struct, e.g.,
type SessionHandler struct {
mgo.Session
}
func (s SessionHandler) Handle(ctx context.Context, req *jrpc2.Request) (interface{}, error) {
// whatever you want your handler to do, using the s.Session
// as a delegate.
}
Then you can write something like:
a := handler.Map{"DoAThing": SessionHandler{s}}
to plug it into your server.
This might be a can of worms, I'll do my best to describe the issue. We have a long running data processing job. Our database of actions is added to nightly and the outstanding actions are processed. It takes about 15 minutes to process nightly actions. In Vapor 2 we utilised a lot of raw queries to create a PostgreSQL cursor and loop through it until it was empty.
For the time being, we run the processing via a command line parameter. In future we wish to have it run as part of the main server so that progress can be checked while processing is being performed.
func run(using context: CommandContext) throws -> Future<Void> {
let table = "\"RecRegAction\""
let cursorName = "\"action_cursor\""
let chunkSize = 10_000
return context.container.withNewConnection(to: .psql) { connection in
return PostgreSQLDatabase.transactionExecute({ connection -> Future<Int> in
return connection.simpleQuery("DECLARE \(cursorName) CURSOR FOR SELECT * FROM \(table)").map { result in
var totalResults = 0
var finished : Bool = false
while !finished {
let results = try connection.raw("FETCH \(chunkSize) FROM \(cursorName)").all(decoding: RecRegAction.self).wait()
if results.count > 0 {
totalResults += results.count
print(totalResults)
// Obviously we do our processing here
}
else {
finished = true
}
}
return totalResults
}
}, on: connection)
}.transform(to: ())
}
Now this doesn't work because I'm calling wait() and I get the error "Precondition failed: wait() must not be called when on the EventLoop" which is fair enough. One of the issues I face is that I have no idea how you even get off the main event loop to run things like this on a background thread. I am aware of BlockingIOThreadPool, but that still seems to operate on the same EventLoop and still causes the error. While I'm able to theorise more and more complicated ways to achieve this, I'm hoping I'm missing an elegant solution which perhaps somebody with better knowledge of SwiftNIO and Fluent could help out with.
Edit: To be clear, the goal of this is obviously not to total up the number of actions in the database. The goal is to use the cursor to process every action synchronously. As I read the results in, I detect changes in the actions and then throw batches of them out to processing threads. When all the threads are busy, I don't start reading from the cursor again until they complete.
There are a LOT of these actions, up to 45 million in a single run. Aggregating promises and recursion didn't seem to be a great idea and when I tried it, just for the sake of it, the server hung.
This is a processing intensive task that can run for days on a single thread, so I'm not concerned about creating new threads. The issue is that I cannot work out how I can use the wait() function inside a Command as I need a container to create the database connection and the only one I have access to is context.container Calling wait() on this leads to the above error.
TIA
Ok, so as you know, the problem lies in these lines:
while ... {
...
try connection.raw("...").all(decoding: RecRegAction.self).wait()
...
}
you want to wait for a number of results and therefore you use a while loop and .wait() for all the intermediate results. Essentially, this is turning asynchronous code into synchronous code on the event loop. That is likely leading to deadlocks and will for sure stall other connections which is why SwiftNIO tries to detect that and give you that error. I won't go into the details why it's stalling other connections or why this is likely to lead to deadlocks in this answer.
Let's see what options we have to fix this issue:
as you say, we could just have this .wait() on another thread that isn't one of the event loop threads. For this any non-EventLoop thread would do: Either a DispatchQueue or you could use the BlockingIOThreadPool (which does not run on an EventLoop)
we could rewrite your code to be asynchronous
Both solutions will work but (1) is really not advisable as you would burn a whole (kernel) thread just to wait for the results. And both Dispatch and BlockingIOThreadPool have a finite number of threads they're willing to spawn so if you do that often enough you might run out of threads so it'll take even longer.
So let's look into how we can call an asynchronous function multiple times whilst accumulating the intermediate results. And then if we have accumulated all the intermediate results continue with all the results.
To make things easier let's look at a function that is very similar to yours. We assume this function to be provided just like in your code
/// delivers partial results (integers) and `nil` if no further elements are available
func deliverPartialResult() -> EventLoopFuture<Int?> {
...
}
what we would like now is a new function
func deliverFullResult() -> EventLoopFuture<[Int]>
please note how the deliverPartialResult returns one integer each time and deliverFullResult delivers an array of integers (ie. all the integers). Ok, so how do we write deliverFullResult without calling deliverPartialResult().wait()?
What about this:
func accumulateResults(eventLoop: EventLoop,
partialResultsSoFar: [Int],
getPartial: #escaping () -> EventLoopFuture<Int?>) -> EventLoopFuture<[Int]> {
// let's run getPartial once
return getPartial().then { partialResult in
// we got a partial result, let's check what it is
if let partialResult = partialResult {
// another intermediate results, let's accumulate and call getPartial again
return accumulateResults(eventLoop: eventLoop,
partialResultsSoFar: partialResultsSoFar + [partialResult],
getPartial: getPartial)
} else {
// we've got all the partial results, yay, let's fulfill the overall future
return eventLoop.newSucceededFuture(result: partialResultsSoFar)
}
}
}
Given accumulateResults, implementing deliverFullResult is not too hard anymore:
func deliverFullResult() -> EventLoopFuture<[Int]> {
return accumulateResults(eventLoop: myCurrentEventLoop,
partialResultsSoFar: [],
getPartial: deliverPartialResult)
}
But let's look more into what accumulateResults does:
it invokes getPartial once, then when it calls back it
checks if we have
a partial result in which case we remember it alongside the other partialResultsSoFar and go back to (1)
nil which means partialResultsSoFar is all we get and we return a new succeeded future with everything we have collected so far
that's already it really. What we did here is to turn the synchronous loop into asynchronous recursion.
Ok, we looked at a lot of code but how does this relate to your function now?
Believe it or not but this should actually work (untested):
accumulateResults(eventLoop: el, partialResultsSoFar: []) {
connection.raw("FETCH \(chunkSize) FROM \(cursorName)")
.all(decoding: RecRegAction.self)
.map { results -> Int? in
if results.count > 0 {
return results.count
} else {
return nil
}
}
}.map { allResults in
return allResults.reduce(0, +)
}
The result of all this will be an EventLoopFuture<Int> which carries the sum of all the intermediate result.count.
Sure, we first collect all your counts into an array to then sum it up (allResults.reduce(0, +)) at the end which is a bit wasteful but also not the end of the world. I left it this way because that makes accumulateResults be usable in other cases where you want to accumulate partial results in an array.
Now one last thing, a real accumulateResults function would probably be generic over the element type and also we can eliminate the partialResultsSoFar parameter for the outer function. What about this?
func accumulateResults<T>(eventLoop: EventLoop,
getPartial: #escaping () -> EventLoopFuture<T?>) -> EventLoopFuture<[T]> {
// this is an inner function just to hide it from the outside which carries the accumulator
func accumulateResults<T>(eventLoop: EventLoop,
partialResultsSoFar: [T] /* our accumulator */,
getPartial: #escaping () -> EventLoopFuture<T?>) -> EventLoopFuture<[T]> {
// let's run getPartial once
return getPartial().then { partialResult in
// we got a partial result, let's check what it is
if let partialResult = partialResult {
// another intermediate results, let's accumulate and call getPartial again
return accumulateResults(eventLoop: eventLoop,
partialResultsSoFar: partialResultsSoFar + [partialResult],
getPartial: getPartial)
} else {
// we've got all the partial results, yay, let's fulfill the overall future
return eventLoop.newSucceededFuture(result: partialResultsSoFar)
}
}
}
return accumulateResults(eventLoop: eventLoop, partialResultsSoFar: [], getPartial: getPartial)
}
EDIT: After your edit your question suggests that you do not actually want to accumulate the intermediate results. So my guess is that instead, you want to do some processing after every intermediate result has been received. If that's what you want to do, maybe try this:
func processPartialResults<T, V>(eventLoop: EventLoop,
process: #escaping (T) -> EventLoopFuture<V>,
getPartial: #escaping () -> EventLoopFuture<T?>) -> EventLoopFuture<V?> {
func processPartialResults<T, V>(eventLoop: EventLoop,
soFar: V?,
process: #escaping (T) -> EventLoopFuture<V>,
getPartial: #escaping () -> EventLoopFuture<T?>) -> EventLoopFuture<V?> {
// let's run getPartial once
return getPartial().then { partialResult in
// we got a partial result, let's check what it is
if let partialResult = partialResult {
// another intermediate results, let's call the process function and move on
return process(partialResult).then { v in
return processPartialResults(eventLoop: eventLoop, soFar: v, process: process, getPartial: getPartial)
}
} else {
// we've got all the partial results, yay, let's fulfill the overall future
return eventLoop.newSucceededFuture(result: soFar)
}
}
}
return processPartialResults(eventLoop: eventLoop, soFar: nil, process: process, getPartial: getPartial)
}
This will (as before) run getPartial until it returns nil but instead of accumulating all of getPartial's results, it calls process which gets the partial result and can do some further processing. The next getPartial call will happen when the EventLoopFuture process returns is fulfilled.
Is that closer to what you would like?
Notes: I used SwiftNIO's EventLoopFuture type here, in Vapor you would just use Future instead but the remainder of the code should be the same.
Here's the generic solution, rewritten for NIO 2.16/Vapor 4, and as an extension to EventLoop
extension EventLoop {
func accumulateResults<T>(getPartial: #escaping () -> EventLoopFuture<T?>) -> EventLoopFuture<[T]> {
// this is an inner function just to hide it from the outside which carries the accumulator
func accumulateResults<T>(partialResultsSoFar: [T] /* our accumulator */,
getPartial: #escaping () -> EventLoopFuture<T?>) -> EventLoopFuture<[T]> {
// let's run getPartial once
return getPartial().flatMap { partialResult in
// we got a partial result, let's check what it is
if let partialResult = partialResult {
// another intermediate results, let's accumulate and call getPartial again
return accumulateResults(partialResultsSoFar: partialResultsSoFar + [partialResult],
getPartial: getPartial)
} else {
// we've got all the partial results, yay, let's fulfill the overall future
return self.makeSucceededFuture(partialResultsSoFar)
}
}
}
return accumulateResults(partialResultsSoFar: [], getPartial: getPartial)
}
}
I am dealing with geometric types that can be subdivided into instances of themselves. This capability is expressed by the following protocol:
protocol Subdividable {
func subdivision(using block: (Self) -> ())
}
An implementation of Subdividable might look something like this:
struct Tile { ... }
extension Tile: Subdividable {
func subdivision(using block: (Tile) -> ()) {
if condition() {
let (a, b) = createSubTiles()
block(a)
block(b)
} else {
let (a, b, c) = createDifferentSubTiles()
block(a)
block(b)
block(c)
}
}
}
The number of instances a given type subdivides into is not fixed and may or may not depend on properties of the subdivided instance. Once created, every new instance is passed to block.
To create the final result, I need to apply such subdivisions recursively a given number of times:
extension Subdividable {
func subdivision(level: Int, using block: (Self) -> ()) {
switch level {
case 0:
return
case 1:
subdivision(using: block)
default:
precondition(level > 1)
subdivision { value in
value.subdivision(level: level - 1, using: block)
}
}
}
}
As it can't generally be predicted from the onset, how many times block will be called, it is also necessary to keep track of an index. (i.e. to store the result in a buffer)
extension Subdividable {
#discardableResult func subdivision(level: Int, using block: (Int, Self) -> ()) -> Int {
var result = 0
subdivision(level: level) { value in
block(result, value)
result += 1
}
return result
}
}
So far, so good. The resulting geometries can be very complex and may consist of several million elements, making performance a concern. That's why I tried to divide the recursive subdivision into multiple concurrent tasks, it does however not reliably accumulate the results from each of those:
extension Subdividable {
#discardableResult func subdivision(level: (lhs: Int, rhs: Int), using block: #escaping (Self) -> (Int, Self) -> ()) -> Int {
var result = 0
let (queue, group) = (DispatchQueue.global(), DispatchGroup())
subdivision(level: level.lhs) { value in
queue.async(group: group) {
let count = value.subdivision(level: level.rhs, using: block(value))
// not all barrier tasks will have finished after group.wait()
queue.async(group: group, flags: .barrier) {
result += count // accumulate result
}
}
}
group.wait()
return result
}
}
Why does waiting on the group not guarantee the inner async calls to be finished?
I have also tried implementing the above function with a custom serial queue to push the accumulation block onto, and that works, always giving the correct result. I just don't understand why that's not equivalent to pushing it onto the concurrent queue as a barrier task.
What am I missing here? Is there a way to implement this without requiring a custom queue?
Edit No. 1
The issue doesn't seem to be the group not waiting properly but rather the global queue dropping items. Changing queue from DispatchQueue.global() to DispatchQueue(label: "subdivision", attributes: .concurrent) results in correct behavior.
I also ended up wrapping the entire function body in withoutActuallyEscaping(block) { block in ... } so that the input closure doesn't need to be #escaping, which did not result in a crash when using the global queue, which it would have if there actually were tasks executing after group.wait(). To me this reads as the global queue silently dropping items for whatever reason, right?
How can this be and why does a custom concurrent queue not show the same behavior?
Edit No. 2
According to this answer barriers just aren't supported on global queues. They seem to be happy to accept them but then ignore the flag, running the tasks concurrently, resulting in undefined behavior.
I can't find this being properly documented anywhere, does anybody have some pointers?
I'm just getting started with Golang and writing my first test suite.
I have a background in Rails, which has fantastic support for testing tools (Rspec, Cucumber, etc..), so I'm approaching my golang tests with a similar mindset (not sure if that's the right or wrong thing to do)
I have a User data model (basically a struct) that reads records from a users table in postgres and stores an array of them. (Essentially a really simple version of what ActiveRecord does in the Rails world)
I'd like to write a test that checks if the routine correctly reads from the DB and builds the models.
In almost every test suite I'll be connecting to the DB so I have a helper named establish_db_connection. Where can I place this so that it's centrally available to all my tests?
Building off #1 - is there an equivalent of a before block or some setup/teardown method where I can establish a connection before every test?
Lastly, how do I handle fixtures? Right now before each test I call a clear_db function that resets all tables and inserts some static data rows. I'd love to move away from fixtures and use factories to build data as needed (very similar to FactoryGirl in Rails), but not sure how common that is in Golang.
Is the built-in go test framework the best approach, or are there better alternatives?
Go is based on strong package management, meaning a namespace is treated as one single file. If establish_db_connection is used within a single test package, it can begin with a lowercase letter to signify a private instance and use it in the test file with the same package as the code being tested (Note that naming convention in Go is establishDBConnection).
However, most of the time, as in data/sql, you will want to obtain a DB connection once and keep that around until the test is finished (more like a factory and injection pattern).
There is none in the standard testing package. If you like BDD, Goconvey use scopes to define fixtures and a reset function for teardown.
You can use factory and dependency injections in your testing. I think that's pretty idiomatic.
A few includes Goconvey, Ginkgo and Testify They all have pros and cons of their own. The first two often end up with too many nested scopes, but Goconvey has a great browser-based real-time testing server which can be used with Go standard testing.
Since there's no global variables/functions in Go, you might design your project in interface-delegate pattern to help with importing functions cross packages and avoiding cyclic imports when dealing with cross-package testing.
mypackage
type DBOptions struct {
Name, Credentials string
}
func aFunc(db *sql.DB) error {
// do something
return nil
}
func bFunc(db *sql.DB) int, error {
// do something
return 0, nil
}
func establishConn(opts *DBOptions) (*sql.DB, error) {
db, err := sql.Open(opts.Name, opts.Credentials)
if err != nil {
return nil, err
}
return db, nil
}
func destroyConn(conn *sql.DB) {
conn.Close()
}
// test file
mypackage
import "testing"
var myOpt = &DBOptions{
Name: "mysql",
Credentials: "user:password#tcp(127.0.0.1:3306)/hello",
}
var conn, _ = establishConn(myOpt)
func TestAFunc(t *testing.T) {
err := aFunc(conn)
if err != nil {
t.Error(err)
}
}
func TestBFunc(t *testing.T) {
err := aFunc(conn)
if err != nil {
t.Error(err)
}
}
// use `conn` in other tests ...
destroyConn(conn)
About test fixture library similar to FactoryGirl in Rails, There are some choices in go.
those two libraries got stars most.
https://github.com/bluele/factory-go
https://github.com/go-testfixtures/testfixtures
And I also implemented test-fixture library which is type-safe, DRY, and flexible compared to the above libraries!
https://github.com/k-yomo/fixtory
on fixtures: consider passing functions in your testcases:
package main
import "testing"
type testcase struct {
scenario string
before func(string)
after func()
input string
expOutput string
}
var state = ""
func setup(s string) {
state = s
}
func nilSetup(s string) {}
func reset() {
state = ""
}
func execute(s string) string {
return state
}
func TestSetupTeardown(t *testing.T) {
tcs := []testcase{
{
scenario: "blank output when initial state is wrong",
before: nilSetup,
after: reset,
input: "foo",
expOutput: "",
},
{
scenario: "correct output when initial state is right",
before: setup,
after: reset,
input: "foo",
expOutput: "foo",
},
}
for _, tc := range tcs {
tc.before(tc.input)
if out := execute(tc.input); out != tc.expOutput {
t.Fatal(tc.scenario)
}
tc.after()
}
}
I built a tiny utility library to make it easy to create reusable fixtures to tests in go. Check out https://github.com/houqp/gtest to see if it solves your problem.
Here is a quick example on how to create a database transaction fixture for each test in a test group:
type TransactionFixture struct{}
// Construct can take other fixtures as input parameter as well
func (s TransactionFixture) Construct(t *testing.T, fixtures struct{}) (*sqlx.Tx, *sqlx.Tx) {
tx := // create db transaction here
return tx, tx
}
func (s TransactionFixture) Destruct(t *testing.T, tx *sqlx.Tx) {
tx.Rollback()
}
func init() {
// register and make fixture available to all tests
gtest.MustRegisterFixture(
"Transaction", &TransactionFixture{}, gtest.ScopeSubTest)
}
// begin of test definition
type SampleTests struct{}
func (s *SampleTests) Setup(t *testing.T) {
// you can create/initialize DB in this method
// DB instance can also be implemented as a fixture and get injected into Transanction fixture.
}
func (s *SampleTests) Teardown(t *testing.T) {
// you can clean up all DB resources in this method
}
func (s *SampleTests) BeforeEach(t *testing.T) {}
func (s *SampleTests) AfterEach(t *testing.T) {}
func (s *SampleTests) SubTestFoo(t *testing.T, fixtures struct {
Tx sqlx.Tx `fixture:"Transaction"`
}) {
// transaction is available as fixtures.Tx in this test
}
func TestSampleTests(t *testing.T) {
gtest.RunSubTests(t, &SampleTests{})
}
See https://godoc.org/github.com/houqp/gtest and https://github.com/houqp/gtest/blob/master/example_test.go for more advanced and uptodate examples.
I am trying to write a general purpose wrapper for subscriptions, something like:
type Subscriber interface{
Subscribe(addr string) chan interface{}
}
Suppose there is a library I want to use which has a subscribe method in it, but which uses a chan library.Object. I would like to be able to do something like:
func (s *mySubscriber) Subscribe(addr string) chan interface{}{
ch := make(chan library.Object)
library.Subscribe(addr, ch)
return chan interface{}(ch)
}
Currently, I don't believe such a cast is possible. And I don't want to modify the underlying library, since the wrapper should be agnostic to library implementations.
I've seen Is there a way to cast Structs for sending over a channel, but in that case the application can be modified to suit the need. Here, it can't. Is this possible? Is there a better way?
One solution is to pass in a general purpose channel into Subscribe, and to wait indefinetely on chan library.Object and fire anything that comes through on my general channel, but I didn't particularly like having to introduce another channel just to get around the type cast.
No, you can't do this with just a cast. You have to use an extra channel, as you have already considered. Fortunately, there is a helper library for this already (disclaimer: I wrote it). You want the Wrap function.
Documentation: https://godoc.org/github.com/eapache/channels#Wrap
Code: https://github.com/eapache/channels/
For anyone else that stumbles on this issue and wants some inline code:
// wrap a timeout channel in a generic interface channel
func makeDefaultTimeoutChan() <-chan interface{} {
channel := make(chan interface{})
go func() {
<-time.After(30 * time.Second)
channel <- struct{}{}
}()
return channel
}
// usage
func main() {
resultChannel := doOtherThingReturningAsync()
cancel := makeDefaultTimeoutChan()
select {
case <-cancel:
fmt.Println("cancelled!")
case results := <-resultChannel:
fmt.Printf("got result: %#v\n", results)
}
}