HN Reader

Hi everyone. Posting on HN for the first time. I'd like to share Rill - a toolkit for composable channel-based concurrency, that makes it easy to build concurrent programs from simple, reusable parts

Example of what it looks like:

    // Convert a slice into a channel
    ids := rill.FromSlice([]int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}, nil)


    // Read users from API with concurrency=3
    users := rill.Map(ids, 3, func(id int) (*User, error) {
        return api.GetUser(ctx, id)
    })

    // Process users with concurrency=2
    err := rill.ForEach(users, 2, func(u *User) error {
        if !u.IsActive {
            u.IsActive = true
            return api.SaveUser(ctx, u)
        }
        return nil
    })

    // Handle errors
    fmt.Println("Error:", err)

Key features:

  - Makes concurrent code composable and clean
  - Works for both simple cases and complex pipelines
  - Built-in batching support
  - Order preservation when needed
  - Centralized error handling
  - Zero dependencies

The library grew from solving real concurrency challenges in production. Happy to discuss the design decisions or answer any questions.

Hmmm. Some stuff to like, but I do feel like this should have a big, noticeable cautionary note that it does not wait if you end early (e.g. via Err). Any pending actions continue running and draining in background goroutines, and are potentially VERY delayed if internal/core.Delay is ever exposed or your funcs sleep.

I've seen that kind of pattern lead to TONS of surprise race conditions in user-code, because everyone always thinks that "it returned" means "it's done". Which is reasonable, nothing else is measurable on the caller side - changing that won't be noticeable by callers and may violate their expectations and cause crashes.

Thanks for sharing what is working for you in production. I made a somewhat similar library that also does batching (docs are sparse, but I am updating docs on my libraries this week) [1].

I would call this parallelism rather than concurrency.

The main issue I have with this library's implementation is how errors are handled. Errors are retrieved rather than assigned- but assignment is preferable because it gets verified by tools. In my library I used a channel for errors- that gives ultimate flexiblitiy- it can be converted to wait and collect them to a slice or to perform a cancellation on first error.

[1] https://github.com/gregwebs/go-parallel

Looks good, similar to https://github.com/sourcegraph/conc which we've been using for a while. Will give this a look.

Nice! I do a lot of concurrency work with DAG's in https://github.com/purpleidea/mgmt/ and I would love to swap out some of those concurrency runners with a lib if possible.

I was wondering if this could be it... Any thoughts in that direction, please let me know!

I think that using iterators in the public API would have been better than channels.

Love the idea, some weirdness though:

> Here's a practical example: finding the first occurrence of a specific string among 1000 large files hosted online. Downloading all files at once would consume too much memory, processing them sequentially would be too slow, and traditional concurrency patterns do not preserve the order of files, making it challenging to find the first match.

But this example will process ALL items, it won't break when a batch of 5 finds something?

Hi! Looks great. I might use this to fan out/in my RSS reader HTTP calls.

How would I implement timeout? In case a HTTP call takes too long?

This is great. I am working on a robotics application and this seems like a better abstraction than alternatives such local messaging servers. How do you deal with something like back pressure or not keeping up with incoming data?

this is a real problem in go, very easy to have bugs when working with channels and the way it handles errors etc.

This reminds me of rxjs.

The API looks really nice and intuitive! What motivated you to build this?

I think it is time to face the fact CSP style channels are a bad idea if you don’t also have the occam semantics for channel scope. (I know there are other people here that understood that sentence).

The problem in golang is the channel cleanup, particularly. is a mess. In occam they come and go simply by existing as variables in seq or par blocks. Occam is very static though, so the equivalent to goroutines are all allocated at build. (Occam pi attempted to resolve this iirc).

https://en.m.wikipedia.org/wiki/Occam_(programming_language)

Some of the patterns in this library are reminiscent of constructs occam has as basic constructs, such as the for each block, although the occam one must have the number of blocks known at build time.

The fact so many people in golang reach for mutexes constantly is a sign things are not all well there.

Very handy!

How does it compare to RxGo?

:= for assinging a variable sounds and looks weird to me

what sort of environment do you need to be in to have to compose concurrency like this instead of relying on native go's scaling?

any plans to add context support ?

The batching concept is a cool idea and could be useful in the right context. That said, this feels like a JavaScript engineer's take on Go. Abstractions like Map and ForEach don't align with Go's emphasis on simplicity and explicitness. The lack of context.Context handling also seems like an oversight, especially when considering concurrency.

Judging by the praise, I'm probably in the minority, but as a code reviewer, I’d much rather see straightforward loops, channels, and Go's native constructs over something like Rill.

It looks great. What are other existing tools? And how do they compare to them?