real-time synchronous streams #18
Comments
|
First off, awesome work in livejs, and glad you joined. 😄 If I understand correctly: Node streams are good at chunking things for larger amounts of data, but for some usecases in audio they would be too slow regardless, where a synchronous version would be faster. I like your idea of using individual synchronous functions, and to some degree I think we already have that with things like I still think it is important to support some level of Node streams so they can work inside the ecosystem (i.e. with sockets or I'm still kind of new to these packages myself, so what are your thoughts @dfcreative? |
yes, Node streams buffers data by default and in some cases add unnecessary clock ticks in processing a pipeline of streams (in the worst case each transform adds a clock tick), both are not particularly helpful for a real-time audio use case, but i think possible to workaround as it seems y'all have done. in pull streams this becomes explicit, since the default is unbuffered synchronous streams (and waaaaay less magic happening, the source code is so small ✨).
yep agree, for this we have |
|
@ahdinosaur oh thanks for the info and links, I appreciate, I’ve read them, very nice! That delay is indeed bad, and tbh I don’t like how streams perform along with web-audio - occasional glitches etc, but it might be not streams, but syncing, GC etc, scriptProcessor is glitchy regardless of the code we use, I hoped upcoming audio-worker could help with that. Also that might be browserify-caused delays etc, I am going to rewrite mcjs to get smaller and faster bundles to see. Some time ago in audio-through or about I had tests of function calls sequence vs raw/object streams and as I got it, streams in flow mode do not necessary cause processor ticks and can be synchronous, actually performance was identical with functions, so I decided that streams were better choice due to conventional API, inheriting Emitter, builtin pressure control etc. As for pull-stream, zen-observable, rxjs, cycle.js, observable-proposal, etc - there are too many and honestly I am lost. I tried to research, compare, but that was difficult, so I opted for the most standard solution. Actually my hope still is to make as less wrappers and be as less opinionated as possible, in favor of native/polyfill solutions. That is why AudioBuffer :) I like the idea of function interface and pull-streams, they are definitely worth a try. But I don’t see clearly why streams are bad. With audio-through they are reduced to a single function, which can be sync or async decided by user, in mocha-tests style, (tbh I am not sold on with Zalgo, mocha sync/async API is very nice): const Through = require('audio-through');
const Speaker = require('audio-speaker');
const util = require('audio-buffer-utils');
//sync style
Through(buffer => util.fill(buffer, Math.random))
.pipe(Speaker());
//async style
Through((buffer, done) => processOnGPU(buffer, done))
.pipe(Speaker());Async style here is not applicable for realtime, but that might be useful for offline sound rendering, preset-based post-processing, decoding/encoding, network streaming etc. So I don’t see apparent benefits of pull-streams compared to regular streams in terms of performance here, neither in terms API simplicity, but they are definitely nicer in terms of code simplicity and magic. It needs research/comparison/benchmark. But that seems reasonable to provide simple function exports for packages, if possible, and a stream wrapper like So what I would suggest - to hold on any refactoring without tangible digits proving why streams are bad and what is better. |
|
@ahdinosaur hi, could you help with composing a simple pull-stream const pull = require('pull-stream');
//pull-steam
//a stream of random numbers.
function source (n) {
return function (end, cb) {
if(end) return cb(end)
//only read n times, then stop.
if(0 > --n) return cb(true)
cb(null, Math.random())
}
}
//volume changer
function through (read, map) {
//return a readable function!
return function (end, cb) {
read(end, function (end, data) {
cb(end, data != null ? map(data) : null)
});
}
}
//read source and log it.
function sink () {
return function (read) {
read(null, function next(end, data) {
if(end === true) return
if(end) throw end
console.log(data)
read(null, next)
});
}
}
pull(source(), through(), sink());Gives an error "read" is not a function. What should I pass as a read argument? How am I supposed to create through instance? This functional thing creates a bit of confusion which function call is what. UPD. Got it https://github.com/dominictarr/pull-stream-examples/blob/master/pull.js, the readme example is broken |
|
@ahdinosaur here is the setup I have for streams https://github.com/dfcreative/stream-contest/blob/gh-pages/stream.js, I want to come up with the same for pull-stream. I think pull-stream should be better for speaker’s case. |
|
@dfcreative awesome! for most real-world use cases i'd recommend using the helper functions within const pull = require('pull-stream')
//pull-steam
//a stream of random numbers.
function source (n) {
return pull(
pull.infinite(Math.random),
pull.take(n)
)
}
//volume changer
function through (map) {
return pull.map(map)
}
//read source and log it.
function sink () {
return pull.log()
}
pull(source(), through(), sink())you can read those functions' source code for how to do this 'right': |
|
here's a first pass of your const pull = require('pull-stream');
const context = require('audio-context');
const util = require('audio-buffer-utils');
let frameSize = 1024;
function sine () {
return pull.infinite(function () {
return util.noise(util.create(frameSize))
})
}
function volume () {
return pull.map(function (data) {
util.fill(data, v => v * .01);
return data
})
}
//create speaker routine
function speaker (context) {
return function (read) {
var bufferNode = context.createBufferSource()
bufferNode.loop = true;
bufferNode.buffer = util.create(2, frameSize)
var node = context.createScriptProcessor(frameSize)
node.addEventListener('audioprocess', function (e) {
read(null, function (err, data) {
util.copy(data, e.outputBuffer)
})
})
bufferNode.connect(node)
node.connect(context.destination)
bufferNode.start()
}
}
pull(sine(), volume(), speaker(context)) |
|
Alright, some preliminary results show that pull-streams have about 35% less overhead than streams, and ~40% more overhead than plain functions. stream-contest |
|
@ahdinosaur I am trying to add pull-stream-to-stream test, and trying to convert sink stream to node stream, but I can’t wrap my head over it: let toStream = require('pull-stream-to-stream');
toStream.source(sine).pipe(toStream(volume)).pipe(toStream(speaker));Does not work. |
|
it looks like so maybe try: let toStream = require('pull-stream-to-stream');
toStream.source(sine).pipe(toStream.sink(speaker)); |
|
Nope, neither this nor Btw @ahdinosaur @mmckegg @jamen could you look at digits in stream-contest? I can’t understand their significance, whether refactoring of streams to pull-streams is a good idea for audiojs. That is O(c) overhead, it is not going to blow up with the data or processing intensity. Also I really want to estimate the price of pull-stream → stream conversion, in case if we need to output sound in stream-compatible way, but seems that now it is a bit tricky. Probably it is the sum of both latencies. |
|
Those numbers look good to me and for what I would use this for. However, I wouldn't mind transitioning for the little added benefit, but only if everyone else wanted to as well. Edit: If those numbers don't change like you say, then I don't know if it is worth the time, but I still wouldn't personally mind. Pull-streams do look nice though. :'( |
hmm, yeah seems like a bug, will want to get that working if we did make this transition. i think with regards to the benchmarks, a relevant number to compare is much time you have to render each frame given 60 frames per second: 16.67 milliseconds. @dfcreative is a 0.5 ms increase in time per stream in a pipeline or per pipeline? as in, if you had many transform streams in between, would that be an extra 0.5ms per transform? i really appreciate you all listening and even considering something like a transition to pull streams, but i don't want to add more work on you all if it doesn't feel right. on the flip side if it does feel right i'd like to make it work as best as possible, given my own adoption of pull streams. at the very least, i'm very happy with an outcome of exporting more simple functions that are not tied to a particular stream implementation. 😄 |
|
One more thing to note - web-audio-api does not actually care about how long it takes for js to process buffer, it will just spit out the data we managed to put into But it appears that stream is a bit heavy interface for common use, and pull-streams are hip but without certainty. Basically there is no big difference between the two, it is like left or right hand in theory, the first is just with heritage and the second is with errors/end propagation. We can place stream implementation to Many //audio-smth/stream.js
module.exports = Through(require('./index'));
//audio-smth/pull.js
//through wrapper putting ./index.js in use
//audio-smth - direct processor or constructor of processor if stateful impl needed
module.exports = function (buffer, cb) {
return util.noise(buffer);
}@ahdinosaur yeah, I think that would be great to have pull-streams supported, we should try some components, like the main ones are |
|
@dfcreative there are three general types of through streams i'm aware of: sync map function (buffer) {
return buffer
}async map function (buffer, cb) {
cb(null, buffer)
}through (like function write (buffer, cb) {
cb(null, buffer)
}
function end (err, cb) {
cb(true)
}either we export the one that makes the most sense and have |
|
audio-through now supports the both sync and async map format, depending on number of arguments. //sync
module.exports = function (buffer) {
return buffer; //← enough just to modify btw, not necessary to return
}
//async
module.exports = function (buffer, cb) {
cb(null, buffer);
}
//stateful
module.exports = function (opts) {
//some state
count = 0;
return (buffer, cb?) => {
count++;
return buffer;
}
} |
|
It’s alright then? I will try to refactor audio-generator. (audio-generate would be a better name though.) |
|
yep exporting a factory function that returns a sync or async map function is perfect. i'm also happy with i'll add to my TODO: implement
this will give us another alternative stream implementation to play with while exporting modules. although, not sure when i'll do this, since i should be working on a full-time contract at the moment, and am playing with LEDs in my free time 😉 but definitely keen to contribute here! |
|
Sure, I am glad that we came up with some convention, and actually happy to have more-or-less universal interface for modules. I think to make it main package exports and stream/pull a secondary. Though it will make greenkeeper shit bricks again sorry for today's spam. In the meantime I will try to bring these 4 modules to the map fn. |
|
To recap convention: //sync
function write (buffer) {
return buffer;
}
function end (buffer) {
return null; //or true?
}
//async
function write (buffer, cb) {
cb(null, buffer)
}
function end (err, cb) {
cb(true)
}@ahdinosaur right? |
|
@dfcreative if everything works as either a sync or async map, i'd suggest the convention is to export: // sync through
module.exports = function (options) {
return function (buffer) {
return buffer
}
}
// async through
module.exports = function (options) {
return function (buffer, cb) {
cb(null, buffer)
}
}which can be used as node streams in the same way as @dfcreative are there use cases that require the more advanced features that |
|
above comment is for transform / through streams. for a readable / source stream like // sync source
module.exports = function (options) {
return function (size) {
return buffer
}
}
// async source
module.exports = function (options) {
return function (size, cb) {
cb(null, buffer)
}
} |
|
What is the way to end these streams? There are plans for audio-splitter and audio-merger components, also audio-mixer. But the API is not designed yet. |
|
I think you just return without doing |
|
and lastly, for writable / sink streams like // sync sink
module.exports = function (options) {
return function (buffer) {
// do stuff
}
}
// async sink
module.exports = function (options) {
return function (buffer, cb) {
// do stuff
cb()
}
}
what use cases for ending a stream do you have in mind? so far, for sync functions you can end by throwing an error, for async functions you can end by calling back with an error, but i can imagine the stream wrappers could add external
yep, simple example: const pipe = require('pump')
const stream = require('stream')
pipe(
process.stdin,
new stream.Transform({
write: function (chunk, enc, cb) {
this.push(chunk)
this.push(chunk)
cb()
}
}),
process.stdout
) |
|
We need to put this in wiki.
E. g. audio-slice, which ends stream after nth second (similar to take in pull-streams). Generate().pipe(Slice(2)).pipe(Speaker())I usually do that by returning null or |
|
okay cool. so one important difference between node streams and pull streams is how to signal the end without an error. in node streams, you signal end with in pull streams, you signal end with personally i prefer the latter ( |
|
What if in functional code we just consider returned null-data as an end? It would be similar to sync style where we return null for end: //sync
return <data|null>;
//async
cb(null, <data|null>);It takes a bit of parsing in pull-stream wrapper, though |
yeah, i noticed that's what you are doing already, is good with me. 👍 |
|
Added wiki. Feel free to correct me. cb(null) //end stream
cb(err) //propagate error
cb(data) //push chunk |
|
@ahdinosaur I’ve refactored audio-generator, so now it exports simple function as I think to make major switch when all components will have pull-streams. |
i guess return
yeah, i reckon throw errors. personally i'm experimenting with returning errors, but i think it's far more common to throw. however i could be convinced otherwise, since throwing implies a
haha, maybe. but what if you data is an error or null? i personally am okay with this separation, hehe.
woo awesome! 🎉 |
|
I am fine to cast errors and to stick with node convention of 2-arg callbacks, that's just fast fix |
|
So far I’ve rewritten audio-generator, audio-sink and web-audio-stream in functional style, and it feels great. |
|
I’ve faced an issue during implementing web-audio-stream/reader. function Reader(options) {
//...skipped inits
let node = context.createScriptProcessor(samplesPerFrame, channels, channels);
let release;
node.addEventListener('audioprocess', e => {
let cb = release;
release = null;
cb && cb(null, e.inputBuffer);
});
return function read (cb) {
release = cb;
}
}From outside I call it like so: let read = Reader();
let count = 0;
read(function getData(err, buffer) {
if (count++ < 5) {
read(getData);
}
//here we need to end read stream
else {
}
});In that case I would follow the usual pull-stream style, just stop reading, but I want also to disconnect and dispose the Also some sources, like audio-generator, might actually take already created buffer to fill, instead of creating a new buffer each call. That is a practice to avoid memory churn, e. g. scriptProcessorNode just swaps inputBuffer and outputBuffer instead of creating a new one. So I would suggest extend sources convention to something like: // sync source
module.exports = function (options) {
return function read (buffer?) {
//return null to end
return buffer
}
}
// async source
module.exports = function (options) {
return function read (cb, buffer?) {
//cb(null, null) to end
//cb(error) to throw error
cb(null, buffer)
}
}Idk actually, I don’t like the uncertainty of arguments here. But swapping the arguments order to UPD. |
|
@dfcreative great to see your progress. 😃 i like everything you've said above, would be happy with any approach you have in mind. another option is being able to optionally attach an prior art:
|
is it possible for |
|
Sounds very nice! Agreed about generator. |
|
So I changed generator notation, now |
|
Nice work :o and |
|
hey @dominictarr, just wondering: i seem to recall one time you saying you were considering making it part of the pull-stream spec that every sink stream had a |
|
wow, very interesting thread. First let me clear up a few things. @dfcreative in #18 (comment) pull-streams/node-streams compatibility - the problem is currently just converting pull-streams to node-streams. I wrote that module a long time ago, but mostly did it the other way - I converted node-streams to pull-streams with stream-to-pull-stream (which is rock solid) recently people transitioning to pull-streams have found that pull-stream-to-stream had some bugs, but we are fixing those currently. And single arg: #18 (comment) the problem with a single argument is that you then have to use type checking. typechecking in javascript is reliable for primitive types, but because of the way node modules work, you might get two different versions of a module, which can make @ahdinosaur |
|
@dominictarr #18 (comment) is no more a big deal since we decided to provide pull and stream for each audio-component. Btw thanks for pull-streams, they are incredibly good! |
|
oh, awesome! |
|
So seems we decided on this, other concerns in separate tickets and @ahdinosaur’s list seems to be done, we can close the issue? |
hey @audiojs/core @audiojs/devs, thanks for the invite.
i'm Mikey. i do audio / visual programming whenever i feel, i host a local meetup called Art~Hack Wellington, @mmckegg and i collaborate in a
livejsorg, i use ndarray as a common data structure between these modules to represent ndsamples or ndpixels, and (mostly thanks to Matt playing Loop Drop) it turns out like this, this, or this.so anyways, why did i make this issue? basically, i like what's happening here, but i have a tiny concern about exporting modules as node streams, because that's what i used to do as well.
why are node streams bad? a few reasons, most relevant here is that node streams are meant to be asynchronous over many event ticks. this is a problem in my most common use case, where the moment i read a "frame" of audio i want to immediately process it and output pixels to the screen (or LEDs). all of this i want to happen synchronously, because the faster the pixels are displayed the better (enough millisecond delay and everyone will notice it's not "on time", even if subconsciously). even a single
process.nextTickalong the pipeline is wasted time, zalgo be damned, although i can see y'all have worked around this (set highWaterMark to zero, etc).what do i recommend instead? so in my modules, i've started doing two things: either a) export a simple function, even if it will commonly be used as a stream. examples:
pixels-apa102,rainbow-pixels,ndsamples-frequencies,ndpixels-convert. or b) export apull-streamwhich by default are unbuffered synchronous streams (and have many other benefits). examples:read-audio.exporting a simple function is nice, because then it's useful for anyone regardless of which stream implementation they favor (@mmckegg uses
observ-style observables for performance reasons, i want Matt to use my modules 😄). this is becoming my preference now. exporting a pull stream is nice because it's a consistent interface, handles back-pressure and error propagation very well, can be used synchronously or asynchronously, and has an vibrant ecosystem. luckily it's trivial to wrap simple functions as pull streams (usingpull.map,pull.infinite,pull.drain, etc).anyways, lots of text. curious to hear what y'all think. thanks again for the invite. 😺
The text was updated successfully, but these errors were encountered: