Bitwig FFT Resynthesizer and Note Extractor

[00:00:00] Hey folks, welcome back.
[00:00:01] Today I want to show you something that's never been done before inside of Bitwig Studio.
[00:00:06] I built my own FFT Resynthesizer and Node Extractor.
[00:00:12] So this is not based on bandpass filtering and it works kind of like on vocoder without
[00:00:18] bandpass filters.
[00:00:19] It's pure sine, partial or sine oscillator based and I'll show you in a minute how it
[00:00:25] works.
[00:00:26] First I want to show you how it sounds, because I know some people are impatient, they want
[00:00:30] to see results first, so let's do this.
[00:00:33] So here I have some audio tracks.
[00:00:55] Okay, so this is the Rambo theme of course and what I do is I play a sine partial for
[00:01:22] every possible pitch, for every possible frequency and then I combine it with the audio signal
[00:01:28] and then I can extract the amplitude and then I use this amplitude to play back a sine oscillator.
[00:01:35] And because it's playing back a sine oscillator we can do some tricks.
[00:01:39] First up we have a cutoff here and the cutoff is not for the audio signal, it's actually
[00:01:43] for the amplitude signal to smooth it out, right?
[00:01:46] So when we pull this down we get a stretched out kind of sound.
[00:01:57] So it's basically perfect for making ambient out of everything, because you stretch the
[00:02:03] sound out.
[00:02:05] If you push this up here you get much much faster response times, because the amplitude
[00:02:10] is changing faster, but it's very unpleasant to hear, but you can do it, maybe it gives
[00:02:22] you some interesting distortion, it's a spectral distortion device at this point probably.
[00:02:30] So yeah you can do this, then because we are playing back a sine partial to play back the
[00:02:34] audio amplitude we can just shift the octave of this, so it plays back one octifier.
[00:02:51] So this is basically perfect if you want to put the reverb on this and make a spectral
[00:02:58] shimmer reverb.
[00:03:22] So you can overlay the original audio with some pitched up version of itself and then
[00:03:28] put the reverb on it and you have a much fatter layer and it sounds kind of cool and can use
[00:03:34] it for ambient or some other interesting ideas or just here like I did as a shimmer, spectral
[00:03:40] shimmer reverb.
[00:03:43] So that's that, then we have here something like note delay, so when you have fast response
[00:03:47] times we can take each sine partial and delay it slightly longer than the partial before
[00:03:57] and it's more like a disperser working on audio, but here it's working on the amplitude
[00:04:03] or delayed amplitude.
[00:04:25] So it gives you this typical spectral flare, maybe you try it out on bass lines, maybe
[00:04:30] it's perfect for color bass, I have no idea, I haven't tried it out yet.
[00:04:34] So this is the note delay, then cut off and this one here is for smoothing, the amplitude
[00:04:41] is pretty self-explanatory and octave shift here, then we have scale off, so when you
[00:04:47] disable this, when it's off or when the scale off is off, then it removes all the partials
[00:04:55] that are not in the scale of what you are selected here with the global scale selector.
[00:05:01] So if you select a D sharp minor for instance, all the partials that are not in the scale
[00:05:08] detected in here are not played back.
[00:05:23] So it's not pitch correction, it's more like let's remove all the harmonics that are not
[00:05:30] in the scale and leave the rest untouched.
[00:05:32] So this is my idea for this here instead of correcting because I had some problems with
[00:05:39] certain partials moving then to the same note and it doesn't sound really really well in
[00:05:47] my opinion, so I leave just everything untouched that's not or that's in the scale and what's
[00:05:54] out of the scale is just removed.
[00:05:56] So leave this on if you want to have all partials, so by default this is also on.
[00:06:10] But sometimes it can help right when you play something on the piano and you want to have
[00:06:14] a sample playing on top, but it's not really in the scale, but it's also not really out
[00:06:19] of the scale.
[00:06:20] So here you can just remove everything that's not in the scale.
[00:06:24] So that's the resynthesizer.
[00:06:26] Also very unusual with this resynthesizer is that every partial is on a note.
[00:06:32] Usually with FFT is you take a frequency range, let's say 20 Hertz to 20 K and then you divide
[00:06:39] this up into equal parts, I don't know 1000 bins or whatever.
[00:06:46] And then sometimes some of these partials are on a note sometimes out of the out of
[00:06:52] the scale sometimes up between the notes and so on.
[00:06:55] So it's not musically spread out.
[00:06:58] That's what I want to say.
[00:06:59] So this one here is very musical spread out.
[00:07:02] I don't know if this is actually a problem.
[00:07:04] Maybe that's because why it sounds this way.
[00:07:07] But I basically put the sine partial on every note.
[00:07:11] So here I have the first octave C1 to C2 and then that's one patch.
[00:07:15] This patch has 12 voice stacks for every note in this octave.
[00:07:20] So C, C1, C#1, D1 and so on.
[00:07:26] And every note gets this patch.
[00:07:28] And this patch uses a sine partial here, which then is the split up into sine and cosine.
[00:07:36] So it's basically sine just phase shifted by 180 degree of phase inverted, phase flipped.
[00:07:45] And then we multiply this with the audio signal and we get the amplitude out of the audio
[00:07:50] signal of exactly this frequency.
[00:07:52] I show you this in a minute how it works.
[00:07:55] But this is basically here the patch.
[00:07:58] So this is the patch here to extract basically the frequency bin and then I use here a sine
[00:08:03] partial to play back what I extracted and add here the amplitude using then this here
[00:08:13] to play back the sine wave.
[00:08:15] And we can here change the octave and so on.
[00:08:17] You can also change here the sine partial for something else if you want to.
[00:08:22] So this is roughly how it works.
[00:08:26] Very simple actually.
[00:08:28] You can also extract the phase of the frequency here of this of this audio signal if you want
[00:08:35] to because we have the sine and the cosine and then you can you get an amplitude for
[00:08:41] the cosine, you get an amplitude for the sine.
[00:08:44] And then if you compare this to the audio signal, you can kind of triangulate where
[00:08:49] the position is of the of the sign in the audio signal.
[00:08:54] And then you get kind of a phase position out of this and then you can add it here also
[00:08:59] to the sine oscillator if you want to.
[00:09:02] But it needed to you know, I needed to add more modules here and I want to kept it simple
[00:09:07] and it also sounded very okay ish to my ears already without cloning the phase, but you
[00:09:14] can do it.
[00:09:15] And maybe I do also a patch here in the future for that.
[00:09:18] Anyway, I just want to show you this here.
[00:09:21] I show you in a minute how it's been built.
[00:09:23] But I want to focus now on another preset here, which is called FFT note extract.
[00:09:31] And it does exactly what what the name says, it's it tries to extract the notes from an
[00:09:38] audio signal.
[00:09:40] So here we have again, all your signals already here tries to find some notes and you can
[00:09:48] just use your synthesizer and play these notes back with the synthesizer.
[00:09:53] But we can also create another instrument track here, arm this and as an input we use
[00:10:00] audio one, which is this audio track here, right?
[00:10:04] So now we receive MIDI notes from this patch here, but we also need to add your of course
[00:10:13] an instrument, for instance, piano take eight.
[00:10:32] Again here, this is the cutoff works exactly like the cutoff in this preset.
[00:10:38] And yeah, if you push this up here, you get faster response times, but also a lot of
[00:10:43] more garbage.
[00:10:49] If you pull this down, you get more of the harmonics or of the longer drawn out notes.
[00:11:11] It works probably better here on this ambient track on this single audio track.
[00:11:36] So let's choose here different preset, then we can use on this one transpose, no transpose,
[00:11:44] push it up one octave, maybe use a quantizer so we can quantize this to the BPM here one
[00:11:50] on 10.
[00:11:51] It's probably not not the correct BPM for this track here, but anyway, maybe make this
[00:11:58] longer, put the reverb on that, just play it on top.
[00:12:24] Yeah, it's fun.
[00:12:40] It's not always detecting the right notes, of course, it's a very simple implementation,
[00:12:45] my amateurish kind of implementation, but it's fun.
[00:12:48] It kind of works, and I learned something, you probably also learned something.
[00:12:53] It's fun to play around with, and it doesn't cost anything can download this for free on
[00:12:59] my Patreon or on my in the description here.
[00:13:04] So why not do it, right?
[00:13:06] Of course, it's not that precise like a VST or any other plugin, but you get you get
[00:13:12] something out of it, and it's it's fun.
[00:13:15] Okay, so now that we have this, I show you how it works.
[00:13:21] And for that, we need an FX grid.
[00:13:25] This one here.
[00:13:27] And all we have to do, it's actually not very hard, it's very simple.
[00:13:32] If you think about it, we take a sine oscillator here, and you put this on a specific frequencies.
[00:13:38] We use a constant here, and pitch, pitch to Hertz, Hertz to pitch.
[00:13:47] We put this on a specific frequency, let's say one kilohertz, 1000 Hertz.
[00:13:54] And then we multiply this with the audio signal.
[00:14:01] We take this in the first input, and then the audio signal, we use the audio signal
[00:14:05] to amplify actually the sine oscillator.
[00:14:09] And then we get something out, let's use an oscilloscope, when we put audio in, of course.
[00:14:15] So use your test tone, test tone, I go to one kilohertz, and I use here bit of amplitude,
[00:14:29] and use your sine, okay.
[00:14:32] So we get something out, and if we change the frequency, you can see it's very wild,
[00:14:40] but when we go back to one kilohertz, actually it looks different, right?
[00:14:46] It looks a bit different.
[00:14:48] So with this knowledge, we can already use a filter, so we try to smooth the signal down.
[00:14:56] So we go into a filter here, maybe use a different color here, to see how long.
[00:15:06] Maybe use a low pass, maybe a very steep one, and we try to use the lowest frequency there
[00:15:11] is.
[00:15:12] At the moment, the lowest one is here, 19.4 Hertz, which is okay, I guess, for this experiment.
[00:15:23] But you can go lower, so you can take a constant here, go back to C3, it's the middle position,
[00:15:30] go in here and say minus one, then push this down.
[00:15:34] So now this is the lowest, I think this goes down to at least zero Hertz, so we can go up
[00:15:42] here to maybe six, or six.
[00:15:46] Let's try this first.
[00:15:47] You can see here, it's actually not that wild anymore as a signal compared to this one,
[00:15:52] maybe use a second oscillator here.
[00:15:55] So it's a different signal just by filtering this.
[00:16:00] So now we change the frequency, and you can see it goes back to zero.
[00:16:06] So we have no amplitude here, except when we go to one kilohertz, then we get something.
[00:16:15] So we can extract or can sense or detect that there is some kind of input on exactly this
[00:16:23] frequency.
[00:16:24] Here it's just one single test tone, of course, but when you have an audio signal, you can
[00:16:30] extract basically single frequencies just with this.
[00:16:34] In my patch, then, I used instead of just using here the sign, I'm also using a second
[00:16:42] sign oscillator here, but this one is offset by 180 degrees, so it's the cosine, basically.
[00:16:51] I can actually take an oscilloscope, it's very hard to see, but it's actually exactly
[00:16:59] the opposite.
[00:17:00] So it fills in the gaps, so when this first oscillator goes down, this one goes up and
[00:17:06] so on.
[00:17:07] So we have the sine and the cosine, but instead of using two oscillators here and one is offset,
[00:17:15] I'm using just one, and I used the dome filter here, which is a new module, which gives us
[00:17:22] here the sine and the cosine as a second output.
[00:17:26] So here I'm doing the same thing with the audio signal, but here with the cosine, and
[00:17:31] go also into a filter with the same position here, same filter position, and here we get
[00:17:42] a different position from this one, and we also use a different color.
[00:17:51] So this is the sine, this is the cosine here, and you can extract actually the phase from
[00:17:56] this.
[00:17:57] So because we have the sine and the cosine, we multiply then this with the audio signal,
[00:18:03] and dependent on the level from the sine and the cosine, we can kind of triangulate where
[00:18:09] the original audio phase position is.
[00:18:15] I don't do this in my patch because it needs more modules, and I don't think it's needed,
[00:18:21] but you can do it.
[00:18:22] So this is how we can extract actually the phase.
[00:18:26] But what we do then is we multiply the signal with itself, so we have only positive value.
[00:18:39] So here, and then we add these two together, just an addition, and then we take the square
[00:18:52] root, what's it called in English, square root from this, and this is our amplitude here.
[00:18:59] So you can see we have the test frequency on one kilohertz, and we have an amplitude,
[00:19:05] and we change the frequency, it goes down.
[00:19:08] We are not on 1k anymore, and we don't get an amplitude or value here until we hit 1k.
[00:19:18] Then we get something.
[00:19:19] And the filtering gives you a response time.
[00:19:23] So when you have this open, you get much, much more wider results here, as you can see.
[00:19:30] It responds much faster, but you get also garbage here for frequencies that are not 1k.
[00:19:36] We try to filter everything out, 1k.
[00:19:43] So that's kind of the thing you try to find, the sweet spot between responsiveness and getting
[00:19:54] actually a real pitch harmonic out of this.
[00:19:57] But it kind of works.
[00:19:58] So this is what I did basically for one frequency, and we can of course, instead of using here
[00:20:04] a test tone, we can use an audio signal here, let this play back, and then see what's going
[00:20:12] on at 500 hertz, there's probably something, and you can see here there's something happening
[00:20:19] at some point, probably more 200 hertz, there's more action.
[00:20:32] And then you can extract basically the amplitude of exactly this frequency out of an audio
[00:20:37] signal.
[00:20:38] So this is what I did.
[00:20:39] This is how it works roughly.
[00:20:42] Like I said, I put you here the presets in the description on my Patreon, you can download
[00:20:49] this, leave a thumbs up, leave a subscription, maybe think about the Patreon subscription
[00:20:55] would be nice, but you can download this for free.
[00:20:58] Okay, you don't need to.
[00:21:00] That's it.
[00:21:01] That's it.
[00:21:02] Thank you so much for watching.
[00:21:03] See you in the next video.
[00:21:04] Bye.
[00:21:04] Thanks for watching.
[00:21:05] See you in the next video.
[00:21:06] Bye.
[00:21:06] Thanks for watching. See you in the next video. Bye.