remove zero value for head and tail

hello everyone i’m new user in Knime, i have a problem in removing all the row with zero value for head and tail in audio data table. btw, i’m working on audio recognition using Knime, so i want to filter head and tail for all the zero value. How can i filter out using Rule-Based Row Filter. please any comment will help. Thank you in advance.

Do you know which row is the last header and which is the first tail? If you have this information, you can use the simple Row Filter node but include/exclude by row ID or by row number.

If your problem is more complex than that, then please attach some data and a workflow as well as for us to find a solution to your problem. Thank you.

1 Like

yes, my problem is i want to apply on other files also. i have so many data to filter out the tail and head. which is the best nodes i can use ?

Hi @raihns & welcome to the KNIME community !

If you do not know which row is the last header and which one is the first tail, it would be easier for us to help you if you could post an example of your record files. Otherwise we may misunderstand the nature of your data. For instance, you are showing “zero” values, but since they are “Double” coded, most probably they are not true zeros which would make “zero” comparison (i.e. $Channel 0$ = 0) to fail. There maybe be glitches in your sampled data already within the head and tail of your sound records, etc.

Hence, an example of your data would greatly help us to do our best on helping you.

Is it possible for you to upload here at least a meaningful portion of it ? Thanks in advance.

Best

Ael

2 Likes

Here is a sample workflow which does what I think you are asking using a table editor node (you can replace this node with your data and it should still run, but if not you may have to modify the workflow). If you encounter issues, post your data (or a fake sample of your data) so we can further assist you.

remove_head_tail_zeroes.knar.knwf (19.7 KB)

1 Like

The moving aggregation (cum sum) can give you the first value different from zero, row filter with lower bound set can filter out the beginning zeros, the math formula (col max) should give you the maximum of the moving aggregation column. You split based on the max value and filter out the first (which is the only max value before the zeros you want to keep) and concatenate this one row back to your first dataset)

That sounds way more confusing when I read it as when I thought about it but I saw @victor_palacios already provided help.
br

1 Like

@victor_palacios @aworker

here i attached my workflow on audio processing. As you can see, i want to filter out my head and tail of my sound record. but i didn’t know how to filter out the audio before the window slider node apply. .

AI Signal.knwf (28.3 KB)

Hi @raihns

Thanks for posting the workflow. Interesting workflow using a plugin for Signal Processing. If I’m not wrong, your aim is to extract Frequency & Time Domain features for later processing.

As I said in my previous post, if you know beforehand the head and tail sampled data starting and ending index then you can go for @victor_palacios solution.

If you don’t have this index information, you can still do a cumulative operation using a -moving aggregation- node to detect when the aggregated values go over a given threshold, as suggested by @Daniel_Weikert . This can be done backwards from the end too (you would need to calculate two different aggregated columns, one forward & another backward). In signal processing (since you are in this field), this is the equivalent of integrating your discrete signal. This is feasible if your signal is always positive. Otherwise you would need to unsign or square your signal using the -math operation- node. This latter case is the same as calculating the energy of your signal.

As I said, it would be nice if you could post and example of your data so that we can implement something sensible based on it. If your head and tail data is noisy, even without music (for instance, if your records come from digital transcriptions of old records without filtering preprocessing), then the aggregation approach may find false “ending of head” & “beginning of tail” indexes. That’s why a data sample would help here to refine the solution.

If your head & tail signals are noisy, a low past filter before calculating the energy of the signal may help to better determine a threshold for these index values.

Hope this helps.

Best

Ael

2 Likes

Hi @victor_palacios,

I had a similar problem some time back Find frist date with non zero value in a time series and there where a KNIME solution and a python script solution. Maybe you want to check this out. You can easily modify the scripting solution to also remove the tail.

Best Regards,

Paul

1 Like

Thankyouu. That helps a lot. This is example of my data. I used 10 commands for 10 speakers.
https://drive.google.com/drive/folders/1nnf78azkPyZv2arVH8ImKCWzCUzYPiip?usp=sharing

1 Like

My pleasure @raihns. Thanks for the files. I just downloaded them and will come back with my thoughts and a possible solution.

Best

Ael

Thank you so much for your concern. i’ll wait for your possible solution while i’m trying on @victor_palacios 's sample workflow.

1 Like

Hi @raihns

My pleasure. Please find below a possible solution which is based on an automatically estimated threshold of the level of Head & Tail Noise:

The “Zero” data is noisy and hence this needs to be taken into account. Please have a look at it and just let us know if you have any questions about it. I guess from what you already implemented that understanding this tail & head suppression should be easy for you :wink:

20220125 Pikairos remove zero value for head and tail.knwf (2.1 MB)

Best wishes,

Ael

6 Likes

Thankyou so much. That helps me a lot. Yes, your worklow filter out my head and tail data.

Thank you to @victor_palacios & @Daniel_Weikert for helping me out. I appreciate it so much.

Best

Rai

2 Likes

Hi @raihns , this is quite an interesting project.

@aworker quite impressive.

I love comparing the before and after chart:
Before:

After:

How do you convert the result back to wav :slight_smile: ? It would be interesting to hear the result without the head and tail noise

3 Likes

I think remove 0s at head and tail just shorten the length of it :innocent:

Thanks @bruno29a for your kind comments !

I had the same thought when I completed @raihns’s workflow with this head & tail signal suppression since nothing better than checking by ear. Unfortunately, I couldn’t find a -WAV writer- node :pensive:

It would be nice to have one lol :smiley: :+1:!

Best

Ael

2 Likes

This is indeed interesting. I never worked with sound data before. Also great idea to use the MA Node backwards @aworker . I did not thought about that. Thanks for sharing
br

1 Like

I’m so sorry for the late reply. Yes, removing zero at head and tails just to shorten the length @Kenyx.
also i cannot find to remove the -wav nodes. @aworker. But its okay, i just want to shorten a little bit. You have helped me a lot.

@bruno29a haha I have many more steps to do. i working on automatic speech recognition in-vehicle using knime. yes this is interesting but quite hard to do :rofl: but i’ll try my best.

best,
rai

2 Likes

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.