Knime Analytics Platform - Batch script & Reset Selected Nodes, and brainstorming

Hello, thanks for your time. After reading FAQ and the first blog post, realize I rushed ahead and posted lol. My bad. Will start there before using user forums!

I’m very new to knime but already digging in deep and I hope this is a simple question. I dig the forum, found answer to my core question immediately.

Couple questions, knime workflow to scrape LinkedIn Job data attached below, and some brainstorming.

Today, I was lucky to have a KNIME data scientist chat w/ me about KNIME on LinkedIn & I had a question about batch scripting.

She sent me to the community, glad she did, answered my question by typing a title! Nice work :slightly_smiling_face:

So here goes, nodes needing to be reset in the UI, to “restart” a create date time, VS leaving it cached (although I’m a big fan of every tool auto caching)…

Saving the file with things reset cleared all this up, silly user error lol.

and…
-reset

answers that too.

Thinking too far ahead… So, I can use KNIME to make a batch script creator, each knime file has evidence of the need to reset, and that could be automated from the file code, which could automatically funnel to building the batch schedule too…(assuming based on limited experience) which would enable me to automate build schedules based on conditions. so rad… :smiley:

I’m wondering if anyone is doing any - Voice to text - and calling the script/macro?

Sorry I feel like im all over the place, not enough coffee, sleep, and having too much fun playing in knime.

Started last week and still can’t find the end of the tunnel. Wish i started using knime sooner. Been getting all my college mates excited about it because we all had to deal with SAP BOBJ data services collectively and we all understand the life of living within advanced etl products.

Thanks for the input, comments, links to your blogs - im eager to learn…

And I’m excited to start writing about the product, this is the first time I feel a company is trying to un-democratize solutions & I’m a big fan.

CANADA_Analytics_v1.001.knwf (141.5 KB)
Attached my first workflow, it’s is a simple api/web scrape that hits LinkedIn job data with some pretty decent success. You can swap out the data/keyword and fire away. Robots.txt and LinkedIn says it’s OK as long as you’re not selling the data or hosting it privately for profit. ENJOY

Best,
Tyler Garrett
www.tylergarrett.com

2 Likes

Hi there!

Welcome to KNIME community Tyler! Glad you like KNIME :slight_smile:

You are really all over the place :smiley:

As I have seen you answered pretty much all questions yourself (with help of this great forum of course) I have checked you workflow and think it is pretty cool. Although I did get an error in first Cell Splitter caused by a TimeoutException in Get Request node. Couple of suggestions from my side:

  • use metanodes(!) - they really help your workflow being tidy and more user friendly
  • use flow variables for workflow parameterization - https://www.knime.com/wiki/flow-variables & https://www.knime.com/knime-introductory-course/chapter7/section1/creation-and-usage-of-flow-variables - must check possibilities that wrapped metanodes offer combined with Quickform nodes(!) and visualization nodes
  • use curved connections - I just think they are so cool
  • instead of using lot of annotations for description add description under the node instead of “Node xx” text
  • for TimeoutException there is Timeout option in Get Request node or using Error handling nodes is pretty handy in this scenario because of workflow dependency on network connection
  • check Column Expression node for substituting other multiple nodes
  • create input files (data) outside of KNIME and read it at the beginning of workflow - this way changing input data does not requires opening KNIME and changing workflow

In links you will see a link to NodePit which is pretty useful. At least for me. Also if you haven’t figure out (or wasn’t aware they exists!) there are lots of KNIME extensions so check them out. The answer to the only unanswered question if I got it right is here (or might help): https://medium.com/@craigcullum/hey-siri-run-my-knime-workflow-b7da176ee243

Happy KNIMEing!

Have a nice day,
Ivan

7 Likes

Love it. Thanks a bunch. Some comments.

  • use metanodes(!) - they really help your workflow being tidy and more user friendly
    awesome, thank you.
  • use flow variables for workflow parameterization - https://www.knime.com/wiki/flow-variables & https://www.knime.com/knime-introductory-course/chapter7/section1/creation-and-usage-of-flow-variables - must check possibilities that wrapped metanodes offer combined with Quickform nodes(!) and visualization nodes
  • use curved connections - I just think they are so cool
    i agree, but never! lol
  • instead of using lot of annotations for description add description under the node instead of “Node xx” text
    i prefer lots of annotations, better explanations, more to come… I come from a time where tools had no annotations, comments, etc. it’s a best practice for mates and myself.
  • for TimeoutException there is Timeout option in Get Request node or using Error handling nodesis pretty handy in this scenario because of workflow dependency on network connection
    awesome, thank you! lots to learn.
  • check Column Expression node for substituting other multiple nodes
    thank you! will do.
  • create input files (data) outside of KNIME and read it at the beginning of workflow - this way changing input data does not requires opening KNIME and changing workflow.
    **im still learning, hope to start making everything this smart soon :smiley: **

Huge help, learned a lot.

Best,
T

2 Likes

Tyler Ive recently made a Meta node with a counting loop to work around the Google Analytics limitation of pulling 10k rows at at time. I pass the query a dynamic start index parameter and right now brute force 10 iterations which will bring in 100k rows. I could up this to any number.
The loop is cool because otherwise you have to daisy chain them together. staring at 10,001 20,001, 30,001 etc. The loop end node does an automatic table concatenation which is awesome.

This flow is a bit rough in terms of being pretty but ive production tested it - without fail. Your blog on this saved my butt - so lmk if you want me to send your way.

3 Likes

Hi webstar,
Looks awesom what you’ve done with your loop. Love the chew on that. Is it possible to send me the workflow? Thanks in advance.
Grz Ruud

HI @k2marketing

If you are after a GA connector that will handle pagination check out - https://hub.knime.com/search?q=120feet

1 Like

Drop that information over here: KNIME Google Authentication API Key Walkthrough – to help others who find things via search/algorithms.

It’s a subsequent thread to a blog i wrote here: https://knime.dev/google-api/knime-google-analytics-connection/

Also, i remember this post was awhile ago. And since then i’ve learned a lot about knime.

Here’s the piece about ; Batch script & reset nodes; decided to write a blog about how to do it on macbook here; https://knime.dev/updates/how-to-automate-knime-apps-in-batch-mode-on-a-mac/

Best, T

1 Like