git version control of KNIME workflows

Hello KNIME community.

I’m looking at utilizing git, and GitHub specifically, for version control of KNIME workflows. I read from this forum post https://www.knime.com/forum/knime-general/using-git-or-another-revision-control-system-for-knime-workflows?destination=/forum/knime-general/using-git-or-another-revision-control-system-for-knime-workflows that this is feasible with a proper gitignore file. Does anyone have experience with this?

If this is truly a workable solution, are there any thoughts about adding it to GitHub as a template .gitignore? I recently made a new repo through the GitHub web interface for a project, and there is a pulldown menu there with gitignore files, but KNIME is not included. I’m not sure exactly how to go about this, possibly adding to this repo https://github.com/github/gitignore? I’m also hesitant to do this myself as I have not worked with it yet to know if it is an actual solution or if it needs modification, and I wouldn’t want to appear to be passing off other people’s effort as my own. Anyone have thoughts on this?

Thanks!

  • Jeff
6 Likes

Hey @JeffB,

I read about some people trying to make use of git to version control KNIME workflows and as far as I know the .gitignore in the thread is a good starting point. I will have a deeper look and try to get it working.

However, it will probably only work for versioning the workflow (node settings, workflow structure), but without data. I would not recommend methods like diff, merge etc., since it can have undesirable effects. It might also not work on KNIME Server, since there were issues regarding access rights when switching between versions.

Cheers,

Julian

Hey again,

as far as I can see, the .gitignore still seems to work, but before switching between versions, all files covered by the .gitignore file should be removed. Be aware, that the workflow will be reset and you have to reexecute it, if you do it. This is necessary though, otherwise the tables will contain output data that doesn’t fit to the settings.

In general, I’d say you have to be really careful with this. Only use it for versioning to keep track of node settings and workflow structure, but nothing else.

Cheers,

Julian

2 Likes

Hey Julian,

Thanks for the replies, I apologize for my delayed response. Figured I’d update here in case this is helpful for anyone else.

I have been using git and GitHub successfully for version control of my KNIME workflows! The way I have been doing it is to go to my command line and add the workflow folder, because each individual node is stored as a folder within that workflow folder, and git add on the workflow folder adds, deletes, or updates all the nodes that have changes. I have multiple workflow folders in the git repo, which is also the folder I use as the KNIME workspace, so I could probably just add the whole git repo each time, but I like having the control at the workflow level. Then I commit this and push it to GitHub. I haven’t rigorously tested it, but I have been able to load previous versions of workflows that are still functional, which is all I was looking for.

Recently I encountered the GitHub limit, apparently there is a 100MB limit for individual files, and if you reach that attempts to push to GitHub will fail. The data I work with are microscopy movies, which easily exceed 100MB, and I wasn’t careful about resetting the workflows because I thought the .gitignore would take care of not including the data itself under version control. I’m not an advanced git user, I really just care about using it for versioning as I’m building things in case I make some alteration that breaks things I can get back to a working version and understand how to fix things. With that, I just saved my current working workflows somewhere else, and burned the repo down by deleting the whole thing, and then created a new repo and added my latest working versions. I lost the previous versioning, which is OK for my case, but have versioning again going forward. I will be careful about resetting workflows before I commit and push from now on, and I will update here again at some point with my experiences.

  • Jeff
2 Likes

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.