How to Backup Your KNIME Workflows v2? A Christmas Present for the Community 🎁

Ho ho ho Knimers,

finally, after the Knime Community Hacking days have concluded, I am proud to share with you version 2, with a wealth of improvements and new features, of my backup workflow, previously published in this post:

What’s new?
In short, abandoning build in Windows zip in favor of 7Zip, improving interoperability and providing a feature rich, much faster and more reliable, backup method!

In detail

  1. Biggest change: Restore Feature (Restore from AWS S3 WIP)
  2. Chose between backup locations Workflow Data Directory (default), Local (can be a network or thumb drive) or AWS S3
  3. True delta Processing even with enabled compression
  4. Ability to set a password
  5. CRC Checks by default to ensure data integrity
  6. Compression level can be selected to balance size and speed
  7. Parallel processing / compression if individual / per workflow archives are chosen
  8. Log details
  9. Various bug fixes
  10. And maybe more which I forgot about

About the restore

  1. Interactive View
  2. Select your backup source root
  3. Select your destination root where the data should be extracted into
  4. Knime automatically searches for the archive file “workflow-lastmod.table”, updates the filter and displays the preview
  5. If a password is set, the input automatically appears
  6. Knime generates the command and you only have to execute everything

If data exists in the destination, it is overwritten BUT only the delta. Meaning, any temp data will remain. I have to integrate a process that offers to:

  1. Keep a backup by appending a “-bak”
  2. Wipe the destination root before extracting

As before, I’d love to hear your feedback and hope you find the backup workflow as useful as I do i.e.:

  1. when synchronizing between multiple workstations
  2. backing up before major Knime releases
  3. Restore if Windows Update or a ransomware attack, virus, you name it bricks your system

Cheers
Mike

12 Likes

Merry Christmas Knimers :gift: :christmas_tree: :santa:

It is to my great pleasure sharing with you, just in the nick of time to the 24th of December, an updated version of my Knime Workspace Backup Workflow. With this, full AWS S3 support for backup and restore is introduced!

In a nutshell:

  1. Auto download 7Zip if it ain’t present, easing “Plug&Play” approach!
  2. Check for missing restore parent folder and auto-crate it if necessary
  3. Auto-Testing AWS Credentials and provide error response in interactive view (prev. it never rendered upon error)
  4. Auto preview and download from AWS S3 upon restore
  5. Other bug fixes and enhancements

I’d be happy to hear your feedback, if you liked it, encountered some issues, would like to suggest further enhancements or new features being added.

Enjoy the Christmas time and a happy new year in advance!

Best
Mike

PS: My next challenge is to resume work on the POC to put the whole Knime Workspace under a revision system / repository like GitHub incl. a Large File Storage Server. Stay tuned!

5 Likes