Lookign for ideas on how to manipulate home automation data

Hi folks. I have a house that every second logs the state of 150 inputs. The majority of the inputs are on/off (wall switches, lights, infrared & magnetic sensors). I also have a bunch of analog sensors (energy usage/generation, temperature, etc). I'm starting to do some initial analysis to debug the house's controllers so I'm looking at 2 months worth of data . I have 5M rows spread across 850 hourly CSV files (1GB in total). I have done the basic loading and preprocessing of timestamp so I can now plug analysis and visualization nodes. The first problem I found is that all the plotting nodes cannot handle more than 2.5k rows. I have done some random sampling (coupled with moving aggregation) to reduce the time series granularity so I can plot a big picture view of the whole two months.

This works ok, but overall the experience is slow. I look at the aggregated data then high lite something and go to an Interactive Table node that holds the 5M rows to look into the details of what I highlighted.

Is there a better approach to look at large time series and dynamically zoom in and out?

Thanks for your help


PS: Related question I want to use the GroupBy node to aggregate rows into time buckets. Idealy I would extract a UNIX epoch the divide it to get a minute granularity and then group all the 60 samples I have on that minute. How do I convert a TS to a UNIX epoch?

No sugestions? I review the IoT white paper but it did not help...

FYI, I figured out how to aggregate information in time buckets. I first use this Java snippet to add an index/epoch in minutes

DateFormat format = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.S"); //This is the internal format
format.setTimeZone(new SimpleTimeZone(SimpleTimeZone.WALL_TIME, "ARG"));
Date dt = new Date();
 dt = format.parse($ArgDate$);
}catch(Exception e){}
return Long.toString(dt.getTime()/(1000*60)); //Convert from ms to epoch to min

Then I use a GroupBy node to aggregate all the readings that falll into the same minute bucket. Finally I do row sampling to get at most 2.5k reading. This allows me to navigate a couple of month worth of data with reasonable detail 






1. did you consider using the jfreechart nodes for plotting? As they generate a static image they can handle more datapoints

2. However, in all of our visualization nodes you can change the number of plotted rows in the configuration dialog.

3. you can convert it into a "unix" epoch as follows. Convert to Date and Time Cell. Use the Time Difference node on a fixed date (1.1.1970 gets the unix time) with granularity miliseconds.

Cheers, Iris


for the epoch translation, I use the following snippet:

out_ts = new Date(((long) c_epoch) * 1000);

The output is of type DateAndTimeCell.