Managing dataset previews, on dynamic data #53

Activity

1. billhowe commented on Apr 14, 2015

   

   billhowe

   on Apr 14, 2015

   More actions

   On Mon, Apr 13, 2015 at 8:41 AM, Patrick Michaud notifications@github.com
   wrote:

   Currently when a dataset is created, I select the top 100 rows from the
   view of the dataset, and create a new table with that data. That table is
   the source of the sample_data field for a dataset in the REST resource.

   Some (maybe many) datasets are built on top of other datasets, that may
   have their data changed. Is it possible to know when a dataset has changed,
   given that it is N views away from the data that changed?

   Yes, but it's a bit wonky.

   I thin this should be an API call, if it isn't already:
   dependencies(viewname)

   To implement it, we should be able to do this to get the direct
   dependencies (the "children"):

   SELECT * FROM sys.sql_expression_dependencies
   WHERE referencing_id = OBJECT_ID(N'schema.tablename');

   You can make this recursive with a common table expression. There's an
   example here:
   https://technet.microsoft.com/en-us/library/bb677168%28v=sql.105%29.aspx

   Then, whenever a view is modified, we call this function, get the list of
   dependencies, invalidate their previews, recreate the views, and kick off
   the jobs to recompute the previews. (We want view modification to be
   interactive, so I figure we can have previews be computed asynchronously.
   Need to be careful about what happens when we edit the same view over and
   over.)

   This would be a great feature.

   —

   Reply to this email directly or view it on GitHub
   #53.
2. vegitron commented on Apr 14, 2015

   

   vegitron

   on Apr 14, 2015

   Author

   More actions

   I currently have dataset previews happening as an out of process queue. The details there are still pretty handwavy - but that would be a good place to invalidate the dependency chain and rebuild their previews as well, without dragging down the front end.
3. billhowe commented on Apr 14, 2015

   

   billhowe

   on Apr 14, 2015

   More actions

   This seems great, but we need to trigger something when the view is
   actually modified, as I'm not sure there's a "modified date" in the catalog
   we can use.

   On Tue, Apr 14, 2015 at 10:50 AM, Patrick Michaud notifications@github.com
   wrote:

   I currently have dataset previews happening as an out of process queue.
   The details there are still pretty handwavy - but that would be a good
   place to invalidate the dependency chain and rebuild their previews as
   well, without dragging down the front end.

   —
   Reply to this email directly or view it on GitHub
   #53 (comment).

   Bill Howe
   Associate Director and Senior Data Science Fellow, UW eScience Institute
   Affiliate Faculty, Computer Science & Engineering
   University of Washington
   To acknowledge eScience: "Supported in part by the University of Washington
   eScience Institute"
4. vegitron commented on Apr 14, 2015

   

   vegitron

   on Apr 14, 2015

   Author

   More actions

   The view modifications happen through the rest interface, so that can trigger it.