2010-05-21T02:07:06 *** epifanio has joined #opticks 2010-05-21T03:39:49 *** epifanio has quit IRC 2010-05-21T03:41:05 *** epifanio has joined #opticks 2010-05-21T04:44:18 *** goffena has quit IRC 2010-05-21T04:44:23 *** goffena has joined #opticks 2010-05-21T05:40:14 *** dsulgrov has joined #opticks 2010-05-21T05:45:39 *** tclarke has joined #opticks 2010-05-21T05:45:39 *** ChanServ sets mode: +o tclarke 2010-05-21T05:50:18 *** tclarke has quit IRC 2010-05-21T05:50:25 *** tclarke has joined #opticks 2010-05-21T05:50:25 *** ChanServ sets mode: +o tclarke 2010-05-21T05:54:58 *** tclarke has quit IRC 2010-05-21T05:58:47 *** tclarke has joined #opticks 2010-05-21T05:58:47 *** ChanServ sets mode: +o tclarke 2010-05-21T06:30:31 *** epifanio has quit IRC 2010-05-21T07:06:25 *** epifanio has joined #opticks 2010-05-21T08:18:06 *** rforehan has joined #opticks 2010-05-21T10:33:38 . 2010-05-21T10:33:39 '. 2010-05-21T10:33:39 . 2010-05-21T10:33:43 session save question... 2010-05-21T10:35:15 My extension will start allowing processing of multi-gigabyte datasets (too large to fit into memory). 2010-05-21T10:35:21 So the datasets will be on disk. 2010-05-21T10:35:48 Datasets could be over 10 gigabytes. 2010-05-21T10:37:09 Saving those large datasets to session could take over 10 minutes. 2010-05-21T10:37:13 likly longer 2010-05-21T10:37:16 likely longer 2010-05-21T10:37:41 Would it be possible to rename/move/update the on disk temporary files into the save session directory? 2010-05-21T10:38:06 Thusly saving considerable amount of time, when saving a session with 10GB+ size files? 2010-05-21T10:50:02 back 2010-05-21T10:50:21 the raster elements will only be saved to the session when they are modified (update called on the raster element) 2010-05-21T10:50:26 otherwise, only the import information is saved 2010-05-21T10:51:47 if you are on-disk read-only it will reload the original file...if it's on-disk read/write, it will have to recreate the temp file 2010-05-21T10:52:02 the problem with moving that temp file to the session save directory is that most Windows file systems don't move files like *NIX does 2010-05-21T10:52:14 *NIX will allow you to create a new file metadata block and link to the same data 2010-05-21T10:52:28 Windows will copy the entire thing, so a muli-gigabyte on-disk file might cause session save to take multiple minutes 2010-05-21T10:52:45 if the raster data has been modified, it is "exported" as a raw file to the session save directory and that file is re-imported when the session is loaded 2010-05-21T10:54:48 On my Windows XP development machines, i can cut and paste a file into a different directory very quickly (8GB file took 0-4 seconds), while copying the same 8GB file takes 5+ minutes 2010-05-21T10:56:21 Is that somehow optimized for Windows Explorer, and not optimized when calling code to perform the copies/moves? 2010-05-21T10:56:30 there may be an alternate Win API function for this....there are some other complications...we don't support specifying the temp file when loading so there would need to be changes to the importers as well 2010-05-21T10:56:38 it's likely doable but would require a JIRA issue an rack and stack 2010-05-21T11:00:11 ok. i'll probably submit a jira issue on Monday 2010-05-21T11:05:43 I assume this is for work plug-ins and not off hour ones? I need to know so we can set the priority and charge information appropriately 2010-05-21T12:00:03 for work plugins 2010-05-21T12:00:08 probably will not submit a jira issue afterall. 2010-05-21T12:00:16 after speaking with tjohnson 2010-05-21T12:00:32 we will continue thinking about the large tmp files and session save 2010-05-21T12:03:25 if you may still want the change, submit it and we'll set it as a lower priority...that way it will be in the system and we won't forget it 2010-05-21T13:13:24 *** tclarke has quit IRC 2010-05-21T13:59:41 *** dsulgrov has quit IRC 2010-05-21T17:32:14 *** rforehan has left #opticks 2010-05-21T18:20:01 *** goffena has quit IRC