Meeting on August 9, 2010
Agenda
- Attendance
- Announce agenda
- Approve July 26, 2010 Meeting Notes. Ask for review and address any problems seen in the notes.
- Update on Google Summer of Code
- Open Questions
- Summary and closing
Notes
Attendance
- tclarke
- dadkins
- goffena
- rforehan
- dsulgrov
- tjohnson
- mconsidi
Summary
Pencils down for Summer of Code is today. Watch for student final deliverables.Pencils down for Summer of Code is today. Watch for student final deliverables.
There are some issues with Visual Studion 2010 which might delay the update. There is concern about the end of life for VS2010 in April. The upgrade will progress and a decision about how to proceed will happen when we find out if Microsoft has a hotfix for the bug.
There is a performance issue related to the import options dialog's metadata tab which is causing problems for tjohnson and goffena. Some developers have begun an evaluation during this meeting which will continue after the meeting. A summary will be posted when the problem is discovered. A key question is if this will drive a 4.4.0rc4 release. The answer is still not clear.
There are some issues with Visual Studion 2010 which might delay the update. There is concern about the end of life for VS2010 in April. The upgrade will progress and a decision about how to proceed will happen when we find out if Microsoft has a hotfix for the bug.
There is a performance issue related to the import options dialog's metadata tab which is causing problems for tjohnson and goffena. Some developers have begun an evaluation during this meeting which will continue after the meeting. A summary will be posted when the problem is discovered. A key question is if this will drive a 4.4.0rc4 release. The answer is still not clear.
Decisions
Logs
2010-08-09T11:00:55 <tclarke> here
2010-08-09T11:00:57 <dadkins> here
2010-08-09T11:01:00 <goffena> here
2010-08-09T11:01:01 <rforehan> here
2010-08-09T11:01:02 <dsulgrov> here
2010-08-09T11:01:02 *** tjohnson has joined #opticks
2010-08-09T11:01:07 <tclarke> 1. Attendance
2010-08-09T11:01:07 <tclarke> 2. Announce agenda
2010-08-09T11:01:07 <tclarke> 3. Approve July 26, 2010 Meeting Notes. Ask for review and address any problems seen in the notes.
2010-08-09T11:01:07 <tclarke> 4. Update on Google Summer of Code
2010-08-09T11:01:07 <tclarke> 5. Open Questions
2010-08-09T11:01:07 <tclarke> 6. Summary and closing
2010-08-09T11:01:08 *** mconsidi has joined #opticks
2010-08-09T11:01:09 <tjohnson> here
2010-08-09T11:01:11 <mconsidi> here
2010-08-09T11:01:12 <tclarke> last meeting notes are here: http://opticks.org/confluence/display/opticksChat/July+26%2C+2010+Meeting+Notes
2010-08-09T11:02:09 <tclarke> Google SoC pencils down date is today...the students have one more week to put together their final deliverables and tie up loose ends
2010-08-09T11:02:51 <tclarke> there were some good plug-ins created so be on the lookout for the final AEBs and reports
2010-08-09T11:03:04 <tclarke> anything else to discuss about SoC?
2010-08-09T11:03:25 <tclarke> ok, the other topic I would like to bring up is the conversion to Visual Studio 2010
2010-08-09T11:03:34 <tclarke> I've been working with Qt v4.7.0 beta 2 since 4.7.0 is the first Qt version to support the new compilers
2010-08-09T11:03:39 <tclarke> I've run into a few problems. one in particular is troubling
2010-08-09T11:03:42 <tclarke> http://bugreports.qt.nokia.com/browse/QTBUG-11445
2010-08-09T11:03:49 <tclarke> 64-bit VS2010 compiles crash in a number of places
2010-08-09T11:04:05 <tclarke> it appears to be a problem in the compiler with misaligned data and SSE. A workaround seems to be disabling optimizations
2010-08-09T11:04:18 <tclarke> I spoke with some of the Qt developers on #qt-labs and they have informed microsoft
2010-08-09T11:06:07 <tclarke> they told me MS is planning on having a hot fix available in 3-4 weeks
2010-08-09T11:06:18 <tclarke> this may impact the timeline for the conversion to 2010 and we may not be able to make the 4.5.0 deadline
2010-08-09T11:06:23 <tclarke> how will this impact folks?
2010-08-09T11:06:24 <tclarke> .
2010-08-09T11:06:46 <kstreith> makes it harder for people in the open-source to build plug-ins if we can't upgrade the compiler
2010-08-09T11:06:48 <tjohnson> Does that mean staying with 2005 for 4.5.0 or slipping the 4.5.0 deadline?
2010-08-09T11:06:52 *** gmartin_cn has joined #opticks
2010-08-09T11:08:11 <dadkins> There is a slight impact on OPTICKS-516. The short term workaround is to update Xerces and XQilla using VS 2005 builds for the time being. When we update to VS 2010 they will need to be rebuilt there. TBD whether we will keep versions or upgrade again at that point (probably keep them unless there is a compelling reason otherwise).
2010-08-09T11:08:22 <tclarke> that has not been dicussed...at this point it would mean sticking with 2005 for 4.5.0...if the slip would be a short one we could possibly slip by a week or two but there would need to be a fair bit of deconfliction, etc. for that to happen
2010-08-09T11:09:00 <goffena> Is there any reason not to let OPTICKS-516 slip past 4.5.0?
2010-08-09T11:10:21 <tclarke> tjohnson: do you have a preference which way it goes?
2010-08-09T11:10:24 <tjohnson> I am not aware of any direct impact on us with staying with 2005 through 4.5.X.
2010-08-09T11:10:29 <kstreith> i'm concerned about staying on 2005
2010-08-09T11:10:32 <kstreith> at some point we hit end of life on that version
2010-08-09T11:10:40 <tclarke> OPTICKS-516 does fix some memory leaks so we'd like to get it in 4.5.0 and most of the work for a 2005 based implementation is done
2010-08-09T11:10:41 <tjohnson> What is the end of life date for 2005?
2010-08-09T11:10:41 <kstreith> what happens when that occurs and we are still in binary compat mode?
2010-08-09T11:11:06 <tclarke> kstreith: I agree but we may have little choice...stay with an EOL 2005 or upgrade to a version that either crashes or runs without optimizations
2010-08-09T11:11:07 <kstreith> http://support.microsoft.com/lifecycle/search/?sort=PN&alpha=Visual+Studio
2010-08-09T11:11:11 <kstreith> mainstream support ends 4/12/201
2010-08-09T11:11:13 <kstreith> mainstream support ends 4/12/2011
2010-08-09T11:11:13 <tclarke> we can still get it in the trunk ASAP but it might not make the 4.5.0 release
2010-08-09T11:11:31 <kstreith> there is 2008, don't forget
2010-08-09T11:11:41 <tclarke> I am still moving forward with the 2010 upgrade branch and with luck it will be ready to merge as soon as the fix and final Qt 4.7.0 are ready
2010-08-09T11:11:49 <tclarke> There is 2008 but it is a lot of work for very little benefit
2010-08-09T11:12:01 <tclarke> I'd rather run on EOL for a couple of months as it could take the better part of a month to get a 2008 upgrade done
2010-08-09T11:12:12 <goffena> what about releaseing both VS2005 and VS2010 builds of Opticks 4.5.0?
2010-08-09T11:14:31 <kstreith> that is a VERY big task
2010-08-09T11:14:37 <kstreith> and one I'd avoid at all costs
2010-08-09T11:14:48 <tclarke> goffena: that would require maintining two separate 4.5.0 trunks...as mentioned, it is a very large undertaking
2010-08-09T11:15:16 <tclarke> ok, any more discussion or ideas? we can continue on the mailing list if people would like a more in depth or prolonged discussion
2010-08-09T11:16:40 <goffena> i would place something on the mailing list, just so others are aware of the situation.
2010-08-09T11:16:47 <tclarke> goffena: I will do that after the meeting
2010-08-09T11:16:49 <tclarke> any other topics for today?
2010-08-09T11:16:52 <goffena> yes
2010-08-09T11:16:53 <dadkins> wait & see – chances are the problem fixes itself
2010-08-09T11:17:00 <goffena> will there be a Opticks 4.4.0 RC4?
2010-08-09T11:17:05 <dadkins> NO
2010-08-09T11:17:08 <tclarke> no current plans
2010-08-09T11:17:15 <tclarke> as indicated by the exclamation that just came out of mconsidi and kstreith's office 
2010-08-09T11:17:18 <goffena> i sent dsulgrov an email outlining a problem i discovered on Friday
2010-08-09T11:18:39 <goffena> at one point sounded like tjohnson would like it fixed
2010-08-09T11:18:45 <goffena> not sure what tjohnson's current thoughts are
2010-08-09T11:18:55 <goffena> The problem involved performance degrading on the Import Options Dialog
2010-08-09T11:18:57 <tclarke> what version does it need to be fixed in? I'm aware of the problem and I'm not sure if we have a solution yet...certainly not one that can make the 4.4.0 timeline
2010-08-09T11:19:00 <goffena> when selecting datasets
2010-08-09T11:19:01 <tclarke> it is something that could be in 4.4.1
2010-08-09T11:19:14 <tjohnson> The impact is severe enough that we would need to stay with 4.3.4 until it is fixed.
2010-08-09T11:19:31 <tclarke> dsulgrov: thoughts?
2010-08-09T11:19:34 <goffena> would adding a config setting to ignore wavelengths be a sufficent short term solution, so that it could be placed into a RC4 build?
2010-08-09T11:21:21 <dsulgrov> I'm not sure what the cause of the slowdown is
2010-08-09T11:21:27 <kstreith> trying now...
2010-08-09T11:21:32 <dsulgrov> it appears to be somewhere either in Qt code, or how we are using the Qt widgets
2010-08-09T11:23:11 <dsulgrov> I think that a short term workaround, like what goffena suggests may take some time to design and implement
2010-08-09T11:23:44 <kstreith> just tried cube creator cube with 4000 bands
2010-08-09T11:23:46 <kstreith> and no special metadata
2010-08-09T11:23:49 <kstreith> takes about 14 seconds to switch datasets
2010-08-09T11:23:55 <kstreith> even if you never look at the wavelengths tab
2010-08-09T11:25:49 <dadkins> kstreith: Humor me and click the "Apply" button. How long does it take before the GUI responds after that?
2010-08-09T11:25:50 <tjohnson> how much did your memory usage go up? We are seeing 30-50 MB on a 3000 frame dataset.
2010-08-09T11:25:53 <kstreith> for comparison, same files in 4.3.3, the switch is instant
2010-08-09T11:26:03 <kstreith> dadkins: apply button on the 4000 frame dataset?
2010-08-09T11:26:08 <dadkins> Yes
2010-08-09T11:26:13 <kstreith> import options dialog doesn't have an apply button
2010-08-09T11:27:35 <dadkins> My mistake – import it and try the properties dialog
2010-08-09T11:27:55 <kstreith> looking into mem usage question first
2010-08-09T11:30:09 <kstreith> yes, i'm seeing 50 MB RAM usage increase
2010-08-09T11:30:13 <kstreith> every time i click back to the 4000 frame dataset
2010-08-09T11:30:26 <kstreith> so I click over to the other dataset, back to 4000 frame dataset, another 50 MB of RAM consumed
2010-08-09T11:30:56 <rforehan> Any chance the wavelength info is being repeatedly added to the metadata dynamic object?
2010-08-09T11:31:14 <kstreith> these cubes have no metadata
2010-08-09T11:31:15 <tclarke> I don't believe there is any wavelength info in the situation tested
2010-08-09T11:31:18 <kstreith> and no it isn't appearing on the metadata tab
2010-08-09T11:34:02 <rforehan> Does a 2000 band cube eat ~25MB per change?
2010-08-09T11:34:18 <kstreith> closing the import options dialog has now pegged my CPU to 100% on one core
2010-08-09T11:36:08 <kstreith> RAM is re-claimed once the file is imported
2010-08-09T11:36:28 <tjohnson> we didn't have any wavelength info either.
2010-08-09T11:36:53 <tclarke> perhaps it's delayed memory cleanup that's somehow causing the slowdown?
2010-08-09T11:38:22 <kstreith> around ~25MB for 2000 band image
2010-08-09T11:38:46 <tclarke> let's finish the meeting and then continue this session so people not interested can leave...are there any other discussion topics for today?
2010-08-09T11:39:07 <tclarke> ok then, meeting adjourned
2010-08-09T11:39:08 <dadkins> one quick thing: I am implementing a K-Means classification algorithm for a class project. Interested parties can contact me offline for details
2010-08-09T11:40:26 <kstreith> ~4 MB for 300 band image again no wavelengths
2010-08-09T11:40:34 <tclarke> if you are interested in the import options dialog problem, stick around
2010-08-09T11:00:55 <tclarke> here
2010-08-09T11:00:57 <dadkins> here
2010-08-09T11:01:00 <goffena> here
2010-08-09T11:01:01 <rforehan> here
2010-08-09T11:01:02 <dsulgrov> here
2010-08-09T11:01:02 *** tjohnson has joined #opticks
2010-08-09T11:01:07 <tclarke> 1. Attendance
2010-08-09T11:01:07 <tclarke> 2. Announce agenda
2010-08-09T11:01:07 <tclarke> 3. Approve July 26, 2010 Meeting Notes. Ask for review and address any problems seen in the notes.
2010-08-09T11:01:07 <tclarke> 4. Update on Google Summer of Code
2010-08-09T11:01:07 <tclarke> 5. Open Questions
2010-08-09T11:01:07 <tclarke> 6. Summary and closing
2010-08-09T11:01:08 *** mconsidi has joined #opticks
2010-08-09T11:01:09 <tjohnson> here
2010-08-09T11:01:11 <mconsidi> here
2010-08-09T11:01:12 <tclarke> last meeting notes are here: http://opticks.org/confluence/display/opticksChat/July+26%2C+2010+Meeting+Notes
2010-08-09T11:02:09 <tclarke> Google SoC pencils down date is today...the students have one more week to put together their final deliverables and tie up loose ends
2010-08-09T11:02:51 <tclarke> there were some good plug-ins created so be on the lookout for the final AEBs and reports
2010-08-09T11:03:04 <tclarke> anything else to discuss about SoC?
2010-08-09T11:03:25 <tclarke> ok, the other topic I would like to bring up is the conversion to Visual Studio 2010
2010-08-09T11:03:34 <tclarke> I've been working with Qt v4.7.0 beta 2 since 4.7.0 is the first Qt version to support the new compilers
2010-08-09T11:03:39 <tclarke> I've run into a few problems. one in particular is troubling
2010-08-09T11:03:42 <tclarke> http://bugreports.qt.nokia.com/browse/QTBUG-11445
2010-08-09T11:03:49 <tclarke> 64-bit VS2010 compiles crash in a number of places
2010-08-09T11:04:05 <tclarke> it appears to be a problem in the compiler with misaligned data and SSE. A workaround seems to be disabling optimizations
2010-08-09T11:04:18 <tclarke> I spoke with some of the Qt developers on #qt-labs and they have informed microsoft
2010-08-09T11:06:07 <tclarke> they told me MS is planning on having a hot fix available in 3-4 weeks
2010-08-09T11:06:18 <tclarke> this may impact the timeline for the conversion to 2010 and we may not be able to make the 4.5.0 deadline
2010-08-09T11:06:23 <tclarke> how will this impact folks?
2010-08-09T11:06:24 <tclarke> .
2010-08-09T11:06:46 <kstreith> makes it harder for people in the open-source to build plug-ins if we can't upgrade the compiler
2010-08-09T11:06:48 <tjohnson> Does that mean staying with 2005 for 4.5.0 or slipping the 4.5.0 deadline?
2010-08-09T11:06:52 *** gmartin_cn has joined #opticks
2010-08-09T11:08:11 <dadkins> There is a slight impact on OPTICKS-516. The short term workaround is to update Xerces and XQilla using VS 2005 builds for the time being. When we update to VS 2010 they will need to be rebuilt there. TBD whether we will keep versions or upgrade again at that point (probably keep them unless there is a compelling reason otherwise).
2010-08-09T11:08:22 <tclarke> that has not been dicussed...at this point it would mean sticking with 2005 for 4.5.0...if the slip would be a short one we could possibly slip by a week or two but there would need to be a fair bit of deconfliction, etc. for that to happen
2010-08-09T11:09:00 <goffena> Is there any reason not to let OPTICKS-516 slip past 4.5.0?
2010-08-09T11:10:21 <tclarke> tjohnson: do you have a preference which way it goes?
2010-08-09T11:10:24 <tjohnson> I am not aware of any direct impact on us with staying with 2005 through 4.5.X.
2010-08-09T11:10:29 <kstreith> i'm concerned about staying on 2005
2010-08-09T11:10:32 <kstreith> at some point we hit end of life on that version
2010-08-09T11:10:40 <tclarke> OPTICKS-516 does fix some memory leaks so we'd like to get it in 4.5.0 and most of the work for a 2005 based implementation is done
2010-08-09T11:10:41 <tjohnson> What is the end of life date for 2005?
2010-08-09T11:10:41 <kstreith> what happens when that occurs and we are still in binary compat mode?
2010-08-09T11:11:06 <tclarke> kstreith: I agree but we may have little choice...stay with an EOL 2005 or upgrade to a version that either crashes or runs without optimizations
2010-08-09T11:11:07 <kstreith> http://support.microsoft.com/lifecycle/search/?sort=PN&alpha=Visual+Studio
2010-08-09T11:11:11 <kstreith> mainstream support ends 4/12/201
2010-08-09T11:11:13 <kstreith> mainstream support ends 4/12/2011
2010-08-09T11:11:13 <tclarke> we can still get it in the trunk ASAP but it might not make the 4.5.0 release
2010-08-09T11:11:31 <kstreith> there is 2008, don't forget
2010-08-09T11:11:41 <tclarke> I am still moving forward with the 2010 upgrade branch and with luck it will be ready to merge as soon as the fix and final Qt 4.7.0 are ready
2010-08-09T11:11:49 <tclarke> There is 2008 but it is a lot of work for very little benefit
2010-08-09T11:12:01 <tclarke> I'd rather run on EOL for a couple of months as it could take the better part of a month to get a 2008 upgrade done
2010-08-09T11:12:12 <goffena> what about releaseing both VS2005 and VS2010 builds of Opticks 4.5.0?
2010-08-09T11:14:31 <kstreith> that is a VERY big task
2010-08-09T11:14:37 <kstreith> and one I'd avoid at all costs
2010-08-09T11:14:48 <tclarke> goffena: that would require maintining two separate 4.5.0 trunks...as mentioned, it is a very large undertaking
2010-08-09T11:15:16 <tclarke> ok, any more discussion or ideas? we can continue on the mailing list if people would like a more in depth or prolonged discussion
2010-08-09T11:16:40 <goffena> i would place something on the mailing list, just so others are aware of the situation.
2010-08-09T11:16:47 <tclarke> goffena: I will do that after the meeting
2010-08-09T11:16:49 <tclarke> any other topics for today?
2010-08-09T11:16:52 <goffena> yes
2010-08-09T11:16:53 <dadkins> wait & see – chances are the problem fixes itself
2010-08-09T11:17:00 <goffena> will there be a Opticks 4.4.0 RC4?
2010-08-09T11:17:05 <dadkins> NO
2010-08-09T11:17:08 <tclarke> no current plans
2010-08-09T11:17:15 <tclarke> as indicated by the exclamation that just came out of mconsidi and kstreith's office
2010-08-09T11:17:18 <goffena> i sent dsulgrov an email outlining a problem i discovered on Friday
2010-08-09T11:18:39 <goffena> at one point sounded like tjohnson would like it fixed
2010-08-09T11:18:45 <goffena> not sure what tjohnson's current thoughts are
2010-08-09T11:18:55 <goffena> The problem involved performance degrading on the Import Options Dialog
2010-08-09T11:18:57 <tclarke> what version does it need to be fixed in? I'm aware of the problem and I'm not sure if we have a solution yet...certainly not one that can make the 4.4.0 timeline
2010-08-09T11:19:00 <goffena> when selecting datasets
2010-08-09T11:19:01 <tclarke> it is something that could be in 4.4.1
2010-08-09T11:19:14 <tjohnson> The impact is severe enough that we would need to stay with 4.3.4 until it is fixed.
2010-08-09T11:19:31 <tclarke> dsulgrov: thoughts?
2010-08-09T11:19:34 <goffena> would adding a config setting to ignore wavelengths be a sufficent short term solution, so that it could be placed into a RC4 build?
2010-08-09T11:21:21 <dsulgrov> I'm not sure what the cause of the slowdown is
2010-08-09T11:21:27 <kstreith> trying now...
2010-08-09T11:21:32 <dsulgrov> it appears to be somewhere either in Qt code, or how we are using the Qt widgets
2010-08-09T11:23:11 <dsulgrov> I think that a short term workaround, like what goffena suggests may take some time to design and implement
2010-08-09T11:23:44 <kstreith> just tried cube creator cube with 4000 bands
2010-08-09T11:23:46 <kstreith> and no special metadata
2010-08-09T11:23:49 <kstreith> takes about 14 seconds to switch datasets
2010-08-09T11:23:55 <kstreith> even if you never look at the wavelengths tab
2010-08-09T11:25:49 <dadkins> kstreith: Humor me and click the "Apply" button. How long does it take before the GUI responds after that?
2010-08-09T11:25:50 <tjohnson> how much did your memory usage go up? We are seeing 30-50 MB on a 3000 frame dataset.
2010-08-09T11:25:53 <kstreith> for comparison, same files in 4.3.3, the switch is instant
2010-08-09T11:26:03 <kstreith> dadkins: apply button on the 4000 frame dataset?
2010-08-09T11:26:08 <dadkins> Yes
2010-08-09T11:26:13 <kstreith> import options dialog doesn't have an apply button
2010-08-09T11:27:35 <dadkins> My mistake – import it and try the properties dialog
2010-08-09T11:27:55 <kstreith> looking into mem usage question first
2010-08-09T11:30:09 <kstreith> yes, i'm seeing 50 MB RAM usage increase
2010-08-09T11:30:13 <kstreith> every time i click back to the 4000 frame dataset
2010-08-09T11:30:26 <kstreith> so I click over to the other dataset, back to 4000 frame dataset, another 50 MB of RAM consumed
2010-08-09T11:30:56 <rforehan> Any chance the wavelength info is being repeatedly added to the metadata dynamic object?
2010-08-09T11:31:14 <kstreith> these cubes have no metadata
2010-08-09T11:31:15 <tclarke> I don't believe there is any wavelength info in the situation tested
2010-08-09T11:31:18 <kstreith> and no it isn't appearing on the metadata tab
2010-08-09T11:34:02 <rforehan> Does a 2000 band cube eat ~25MB per change?
2010-08-09T11:34:18 <kstreith> closing the import options dialog has now pegged my CPU to 100% on one core
2010-08-09T11:36:08 <kstreith> RAM is re-claimed once the file is imported
2010-08-09T11:36:28 <tjohnson> we didn't have any wavelength info either.
2010-08-09T11:36:53 <tclarke> perhaps it's delayed memory cleanup that's somehow causing the slowdown?
2010-08-09T11:38:22 <kstreith> around ~25MB for 2000 band image
2010-08-09T11:38:46 <tclarke> let's finish the meeting and then continue this session so people not interested can leave...are there any other discussion topics for today?
2010-08-09T11:39:07 <tclarke> ok then, meeting adjourned
2010-08-09T11:39:08 <dadkins> one quick thing: I am implementing a K-Means classification algorithm for a class project. Interested parties can contact me offline for details
2010-08-09T11:40:26 <kstreith> ~4 MB for 300 band image again no wavelengths
2010-08-09T11:40:34 <tclarke> if you are interested in the import options dialog problem, stick around