Thursday, 2009-05-21

*** tpb has joined #melange00:00
*** madrazr has joined #melange00:41
*** schultmc1 has joined #melange02:15
*** madrazr has left #melange02:23
*** schultmc has quit IRC02:26
*** dmitrig01|afk has quit IRC02:36
*** dmitrig01|afk has joined #melange02:37
*** ChanServ sets mode: +v dmitrig01|afk02:37
*** dr__house has joined #melange02:40
*** dhaun has joined #melange02:43
*** tpb has joined #melange07:23
*** SRabbelier has joined #melange07:26
*** ChanServ sets mode: +o SRabbelier07:26
*** madrazr has joined #melange08:05
*** dhans has joined #melange08:25
*** dhans has quit IRC08:38
*** schultmc1 is now known as schultmc09:02
*** dhans has joined #melange09:08
madrazrSRabbelier: Hi09:58
madrazrsolydzajs: Hi09:58
solydzajsmadrazr: hi there, I will be back after food, in 50 minutes09:59
madrazrsolydzajs: sure take time10:00
madrazrwill be talking to SRabbelier in mean time if he is here10:01
MerioHi madrazr ^_^10:01
madrazrMerio: Heya!10:02
madrazrMerio: how are you? how are the preps going?10:02
Meriomadrazr: hi ^_^ I think all well, I've studied a lot about the data model.. and about Python :P Still need more study of GAE, but two days are enough :)10:03
Meriomadrazr: What about you?10:04
madrazrprevious 2 I had some coding experience, have some experience with GAE too, but never wrote code to project that is as big as Melange10:05
madrazrso have may have to refer the docs very often10:05
madrazrMerio: ^10:06
Meriomadrazr: we all have to do it ... and also I think this is an opportunity for us to help the wiki docs. We're "beginners" (each of us in some field) so we can understand better which parts of the wiki can be enhanced to help.. well... beginners ^__^10:07
madrazrMerio: definitely10:08
dr__houseMerio: SRabbelier: There has been a power cut at madrazr's place and has lost the connection10:17
dr__houseMerio: SRabbelier: sorry for the inconvenience10:18
Meriodr__house: oh sorry about that :(10:19
dr__houseMerio: heh, there's nothing to be sorry about, its just a power cut :P10:19
Meriodr__house: Yes, but it's annoying :) Everything that makes me loose or having an hiccup Internet connection always makes me feel angry :)10:22
Meriodr__house: too much addicted :P10:22
dr__houseMerio: same here and same for madrazr10:22
*** madrazr has quit IRC10:27
*** forever has quit IRC10:27
*** dr__house has quit IRC10:49
*** dr__house has joined #melange10:50
*** madrazr has joined #melange11:16
*** solydzajs has quit IRC11:18
madrazrMerio: sorry for what happened11:21
madrazrback now11:21
*** dr__house has quit IRC11:28
*** dr__house has joined #melange11:28
SRabbeliermadrazr: wb11:50
madrazrSRabbelier: thanks :D11:56
Meriomadrazr: np wb :)11:59
madrazrMerio: thanks!12:01
*** MatthewWilkes has quit IRC14:02
dhansMerio: ping? :)14:13
Meriodhans: pong!14:23
Meriodhans: how do you do? :)14:24
dhansMerio: great, thank you:)14:24
dhansMerio: at last we may talk about our projects...:)14:24
Meriodhans: yeah, sounds great ^__^14:24
dhansMerio: if you have some time of course...14:24
Meriodhans: just wait 10 minutes and I'll be right back :)14:26
dhansMerio: sure:)14:26
Meriodhans: can't explain in public my current needs :P14:26
Meriodhans: brb :)14:26
dhansMerio: :D14:26
*** madrazr1 has joined #melange14:28
*** madrazr has quit IRC14:28
*** madrazr1 is now known as madrazr14:29
*** madrazr1 has joined #melange14:33
*** madrazr has quit IRC14:34
*** madrazr1 is now known as madrazr14:34
*** tuxmaniac has joined #melange14:37
*** tuxmaniac has left #melange14:38
*** James--Crook has joined #melange14:43
James--CrookHi dhans.  How are things going?14:44
Meriodhans: back :)14:44
MerioJames--Crook: hi!!! Good evening :)14:44
dhansMerio: ok:)14:45
James--CrookHi merio.  At last you and dhans meet on irc.14:45
MerioJames--Crook: yes we did it, we're going to talk about the project :)14:45
James--CrookSo, is it helpful for me to be involved, or just let you two talk and come back later?14:46
* James--Crook waves to dhans14:46
dhansJames--Crook: hello:)14:46
MerioJames--Crook: to me it would be very fine if you want to be involved :)14:46
dhansJames--Crook: the same for me14:46
James--CrookOK, so dhans, first an update.... we're going to get a simpler stat from GAE as our first goal.14:47
James--CrookThe one on the wiki involves more than one model.14:47
James--CrookWE're going to go instead for the percentage of students undergrad/msc/phd14:48
James--Crookso just one model to look at.14:48
James--Crookdhans: how far are you into undertsnading what is needed to extarct a statistic like that and get it into memcahce?14:49
James--Crookdhans???  ^^14:50
dhansJames--Crook: I am far with extracting this kind of data from model and putting it in our new model, but it shouldn't be so difficult to put it into memcache, i think...14:50
dhansJames--Crook: so we are not going to put our json data into 'standard' model? just into memcache?14:51
Meriodhans: yes, it's a sort of one-liner14:51
Meriodhans: I think we'll be doing both14:51
James--CrookWe have a choice.14:52
James--Crookdhans standard model - please elaborate.14:52
dhansJames--Crook: I mean database14:53
James--CrookWe will have a model for collected stats.14:54
James--CrookIt will contain JSONified stats.14:54
dhansBecause when I saw the json data format on the wiki page, I thought we could add just one new model and put there a lot of various stats14:54
dhansJames--Crook: exactly:)14:54
Meriodhans: I think we're going to check if the JSON it's in the memcache, if it's not we'll fallback to a "JSONstat" instance14:54
dhansMerio: ok14:55
Meriodhans: actually, it'd be more elaborate, but that's the basic thing14:55
James--CrookSo, what are the fields of this model?14:55
James--CrookDo either of you chip in....14:56
dhansJames--Crook: I thought about something like 'stat name', 'stat data' (json string) and maybe date14:56
James--Crookstat-name will work fine - we can use it to select what stat we are collecting.14:56
James--CrookI think there are two json strings, one for the full JSON object, and one for the stats as built so far.14:57
*** madrazr1 has joined #melange14:57
James--CrookThere may or may not be an additional field that tells us how far through collecting we are.14:58
James--CrookWe could have that as a separate field, or we could put it into the JSON string.14:58
James--CrookI'm inclined to have it as a separate field so that we don't have to parse the JSON string within the python code.14:59
dhansJames--Crook: 'and one for the stats as built so far' - I am not sure if I get it...14:59
MerioJames--Crook: I think it would be good if we put it in both14:59
James--CrookWell, we're building the JSON up say 50 records at a time.14:59
James--CrookSo let us say there are 1000 records.14:59
MerioJames--Crook: that way we have it available for Python, and we don't have to send a separate "context variable" to JavaScript if we're going to have a check panel14:59
*** madrazr has quit IRC15:00
*** madrazr1 is now known as madrazr15:00
James--CrookMerio: OK sounds good.15:00
James--CrookThen one JSON field will have the full 1000 records, as collected last time.15:00
James--CrookThe othe rmight only have 350 - those collected so far.15:00
James--CrookAfter collecting another batch it goes up to 400.15:01
James--CrookEventually it reaches 1000 and is copied into the 'master' copy too.15:01
James--Crookdhans: does that make sense to you?15:01
dhansJames--Crook: ok, i get it know. I thought it would be possible to prepare the whole json object first and then put it into a model field15:02
James--CrookWould be nice if we could...  but I think GAE may not give us enough time.15:02
James--CrookAt least for some stats at any rate.15:02
James--CrookSo this gives us a way to store it between getting turns at collecting....15:03
dhansJames--Crook: I thought that cron jobs don't have those limitations, but sure15:03
James--CrookNope, I think they do.  I could easily be wrong.  If I am please point me at documentation that shows it...15:04
James--CrookThe thing we really really need to get started is some JSON object in memcache, and some page that serves it.15:05
dhansJames--Crook: I am not sure either:) but I will look into it15:05
James--CrookIt is then possible for the two of you to develop new features with less dependency on each other.15:05
James--CrookEither can add new stats to collect or new ways to display stats.15:05
James--CrookSo, it is how to get to that stage.  And I suppose more importantly, what if any blocks are there to getting there that we can deal with....15:06
MerioJames--Crook: what do you mean?15:08
*** madrazr1 has joined #melange15:08
James--CrookLet me put it as a question.  Could you code that new model (to hold the JSON) and get it putting that data into a memcache right now?15:10
dhansJames--Crook: i think it's not so difficult;)15:10
James--CrookGood.  Merio?15:11
James--Crookdhans: would you be able to start on it tonight?15:11
MerioJames--Crook: mee too15:11
James--CrookMerio about how long would it take you, do you reckon, assuming no unexpected difficulties.15:12
dhansJames--Crook: I thought about starting tomorrow in the morning, but I think I can do some stuff tonight15:12
James--Crookdhans: OK, if you started tomorrow morning, about how long would you expect it to take?15:12
James--CrookMerio: are there any missing pieces of information that you would need?15:13
dhansJames--Crook: assuming no unexpected difficulties I think it's one day job15:13
dhansJames--Crook: another question about the new model...15:13
James--CrookDhans are there any missing pieces of information that you would need?15:13
James--Crookdhans: go on...15:14
dhansJames--Crook: as I understand, we have one field with the stats that were completely collected some time ago15:14
MerioJames--Crook: never coded a line of python, so I don't really know how much time it would get... perhaps one day or two, because... well, getting to work on a new language is an expected difficulty ^__^15:14
dhansJames--Crook: and the second one is some kind of temporary field for the data that is collected right now, ok?15:15
MerioJames--Crook: but the logic behind is very simple, language it's my variable ^_^15:15
James--Crook(what I am currently thinking is that Merio writes up the model on the wiki, and dhans puts the code in, then merio adds a second statistic and the whole thing gets a code review)15:15
*** dbentley-laptop has joined #melange15:15
James--Crookall in about 3 days time...15:15
James--Crookdhans: yes.15:16
dhansJames--Crook: ok, but I also thought about projecting the model so that it stores history of statistics (that's why I wanted to add creation date)15:17
James--Crookdhans: is JSON the right choice for the temporary?15:17
dhansJames--Crook: not really:)15:18
Meriodhans: yes, timestamp is needed15:18
James--Crookdhans: most of our stats are constant once created and don't have history.  I think we should do a 'spike solution' that does not support time varying stats, just a snapshot of current state.  Then later we elaborate.15:19
dhansJames--Crook: ok15:19
James--Crookdhans: is there a better choice for the temporary?15:20
MerioJames--Crook: we can put it in the data model, even if not used at the moment15:20
dhansJames--Crook: pure string?:)15:20
James--CrookMerio: yes.  But that is time of collection isn't it?15:21
dhansJames--Crook: or better some kind of object15:21
MerioJames--Crook: yes, then we'll have the timestamp in the JSON15:21
James--Crookdhans: but then different object types for different stats?  Can we put different types of objects in the same field?15:21
dhansJames--Crook: can we put a dictionary in a model field?15:22
James--CrookVery probably yes.  Of course we shouldn't then try and index it :-)15:23
James--Crook(GAE index that is)15:23
James--CrookIf we can put a (generic) python object in in some way, then fine.15:24
Meriodhans: is the blob type suitable?15:24
James--CrookIf not, we encode the information in a string somehow.15:24
Meriodhans: I don't really know15:24
James--Crookdhans: can you look into it and make a choice and write it up on the wiki?15:24
dhansJames--Crook: ok15:24
*** madrazr has quit IRC15:25
Meriodhans: perhaps after a serialization or something15:25
dhansMerio: yes15:26
James--CrookRight.  I think we are clear than as to how things progress - dhans is doing the first cut, meanwhile mario is updating the wiki with the model, then mario is adding one more stat, then we check in with each other and see how things are going.15:26
James--Crooksound right to everyone?15:26
MerioJames--Crook: to me yes15:26
dhansJames--Crook: Just to clarify for me...:)15:27
dhansJames--Crook: do you want me to write some kind of logic that gather data from model, count stats (for example student/country), and finally puts them into a json object?15:28
James--CrookYes.  And it will be a stat that can be gathered form just one model.15:28
Meriodhans: out spike solution it's simply to gather statistics on students deegree. That way we'll be looking only in one entity model15:29
James--CrookSo in a bit more detail...  There will be a page15:29
Meriodhans: So it's simpler15:29
dhansMerio: ok15:29
James--Crookwith a button on it...15:29
James--Crookactually two buttons (or more)15:29
James--Crookclick to clear stats (for test purposes only.  Will not normally be needed otherwise)15:30
James--Crookclick to collect one batch of stats, using the name we chose for the stats.  will collect say 50 records and combine the information into the working data field.15:31
dhansJames--Crook: and I think there should be also some info how much records have been collected15:32
dhansJames--Crook: just to tell the admin when he or she should stop clicking:)15:32
James--CrookYep.  Logic updates it, and ideally we see it on the page with the buttons.15:33
James--CrookWe also need a page to show the contents of the final JSON field.15:33
James--CrookAnd I think (for our own development and diagnostics) we should have a page for viewing the page of work in progress.15:33
MerioIf we go communicating all the way with JSONs we can have a single page and use it Ajaxy way15:33
James--CrookSo that will be JSONiified then , even if stored in some other format.15:33
James--CrookMerio - yes.  Dual function.  Works in a job way and ain an ajaxy way.  win-win.15:34
James--Crookdhans: so three pages currently...15:34
James--CrookOne other reason for the work in progress page is...15:35
dhansJames--Crook: ok, but i am not string with ajax ;/15:35
Meriodhans: what do you mean?15:35
James--Crookthat it means testing of any stats display can be done with partial stats too...  so quicker to do quick tests....15:36
James--Crookdhans: Mario will be showing you ajax tricks, and you will be teaching him about python :-)15:36
James--CrookIt's a very good combination.15:37
dhansMerio: I mean that a single page that uses JSONs in Ajaxy way wouldn't be simple for me to write15:37
James--Crookdhans: in the end it will be, but not at the moment.  Hence the division of work in the first stages.15:37
dhansJames--Crook: yes, indeed:)15:37
Meriodhans: no problems, James--Crook is right... will be exciting working together for that reason :)15:37
dhansMerio: me too:)15:38
James--CrookOK.  I think we are clear about the next steps, and I'm expecting something in a memcache in about 3 days :-)15:38
James--CrookFirst any questions, then if not I want to say a little more about related things later...15:38
MerioJames--Crook: I'm finishing something for the seed_db tomorrow, so I can't work on it till Saturday TBH15:39
James--CrookThat's fine.  So do the wiki update on saturday.15:39
dhansJames--Crook: i'll try to show something like tomorrow or on Saturday to check if I'm not doing everything completely wrong:)15:39
James--Crookcool.  It's unlikely you'll be able to do everything completley wrong :-)15:39
dhansJames--Crook: we'll see...:)15:40
*** florinciu has joined #melange15:40
James--CrookOK.  so...  what I wanted to say a bit more about is future of stats collecting.  Dhans, you mentioned timestamps and it raises an interesting point...  (one reason not to do time varying stats yet).15:41
James--CrookThe problem is that there are different ways of 'putting stats into buckets'.15:41
dhansJames--Crook: and I'll ask sverre about those time limitations, because I am wondering why we have to use jobs if we want to process only 50 records one time15:41
James--Crook+1.  (on asking sverre)15:42
James--CrookInitially the stats collection page is going to have string names for the different kinds of stats.15:42
James--CrookNow the developers do not want to push a new version of Melange every time someone dreams up a new stat that they want to collect.15:43
James--Crooktime varying stats is a particualr problem area here.15:43
James--CrookHow often do we collect?15:43
James--Crook(for example).15:43
James--CrookSuppose we want to see arrival times of GSoC project proposals....15:44
James--CrookSo we will draw a graph of the stats for projects recieved against time.15:44
James--CrookIt turns out that the best data to work from, in that case, is not to colelct stats once a day (say) but instead to have a table15:45
James--Crookwith all the timestamps at which project proposals arrived.15:45
James--Crookdhans: Merio: are you following?15:45
MerioJames--Crook: yes15:45
dhansJames--Crook: yes15:45
James--CrookOK.  So as I see it, stats collection is going to eveolve from collecting 'condensed stats' with small JSON obejcts...15:46
James--Crookto collecting stats with much larger JSON objects...15:46
MerioJames--Crook: this would mean having a sort of 1:1 correspondence between the JSON object and the actual data, at least for the two fields involved in the stat :)15:46
James--CrookSo I just wanted to make you both aware of that, especially in the context of how to do time varying stats.15:47
James--CrookMerio: It is more complex than that.  But yes, under the hood a lot of what stats collection is doing is solving the problem that we cannot see all the records at once normally....15:48
James--Crookso stats can be thought of as an aspect of filtering and conversion...  and it will have tie ins to tabular views too, all in due course....15:49
James--Crook(at least this is how I see it)15:49
MerioJames--Crook: Agree on that15:50
James--Crookdhans: Have I said enough on this to give a sense of how time varying stats is likely to work?15:51
dhansJames--Crook: I see the problem and I think I get it how you imagine it15:52
James--CrookThat's good enough for this stage.  Lots of this will get talked through and sorted (and I will understand it better too later) on wiki/e-mail/IRC.15:52
James--CrookOK.  I've talked enough.  Anything else?15:52
MerioJames--Crook: all ok my side15:53
dhansJames--Crook: me the same15:53
James--CrookOK.  Thanks both.15:53
James--Crookbye for now.15:53
* James--Crook wanders off to make some tea.15:53
dhansJames--Crook: bye15:53
MerioJames--Crook: have a nice... tea ^_^15:54
dhansMerio: i think it's clearer now:) tomorrow (once I get response from sverre) i'll start working on this data gathering15:54
Meriodhans: ok, will you be idling on irc too?15:55
dhansMerio: I need to leave for about half an hour (my friend's been waiting for like half an hour:), but i'll be then15:56
Meriodhans: I was meaning tomorrow :)15:56
dhansMerio: so yes:P15:57
Meriodhans: but then... will see tomorrow, enjoy ^_^15:57
dhansMerio: see you:)15:57
*** dhans has left #melange15:58
*** dr__house has quit IRC16:33
*** dr__house has joined #melange16:35
*** madrazr has joined #melange16:35
*** madrazr1 has quit IRC16:37
SRabbelierJames--Crook: I thought you drank hot water, rather than tea?16:39
James--CrookSRabbelier: Correct.  Am drinking it now (it has cooled down enough).16:40
SRabbelierJames--Crook: then why call it tea? :P16:41
James--CrookSRabbelier: can you look through the backlog?  dhans had a couple of questions for you I think...16:41
SRabbelierJames--Crook: yes, I will in a moment16:41
James--Crook( SRabbelier it is very weak tea )16:41
SRabbelierJames--Crook: tea without tea in it does not count as tea :P16:41
James--CrookSRabbelier: And I did not want to explain....16:41
SRabbelierJames--Crook: hehe, too late!16:42
James--CrookIt is homeopathic tea... less than one molecule of tea in it...  so according to the homeopathic theory it is actually very very strong.16:42
James--Crook(statistically speaking)16:42
*** James--Crook has left #melange16:45
ErantSRabbelier: Ohai. So have all the recent fixes to HTML shit been pushed to testing yet?16:48
SRabbelierErant: iunno16:48
ErantI'm fully expecting to fail miserably by now, but I'd like to prod some more crap16:50
SRabbelierErant: lol, I see16:50
ErantIt's very therapeutic you see.16:50
ErantAs will this cup of tea. Which actually contains tea.16:51
ErantAscension day is a holiday in America too, right?16:52
SRabbelierErant: I would assume so16:54
*** Merio has quit IRC17:05
*** dr__house has quit IRC17:21
*** dhaun has quit IRC17:23
*** madrazr has left #melange17:45
*** penyaskito has joined #melange18:01
*** Kraln has quit IRC18:48
*** Kraln has joined #melange18:49
*** florinciu has quit IRC19:19
*** SRabbelier has quit IRC19:43
*** dbentley-laptop has quit IRC19:46
*** jameslevy has joined #melange20:19
jameslevyajaksu: i think you forgot to add the midterm logic file before commiting?20:19
ajaksujameslevy: probable, I was too focused on caching when I did the commit... I've figured half of the caching issue (now schema and field values are updated or deleted correctly), going for the other half if you haven't fixed the 'shows field ghosts' problem yet :)20:21
jameslevyI couldn't figure out the root of the problem, but I effectively fixed it20:22
jameslevyIt's not an elegant fix, but it will work for now20:22
ajaksujameslevy: have you commited that somewhere?20:23
*** jameslevy has quit IRC20:23
ajaksu(btw, I've mailed you a fix for the StopIteration bug I created)20:23
ajaksuer... he's gone :/20:23
*** jamtoday has joined #melange20:28
ajaksujamtoday: I've mailed you a fix for the StopIteration bug I created20:29
jamtodayajaksu: I'm rebasing a new head20:29
ajaksujamtoday: does that mean you have changes I don't?20:29
jamtodayNot in the same files20:30
jamtodayI'm using your versions of the latest files you've changed20:30
jamtodaybut I don't have the midterm logic20:30
jamtodayAnd it's throwing up an exception :)20:30
ajaksuI'll add and commit it right now :)20:31
ajaksubut I still want to know if you've fixed the caching further, because I think I've found the main issue20:31
jamtodayI fixed it for now with a manual check in JS20:32
jamtodayand it empties the form if its a new survey20:32
jamtodayso no20:32
ajaksujamtoday: when was that? ?20:36
tpb<> (at
ajaksujamtoday: the logic/models/ is present and has a log in my local git, but is missing from github... maybe I should've passed something like a '-a' parameter  more when I pushed?20:38
jamtodayYou may need to commit again20:39
jamtodaygit status will show you20:39
jamtodaygit commits sometimes miss things I've found20:39
jamtodayso I always do a git status to make sure20:40
ajaksuyep, it was there as an untracked file, even if git log on it showed history :)20:43
ajaksumissing other files too20:43
jamtodayok push them and I'll try pulling them down20:45
ajaksujamtoday: check , now I'm trying to find out where the zombie form fields come from... if your changeset above was supposed to fix this, I can't make it work here :/20:55
tpb<> (at
ajaksuI do have a suspicion we're seeing a sort of globals-related problem, and I want to figure out if "SurveyForm.base_fields.insert(position, property, fields[property])"  causes it :)20:57
*** dbentley-laptop has joined #melange21:04
jamtodayThat actually was supposed to be a temporary thing, I don't think I should have been adding to the base_fields attribute21:04
jamtodayI'll look at that21:04
jamtodayI just pushed a rebased merge of everyone's changes21:04
jamtodayI agree that there is too much duplication21:06
jamtodaythe subclass should properly only include things that are very specific to midterms21:06
ajaksuyes, and that might need a few changes in the base class, but I was chasing caching bugs thinking they were midterm issues, so there's a lot of unnecessary stuff there :)21:08
ajaksuright now, editing or adding a field in a survey doesn't trash the rest of it anymore, but the deleted field still shows (but missing its custom prompt, other fields keep theirs)21:09
ajaksujamtoday: git pull says I'm up-to-date... can you confirm no other commits after ?21:12
tpb<> (at
ajaksujamtoday: if so, deleting a survey field still erases its schema and its custom prompts and still shows the ghost field here after that commit :(21:16
*** jamtoday has quit IRC22:03
*** jamtoday has joined #melange22:50
*** dmitrig01|afk has joined #melange23:04
*** ChanServ sets mode: +v dmitrig01|afk23:04
*** forever has joined #melange23:14
jamtodayajaksu: What a stupid mistake on my part! Why was I adding to the base_fields instead of self.fields?23:26
jamtodayAnd I wasted 15 minutes coding up javascript to sweep it under the rug....23:27
*** dmitrig01|afk has quit IRC23:33
*** dmitrig01_ has joined #melange23:33
*** ChanServ sets mode: +v dmitrig01_23:33

Generated by 2.13.1 by Marius Gedminas - find it at!