Sunday, 2009-10-18

*** tpb has joined #melange00:00
*** SRabbelier1 has joined #melange00:31
*** SRabbelier has quit IRC00:31
*** SRabbelier1 is now known as SRabbelier01:01
*** ChanServ sets mode: +o SRabbelier01:02
*** madrazr has joined #melange01:57
*** ChanServ sets mode: +v madrazr01:57
*** madrazr has quit IRC03:58
*** madrazr has joined #melange04:43
*** ChanServ sets mode: +v madrazr04:43
*** madrazr has quit IRC05:55
*** madrazr1 has joined #melange05:55
*** madrazr1 is now known as madrazr05:55
*** ChanServ sets mode: +v madrazr05:56
*** scorche has quit IRC07:02
*** scorche has joined #melange07:24
*** madrazr1 has joined #melange08:51
*** madrazr has quit IRC08:51
*** madrazr1 has quit IRC09:01
*** madrazr has joined #melange09:01
*** ChanServ sets mode: +v madrazr09:01
*** Lennie has joined #melange09:19
*** ChanServ sets mode: +o Lennie09:19
*** Lennie is now known as Lennie|Gone09:24
dandersonSRabbelier: ping?10:16
Lennie|Gonedanderson, not sure if he's awake yet since he's in the states :)11:18
SRabbelierdanderson: pong12:05
SRabbelierdanderson: export yes?12:05
dandersonyes, if possible :)12:05
SRabbelierdanderson: will do so now12:05
dandersonI'm working on getting the 2005-2008 data out, but the 09 one is the most important :)12:05
danderson(reminder: successful students + all accepted mentors plz)12:06
dandersonand I owe you a beer :P12:06
SRabbelierdanderson: oh?12:06
dandersonoh, and just their google account (email) is all I need12:06
dandersonwell, for helping me out on this :)12:06
SRabbelierdanderson: ah, heh, I'd figure it'd be the other way around, :P12:06
dandersongotta run now, laterz12:07
*** Lennie|Gone is now known as Lennie13:45
SRabbelierdanderson: mailed13:52
dandersonSRabbelier: just got back, you win, beer is owed.13:54
Lenniedanderson, I think SRabbelier would be happier with food or a cherry coke :P13:58
dandersonwhatever, that works too13:58
SRabbelierdanderson: or some buttershots with baileys :P14:04
SRabbelierLennie: how far did you test the conversion?14:04
LennieSRabbelier: baileys? really :)14:10
Lenniewhat made you change your midn :P14:10
SRabbelierLennie: buttershots + baileys + grand marnier = a shot I drank with Jack, it's tasteh :P14:11
Lennieok )O14:11
Lenniejust don't feed SRabbelier beer14:11
Lenniethat's all then :)14:11
Lennieoh and keep all liqour away from Erant :p14:12
ErantDon't you dare.14:13
SRabbelierLennie: good idea, or he'll puke all over us again14:13
ErantSo, who epic failed on the security.14:14
Lenniedo i care :)?14:15
Lennieoh damn I do :P14:15
ErantCalls himself a project manager.14:16
ErantSpeaking of managers, LH still in Vegas?14:16
Lennieno clue :P14:16
*** Lennie has quit IRC14:17
ErantRight. Bye?14:17
*** madrazr has quit IRC17:09
dandersonSRabbelier: ping?18:05
SRabbelierdanderson: pong18:06
dandersonSRabbelier: what does the mentors list you sent include?18:06
SRabbelierdanderson: all active mentors for this year18:06
dandersongetting lots of 2009 mentors saying I misclassified them as invalid18:07
dandersonoh gah18:07
dandersonthe list has [email protected]18:07
dandersonbut the account login is [email protected]18:07
SRabbelierdanderson: tell them to use the same account that they use on Melange :P18:08
dandersonI think it is the same18:08
dandersonbut that is the primary email registered for the account or something18:08
dandersonanyway, we'll see how bad it gets18:08
* SRabbelier nods18:08
dandersonI was just wondering if there was something obvious I misunderstood about your data dump :)18:09
*** MatthewWilkes has joined #melange18:09
* SRabbelier shakes18:09
SRabbelierdanderson: no i think you got it right18:09
MatthewWilkesdanderson: Cheers :) (didn't want to email and clog up your inbox even more)18:22
dandersonSRabbelier: does the admin data dump also list co-admins? (or backup admins or whatever it is they're called)18:36
dandersonphew, finished beating back the first wave (pun not intended but kept after noticing :P) of "ohnoes i haz no wave invite" email19:05
dandersonworse than a zergling rush19:05
* danderson builds more bunkers19:06
MatthewWilkesWe'll get in either way :)19:06
dandersonI'm already at 2% of my daily app push quota19:07
dandersonjust pushing fixes and updates :P19:07
MatthewWilkesdanderson: Well, we're all very thankful! :)19:10
dandersonokay and wtf is happening to my internet connection19:13
dandersonit's laggy as shit all of a sudden19:13
dandersonI think I'm going to have to blacklist my flatmate's MAC address again...19:13
*** mithro has quit IRC19:23
SRabbelierdanderson: push to where quota?19:34
dandersonto app engine19:34
dandersonyou have a quota of 1000 app pushes/day19:35
dandersonI'm currently at 3% of that quota :P19:35
danderson(it's just simple DoS protection, nothing fancy)19:35
MatthewWilkesAwesome, "We have verified that you participated in SoC"19:35
* danderson says "Access granted" in a very neutral robotic tone19:37
SRabbelierdanderson: what are you pushing?19:37
dandersonupdates to
tpbTitle: GSoC wave invite checker (at
dandersonthe valid/invalid lists are just two .txt files bundled in the app19:38
dandersonand the app python module just reads them into a set on startup :P19:38
dandersonepic cheap database!19:38
dandersonbut it means I have to repush when I tweak the lists, or fix bugs19:38
MatthewWilkesdanderson: You should put ads on the sign-in page and leak the address19:38
dandersonpost it to reddit19:39
MatthewWilkesnow, come on19:39
SRabbelierdanderson: lol, cute, you have no life :P19:39
dandersonand exceed my daily app engine quota in 5 minutes19:39
MatthewWilkesreddit is only used for bitching about python infighting19:39
MatthewWilkes(anyone on python-dev or distutils-sig will know what I mean)19:39
SRabbelierdanderson: why not use the database though? :P19:40
dandersonbecause it's 1000 entries, ridiculously small19:41
dandersonbecause I need a .txt with one email per line to send to the wave team19:41
dandersonand because I didn't want to waste time fucking around with the datastore API19:42
dandersonmy "database access layer" is the following line of python:19:42
SRabbelierdanderson: create one entity per list?19:42
dandersonVALID = set(l.strip() for l in open('data/valid.txt'))19:42
SRabbelierdanderson: lol :)19:43
dandersonmy "data access method" to verify an account is:19:43
dandersonif in VALID:19:43
dandersontry that with the datastore19:43
dandersonplus, with this, I can respond to queries in-memory19:43
MatthewWilkesclass DataStoreWrapper(object): def __contains__ …19:43
dandersonno external round-trips, just a lookup in a local hashtable, and boom19:43
dandersonplus, RAM is free19:43
dandersondatastore lookups are slow, prone to long-tail failures, and are charged against quota19:44
SRabbelierdanderson: replace open() with db.get('valid').data19:44
SRabbelierdanderson: cache the result19:44
dandersonyes, but you're missing the critical point: why?19:44
dandersonI have a .txt that fits in memory19:44
SRabbelierdanderson: because you can?19:44
MatthewWilkesSRabbelier: I'm with danderson on this, flat text files are my 2nd favourite database format19:44
dandersonI don't need to scale, I'll be deleting the app in 2 days19:44
dandersonthe datastore is awesome at scaling19:44
SRabbelierdanderson: I guess this does deserve some 'awesomeness' credits :P19:45
dandersonbut to present an interface to a 1k element hashtable? Seriously?19:45
dandersonmemory wins.19:45
dandersonI could also cite our internal document reminding engineers of orders of magnitude in computing19:45
SRabbelierdanderson: oooh?19:45
SRabbelier0, 1, many?19:45
dandersonwell, except I'm not allowed to, obviously19:46
dandersonbut I can give you one extract19:46
* MatthewWilkes wishes there was a site to sign up for the chance to hug Lars Rasmussen…19:46
dandersonmain memory reference: ~100ns19:46
*** mithro has joined #melange19:46
*** ChanServ sets mode: +v mithro19:46
dandersonround trip within local network: ~500 000ns19:46
dandersonjust saying :P19:46
SRabbelierdanderson: point, set, match :P19:47
dandersonthe most awesome part of that is that disk space is about 3x slower than reading the data from RAM on another machine :)19:48
SRabbelierdanderson: oh, how slow is disk access then?19:48
dandersonso, if you have the choice between a cache on disk, or an in-memory cache on a LAN machine, take the LAN machine, every time19:48
dandersonit's actually around 100x slower because it has high setup costs19:49
dandersonit's pretty good at streaming lots of data once it's wasted a long time setting up19:49
dandersonthe problem is you don't usually want to read more than <1M at a time19:49
dandersonin which case the setup cost of moving the heads, waiting for the data to arrive, etc. are actually really significative19:49
SRabbelierdanderson: in your case most certainly so :P19:49
dandersonwell, in my case yes, but also in most cases19:50
dandersonthere are two forms of data access in general19:50
dandersonsequentially stream lots of data (eg. transfer 10G of video from disk to disk)19:50
dandersonor randomly stream small pieces of data19:50
dandersondisks are not too bad at the former19:50
dandersonbut suck rocks at the latter19:50
dandersonbecause the setup cost takes longer than actually reading the data19:51
SRabbelierdanderson: even with SSD?19:51
dandersonthings to keep in mind while designing a distributed system ;)19:51
dandersonwith SSD, not sure what the setup cost is exactly, but it does change the game quite a lot19:51
dandersonit'll still be slower than ram access, but dunno about remote ram access19:52
SRabbelierdanderson: interesting19:52
dandersonbut then you can just change the problem into randomly writing small pieces of data19:52
dandersonand end up in a world of shit again :)19:52
dandersonSSDs are catching up, but you can't beat ram's bandwidth atm19:52
SRabbelierdanderson: uhm19:52
SRabbelierdanderson: I thought the newer SSD's don't have that problem19:52
tpb<> (at
dandersonthey don't have that problem when you consider them to be disks, yes19:53
dandersonso for disks, they have decent read/write performance19:53
dandersonbut compared to ram, I'm pretty sure the write performance still sucks :)19:53
SRabbelierdanderson: ah, but compared to RAM or even network still sucky19:53
* SRabbelier nods19:53
dandersonwell, maybe not network, that's the thing19:53
dandersonthey may be good enough to become an alternative to the network19:54
dandersonbut then you have different problems19:54
dandersonwhat sizes are SSDs these days?19:54
dandersonI mean, what's the cost per MB?19:54
dandersonie. how much is it going to cost you to scale up?19:55
SRabbelier128GB costs around $36019:56
dandersonok, not too bad, still expensive compared to ram though19:58
dandersonit'd be interesting to update our numbers to include read from local SSD and read from remote SSD19:58
SRabbelierand I reckon those prices get lower if you buy in bulk :P19:58
dandersonbecause I don't know how much of the disk cost is also caused by the slowness of the data bus19:58
dandersoncompared to the FSB for ram19:58
* SRabbelier nods19:58
dandersonok, seriously, wtf20:21
dandersonit looks like my ISP is dropping all UDP traffic or something20:21
dandersonzero DNS resolution20:21
SRabbelierdanderson: nice, try OpenDNS instead?20:55
*** tansell_laptop has joined #melange22:12
durin42danderson: this is the internet telling you to go to sleep ;)23:40

Generated by 2.13.1 by Marius Gedminas - find it at!