| *** tpb has joined #symbiflow | 00:00 | |
| duck2 | well, by 1) using pypy magic, 2) not using prjxray.db_cache, 3) not using threads in prjxray_create_edges.py and also rearranging some code in build_channels, i could bring the build into the vpr stage in a LXC container with 2gb RAM and 1gb swap. it takes roughly 40 mins or maybe a hour | 00:11 |
|---|---|---|
| *** OmniMancer has joined #symbiflow | 00:12 | |
| duck2 | honestly i don't think i can get away with vpr with this much ram but let's see | 00:17 |
| duck2 | litghost: indeed, vtr uses pugixml which needs 2x the file size in memory to build a dom tree https://github.com/zeux/pugixml/issues/166 | 01:22 |
| tpb | Title: Incrementally reading very large files · Issue #166 · zeux/pugixml · GitHub (at github.com) | 01:22 |
| duck2 | in fact, rr_graph.xml would be a very neat json file but i guess that would be hacking too much | 02:06 |
| litghost | duck2: I think a zero parse format is worth investing in the future (e.g. captain proto or flatbuffers). I have a VPR branch with flatbuffer support, but it turns out the flatbuffer support in python is actually really terrible :( | 03:01 |
| *** _whitelogger has quit IRC | 03:46 | |
| *** _whitelogger has joined #symbiflow | 03:49 | |
| *** Bertl_oO is now known as Bertl_zZ | 05:27 | |
| *** i8hantanu has joined #symbiflow | 05:32 | |
| *** _whitelogger has quit IRC | 06:46 | |
| *** _whitelogger has joined #symbiflow | 06:49 | |
| *** i8hantanu has quit IRC | 07:41 | |
| *** blurSong has joined #symbiflow | 08:03 | |
| *** _whitelogger has quit IRC | 09:34 | |
| *** _whitelogger has joined #symbiflow | 09:37 | |
| *** _whitelogger has quit IRC | 09:58 | |
| *** _whitelogger has joined #symbiflow | 10:01 | |
| *** riha has joined #symbiflow | 10:33 | |
| *** riha has quit IRC | 10:36 | |
| *** i8hantanu has joined #symbiflow | 11:29 | |
| *** proteusguy has quit IRC | 12:45 | |
| duck2 | what is the problem with python's flatbuffers support? i looked at the tutorial and it does not seem worse than, say, javascript | 13:11 |
| *** Bertl_zZ is now known as Bertl | 13:52 | |
| *** i8hantanu has quit IRC | 14:49 | |
| litghost | duck2: It is implemented in python, using ctypes. It is very very slow. It actually turns it to be faster to write out the rr graph in XML, and have VPR load the XML and write it back to flatbuffers. | 15:05 |
| litghost | duck2: It also doesn't support incremental writing, which may become important for larger graphs. | 15:06 |
| *** OmniMancer has quit IRC | 15:22 | |
| *** _whitelogger has quit IRC | 15:28 | |
| *** _whitelogger has joined #symbiflow | 15:31 | |
| duck2 | i think converting to libxml2 SAX would be good in terms of memory usage and ease of impl, since the current code is suitable for sax: processes one element at a time | 15:43 |
| duck2 | litghost: it's very sad. maybe pypy can help with that though, it runs pure python code quite fast. | 15:52 |
| *** citypw has quit IRC | 16:26 | |
| *** kraiskil has joined #symbiflow | 16:34 | |
| litghost | Duck2: yes, sax parsing was what I was implying above | 17:16 |
| *** kraiskil has quit IRC | 23:04 | |
Generated by irclog2html.py 2.13.1 by Marius Gedminas - find it at mg.pov.lt!