| *** tpb has joined #symbiflow | 00:00 | |
| *** citypw has joined #symbiflow | 02:08 | |
| mithro | syed.ahemd: At the moment is it pretty quiet with people on holidays, I expect that acomodi and litghost will be back in Jan and able to help then | 02:22 |
|---|---|---|
| *** hansfbaier has joined #symbiflow | 02:23 | |
| mithro | hansfbaier: You are correct that the 071-dump_all takes a pretty large amount of disk space | 02:24 |
| hansfbaier | mithro: I now really understand the reason for the ticket: In order to build the part for the Nitefury, the fully bonded part had to be built first. It creates a TON of JSON files and hung my linux system because create_node_tree consumed all my 32GB of RAM. | 02:35 |
| hansfbaier | mithro: I think if we could use sqlite instead of JSON that would save a HUGE amount of disk space and memory | 02:36 |
| hansfbaier | mithro: because the data is inherently relational and thus a relational database would be a great fit | 02:36 |
| hansfbaier | mithro: Also sqlite has simple python bindings | 02:36 |
| hansfbaier | mithro: I will look into moving 071-dump_all to sqlite | 02:37 |
| mithro | hansfbaier: Dunno how those work, but it might be a quick solution to put a compression filter on the json output? | 02:39 |
| hansfbaier | mithro: it would be the proper way to deduplicate (= normalize) the data, because each item is stored exactly once | 02:39 |
| hansfbaier | mithro: But in order to to make a database schema I have to understand the logical structure of the data | 02:40 |
| mithro | hansfbaier: Probably - but sometimes starting with an easier solution gets you to the end goal faster :-) | 02:40 |
| mithro | hansfbaier: https://pypi.org/project/compress-json/ | 02:41 |
| tpb | Title: compress-json · PyPI (at pypi.org) | 02:41 |
| hansfbaier | mithro: The compression does not solve my problem: create_node_tree still would have to uncompress everything and then it would take even more memory | 02:42 |
| hansfbaier | mithro: And I am quite reluctant to buy more memory because, now I have 32GB and I even rarely maxed out 16G | 02:42 |
| mithro | hansfbaier: Spin up a cloud machine? | 02:43 |
| mithro | hansfbaier: https://www.lucidchart.com/techblog/2019/12/06/json-compression-alternative-binary-formats-and-compression-methods/ seems like an interesting page | 02:43 |
| tpb | Title: JSON Compression: Alternative Binary Formats and Compression Methods - Lucidchart (at www.lucidchart.com) | 02:43 |
| mithro | I wonder if we should be using ujson in some places too - https://pypi.org/project/ujson/ | 02:45 |
| tpb | Title: ujson · PyPI (at pypi.org) | 02:45 |
| hansfbaier | mithro: saving space is not the only advantage, you also can create indices for oft used tables, this will speed up the processing of the data by orders of magnitude | 02:45 |
| mithro | hansfbaier: I'm sure that litghost might have some thoughts on that but is currently on holidays until Jan | 02:46 |
| mithro | Anyway, I'm heading out | 02:47 |
| hansfbaier | mithro: I will discuss it in the ticket. I would aim at a proof of concept limited to 071-dump_all | 02:47 |
| *** andrewb1999 has quit IRC | 02:53 | |
| hansfbaier | mithro: sqlite seems to handle large datasets quite well: https://stackoverflow.com/questions/1033309/sqlite-for-large-data-sets | 02:59 |
| tpb | Title: SQLite for large data sets? - Stack Overflow (at stackoverflow.com) | 02:59 |
| *** hansfbaier has quit IRC | 03:30 | |
| *** hansfbaier has joined #symbiflow | 03:32 | |
| *** hansfbaier has quit IRC | 04:00 | |
| umarcor | mithro: what about using yaml and anchors? | 04:02 |
| *** Degi_ has joined #symbiflow | 04:42 | |
| *** Degi has quit IRC | 04:44 | |
| *** Degi_ is now known as Degi | 04:44 | |
| *** ASHR has joined #symbiflow | 05:03 | |
| *** frubbl has quit IRC | 07:25 | |
| *** hansfbaier has joined #symbiflow | 07:25 | |
| *** frubbl has joined #symbiflow | 07:43 | |
| *** kraiskil has joined #symbiflow | 07:51 | |
| *** frubbl has quit IRC | 08:16 | |
| *** bjorkintosh has joined #symbiflow | 08:28 | |
| *** frubbl has joined #symbiflow | 08:43 | |
| *** kraiskil has quit IRC | 09:02 | |
| *** hansfbaier has quit IRC | 09:57 | |
| *** frubbl has quit IRC | 10:09 | |
| *** frubbl has joined #symbiflow | 10:30 | |
| *** kraiskil has joined #symbiflow | 11:37 | |
| *** ASHR has quit IRC | 11:38 | |
| *** kraiskil has quit IRC | 11:42 | |
| *** kraiskil has joined #symbiflow | 11:42 | |
| *** mkru has joined #symbiflow | 12:00 | |
| *** mkru has quit IRC | 12:04 | |
| *** mkru has joined #symbiflow | 12:58 | |
| *** mkru has quit IRC | 13:27 | |
| *** kraiskil has quit IRC | 14:41 | |
| mithro | umarcor: I think I would take json+compression over yaml+anchors personally... | 14:42 |
| *** frubbl has quit IRC | 15:03 | |
| *** kraiskil has joined #symbiflow | 15:26 | |
| *** FFY00 has quit IRC | 15:36 | |
| *** FFY00 has joined #symbiflow | 15:46 | |
| *** FFY00 has quit IRC | 16:05 | |
| *** FFY00 has joined #symbiflow | 16:06 | |
| *** FFY00 has quit IRC | 16:20 | |
| *** andrewb1999 has joined #symbiflow | 16:24 | |
| *** citypw has quit IRC | 16:58 | |
| *** ASHR has joined #symbiflow | 18:54 | |
| *** kgugala has joined #symbiflow | 18:56 | |
| *** kgugala_ has quit IRC | 18:56 | |
| *** andrewb1999 has quit IRC | 19:27 | |
| *** andrewb1999 has joined #symbiflow | 19:27 | |
| *** FFY00 has joined #symbiflow | 19:38 | |
| *** FFY00 has quit IRC | 19:49 | |
| *** FFY00 has joined #symbiflow | 19:54 | |
| *** rvalles_ has quit IRC | 20:06 | |
| *** rvalles_ has joined #symbiflow | 20:07 | |
| *** ASHR has quit IRC | 21:27 | |
| *** tannewt has quit IRC | 21:27 | |
| *** elms has quit IRC | 21:28 | |
| *** elms has joined #symbiflow | 21:29 | |
| *** tannewt has joined #symbiflow | 21:29 | |
| *** kraiskil has quit IRC | 21:41 | |
| *** ASHR has joined #symbiflow | 21:55 | |
| *** ASHR has quit IRC | 23:20 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!