| *** tpb has joined #symbiflow | 00:00 | |
| *** citypw has joined #symbiflow | 03:42 | |
| *** proteusguy has joined #symbiflow | 06:12 | |
| *** kraiskil has joined #symbiflow | 08:58 | |
| nats` | litghost, I ran the tiles part in 15 minutes and the nodes part in 55 minutes | 09:24 |
|---|---|---|
| nats` | without tinkering the ratio process blocksize we can do the 074 in 1h10min + few minutes for all file remerge | 09:25 |
| nats` | is it a progress ? | 09:25 |
| *** citypw has quit IRC | 09:45 | |
| *** kraiskil has quit IRC | 09:57 | |
| *** kraiskil has joined #symbiflow | 10:10 | |
| *** kraiskil has quit IRC | 10:16 | |
| *** kraiskil has joined #symbiflow | 10:29 | |
| kgugala | @nats` we can discuss 07x fuzzers here, but I have a meeting in about 2 minutes I'll bbl | 11:06 |
| nats` | no problem :) | 11:06 |
| nats` | ttyl | 11:06 |
| nats` | I'm at work too | 11:06 |
| nats` | I started a run with a full 074 new version script | 11:43 |
| nats` | let's see how long it take | 11:44 |
| *** proteusguy has quit IRC | 12:20 | |
| *** kraiskil has quit IRC | 12:31 | |
| *** kraiskil has joined #symbiflow | 12:44 | |
| kgugala | at my side it finished | 12:52 |
| kgugala | I'll push the fix | 12:52 |
| *** kraiskil has quit IRC | 12:54 | |
| *** kraiskil has joined #symbiflow | 13:07 | |
| nats` | kgugala, I saw your new bug on the 074 | 13:29 |
| nats` | shoudl I do something in the new version or is it something solved by ignoring those net ? | 13:30 |
| nats` | I didn't fix the working part of the 074 | 13:30 |
| nats` | the other point to check wouldbe to see if it's because of the 072 patch | 13:30 |
| nats` | Command being timed: "python3 run_fuzzer.py" | 13:31 |
| nats` | User time (seconds): 14816.12 | 13:31 |
| nats` | System time (seconds): 755.35 | 13:31 |
| nats` | Percent of CPU this job got: 346% | 13:31 |
| nats` | Elapsed (wall clock) time (h:mm:ss or m:ss): 1:14:53 | 13:31 |
| nats` | Average shared text size (kbytes): 0 | 13:31 |
| nats` | Average unshared data size (kbytes): 0 | 13:31 |
| nats` | Average stack size (kbytes): 0 | 13:31 |
| nats` | Average total size (kbytes): 0 | 13:31 |
| nats` | Maximum resident set size (kbytes): 1201000 | 13:31 |
| nats` | Average resident set size (kbytes): 0 | 13:31 |
| nats` | Major (requiring I/O) page faults: 0 | 13:31 |
| nats` | Minor (reclaiming a frame) page faults: 35432521 | 13:31 |
| nats` | Voluntary context switches: 66357 | 13:31 |
| nats` | Involuntary context switches: 10470447 | 13:31 |
| nats` | Swaps: 0 | 13:31 |
| nats` | File system inputs: 11168 | 13:31 |
| nats` | File system outputs: 27366400 | 13:31 |
| nats` | Socket messages sent: 0 | 13:31 |
| nats` | Socket messages received: 0 | 13:31 |
| nats` | Signals delivered: 0 | 13:31 |
| nats` | Page size (bytes): 4096 | 13:31 |
| nats` | Exit status: 0* | 13:31 |
| nats` | here is the runtime of the new version of the 074 | 13:31 |
| nats` | 1h14 | 13:31 |
| nats` | I'm wondering how long it take ont he current version | 13:31 |
| nats` | uhhmm kgugala do yo uhave the csv from your zynq run ? | 13:37 |
| nats` | I would like to know if it's normal to have a lot of NULL inside it | 13:38 |
| kgugala | the bug is for 054 | 13:38 |
| kgugala | TBH I think that 054 is redundant | 13:39 |
| kgugala | it runs the same tcl as 56 | 13:39 |
| nats` | oh yes sorry | 13:40 |
| nats` | I red 074 | 13:40 |
| nats` | my bad | 13:40 |
| kgugala | IMO we can drop 54 and use 56 to calculate those | 13:40 |
| nats` | I didn't look at 05x serie | 13:40 |
| kgugala | I'm testing this right now | 13:40 |
| nats` | can you look in the csvoutput of the 074 ? | 13:40 |
| kgugala | sure | 13:40 |
| nats` | I have a lot of tile NULL | 13:40 |
| nats` | but maybe that's ok | 13:41 |
| nats` | filetype,subtype,filename | 13:41 |
| nats` | tile,NULL,tile_NULL_X0Y156.json5 | 13:41 |
| nats` | tile,NULL,tile_NULL_X1Y156.json5 | 13:41 |
| nats` | tile,NULL,tile_NULL_X2Y156.json5 | 13:41 |
| nats` | tile,NULL,tile_NULL_X3Y156.json5 | 13:41 |
| nats` | tile,T_TERM_INT,tile_T_TERM_INT_X4Y156.json5 | 13:41 |
| nats` | tile,T_TERM_INT,tile_T_TERM_INT_X5Y156.json5 | 13:41 |
| kgugala | I have them too | 13:41 |
| nats` | oky | 13:41 |
| nats` | do you have a place to upload it ? | 13:41 |
| kgugala | I may have | 13:42 |
| nats` | I'm interested in having it to compare with my next run of zynq 074 | 13:42 |
| nats` | starting the zynq run of my new 074 | 13:49 |
| nats` | will let you know the time taken :) | 13:49 |
| nats` | is there a good to make something like PR on github but only for testing before doing a real PR | 13:57 |
| kgugala | nats`, you can always mark a PR as WIP | 13:58 |
| kgugala | and use it for discussion | 13:58 |
| nats` | ahhh sure ! | 13:58 |
| nats` | thanks | 13:58 |
| nats` | I'll do that for the new version of 074 | 13:59 |
| nats` | I'll need to have people testing it | 13:59 |
| nats` | kgugala, | 14:27 |
| nats` | Work done ! | 14:27 |
| nats` | Command being timed: "python3 run_fuzzer.py" | 14:27 |
| nats` | User time (seconds): 6608.01 | 14:27 |
| nats` | System time (seconds): 442.71 | 14:27 |
| nats` | Percent of CPU this job got: 361% | 14:27 |
| nats` | Elapsed (wall clock) time (h:mm:ss or m:ss): 32:30.47 | 14:27 |
| nats` | Average shared text size (kbytes): 0 | 14:27 |
| nats` | Average unshared data size (kbytes): 0 | 14:27 |
| nats` | Average stack size (kbytes): 0 | 14:27 |
| nats` | Average total size (kbytes): 0 | 14:27 |
| nats` | Maximum resident set size (kbytes): 1020264 | 14:27 |
| nats` | Average resident set size (kbytes): 0 | 14:27 |
| nats` | Major (requiring I/O) page faults: 0 | 14:27 |
| nats` | Minor (reclaiming a frame) page faults: 31386362 | 14:27 |
| nats` | Voluntary context switches: 37120 | 14:27 |
| nats` | Involuntary context switches: 6036505 | 14:27 |
| nats` | Swaps: 0 | 14:27 |
| nats` | File system inputs: 71360 | 14:28 |
| nats` | File system outputs: 16468960 | 14:28 |
| nats` | Socket messages sent: 0 | 14:28 |
| nats` | Socket messages received: 0 | 14:28 |
| nats` | Signals delivered: 0 | 14:28 |
| nats` | Page size (bytes): 4096 | 14:28 |
| nats` | Exit status: 0 | 14:28 |
| nats` | I'm surprised it went really fast for the zyn | 14:28 |
| nats` | zynq | 14:28 |
| nats` | same csv file size | 14:30 |
| nats` | good start | 14:31 |
| nats` | by eye first and last lines are the same | 14:32 |
| nats` | kgugala, are you interested that I push and PR that so you could test it ? | 14:32 |
| kgugala | sure | 14:44 |
| kgugala | you can tag me in the PR | 14:44 |
| *** proteusguy has joined #symbiflow | 14:50 | |
| nats` | kgugala, I'm running it from the directory with the makefile to be sure and them I push :) | 15:12 |
| nats` | kgugala, I have an other problem with directory in 074 I think | 15:41 |
| nats` | in generate_dump_after | 15:42 |
| nats` | all the python script certainly need to be prefixed by the $FUZDIR path | 15:43 |
| nats` | ah wait nop | 15:43 |
| nats` | the cd ../.. in generate.sh should make that ok | 15:44 |
| *** _whitelogger has quit IRC | 15:45 | |
| *** citypw has joined #symbiflow | 15:47 | |
| kgugala | it is fixed now in my PR | 15:47 |
| kgugala | https://github.com/SymbiFlow/prjxray/pull/516 | 15:47 |
| tpb | Title: fuzzers: 074: fix path to generate_after_dump by kgugala · Pull Request #516 · SymbiFlow/prjxray · GitHub (at github.com) | 15:47 |
| kgugala | this one | 15:47 |
| *** _whitelogger has joined #symbiflow | 15:48 | |
| nats` | yep so that but it failed on my test because I didn't have it :) | 15:50 |
| nats` | fixed like your PR and rerun :) | 15:50 |
| nats` | s/so/saw/ | 15:50 |
| nats` | just output of the tcl script I made to fix (bad directory) and I should be good :) | 16:29 |
| nats` | the final generation seems to work | 16:29 |
| nats` | 2019-01-15 17:18:51.938046 Generating reduced tile for CLBLM_L | 16:29 |
| nats` | 2019-01-15 17:18:51.938328 Using pool.imap_unordered | 16:29 |
| nats` | 100% (600 of 600) |###################################################################################################################################################################################| Elapsed Time: 0:02:00 Time: 0:02:00 | 16:29 |
| nats` | 2019-01-15 17:21:10.662660 Generating reduced tile for CLBLM_R | 16:29 |
| nats` | I think I'll try to do the same progress bar for the multithread version of fuzzer | 16:29 |
| nats` | it's better than nothing or ton of line of text | 16:29 |
| *** powerbit has quit IRC | 16:41 | |
| nats` | the reducing stage is awfully long too ! | 16:47 |
| nats` | litghost, I think digshadow is right it's not possible to put 072 and 074 before other fuzzers... the 074 reduce stage is a nightmare | 16:47 |
| *** citypw has quit IRC | 17:12 | |
| *** kraiskil has quit IRC | 17:56 | |
| nats` | 2019-01-15 18:07:30.598128 Creating wire<->node index | 18:04 |
| nats` | 100% (3824221077 of 3824221077) |#####################################################################################################################################################################| Elapsed Time: 0:00:59 Time: 0:00:59 | 18:04 |
| nats` | 100% (5153171150 of 5153171150) |#####################################################################################################################################################################| Elapsed Time: 0:01:00 Time: 0:01:00 | 18:04 |
| nats` | 2019-01-15 18:09:33.945337 Creating node tree | 18:04 |
| nats` | 35% (398115 of 1122477) |############################################################# | Elapsed Time: 0:00:07 ETA: 0:00:15Traceback (most recent call last): | 18:04 |
| nats` | File "create_node_tree.py", line 279, in <module> | 18:04 |
| nats` | main() | 18:04 |
| nats` | File "create_node_tree.py", line 271, in main | 18:04 |
| nats` | if node in uphill_wire_node_index else []))) | 18:04 |
| nats` | File "create_node_tree.py", line 103, in create_ordered_wires_for_node | 18:04 |
| nats` | assert len(wires_in_node) >= len(all_wires) | 18:04 |
| nats` | AssertionError | 18:04 |
| nats` | 100% (1122477 of 1122477) |###########################################################################################################################################################################| Elapsed Time: 0:00:10 Time: 0:00:10 | 18:04 |
| nats` | something failed | 18:04 |
| nats` | ... | 18:05 |
| nats` | I guess I may need some help to sort that | 18:39 |
| nats` | it seems to be unhappy of results of the 072 fuzzer and something else | 18:40 |
| litghost | nats: Do put up the PR | 18:45 |
| litghost | I'll take a look | 18:45 |
| nats` | oky | 19:25 |
| nats` | litghost kgugala https://github.com/SymbiFlow/prjxray/pull/526 | 19:42 |
| tpb | Title: WIP: reworking Fuzzer 074 for multi thread process - Dont MERGE ! by natsfr · Pull Request #526 · SymbiFlow/prjxray · GitHub (at github.com) | 19:42 |
| nats` | PR pushed | 19:42 |
| nats` | not for merging until we are sure :) | 19:42 |
| nats` | ohhhh shit | 19:42 |
| nats` | rubber ducky :D | 19:43 |
| nats` | I think it fails because the 072 was generated for artix 7 and the 074 for zynq | 19:43 |
| nats` | :D | 19:43 |
| nats` | let me try | 19:43 |
| litghost | ya, that'll totally bork it | 19:46 |
| litghost | FYI WIP: in the title is enough to prevent a merge, so you don't need a giant "don't merge", that's what WIP means | 19:46 |
| nats` | ah oky sorry | 19:46 |
| nats` | I wante dot be sur e:) | 19:46 |
| nats` | duuh | 19:47 |
| nats` | I wanted to be sure | 19:47 |
| nats` | I rerun a full 072/074 | 19:48 |
| nats` | but feel free to test :) | 19:48 |
| nats` | do you remember how long is the 074 run ? | 19:48 |
| litghost | I usually leave it overnight | 19:57 |
| litghost | :/ | 19:57 |
| litghost | There is nothing in the processing that truely requires that length, it is just severly unoptimized | 19:58 |
| litghost | Let's first fix the vivado leak per your PR, then maybe consider attacking the post-processing script. I believe it should be possible to speed it up significantly with only small modifications | 19:58 |
| nats` | I didn't look into that | 20:28 |
| nats` | but I was wondering if the json processing is optimized | 20:28 |
| nats` | I have pretty bad souvenir of json lib | 20:29 |
| nats` | but that was really long time ago | 20:29 |
| nats` | anyway all the vivado processing for zynq take less than 50 minute for the 074 | 20:29 |
| nats` | and I started one with time to see the full process | 20:29 |
| litghost | "but I was wondering if the json processing is optimized" -> Depends on what you mean. The json parsing library is reasonable fast, but problem I believe is some inefficient data passing methods. When I wrote the post-processing for 074, I proritized something working, so it isn't as clean as it might have been. Some profiling and refactoring would probably go a long way. | 20:31 |
| nats` | I guess you'll be more efficient than me as you could see my python knowledge is long outdated now | 20:32 |
| nats` | so performance wise..... :D | 20:33 |
| litghost | reducing the runtime and memory usage of the vivado portion is of immediate benefit, so the PR is still very valuable | 20:33 |
| nats` | sure sure :) | 20:34 |
| nats` | and I'm happy to help on that | 20:34 |
| nats` | uhhmmmmm got an idea ! | 20:58 |
| nats` | maybe I can make a bug report on that | 20:58 |
| nats` | we should store in the run.ok the target of the last run ! | 20:58 |
| nats` | you would avoid stupid mistake like I made | 20:58 |
| nats` | https://github.com/SymbiFlow/prjxray/issues/528 | 21:01 |
| tpb | Title: Fuzzer storing last target · Issue #528 · SymbiFlow/prjxray · GitHub (at github.com) | 21:01 |
| nats` | uhhhh litghost the post processing is about to blow my RAM away :D | 21:15 |
| nats` | it seems it's really tight in 16GB | 21:15 |
| litghost | nats: Ya, that seems about right | 21:20 |
| litghost | I'm currently working on testing a couple INT pip CLs, and I'll move over to checking out your PR | 21:20 |
| nats` | I'll try to get 16GB more next month | 21:21 |
| nats` | it's too expensive for this one :D | 21:21 |
| kgugala | nats`: I also started a 072/074 build | 21:33 |
| nats` | oky | 21:34 |
| nats` | with the PR ? | 21:35 |
| nats` | and which target ? | 21:35 |
| kgugala | artix | 21:35 |
| kgugala | should I try zynq instead? | 21:35 |
| nats` | oky | 21:35 |
| nats` | nop | 21:35 |
| nats` | should work on all | 21:35 |
| nats` | the artix is longer | 21:35 |
| nats` | zynq < artix < kintex | 21:35 |
| kgugala | ok | 21:36 |
| nats` | count about 30minute 072 for artix | 21:36 |
| nats` | and I would say 1h15 074 BEFORE post processing | 21:36 |
| nats` | I couldn't run the post proc | 21:36 |
| nats` | I'm trying it for zynq right now | 21:36 |
| nats` | I think we really have to stabilize all of that and generate as few as possible | 21:38 |
| nats` | uhhmm how hard would it be to distribute the build over network | 21:38 |
| nats` | there are python lib for that ? | 21:38 |
| nats` | duhhh we really need to optimise the post processing | 21:50 |
| nats` | it's awfully slow :D | 21:50 |
| litghost | In general 072-074 weren't optimized because they are very deterministic, and don't need to be re-run often. | 22:12 |
| litghost | This is unlike the other fuzzers which are fairly stochastic | 22:12 |
| nats` | Traceback (most recent call last): | 22:15 |
| nats` | File "generate_grid.py", line 676, in <module> | 22:15 |
| nats` | main() | 22:15 |
| nats` | File "generate_grid.py", line 578, in main | 22:15 |
| nats` | with open(os.path.join(prjxray.util.get_db_root(), 'tilegrid.json')) as f: | 22:15 |
| nats` | FileNotFoundError: [Errno 2] No such file or directory: '/home/nats/project/symbiflow/prjxray/database/zynq7/tilegrid.json' | 22:15 |
| nats` | Makefile:16: recipe for target 'build/specimen_001/OK' failed | 22:15 |
| nats` | make[1]: *** [build/specimen_001/OK] Error 1 | 22:15 |
| nats` | make[1]: Leaving directory '/home/nats/project/symbiflow/prjxray/fuzzers/074-dump_all' | 22:15 |
| nats` | Makefile:20: recipe for target 'run' failed | 22:15 |
| nats` | make: *** [run] Error 2 | 22:15 |
| nats` | Command exited with non-zero status 2 | 22:15 |
| nats` | Command being timed: "make run" | 22:15 |
| nats` | User time (seconds): 17837.88 | 22:15 |
| nats` | System time (seconds): 1258.89 | 22:15 |
| nats` | Percent of CPU this job got: 305% | 22:15 |
| nats` | Elapsed (wall clock) time (h:mm:ss or m:ss): 1:44:09 | 22:15 |
| nats` | Average shared text size (kbytes): 0 | 22:15 |
| nats` | Average unshared data size (kbytes): 0 | 22:16 |
| nats` | Average stack size (kbytes): 0 | 22:16 |
| nats` | Average total size (kbytes): 0 | 22:16 |
| nats` | Maximum resident set size (kbytes): 4110148 | 22:16 |
| nats` | Average resident set size (kbytes): 0 | 22:16 |
| nats` | Major (requiring I/O) page faults: 44 | 22:16 |
| nats` | Minor (reclaiming a frame) page faults: 659168440 | 22:16 |
| nats` | Voluntary context switches: 5940756 | 22:16 |
| nats` | Involuntary context switches: 10981805 | 22:16 |
| nats` | Swaps: 0 | 22:16 |
| nats` | File system inputs: 13773992 | 22:16 |
| nats` | File system outputs: 18203112 | 22:16 |
| nats` | Socket messages sent: 0 | 22:16 |
| nats` | Socket messages received: 0 | 22:16 |
| nats` | Signals delivered: 0 | 22:16 |
| nats` | Page size (bytes): 4096 | 22:16 |
| nats` | Exit status: 2 | 22:16 |
| nats` | ar sorry bad copy paste | 22:16 |
| nats` | it failed at 1h44 -_- | 22:16 |
| nats` | I think we really should do a sanity check before running | 22:16 |
| nats` | ahhh I didn't run previous fuzzer | 22:17 |
| nats` | :| | 22:17 |
| nats` | https://github.com/SymbiFlow/prjxray/issues/529 | 22:19 |
| tpb | Title: Fuzzer pre-run sanity check · Issue #529 · SymbiFlow/prjxray · GitHub (at github.com) | 22:19 |
| nats` | oky I started a full run of all fuzzer from the main directory | 22:22 |
| nats` | it'll run all night hopefully | 22:22 |
| nats` | good night ! | 22:22 |
| nats` | uhhmm it stopped right before going to be | 22:23 |
| nats` | I'll check that tmorrow | 22:24 |
Generated by irclog2html.py 2.13.1 by Marius Gedminas - find it at mg.pov.lt!