Thursday, 2015-06-04

*** tpb has joined #timvideos00:00
*** CarlFK has joined #timvideos00:24
*** ChanServ sets mode: +v CarlFK00:24
*** thaytan has quit IRC01:22
*** thaytan has joined #timvideos01:23
*** ChanServ sets mode: +v thaytan01:23
*** thaytan has quit IRC01:35
*** thaytan has joined #timvideos01:36
*** ChanServ sets mode: +v thaytan01:36
*** CarlFK has quit IRC02:25
*** CarlFK has joined #timvideos02:45
*** ChanServ sets mode: +v CarlFK02:45
*** hyades_ has joined #timvideos03:25
*** Niharika__ has joined #timvideos04:29
*** Niharika__ has quit IRC04:34
*** CarlFK has quit IRC04:43
*** Niharika__ has joined #timvideos04:59
*** Niharika__ has quit IRC05:03
mithroMaZderMind: there are a couple of ways to get strong synchronization with inter* elements05:16
mithroMaZderMind: I'm interested to know how you are measuring audio sync?05:20
*** hyades_ has quit IRC06:18
MaZderMindmithro: I'm playing back a testclip into the system (http://c3voc.mazdermind.de/avsync-30min.mp4) and record it on the other end06:29
MaZderMindI then import both into kdenlive and play back06:30
MaZderMindI align them by video frame06:30
MaZderMindone can now hear that a) the audio does not lign up at the start and b) the clicks (one from the source and one from the recording) are getting closer and closer to each other until they lign up and finally drift away again06:31
MaZderMindwhile the video-frames are exactly identical06:31
mithroMaZderMind: and the audio is behind or ahead of the video?06:34
MaZderMindthe starts ahead, catches uo and then drifts to beeing behind06:35
MaZderMindi hat a conversation on #gstreamer with heftig, he said beside other things:06:36
MaZderMind10:19 < heftig> for swapping sinks I just use the multifdsink06:36
MaZderMind10:19 < heftig> for swapping sources i have a system which can swap between a videotestsrc+audiotestsrc and a bin containing fdsrc+matroskademux06:36
MaZderMindunfortunately he can't offer source code. I currently try to replicate that06:36
MaZderMindit would result in one big pipeline06:37
mithroone big pipeline is really dangerous in my opinion as the more complicated a pipeline is, the easier it is for something to go wrong in it06:37
mithroHowever, in your case with a very fix configuration and very controlled environment it might be reliable enough06:38
MaZderMindyes, my oppinion too, but driftig audio/video is a blocker06:38
*** Guest32262 is now known as aps06:38
mithroMaZderMind: are you going to have time on the weekend to work on this?06:39
MaZderMindmithro: yes, I'm planning to work on this06:39
MaZderMindI'll too have to do some InDesign juggeling for my wedding in 3 months but nothing more :)06:40
mithroMaZderMind: How about we schedule a 4-6 hour block on the weekend were we both sit down and do some pair coding and analysis? I'm pretty sure we can fix your drifting problem with inter stuff but I'm likely to get distracted if I don't actually schedule a time to work on it06:41
mithrobtw hardware hacker - I started the following for a "choosing the right FPGA board" talk I'm doing at work https://docs.google.com/document/d/14MVW0ncpYKrkJtC4wGY7wMX-CbNFKmcFpsHzXl1khug/edit06:45
tpbTitle: FPGA Development Boards - Google Docs (at docs.google.com)06:45
MaZderMindmithro: those timezones… how about Sunday, 6pm to 8pm Sydney Time?06:48
mithroMaZderMind: yeah I was thinking reserving like 6pm -> midnight ?06:48
MaZderMindmithro: I missed that I have some appointments over the weekend I wasn't aware of…06:49
*** Niharika__ has joined #timvideos06:50
mithroI can do Saturday or Sunday06:50
MaZderMindmithro: yes, I'm planning around… but (unfortunately?) the wedding preparations take a lot of time06:52
mithroMaZderMind: yeah, no worries06:52
mithroMaZderMind: this is also why I like scheduling things because it uncovers things like this :)06:52
MaZderMindSaturday I planned with my witness (is that the correct word?) to look for Suit and I guess we'll be in the city the whole day and after that I'll be really done06:53
MaZderMindSunday I just found hat I have to leanr dancing and we have a dancing class booked :->06:54
MaZderMindBut what interests me most: how were you plannung to "fix" the inter*-elements?06:55
*** [d__d] has quit IRC06:56
MaZderMindI could imagine that it may be possible to rewrite the interaudiosrc to pass on timestamps but as easy as this sounds - it may break a whole lot of things like syncing correctly to the target playlist's clock and such06:56
mithroMaZderMind: before I figure out how to "fix" the inter*-elements I need to first understand exactly what is going on that is causing the drift07:00
MaZderMindok, sorry for my ungraceful wording – I assumed you knew about that problem07:04
*** [d__d] has joined #timvideos07:09
mithroMaZderMind: I want to trace the buffers going through the inter elements and understand when they are getting dropped / duplicated and how they are getting restamped07:10
MaZderMind  GST_BUFFER_PTS (buffer) = intervideosrc->timestamp_offset +07:11
MaZderMind      gst_util_uint64_scale (GST_SECOND * intervideosrc->n_frames,07:11
MaZderMind      GST_VIDEO_INFO_FPS_D (&intervideosrc->info),07:11
MaZderMind      GST_VIDEO_INFO_FPS_N (&intervideosrc->info));07:11
MaZderMindthere is similar code in interaudiosrc07:11
MaZderMindthe PTS is here calculated based on the fps * the numer of frames passed through + the startup-offset -- as far as I can read that07:13
MaZderMindfor audio that is07:13
MaZderMind  GST_BUFFER_TIMESTAMP (buffer) = interaudiosrc->timestamp_offset +07:13
MaZderMind      gst_util_uint64_scale (interaudiosrc->n_samples, GST_SECOND,07:13
MaZderMind      interaudiosrc->info.rate);07:13
MaZderMindso number of samples times sample length + offset07:14
MaZderMindmaybe involving the original buffers' timestamp + offset could change that, but when there is no frame to read from the input the element need to create a buffer with a timestamp exactly ligning up with the last one received from upstream07:16
MaZderMindI think I have to write me a better testbench for hacking on these elements07:16
MaZderMindI'll have to go afk now but will report back soon07:17
mithroMaZderMind: okay, so if we record the buffer timestamp when it goes into the inter component and then record the buffer timestamp out of the inter we can understand any change in the timestamps07:17
MaZderMindyes, I already had a sample for that. I used an identity element and its handoff-signal to get notified about each passing buffer and report its pts07:18
mithroMaZderMind: so you have a log of the timestamps on both sides somewhere I can look at?07:19
MaZderMindnot at hand07:19
mithroMaZderMind: if you can get me one, I'd love to look at it07:19
MaZderMindbut to to real testing I'll try to construct sth. with appsink/appsrc which would allow me to pass in buffers with arbitrary timestamps and see how they get transformed07:20
MaZderMinddid you see the screenshots in my mailinglist-post?07:20
MaZderMindHere is a comparison of a 15 Minute Test-Clip, on the top is the07:20
MaZderMindinput, on the bottom the output:07:20
MaZderMindMinute  0: http://pasteboard.co/VeEzzF0.png07:20
tpbTitle: Pasteboard Uploaded Image (at pasteboard.co)07:20
MaZderMindMinute 20: http://pasteboard.co/VeHxCU0.png07:20
tpbTitle: Pasteboard Uploaded Image (at pasteboard.co)07:20
MaZderMindthe top clip is the source, the bottom is the recording, they are kigned up by video frame07:21
mithroMaZderMind: yeah - it doesn't explain why it is happening though07:23
*** Niharika__ is now known as Niharika_07:26
MaZderMindin the code-quotes above buffer may be an input buffer from the source. Isl PTS (video) and TIMESTAMP (audio) is completely overwritten with values calculated from the nominal fps/sample-length. The *actual* timestamp of the source buffer is completely discarded07:26
MaZderMindthat is why it happens07:26
MaZderMindI'll see if I can change that later :)07:27
mithroMaZderMind: which means the input values have an actual timestamp which doesn't match those values, right?07:27
MaZderMindyes07:28
MaZderMindthe real timestamps of real files are never exactly as calculated, because of clock drift in cameras, temperature differences of hardware clocks, …07:28
MaZderMindthis is why the original timestamps need to be taken into account when calculating the new timestamps in the inter*src07:29
mithroMaZderMind: if the input values are running fast or slow then there will be an eventual discontinuity where the inter element drops or duplicates a buffer?07:29
MaZderMindif they exceed the timeout, yes07:30
MaZderMindthe timeouts defaults to 1s07:30
MaZderMindI have to go afk now, sorry07:31
mithroNo worries07:31
mithroSo, I can't recall of the top of my head how much data is inside a single buffer07:31
mithroI think we can look at those timeouts07:32
*** Niharika_ has quit IRC08:41
*** Niharika_ has joined #timvideos08:49
*** Niharika_ has quit IRC09:16
*** Niharika_ has joined #timvideos09:20
*** Niharika_ has quit IRC09:24
*** Niharika_ has joined #timvideos11:20
*** Niharika_ has quit IRC11:25
*** Niharika_ has joined #timvideos11:36
*** tariq786 has quit IRC11:43
*** tariq786 has joined #timvideos11:46
*** tija has joined #timvideos12:22
*** f15h has joined #timvideos12:53
*** CarlFK has joined #timvideos12:55
*** ChanServ sets mode: +v CarlFK12:55
MaZderMindthe timeouts do apply now, too12:59
MaZderMindthey wouln't change12:59
*** f15h has quit IRC13:13
*** Niharika_ has quit IRC15:07
*** CarlFK has quit IRC15:10
*** CarlFK has joined #timvideos15:34
*** ChanServ sets mode: +v CarlFK15:34
*** Niharika_ has joined #timvideos15:39
*** f15h has joined #timvideos15:49
*** f15h has quit IRC15:52
*** Niharika_ has quit IRC16:21
*** tija has quit IRC16:54
*** CarlFK has quit IRC16:59
*** hyades_ has joined #timvideos17:47
*** [d__d] has quit IRC18:33
*** [d__d] has joined #timvideos18:33
MaZderMindi have a working prototype of a pts synced interaudiosrc18:35
MaZderMindbut it seems the duration needs to be transported as well18:37
*** hyades is now known as Guest5894918:38
*** hyades_ is now known as hyades18:38
*** hyades_ has joined #timvideos18:38
*** hyades_ is now known as hyades_zombie18:55
MaZderMindfirst test looks good so far, although i have not implemented all possibilities yet19:19
*** _florent_ has joined #timvideos19:53
MaZderMindhttps://github.com/MaZderMind/gst-plugins-bad/compare/inter-avsync20:28
tpbTitle: Comparing master...inter-avsync · MaZderMind/gst-plugins-bad · GitHub (at github.com)20:28
*** CarlFK has joined #timvideos20:35
*** ChanServ sets mode: +v CarlFK20:35
*** _florent_ has quit IRC21:13
MaZderMindhttp://lists.freedesktop.org/archives/gstreamer-devel/2015-June/053039.html21:36
tpbTitle: debugging a/v-sync issues, possibly inter*-element related [patch included] (at lists.freedesktop.org)21:36
MaZderMindi asked on the ml for feedback, we'll see what they say21:36
CarlFKMaZderMind: what triggers the problem?  (I am wondering  if it can be worked around like restart everything between talks )21:39
MaZderMindthe problem was that the audio-buffer-timestamps as generated by the source are not used in the target pipeline. they are recalculated based on the number of samples seen. but in real physics world, the samples are not always exactly same length, they drift over time.21:41
MaZderMindthe patch ove uses the timestamps known in the source to build the timestamps in the target21:41
MaZderMindrestarting is not an option for a running live-stream, btw.21:41
CarlFKyes it is ;)21:43
mithroMaZderMind: I believe your patch will solve the issue when playing back already recorded files, but I think it breaks the way the inter parts are non blocking.22:42
mithroBut I'm currently on the way to the dentist and haven't actually looked at your code yet.22:43
mithroBTW playing back files and recording live video is actually slightly different scenarios when talking about clocks22:45
mithroMaZderMind: although in your case where you are using mkv and ffmpeg for capture things are a little confusing.22:50
mithrothaytan: can you tell me how long (in ms) a gstreamer buffer which contains audio is?22:55
*** hyades has quit IRC23:28

Generated by irclog2html.py 2.13.1 by Marius Gedminas - find it at mg.pov.lt!