Analiza plików binarnych i ich przetwarzanie za pomocą PHP CLI

Ostatnio podjąłem się misji tłumaczenia pewnej gry na język polski. Nie są to zwykłe pliki tekstowe, a niestandardowe skrypty zakodowane w postaci binarnej. Na samym dole plików znajduje się tekst w jednym ciągu. Nie można tak naprawdę odróżnić, która część tekstu będzie wyświetlana w którym momencie gry. Poprzez dogłębną analizę tychże plików możemy jednak wywnioskować pewne rzeczy. Podglądając jeden z nim w edytorze hexadecymalnym możemy zauważyć pewną powtarzalność występujących w nim znaków. Można dostrzec, że występują one co 4 bajty, czyli prawdopodobnie są to zakodowane liczby 32-bitowych zniennych (long int). Jednakże, skoro jest to skrypt i składa się z samych liczb i jest ich dość wiele, to możemy założyć, że duża ilość z nich nie będzie nam tak naprawdę do niczego potrzebna. Także przyglądając się szczegółowo można zauważyć powtarzające sie bajty [FF] koło siebie, czyli zmienne te będą najprawdopodobniej w trybie signed (dodatnie i ujemne), ponieważ nie wydaje mi się, aby w oprogramowaniu, którego firma używała do edycji tychże skryptów, ktokolwiek wpisywał dodatnie numery rzędu 4 milardów, raczej były to negatywne cyfry takie jak -1 czy -3.

Jednak po co nam ta wiedza?

Takie szczegółowe zapoznanie się z plikiem jest nam potrzebne aby mieć całkowitą kontrolę nad długością tekstu, który chcemy modyfikować. Tak jak wcześniej wspominałem sam tekst na końcu pliku nie ma wyraźnie widocznych podziałów, w którym momencie w grze dana jego część będzie się wyświetlać. W takim razie coś innego musi kontrolować ich początek wczytywania i ilość znaków jaka będzie wyświetlana.

Zacznijmy od początku pliku, widać tam tag DCPB, który identyfikuje rodzaj pliku wykorzystywany przez grę. Następnie spróbujmy zdekodować 4 kolejne bajty, ponieważ może to być oznaczenie offsetu lub rozmiar pewnej ilości danych w tym pliku. Czyli [BC] [BA] [01] [00] to: 188 + 186 * 256 + 1 * 256^2 + 0 * 256^3 (little endian) = 113 340. Teraz spójżmy na rozmiar pliku: 117 150. Czyli tak jak pisałem jest to prawdopodobnie oznaczenie długości pewnej części pliku. Spróbujmy w takim razie zobaczyć czy coś ciakawego znajduję się po przeskoczeniu na adres decymalny 113 348 w pliku. Znajduje się tam [64] [00] [00] [00], czyli wartość 100. Tylko co to może oznaczać? Offset? Wartość? Coś innego? Jeśli jest to offset to na pewno w odległości +/-100 bajtów od tego miejsca nic ciekawe nie znajduje się. W takim razie przeskoczmy 4 bajty do przodu. Znajduje się tam [20] [03] [00] [00], czyli wartość 800, zobaczmy czy w odległości 800 bajtów od tego miejsca znajduje się coś ciekawego. Bingo! Dodając 800 do aktualnego offsetu, z którego odczytaliśmy tę własnie wartość, znajduje się pierwsza litera/znak tekstu. Czyli możemy prawie w 100% potwierdzić, że jest to oznaczenie offsetu pierwszej części teksu, która będzie wyświetlana w grze. Cofnijmy się spowrotem o 800 bajtów i odczytajmy następne 4 bajty jako signed long int. Znajduje się tam [12] [00] [00] [00] czyli wartość 18. Czyżby było to 800 + 18 jako offset? Hmm. Sprawdźmy następne 4 bajty, [32] [03] [00] [00] czyli 818. Wygląda na to, że własnie W TYM miejscu jest tak naprawdę offset. Co w takim razie oznacza ta wcześniejsza liczba 18? 800 + 18 = 818! Jeśli przyjąć, że 800 i 818 to offsety, to w takim razie to co znajduje się po każdym offsecie (w tym przypadku 18) to pewnie jest długość (ilość bajtów) wczytywanego tekstu przez grę w danym miejscu. Czyli każde 8 bajtów jest używane do wczytania parametrów danego tekstu wyświetlanego w grze. W takim razie policzmy ile razy takie 8 bajtów występuje zanim zacznie się tekst. Wygląda na to, że jest 100 wystąpień, czyli 4 bajty zdekodowane przed pierwszym offsetem ([64] [00] [00] [00] = 100) oznaczają najprawdopodobniej ilość linii tekstu do przetłumaczenia.

Zaczynamy?

Znająć już mniej więcej strukturę pliku, która nas interesuje możemy przejść do tłumaczenia tekstu na język polski. Jednak pomyślmy przez chwilę. Jeśli mamy 100 wystąpień, każde z nich musi mieć dopasowany offset i długość to czy nie będzie to dla nas problemem jeśli wydłuży nam się pierwszy tekst przy tłumaczeniu? Wiadomym jest, że w języku polskim BARDZO często tłumaczony tekst jest dłuższy niż w innych językach. Będziemy musieli wtedy ręcznie zmieniać wszystkie offsety! Ogromna robota i duża ilość czasu potrzebna. Czy jest może inny sposób?

Skrypty PHP CLI jako pomocna dłoń przy konwertowaniu plików na bardziej przyjazny dla użytkownika.

Najprostrzym sposobem jest "przerobić" plik w taki sposób abyśmy mieli tekst ładnie podzielony linia po linii i nie przejmowali się jakimiś offsetami tylko samym tłumaczeniem tekstu. Pamiętać jednak należy, że taki plik musi być poprawnie zakodowany spowrotem do poprzedniego formatu wraz z odpowiednim nałożeniem zmian w binarnej części oznaczającej offsety i długości danych części tekstu/linii. Oczywiście piszac w języku C możemy osiągnać znacznie więcej, na przykład bezpośrednio wczytując plik dynamicznie tworzyć pola do wpisywania/zmiany tekstu bazująć na oryginalnym pliku skryptu gry. Może to jednak zająć więcej czasu, zależy czy mamy części kodu z naszych wcześniejszych projektów, które możemy wykorzystać, czy nie.

Analiza i przetwarzanie pliku skryptami PHP.

Skrypty PHP można pisać w zwykłym notatniku korzystając z manuala. Na początek jednak musimy stworzyć sobie algorytm lub mieć zdolność tworzenia ich w locie w naszej głowie. Musimy pamiętać o naszej wcześniejszej analizie. Rozpiszmy sobie jak mniej więcej wygląda struktura pliku:

String: "DCPB",

4 bajty signed long int: długość danych, których nie modyfikujemy,

dane, których nie modyfikujemy,

4 bajty signed long int: ilość wystąpień tekstu,

{4 bajty signed long: offset tekstu n, 4 bajty signed long: długość tesktu n} * ilość wystąpień,

tekst w jednym ciągu.

Patrząc na powyższy podział na pewno będziemy musieli mieć pętle/funkcję wczytywania 4 bajtów i tłumaczenia ich na long int. Na pewno takżę będziemy musieli wczytać część danych, których nie modyfikujemy, a w ziązku z tym, że muszą one być zachowane w trybie bezpiecznym do edycji tekstowej, proponuję wykorzystać base64_encode, a przed tym kompresję deflate:

base64_encode(gzcompress($buf,9));

W ten sposób nasze pliki do edycji będą znaczne mniejsze i nie będą na początku zawierać nadmiernej ilości nieinteresującego nas ciągu cyfr i liter (base64).

Gdy przejdziemy do wczytywania offsetów i dlugości tekstu, to na pewno będziemy potrzebowali dynamiczne arraye, które będą nam przechowywać offsety i długości w 2 tablicach (lub 2-wymiarowej tablicy pod jedną zmienna, jak wolisz). W PHP jest to stosunkowo łatwe, w C byśmy musieli dużo mallocować/reallocować, pamiętać o bajcie NULL na końcu tablicy znaków itp. Dynamiczne arraye najlepiej tworzyć w pętli powtarzającej się [ilość wystąpień tekstu] razy czytając 2 razy po 4 bajty i wrzucając zdekodowane longi do 2 zmienny będących tablicami, na przykład:

for($x=0;$x<$amount;++$x) {

  $buf = fread($plik1,4);

  $pos[] = (ord($buf[3]) << 24) + (ord($buf[2]) << 16) + (ord($buf[1]) << 8) + ord($buf[0]) + 12 + $rawdatalength;

  $buf = fread($plik1,4);

  $len[] = (ord($buf[3]) << 24) + (ord($buf[2]) << 16) + (ord($buf[1]) << 8) + ord($buf[0]);

}

Oczywiście można wczytywać całe 8 bajtów i odpowiednio je bitshiftować lub wykorzystać inne metody. To jaką metodę przyjmiesz zależy od Ciebie, najważniejsze jest aby konwertowanie bezbłednie działało, wydajność kodu potrzebna jest najczęściej tylko w środowiskach produkcyjnych lub przy dużej ilości przetwarzanych danych.

Po zakończonej pętli trzeba rozpocząc następną, która będzie korzystać z danych w poprzednio utworzonych tablicach i wczytywać odpowiednią ilość tekstu oraz zapisywać to w odrębnych liniach do pliku przyjaznego do edycji tekstowej. Czyli dla pewności najlepiej wykorzystać fseek do offsetów i fread do długości, a następnie fwrite. Przykład:

for($x=0;$x<$amount;++$x) {

  fseek($plik1,$pos[$x],SEEK_SET);

  $text = fread($plik1,$len[$x]);

  fwrite($plik2,$text . PHP_EOL);

}

To mniej więcej wszystko co trzeba zrobić aby utworzyć plik przyjazny do edycji tekstowej. Możemy oczywiście na początku pliku zapisać jeszcze długość danych jaką osiągnęliśmy kompresująć i base64-kodując te dane. W ten sposób unikniemy problemów z limitem do 1024 znaków jeśli będziemy korzystać z fgets do czytania pliku, który przygotowaliśmy do zakodowania. W przypadku ciągu danych, które będzie dekodować z base64 i dekompresować, będziemy korzystać z fread($plik,$ilosc_bajtow_do_wczytania).

Kodowanie musi odbywać się odwrotnie do dekodowania, na pewno będziemy musieli odtworzyć tag "DCPB" zapisać 4 bajty długości danych poprzez przeliczenie długości zakodowanego, zdekompresowaneego ciągu bajtów, zapisać ten ciąg i odtworzyć poprawnie offsety i długości z linii, które skrypt będzie czytał z pliku tekstowego, który wykorzystywaliśmy do bezpiecznego tłumaczenia gry. Po napisaniu prawidłowo części kodującej należy przetestować oryginalny zakodowany plik i przetłumaczony, czy różnice między oboma plikami zaczynają się po pierwszym offsecie tekstu i czy wszystkie offsety i długości poprawnie wskazują na części przetłumaczonego tekstu. Jeśli tak rzeczywiście jest, to udało nam się poprawnie stworzyć koder/dekoder skryptów gry, w sposób, który znacznie ułatwia nam nanoszenie zmian na tekst wyświetlający się w niej.

Plik wykorzystany jako przykład: http://virtual.4my.eu/samples/SSS05_01.scripb

Zopfli KrzYmod updates

Zopfli fork also called Zopfli KrzYmod got some updates.

Basically I put there some latest zopfli changes which impact compression, enabled --legacy mode to work with multi-files as well (to provide new Zopfli functionality fully, notably additional splitting last which is unfortunatelly omitted in KrzYmod algorithm with newest changes). Added --brotli switch to use Brotli's Huffman for RLE which provides different block splitting and smaller files at times (on my test case I got 13 bytes reduction on ~2KB gz file of minified JS).

I will put release a bit later after I get rid of all errors, I think there are some with zopflipng crashing or so. You can, however, compile it Yourself from git.

https://github.com/MrKrzYch00/zopfli

Putty Windows x64 and Linux builds (x86, x64, ARMv5, ARMv7)

If anybody is interested, I just compiled Putty for below systems:

- Windows x64 (CLI & GUI),

- Linux x86 (CLI & GUI-GTK),

- Linux x64 (CLI & GUI-GTK),

- Linux ARMv7 (CLI & GUI-GTK),
- Linux ARMv5 (CLI Only).

All builds feature Linker-time optimizations with Linker Plugin (except ARMv5), are static.
ARMv7 build is NEON FPU optimised, built on Odroid U3.
ARMv5 was built on Zyxel NSA-220 (oarm, uClibc).

CLI & GUI builds contain: fuzzterm, pageant, plink, pscp, psftp, pterm, putty, puttygen, puttytel.
CLI only builds contain: fuzzterm, plink, pscp, psftp, puttygen.

I know it's kind of pointless to have Putty on Linux but hey, maybe someone is so used to operate it on Windows he/she is willing to have it on Linux as well. :)

Windows x86 builds are available on the official site, so I'm not compiling those.

Here it is: http://virtual.4my.eu/putty

Why should You care for x64 builds? Well, if You run x64 system usually x64 builds should be faster on those systems. With putty it should make a difference when using tunnels to transfer large amounts of data or copying files with other tools provided. I didn't make any comparison charts myself, but if You care to do them, You can share the information. :)

KrzYVideoFixer v1.05 is out

Few hours ago the new version of KrzYVideoFixer is out which in my opinion contains all possible checks agains video/audio stream to get rid of parts with missing video frames without re-encoding. Remember, You need ffprobe and ffmpeg in the same directory or globally accessible directory. You can download both programs from this site. The only thing left to fix is to support filenames with spaces which I am already testing locally and will be available in v1.06. I did not find any other problems in the last few hours. Here is an example of 3 times run (when video is very broken the output file may still be a bit damaged due to good audio stream being sometimes cut out by ffmpeg if it's near damaged fragment resulting in time skips still occuring, usually 2nd run fixes all problems):

e:\VLCPortable\kvf -s12 349834500000.flv

KrzYVideoFixer v1.05 by Mr_KrzYch00


-> Detecting a minimum of 12 missing frames!

|/ Extracting chunk #01: 0 to 3:25:00.7 (L: 3:25:00.7)
|/ Extracting chunk #02: 3:25:11.5 to 3:55:11.1 (L: 0:29:59.6)
!--> Audio unstable at keyframe: 3:55:12.9, trying next
|/ Extracting chunk #03: 3:55:13.9 to 6:05:13.9 (L: 2:10:00.0)
!--> Audio unstable at keyframe: 6:05:16.6, trying next
|/ Extracting chunk #04: 6:05:17.6 to 6:55:00.7 (L: 0:49:43.1)
!--> Audio unstable at keyframe: 7:45:02.3, trying next
|/ Extracting chunk #05: 7:45:03.4 to 7:45:35.3 (L: 0:00:31.9)
?-> Video drop detected at: 7:45:36.8~7:45:42.52, but audio is stable!
!--> Audio unstable at keyframe: 7:45:42.9, trying next
|/ Extracting chunk #06: 7:45:44.0 to 7:55:04.1 (L: 0:09:20.1)
!--> Audio unstable at keyframe: 7:56:08.0, trying next
|/ Extracting chunk #07: 7:56:09.1 to 8:02:44.5 (L: 0:06:35.4)
!--> Audio unstable at keyframe: 8:02:51.3, trying next
|/ Extracting chunk #08: 8:02:52.4 to 8:05:51.1 (L: 0:02:58.7)
!--> Audio unstable at keyframe: 8:05:53.3, trying next
|/ Extracting chunk #09: 8:05:54.4 to 8:08:36.4 (L: 0:02:42.0)
!--> Audio unstable at keyframe: 8:08:43.3, trying next
?-> Video drop detected at:8:08:44.3~8:08:47.05, but audio is stable!
|/ Extracting chunk #10: 8:08:47.2 to 8:35:43.8 (L: 0:26:56.6)
!--> Audio unstable at keyframe: 8:36:06.1, trying next
|/ Extracting chunk #11: 8:36:07.2 to 8:55:09.3 (L: 0:19:02.1)
!--> Audio unstable at keyframe: 8:55:31.4, trying next
|/ Extracting chunk #12: 8:55:32.5 to 8:57:51.8 (L: 0:02:19.3)
!--> Audio unstable at keyframe: 8:58:08.0, trying next
|/ Extracting chunk #13: 8:58:09.4 to 8:58:16.8 (L: 0:00:07.4)
!--> Audio unstable at keyframe: 8:58:39.0, trying next
?-> Video drop detected at:8:58:24.6~8:58:38.99, but audio is stable!
|/ Extracting chunk #14: 8:58:40.2 to 8:58:50.3 (L: 0:00:10.1)
?-> Video drop detected at:8:59:12.7~8:59:18.99, but audio is stable!
!--> Audio unstable at keyframe: 8:59:19.6, trying next
!--> Audio unstable at keyframe: 8:59:36.2, trying next
|/ Extracting chunk #15: 8:59:37.7 to 8:59:41.6 (L: 0:00:03.9)
?-> Video drop detected at:8:59:43.2~8:59:50.99, but audio is stable!
!--> Audio unstable at keyframe: 8:59:51.6, trying next
|/ Extracting chunk #16: 8:59:52.7 to 9:00:07.7 (L: 0:00:15.0)
!--> Audio unstable at keyframe: 9:00:16.6, trying next
!--> Audio unstable at keyframe: 9:00:17.6, trying next
|/ Extracting chunk #17: 9:00:23.5 to 9:00:24.9 (L: 0:00:01.4)
!--> Audio unstable at keyframe: 9:00:40.1, trying next
|/ Extracting chunk #18: 9:00:41.3 to 9:00:50.4 (L: 0:00:09.1)
!--> Audio unstable at keyframe: 9:01:05.6, trying next
|/ Extracting chunk #19: 9:01:06.7 to 9:01:24.8 (L: 0:00:18.1)
?-> Video drop detected at:9:01:36.2~9:01:42.99, but audio is stable!
!--> Audio unstable at keyframe: 9:01:44.2, trying next
|/ Extracting chunk #20: 9:01:45.7 to 9:34:35.7 (L: 0:32:50.0)
?-> Video drop detected at:9:34:37.3~9:34:39.03, but audio is stable!
!--> Audio unstable at keyframe: 9:34:40.0, trying next
?-> Video drop detected at:9:34:43.6~9:34:55.13, but audio is stable!
!--> Audio unstable at keyframe: 9:34:55.6, trying next
?-> Video drop detected at:9:34:59.2~9:35:11.16, but audio is stable!
!--> Audio unstable at keyframe: 9:35:12.0, trying next
?-> Video drop detected at:9:35:15.6~9:35:27.20, but audio is stable!
!--> Audio unstable at keyframe: 9:35:28.0, trying next
!--> Audio unstable at keyframe: 9:35:44.2, trying next
|/ Extracting chunk #21: 9:35:45.4 to 9:35:49.3 (L: 0:00:03.9)
WARNING! Last audio frame>video frame: 9:36:09.91>9:35:59.16, ignoring audio!
WARNING! Last audio frame>video frame: 9:36:09.91>9:36:07.12, ignoring audio!
!--> Audio unstable at keyframe: 9:36:15.1, trying next
|/ Extracting chunk #22: 9:36:16.2 to 9:36:19.7 (L: 0:00:03.5)
?-> Video drop detected at:9:36:21.2~9:36:31.12, but audio is stable!
!--> Audio unstable at keyframe: 9:36:32.0, trying next
|/ Extracting chunk #23: 9:36:33.1 to 9:44:35.5 (L: 0:08:02.4)
?-> Video drop detected at:9:44:37.0~9:44:39.12, but audio is stable!
!--> Audio unstable at keyframe: 9:44:40.1, trying next
|/ Extracting chunk #24: 9:44:41.2 to 9:44:52.4 (L: 0:00:11.2)
!--> Audio unstable at keyframe: 9:45:05.6, trying next
|/ Extracting chunk #25: 9:45:11.8 to 9:45:25.1 (L: 0:00:13.3)
?-> Video drop detected at:9:45:26.7~9:45:35.23, but audio is stable!
!--> Audio unstable at keyframe: 9:45:35.6, trying next
|/ Extracting chunk #26: 9:45:36.7 to 9:45:46.2 (L: 0:00:09.5)
!--> Audio unstable at keyframe: 9:45:55.6, trying next
|/ Extracting chunk #27: 9:45:56.7 to 9:46:09.2 (L: 0:00:12.5)
?-> Video drop detected at:9:46:10.7~9:46:15.15, but audio is stable!
!--> Audio unstable at keyframe: 9:46:16.0, trying next
|/ Extracting chunk #28: 9:46:17.6 to 9:46:19.6 (L: 0:00:02.0)
?-> Video drop detected at:9:46:21.2~9:46:31.14, but audio is stable!
!--> Audio unstable at keyframe: 9:46:31.5, trying next
|/ Extracting chunk #29: 9:46:32.6 to 9:46:35.6 (L: 0:00:03.0)
!--> Audio unstable at keyframe: 9:46:47.5, trying next
!--> Audio unstable at keyframe: 9:46:48.5, trying next
?-> Video drop detected at:9:46:49.5~9:46:56.14, but audio is stable!
|/ Extracting chunk #30: 9:46:56.4 to 9:47:00.0 (L: 0:00:03.6)
!--> Audio unstable at keyframe: 9:47:20.3, trying next
|/ Extracting chunk #31: 9:47:21.7 to 9:47:35.1 (L: 0:00:13.4)
!--> Audio unstable at keyframe: 9:47:44.2, trying next
?-> Video drop detected at:9:51:35.7~9:51:45.34, but audio is stable!
|/ Extracting chunk #32: 9:47:45.3 to 9:51:46.0 (L: 0:04:00.7)
!--> Audio unstable at keyframe: 9:51:51.4, trying next
!--> Audio unstable at keyframe: 9:51:52.5, trying next
!--> Audio unstable at keyframe: 9:51:53.6, trying next
!--> Audio unstable at keyframe: 9:51:59.2, trying next
!--> Audio unstable at keyframe: 9:52:16.3, trying next
WARNING! Last audio frame>video frame: 9:52:27.71>9:52:17.73, ignoring audio!
?-> Video drop detected at:9:52:18.7~9:52:31.17, but audio is stable!
|/ Extracting chunk #33: 9:52:31.4 to 9:52:33.6 (L: 0:00:02.2)
!--> Audio unstable at keyframe: 9:52:50.4, trying next
|/ Extracting chunk #34: 9:52:51.6 to 9:52:58.7 (L: 0:00:07.1)
!--> Audio unstable at keyframe: 9:53:13.8, trying next
?-> Video drop detected at:9:53:18.6~9:53:19.85, but audio is stable!
|/ Extracting chunk #35: 9:53:14.9 to 9:53:19.9 (L: 0:00:05.0)
!--> Audio unstable at keyframe: 9:53:28.1, trying next
!--> Audio unstable at keyframe: 9:53:29.3, trying next
?-> Video drop detected at:9:53:37.2~9:53:51.17, but audio is stable!
|/ Extracting chunk #36: 9:53:35.5 to 9:53:59.3 (L: 0:00:23.8)
!--> Audio unstable at keyframe: 9:54:16.6, trying next
!--> Audio unstable at keyframe: 9:54:35.0, trying next
|/ Extracting chunk #37: 9:54:36.1 to 9:54:39.3 (L: 0:00:03.2)
?-> Video drop detected at:9:54:40.9~9:54:47.16, but audio is stable!
!--> Audio unstable at keyframe: 9:54:47.2, trying next
|/ Extracting chunk #38: 9:54:48.4 to 9:54:49.6 (L: 0:00:01.2)
!--> Audio unstable at keyframe: 9:55:05.8, trying next
|/ Extracting chunk #39: 9:55:07.1 to 9:55:17.3 (L: 0:00:10.2)
!--> Audio unstable at keyframe: 9:55:30.9, trying next
|/ Extracting chunk #40: 9:55:32.0 to 9:55:35.1 (L: 0:00:03.1)
!--> Audio unstable at keyframe: 9:55:46.2, trying next
|/ Extracting chunk #41: 9:55:47.3 to 9:55:49.7 (L: 0:00:02.4)
!--> Audio unstable at keyframe: 9:56:02.0, trying next
|/ Extracting chunk #42: 9:56:03.1 to 11:15:01.7 (L: 1:18:58.6)
!--> Audio unstable at keyframe: 11:15:18.5, trying next
|/ Extracting chunk #43: 11:15:19.6 to 11:35:20.2 (L: 0:20:00.6)
!--> Audio unstable at keyframe: 11:35:22.0, trying next
|/ Extracting chunk #44: 11:35:23.1 to 12:21:03.8 (L: 0:45:40.7)
!--> Audio unstable at keyframe: 12:21:16.7, trying next
|/ Extracting chunk #45: 12:21:17.8 to 13:46:58.6 (L: 1:25:40.8)
!--> Audio unstable at keyframe: 13:47:05.0, trying next
!--> Audio unstable at keyframe: 13:47:06.0, trying next
?-> Video drop detected at:13:47:17.9~13:47:19.32, but audio is stable!
|/ Extracting chunk #46: 13:47:12.1 to 13:47:22.6 (L: 0:00:10.5)
!--> Audio unstable at keyframe: 13:47:35.4, trying next
!--> Audio unstable at keyframe: 13:47:36.4, trying next
WARNING! Last audio frame>video frame: 13:47:41.16>13:47:38.40, ignoring audio!
?-> Video drop detected at:13:47:38.9~13:47:43.32, but audio is stable!
!--> Audio unstable at keyframe: 13:47:43.8, trying next
|/ Extracting chunk #47: 13:47:44.9 to 13:47:48.2 (L: 0:00:03.3)
!--> Audio unstable at keyframe: 13:48:00.7, trying next
?-> Video drop detected at:13:48:03.8~13:48:07.66, but audio is stable!
|/ Extracting chunk #48: 13:48:01.8 to 14:13:28.1 (L: 0:25:26.3)
!--> Audio unstable at keyframe: 14:13:40.4, trying next
|/ Extracting chunk #49: 14:13:41.5 to 15:00:24.7 (L: 0:46:43.2)
?-> Video drop detected at:15:00:26.2~15:00:31.50, but audio is stable!
!--> Audio unstable at keyframe: 15:00:32.2, trying next
|/ Extracting chunk #50: 15:00:33.3 to 15:06:18.7 (L: 0:05:45.4)
?-> Video drop detected at:15:06:20.3~15:06:23.55, but audio is stable!
!--> Audio unstable at keyframe: 15:06:24.4, trying next
|/ Extracting chunk #51: 15:06:25.5 to 15:10:10.8 (L: 0:03:45.3)
!--> Audio unstable at keyframe: 15:10:17.5, trying next
|/ Extracting chunk #52: 15:10:18.6 to 15:14:45.4 (L: 0:04:26.8)
?-> Video drop detected at:15:14:47.1~15:14:55.61, but audio is stable!
!--> Audio unstable at keyframe: 15:14:56.1, trying next
|/ Extracting chunk #53: 15:14:57.2 to 15:15:53.9 (L: 0:00:56.7)
!--> Audio unstable at keyframe: 15:16:01.1, trying next
|/ Extracting chunk #54: 15:16:02.2 to 15:16:58.5 (L: 0:00:56.3)
!--> Audio unstable at keyframe: 15:17:05.6, trying next
|/ Extracting chunk #55: 15:17:06.7 to 15:18:03.5 (L: 0:00:56.8)
!--> Audio unstable at keyframe: 15:18:11.1, trying next
|/ Extracting chunk #56: 15:18:12.2 to 15:19:50.3 (L: 0:01:38.1)
!--> Audio unstable at keyframe: 15:20:02.9, trying next
|/ Extracting chunk #57: 15:20:04.0 to 15:22:18.0 (L: 0:02:14.0)
!--> Audio unstable at keyframe: 15:22:24.9, trying next
|/ Extracting chunk #58: 15:22:26.0 to 15:28:46.2 (L: 0:06:20.2)
!--> Audio unstable at keyframe: 15:28:58.7, trying next
|/ Extracting chunk #59: 15:28:59.8 to 15:29:40.1 (L: 0:00:40.3)
!--> Audio unstable at keyframe: 15:29:46.6, trying next
|/ Extracting chunk #60: 15:29:47.7 to 15:30:05.5 (L: 0:00:17.8)
!--> Audio unstable at keyframe: 15:30:18.1, trying next
|/ Extracting chunk #61: 15:30:19.2 to 15:45:35.0 (L: 0:15:15.8)
?-> Video drop detected at:15:45:36.5~15:45:43.73, but audio is stable!
!--> Audio unstable at keyframe: 15:45:44.3, trying next
|/ Extracting chunk #62: 15:45:45.4 to 15:54:40.7 (L: 0:08:55.3)
!--> Audio unstable at keyframe: 15:54:48.7, trying next
|/ Extracting chunk #63: 15:54:49.8 to 15:55:10.8 (L: 0:00:21.0)
!--> Audio unstable at keyframe: 15:55:23.1, trying next
|/ Extracting chunk #64: 15:55:24.2 to 15:58:54.5 (L: 0:03:30.3)
!--> Audio unstable at keyframe: 15:59:04.8, trying next
|/ Extracting chunk #65: 15:59:05.9 to 16:04:16.3 (L: 0:05:10.4)
!--> Audio unstable at keyframe: 16:04:28.7, trying next
|/ Extracting chunk #66: 16:04:29.8 to 16:07:06.1 (L: 0:02:36.3)
!--> Audio unstable at keyframe: 16:07:12.8, trying next
|/ Extracting chunk #67: 16:07:13.9 to 16:07:20.9 (L: 0:00:07.0)
?-> Video drop detected at:16:07:22.5~16:07:27.76, but audio is stable!
!--> Audio unstable at keyframe: 16:07:28.0, trying next
?-> Video drop detected at:16:14:15.0~16:14:15.76, but audio is stable!
|/ Extracting chunk #68: 16:07:29.1 to 16:19:24.1 (L: 0:11:55.0)
!--> Audio unstable at keyframe: 16:19:27.8, trying next
?-> Video drop detected at:16:19:25.7~16:19:27.80, but audio is stable!
|/ Extracting chunk #69: 16:19:28.9 to 16:20:54.7 (L: 0:01:25.8)
!--> Audio unstable at keyframe: 16:21:07.2, trying next
|/ Extracting chunk #70: 16:21:08.3 to 16:30:39.6 (L: 0:09:31.3)
!--> Audio unstable at keyframe: 16:30:51.7, trying next
|/ Extracting chunk #71: 16:30:52.8 to 16:33:13.0 (L: 0:02:20.2)
?-> Video drop detected at:16:33:14.6~16:33:19.74, but audio is stable!
!--> Audio unstable at keyframe: 16:33:19.9, trying next
|/ Extracting chunk #72: 16:33:21.0 to 16:39:35.9 (L: 0:06:14.9)
?-> Video drop detected at:16:39:37.5~16:39:43.80, but audio is stable!
!--> Audio unstable at keyframe: 16:39:44.2, trying next
|/ Extracting chunk #73: 16:39:45.3 to 16:41:08.4 (L: 0:01:23.1)
?-> Video drop detected at:16:41:10.0~16:41:11.80, but audio is stable!
!--> Audio unstable at keyframe: 16:41:11.9, trying next
|/ Extracting chunk #74: 16:41:13.0 to 16:46:07.4 (L: 0:04:54.4)
?-> Video drop detected at:16:46:09.0~16:46:15.79, but audio is stable!
!--> Audio unstable at keyframe: 16:46:16.6, trying next
|/ Extracting chunk #75: 16:46:17.7 to 16:47:43.0 (L: 0:01:25.3)
!--> Audio unstable at keyframe: 16:47:55.8, trying next
|/ Extracting chunk #76: 16:47:56.9 to 16:48:51.6 (L: 0:00:54.7)
!--> Audio unstable at keyframe: 16:48:58.4, trying next
|/ Extracting chunk #77: 16:48:59.5 to 16:52:54.2 (L: 0:03:54.7)
!--> Audio unstable at keyframe: 16:53:06.5, trying next
?-> Video drop detected at:16:53:07.9~16:53:11.88, but audio is stable!
|/ Extracting chunk #78: 16:53:07.6 to 16:54:02.4 (L: 0:00:54.8)
?-> Video drop detected at:16:54:04.0~16:54:07.88, but audio is stable!
!--> Audio unstable at keyframe: 16:54:08.2, trying next
|/ Extracting chunk #79: 16:54:09.3 to 17:00:57.1 (L: 0:06:47.8)
?-> Video drop detected at:17:00:58.6~17:01:03.88, but audio is stable!
!--> Audio unstable at keyframe: 17:01:04.5, trying next
|/ Extracting chunk #80: 17:01:05.6 to 17:04:42.5 (L: 0:03:36.9)
?-> Video drop detected at:17:04:44.1~17:04:47.88, but audio is stable!
!--> Audio unstable at keyframe: 17:04:48.2, trying next
|/ Extracting chunk #81: 17:04:49.3 to 17:06:20.5 (L: 0:01:31.2)
?-> Video drop detected at:17:06:22.1~17:06:23.88, but audio is stable!
!--> Audio unstable at keyframe: 17:06:24.0, trying next
|/ Extracting chunk #82: 17:06:25.1 to 17:10:49.0 (L: 0:04:23.9)
?-> Video drop detected at:17:10:50.6~17:10:55.88, but audio is stable!
!--> Audio unstable at keyframe: 17:10:56.2, trying next
|/ Extracting chunk #83: 17:10:57.3 to 17:16:30.5 (L: 0:05:33.2)
?-> Video drop detected at:17:16:32.1~17:16:39.88, but audio is stable!
!--> Audio unstable at keyframe: 17:16:40.0, trying next
|/ Extracting chunk #84: 17:16:41.1 to 17:19:11.4 (L: 0:02:30.3)
!--> Audio unstable at keyframe: 17:19:23.9, trying next
|/ Extracting chunk #85: 17:19:25.0 to 17:23:25.8 (L: 0:04:00.8)
!--> Audio unstable at keyframe: 17:23:37.3, trying next
|/ Extracting chunk #86: 17:23:38.4 to 17:27:21.2 (L: 0:03:42.8)
?-> Video drop detected at:17:27:22.7~17:27:27.86, but audio is stable!
!--> Audio unstable at keyframe: 17:27:28.5, trying next
|/ Extracting chunk #87: 17:27:29.6 to 17:31:26.7 (L: 0:03:57.1)
?-> Video drop detected at:17:31:28.2~17:31:35.88, but audio is stable!
!--> Audio unstable at keyframe: 17:31:36.1, trying next
|/ Extracting chunk #88: 17:31:37.2 to 17:32:38.3 (L: 0:01:01.1)
!--> Audio unstable at keyframe: 17:32:51.0, trying next
|/ Extracting chunk #89: 17:32:52.1 to 17:35:00.0 (L: 0:02:07.9)
!--> Audio unstable at keyframe: 17:35:07.2, trying next
|/ Extracting chunk #90: 17:35:08.3 to 17:41:43.1 (L: 0:06:34.8)
!--> Audio unstable at keyframe: 17:41:55.4, trying next
|/ Extracting chunk #91: 17:41:56.5 to 17:43:28.4 (L: 0:01:31.9)
?-> Video drop detected at:17:43:30.0~17:43:35.88, but audio is stable!
!--> Audio unstable at keyframe: 17:43:36.2, trying next
|/ Extracting chunk #92: 17:43:37.3 to 17:46:40.4 (L: 0:03:03.1)
!--> Audio unstable at keyframe: 17:46:52.2, trying next
|/ Extracting chunk #93: 17:46:53.3 to 17:47:29.7 (L: 0:00:36.4)
?-> Video drop detected at:17:47:31.2~17:47:35.92, but audio is stable!
!--> Audio unstable at keyframe: 17:47:36.8, trying next
|/ Extracting chunk #94: 17:47:37.9 to 17:54:43.5 (L: 0:07:05.6)
!--> Audio unstable at keyframe: 17:54:50.7, trying next
|/ Extracting chunk #95: 17:54:51.8 to 17:59:36.1 (L: 0:04:44.3)
!--> Audio unstable at keyframe: 17:59:47.9, trying next
|/ Extracting chunk #96: 17:59:49.0 to 18:01:19.3 (L: 0:01:30.3)
!--> Audio unstable at keyframe: 18:01:31.9, trying next
|/ Extracting chunk #97: 18:01:33.0 to 18:02:16.2 (L: 0:00:43.2)
?-> Video drop detected at:18:02:17.8~18:02:23.92, but audio is stable!
!--> Audio unstable at keyframe: 18:02:24.0, trying next
|/ Extracting chunk #98: 18:02:25.1 to 18:02:42.2 (L: 0:00:17.1)
?-> Video drop detected at:18:02:43.7~18:02:47.92, but audio is stable!
!--> Audio unstable at keyframe: 18:02:48.2, trying next
|/ Extracting chunk #99: 18:02:49.3 to 18:03:06.3 (L: 0:00:17.0)
?-> Video drop detected at:18:03:07.9~18:03:11.92, but audio is stable!
!--> Audio unstable at keyframe: 18:03:12.2, trying next
|/ Extracting chunk #100: 18:03:13.3 to 18:03:40.4 (L: 0:00:27.1)
!--> Audio unstable at keyframe: 18:03:47.0, trying next
|/ Extracting chunk #101: 18:03:48.1 to 18:06:50.2 (L: 0:03:02.1)
!--> Audio unstable at keyframe: 18:06:56.6, trying next
|/ Extracting chunk #102: 18:06:57.7 to 18:13:48.7 (L: 0:06:51.0)
!--> Audio unstable at keyframe: 18:13:55.3, trying next
?-> Video drop detected at:18:15:03.1~18:15:03.96, but audio is stable!
|/ Extracting chunk #103: 18:13:56.4 to 18:17:24.5 (L: 0:03:28.1)
!--> Audio unstable at keyframe: 18:17:31.9, trying next
|/ Extracting chunk #104: 18:17:33.0 to 18:17:52.3 (L: 0:00:19.3)
!--> Audio unstable at keyframe: 18:18:05.0, trying next
|/ Extracting chunk #105: 18:18:06.1 to 18:20:23.3 (L: 0:02:17.2)
!--> Audio unstable at keyframe: 18:20:33.0, trying next
|/ Extracting chunk #106: 18:20:34.1 to 18:20:44.1 (L: 0:00:10.0)
!--> Audio unstable at keyframe: 18:20:51.2, trying next
|/ Extracting chunk #107: 18:20:52.3 to 18:24:00.2 (L: 0:03:07.9)
!--> Audio unstable at keyframe: 18:24:12.3, trying next
|/ Extracting chunk #108: 18:24:13.4 to 18:24:32.3 (L: 0:00:18.9)
!--> Audio unstable at keyframe: 18:24:44.3, trying next
|/ Extracting chunk #109: 18:24:45.4 to 18:36:20.7 (L: 0:11:35.3)
!--> Audio unstable at keyframe: 18:36:24.0, trying next
?-> Video drop detected at:18:36:22.3~18:36:23.95, but audio is stable!
|/ Extracting chunk #110: 18:36:25.1 to 18:36:53.3 (L: 0:00:28.2)
!--> Audio unstable at keyframe: 18:36:56.0, trying next
?-> Video drop detected at:18:36:54.9~18:36:55.95, but audio is stable!
|/ Extracting chunk #111: 18:36:57.1 to 18:37:57.9 (L: 0:01:00.8)
!--> Audio unstable at keyframe: 18:38:10.4, trying next
|/ Extracting chunk #112: 18:38:11.5 to 18:39:18.8 (L: 0:01:07.3)
!--> Audio unstable at keyframe: 18:39:28.9, trying next
|/ Extracting chunk #113: 18:39:30.0 to 18:46:52.8 (L: 0:07:22.8)
!--> Audio unstable at keyframe: 18:46:59.5, trying next
|/ Extracting chunk #114: 18:47:00.6 to 18:49:44.4 (L: 0:02:43.8)
?-> Video drop detected at:18:49:46.0~18:49:51.93, but audio is stable!
!--> Audio unstable at keyframe: 18:49:52.4, trying next
?-> Video drop detected at:18:54:39.0~18:54:39.93, but audio is stable!
WARNING! pts/dts N/A at frames: 18:59:07.9, 18:59:07.9
|/ Extracting chunk #115: 18:49:53.5 to 18:59:07.9 (L: 0:09:14.4)


|/ Merging chunks -> 349834500000.flv__done__.mp4

The Operation Completed Successfully!

|/ Waiting for audio scan to finish . . .

e:\VLCPortable>kvf -s12 349834500000.flv__done__.mp4

KrzYVideoFixer v1.05 by Mr_KrzYch00


-> Detecting a minimum of 12 missing frames!

?-> Video drop detected at:7:16:52.9~7:16:55.59, but audio is stable!
|/ Extracting chunk #01: 0 to 8:52:21.7 (L: 8:52:21.7)
!--> Audio unstable at keyframe: 8:52:31.5, trying next
!--> Audio unstable at keyframe: 8:52:32.2, trying next
|/ Extracting chunk #02: 8:52:33.4 to 8:52:45.5 (L: 0:00:12.1)
!--> Audio unstable at keyframe: 8:52:47.1, trying next
!--> Audio unstable at keyframe: 8:52:47.2, trying next
!--> Audio unstable at keyframe: 8:53:03.0, trying next
?-> Video drop detected at:8:52:51.0~8:53:02.95, but audio is stable!
?-> Video drop detected at:12:43:59.6~12:44:01.04, but audio is stable!
?-> Video drop detected at:12:44:09.9~12:44:13.81, but audio is stable!
?-> Video drop detected at:15:07:00.4~15:07:01.25, but audio is stable!
?-> Video drop detected at:15:44:16.2~15:44:20.19, but audio is stable!
?-> Audio drop detected at:16:54:20.6, but video is stable!
?-> Video drop detected at:17:02:18.6~17:02:19.43, but audio is stable!
?-> Video drop detected at:17:39:59.2~17:40:00.16, but audio is stable!
WARNING! pts/dts N/A at frames: 17:44:28.1, 17:44:28.1
|/ Extracting chunk #03: 8:53:04.1 to 17:44:28.1 (L: 8:51:24.0)


|/ Merging chunks -> 349834500000.flv__done__.mp4__done__.mp4

The Operation Completed Successfully!

|/ Waiting for audio scan to finish . . .

e:\VLCPortable>kvf -s12 349834500000.flv__done__.mp4__done__.mp4

KrzYVideoFixer v1.05 by Mr_KrzYch00


-> Detecting a minimum of 12 missing frames!

?-> Video drop detected at:7:16:52.9~7:16:55.59, but audio is stable!
?-> Video drop detected at:12:43:29.8~12:43:31.20, but audio is stable!
?-> Video drop detected at:12:43:40.1~12:43:43.97, but audio is stable!
?-> Video drop detected at:15:06:30.6~15:06:31.41, but audio is stable!
?-> Video drop detected at:15:43:46.4~15:43:50.35, but audio is stable!
?-> Audio drop detected at:16:53:50.7, but video is stable!
?-> Video drop detected at:17:01:48.8~17:01:49.59, but audio is stable!
?-> Video drop detected at:17:39:29.4~17:39:30.32, but audio is stable!
WARNING! pts/dts N/A at frames: 17:43:58.2, 17:43:58.3
Video file is not damaged!


The Operation Completed Successfully!

|/ Waiting for audio scan to finish . . .

e:\VLCPortable>

KrzYVideoFixer v1.04 will be delayed

As the title says, v1.04 will be a bit delayed... This is because it turns out there are a bit more factors to take care about when "chunking", "merging" the video in order to get rid of all the damage without re-encoding. If You are interested on what my kvf console output is at the moment when checking very damaged video stream, check below:

KrzYVideoFixer v1.04 by Mr_KrzYch00


-> Saving output file when input is not damaged!

-> Detecting a minimum of 10 missing frames!

WARNING! pts/dts mismatch at frame: 0:38:28.08
WARNING! pts/dts mismatch at frame: 0:38:28.12
|/ Extracting chunk #01: 0 to 0:38:29.5 (L: 0:38:29.5)
WARNING! pts/dts mismatch at frame: 1:21:34.08
WARNING! pts/dts mismatch at frame: 1:21:34.12
|/ Extracting chunk #02: 0:38:33.7 to 1:21:34.0 (L: 0:43:00.3)
!--> Audio unstable at keyframe: 1:21:39.4, trying next
WARNING! pts/dts mismatch at frame: 1:24:04.12
WARNING! pts/dts mismatch at frame: 1:24:04.24
|/ Extracting chunk #03: 1:21:40.5 to 1:24:04.1 (L: 0:02:23.6)
?-> Video drop detected at: 1:24:05.6~1:24:08.64, but audio is stable!
!--> Audio unstable at keyframe: 1:24:09.4, trying next
?-> Video drop detected at: 1:24:13.9~1:24:16.64, but audio is stable!
WARNING! pts/dts mismatch at frame: 1:24:18.64
WARNING! pts/dts mismatch at frame: 1:24:18.68
?-> Video drop detected at: 1:24:18.7~1:24:26.48, but audio is stable!
?-> Video drop detected at: 1:24:27.2~1:24:32.64, but audio is stable!
WARNING! pts/dts mismatch at frame: 3:17:08.80
WARNING! pts/dts mismatch at frame: 3:17:08.84
|/ Extracting chunk #04: 1:24:10.5 to 3:17:08.7 (L: 1:52:58.2)
!--> Audio unstable at keyframe: 3:17:15.9, trying next
WARNING! pts/dts mismatch at frame: 3:42:35.64
WARNING! pts/dts mismatch at frame: 3:42:35.68
|/ Extracting chunk #05: 3:17:17.0 to 3:42:35.6 (L: 0:25:18.6)
!--> Audio unstable at keyframe: 3:42:43.0, trying next
!--> Audio unstable at keyframe: 3:42:44.0, trying next
WARNING! pts/dts mismatch at frame: 3:42:44.08
WARNING! pts/dts mismatch at frame: 3:42:44.12
?-> Video drop detected at: 3:42:44.1~3:42:45.00, but audio is stable!
?-> Video drop detected at: 3:42:45.6~3:42:48.64, but audio is stable!
WARNING! pts/dts mismatch at frame: 3:44:57.36
WARNING! pts/dts mismatch at frame: 3:44:57.40
?-> Video drop detected at: 3:44:57.4~3:44:58.32, but audio is stable!
?-> Video drop detected at: 3:44:59.1~3:45:05.92, but audio is stable!
?-> Video drop detected at: 3:53:35.9~3:53:36.92, but audio is stable!
WARNING! pts/dts mismatch at frame: 3:53:40.00
WARNING! pts/dts mismatch at frame: 3:53:40.04
|/ Extracting chunk #06: 3:42:45.1 to 3:53:39.9 (L: 0:10:54.8)
?-> Video drop detected at: 3:53:41.6~3:53:44.92, but audio is stable!
!--> Audio unstable at keyframe: 3:53:45.4, trying next
?-> Video drop detected at: 3:53:50.1~3:53:52.92, but audio is stable!
WARNING! pts/dts mismatch at frame: 3:53:57.16
WARNING! pts/dts mismatch at frame: 3:53:57.28
|/ Extracting chunk #07: 3:53:46.5 to 3:53:57.2 (L: 0:00:10.7)
!--> Audio unstable at keyframe: 3:54:04.4, trying next
?-> Video drop detected at: 3:54:06.0~3:54:08.92, but audio is stable!
WARNING! pts/dts mismatch at frame: 4:08:27.56
WARNING! pts/dts mismatch at frame: 4:08:27.60
|/ Extracting chunk #08: 3:54:05.5 to 4:08:27.5 (L: 0:14:22.0)
?-> Video drop detected at: 4:08:29.0~4:08:32.92, but audio is stable!
!--> Audio unstable at keyframe: 4:08:33.1, trying next
WARNING! pts/dts mismatch at frame: 4:10:29.40
WARNING! pts/dts mismatch at frame: 4:10:29.44
|/ Extracting chunk #09: 4:08:34.2 to 4:10:29.3 (L: 0:01:55.1)
!--> Audio unstable at keyframe: 4:10:31.1, trying next
WARNING! pts/dts mismatch at frame: 4:40:53.40
WARNING! pts/dts mismatch at frame: 4:40:53.44
/  Video drop detected, Scanning Audio: 4:21:04.19

KrzYVideoFixer & mf2t/t2mf (linux native & armv7, windows)

In the last few weeks I was playing with downloading some publically available video streams in order to upload them to Youtube for better awareness and easy of access. The one thing that bothered me the most was that Youtube converter was getting stuck at some of them (the common infinite 95% processing bug or processing restarting over and over again bug). I was trying to figure out what the heck is going on so I checked few vids but it was hard to pin-point the problem until I watched full video file. Then I discovered that video had small jumping occuring every some time that was caused by server-end keeping real timestamps while it couldn't process video on-time making the stream drop and causing still frames or time jumping ahead in video player (when You watch already downloaded stream, if You watch it in real-time from the internet it usually hangs for a while or looks like broken I-frame).

So I started analysing videos by fast-forwarding with MPC and using ffmpeg to split and merge all chunks together. It worked pretty well at the beginning until I got stuck with it at some video and was really not up to the task of watching ~8 hours material with full attention. 

After being disappointed with one of my vids that couldn't be processed by YT (and I really didn't want to process it myself just to deleted the file afterwards, worsen the quality even more or waste my time by it not being helpful at all), I started searching tha intarwebz and downloaded few programs that turned out to be shit (at least in my case, they may help for some, but they were all a waste of time for me). So I started digging into it by myself...

FFPROBE, some of You may heard about it, cool tool to scan video file and tell You stuff... well then, let's try reading frames... OH shit, 300MB of plain text file... That will take AGES to analyse and my eyes will most likely start bleeding after seeing it in notebad past first 10MB... but hey, there is a way around it... as I like automating stuff, let's try doing my best in this area.

So I opened notepad and started PHPing, vua la, a first automated script to get rid of bad video parts and YT accepted my video! A happy day!

But as I stared with C on my zopfli fork project and was quite successful with it to extend nice Deflate program functionality to even support ZIP files, I thought that I can try my best here too. Opened notepad and started rewritting PHP into C and then successfully GCCed it. Works perfectly! (at least in my sandbox!)

So I wanted to share this solution with You. It's available at http://virtual.4my.eu/KrzYVideoFixer/.

You can find there windows and linux (both x86 and armv7-odroid) builds.

Make sure to read readme.txt file and put ffmpeg and ffprobe in the same directory and run in from that directory (or to be accessed globally, like PATH variable on windows). I was thinking about adding switch to point ffmpeg/ffprobe dir manually but..... is that really needed??

The program is quite simple, takes ~500KB of RAM by itself, uses ffprobe to analyse stuff by itself and split damaged file to chunks then merge them together using ffmpeg. It DOES NOT convert anything. Stream copy is used all the way there, so it's pretty much the original stuff with few damaged parts being cut out of file. However, the output is always saved in MP4 format.

AVCONV WILL NOT BE SUPPORTED. Why? Because it doesn't want to work the way it needs to (avprobe is shit,,,,,, whoooops!!11!), just compile ffmpeg if You are using Ubuntu.

On the other note. I managed to successfully compile mf2t/t2mf (midi file to text / text to midi file) using modern gcc on windows and linux. In case You were searching for one, there is x86, x64 (both windows and linux) and armv7 (linux) versions there! http://virtual.4my.eu/mf2t. As far as I tested, after some sourcecode fixes to properly compile with GCC and actually run as it should, it worked pretty well even with SYSEX data. However, it may have bugs because I'm not that sure if I managed to fix all the code that required fixes. I will maybe add some functionality to it at some point when I have time... I'm personally interested in a switch to always start from tick 0 by shifting all ticks so the resulting MID file starts RIGHT AWAY; a away to make file that is ending too fast to have bit more seconds at the end with a switch for it; aaand somehow to detect/apply proper start/end timings for midi to loop nicely (if it's enabled in, let's say, in_midi... caught... winamp) with a switch.

If You would want to reward me for my work, just process some file with kvf and read summary message or join www.gptchat.com and be interested in some deals there. :)

So suming it all... buuuuuuilds...

KrzYVideoFixer: http://virtual.4my.eu/KrzYVideoFixer (linux x86, armv7; windows x86)

MF2T/T2MF: http://virtual.4my.eu/mf2t (linux x64, x86, armv7; windows x64, x86)

                   check readme first!

Thanks!

Zyxel NSA-210 successfully running 4TB Seagate 4000DM000

This walkthrough will tell You how to make Your >2TB work in Zyxel NSA-210 with newest firmware when disk initialization fails. It does not contain images currently, maybe later. Maybe hard to understand and not enough detailed. I wrote it after I succedded and it was hard to remember all correct steps after 2 days of continous work on this issue. Also I consider that You have a bit of Linux knowledge that's why I skipped details here and there. Feel free to ask questions in comments.

Make Zyxel NSA-210 support Your 4TB HDD.

What will You need:

- Linux x64 (required, I used CentOS, can be in VirtualBox),

- USB->SATA bridge or desktop computer (required)

- Additional tools listed in this walkthrough if You don't have full installation of Linux (required)

- Patience (optional :)

- Clonezilla or other able to read EXT3 and XFS (optional)

- >2TB HD (optional, that is if You have one)

- <2TB HD (required, can be old HD that You kept in NAS)

First You may like to boot up Your NAS with old disk and check if everything works. If You didn't boot it up ever, then You need to do it with HD that is less than 2TB, even old 40GB SATA will do, we just need to initialize it with tools that were shipped with the NAS (or newer ones available on Zyxel site to download. WARNING! MUST BE FOR NSA-210, even if program is the same, newest versions for other models will just not work correctly, even if it appears so).

So init the old HD in NAS with newest firmware so it boots up, You can start setting it up but it's recommended not to make any shares and users yet, let it have admin only (root for shares). You should use as little space HD as possible because clonning it (if You decide to do so) will take a while. If You didn't put 4.41 version of firmware during disk inition (in disk init wizard) then download it now from Zyxel site and update the firmware with web interface. It's important to have newest one due to 4TB disk support, even though You may have trouble making it work at all.

Turn off NAS and take out old HD, use USB->SATA bridge or 2.5 inch USB case (in case You used this size to init Your Zyxel - yes it's possible to put in 2.5 inch HD, it just needs SATA sockets), or connect it inside Your desktop if You have one.

Now the things can get tricky. But You will need linux anyway, clonezilla itself will MOST LIKELY NOT help You, even though it may appear that Your NAS read it.

Easiest way to move firmware data is to just clone old HD onto new HD either using device->device setup with clonezilla or device->image then image->device setup. It's recommended to copy whole harddrive. WARNING! MAKE SURE TO UNCHECK PARTITION RESIZING!!! If You fail in any of steps after this You need to start over doing this.

Alternative way is to make Linux read Your partitions and copy the data on old HD to Your home directory. WARNING! BOTH PARTITIONS MUST BE COPPIED IN SEPARATE DIRECTORIES AND ALL HIDDEN FILES, AND FILE PERMISSIONS MUST BE PRESERVED (all, even when groups/users IDs don't exists on Your linux box/virtualbox).

Access Your old HD and new HD with Linux x64 box or virtual guest machine. In case of using USB->SATA adapters and VirtualBox make sure You don't use USB 3.0 connection, it will fail to read the drive even when 1.1 mode is set in VB - just use 2.0 cable or connect to USB 2.0/ USB 2.0 hub. You need to run terminal as root then type: fdisk -l and using Your sharp eyes and brain to distinguish between disk names like /dev/sda, /dev/sdb, /dev/sdc etc. Now if You already know which one is Your old HD then note down starting and ending point of first partition and starting point of 2nd partition. If You used clonezilla You most likely have same structure on Your new HD. You may already know what I want You to do but STOP, DON'T CREATE ANY PARTITIONS ON NEW HD (or alter them when cloned).

We now need gdisk or parted, because we need to make MBR into GPT partition table on new HD - Yes new firmware reads it all right. Both solutions are valid but I prefer parted myself more, even though it's most likely harder to use but at the same time I know what I'm doing because all things are done manually. Doesn't matter if You cloned Your HD or are starting on fresh HD, Your data will be intact if You DON'T do any mistake here because basically what we are editing is main partition table and XFS file system can be resized pretty nicely.

So now run parted on Your new HD. For example parted /dev/sdc (if sdc is Your new HD THAT IS!). Type in print and confirm that You have cloned partitions or empty table. Now type in mklable gpt. If You had cloned HDs it will give warning information, answer Yes. Now we need to switch to sectors as the measuring unit, type in unit s. Now check Your noted start and end sectors from original disk back when You used fdisk. For me they were: partition 1 - Start: 63, End: 1028159, partition 2 - Start: 1028160, End: not important. Now I would type mkpart primary ext3 63 1028159. Ignore the warning. Now type print and confirm that start and end sector matches Your previously noted range. Also take a look at reported number of sectors by the disk. This is kind of importand because You can't go over that range when creating second partition for data, in fact, You must stay away a bit from the end due to one sector being used for GPT partition table backup and it doesn't exactly means it's the last. Let's make 2nd partition, I typed mkpart primary xfs 1028160 7814033472, and again ignore the warning. I got away by like 2000 or 20000 sectors away, don't remember, but it's not that important if it's that away from the end of HD. You can also name both partiton, first should be named firmware, second is second. Not sure if it's necessary, but I did that just incase.

quit parted.

If You cloned image then You can skip this place. If You didn't then You now need to make ext3 file system on first partition and xfs on 2nd partition. You can search the internet on how to do that. No special settings need to be used here, just make sure they are the same size as place alligned for partition. After making file systems You need to copy all previously backed up data from old HD to new HD or directly copy if You have both HDs connected. WARNING! ALL HIDDEN FILES, AND FILE PERMISSIONS MUST BE PRESERVED. WARNING2! Be sure that Your XFS partition has disabled lazy-count, mount the XFS partition and run xfs_info on the mountpoint. If it has enabled lazy-count then You must umount it and run command that looked like this in my example: xfs_admin -c 0 /dev/md127. I have suspision that if You don't disable it You may run into the same problem as me, disk full messages after writing ~100GB to the HD.

This step is for people that used clonezilla previously. Now You need to mount XFS file system. You need to add -t xfs in the line since system may not detect it automatically. Now let's resize it by typing: xfs_growfs mount_point. It may take a while in case of big HD so go make Yourself a cup of tea or coffee. :)

After this confirm that all files are still visible on both partitions and umount them.

Now You basically should be ready to put Your new HD into Zyxel? Go on and try it... You will be susrpised that all this work is NOT YET ENOUGH! Why? Well You may have noticed something when running fdisk -l on old HD. There is some additional information under old disk. Yes, that's right. A RAID-Linear volume. Why the hell would the use RAID in one HD Zyxel? No idea, but I think it's to avoid errors when booting with USB drive connected, or to show more volumes if there are any (most likely it's possible), because Zyxel doesn't look for sdXY when booting, it searches for MD0 only for data storage.

How the heck then we make RAID stuff. Especially that we just made partitions and checked that data is intact. Well, it won't hurt Your already existing partitions if You do everything RAID, I mean right. :)

First we would like to read all RAID stuff on old HD, by typing mdadm --detail /dev/mdX where X is the thing that fdisk -l reported for old HD. For example my output looked like this:

[root@localhost media]# mdadm --detail /dev/md127

/dev/md127:

        Version : 0.90

  Creation Time : Tue Apr 22 18:36:51 2014

     Raid Level : linear

     Array Size : 155774208 (148.56 GiB 159.51 GB)

   Raid Devices : 1

  Total Devices : 1

Preferred Minor : 127

    Persistence : Superblock is persistent

    Update Time : Thu Apr 24 11:50:17 2014

          State : clean 

 Active Devices : 1

Working Devices : 1

 Failed Devices : 0

  Spare Devices : 0

       Rounding : 64K

           UUID : 8296b7ff:c19f6ddc:7c88728c:d0f54ce9

         Events : 0.29

    Number   Major   Minor   RaidDevice State

       0       8       18        0      active sync   /dev/sdb2

So we now need to create similar thing for new HD, having in mind that most of the info must match and we need to use MDx that is not yet used. For this I ran this command: 

mdadm --create --verbose /dev/md0 --level=linear --raid-devices=1 --rounding=64K /dev/sdc2 --metadata=0.9 --force

It may be important to use 0.9 version metadata otherwise Zyxel may fail to see the volume. After running this command You are basically ready to put Your new HD into Zyxel. What is left is to loosly compare permissions being the same as the old HD ones by looking at few files/directories on both drives. ls -la will as well tell You if new HD consist of hidden directories/files and symlinks.

After this You may try to put Your HD into Zyxel NSA-210 and You should see volume of 3.64TB or similar size in case of 4TB drive. That is Your new data storage. If for some reason Zyxel appears unstable, like adding shares doesn't work or it can't shutdown, then reconnect it to Your linux box and run xfs_check or xfs_repair. Also I noticed that I needed to reboot NAS after adding a share, even though it already existed when I rechecked with my Linux box. So bigger HD may still have some problems with this old device, but it's yet up to test... Finally my 4TB Seagate 4000DM000 has nesting place to rest in. :)

So basically what we done was:

1. Created GPT on new HD (instead of MBR table as on old one),

2. Restored parition structure on new HD,

3. Resized data partition to cover whole free disk area,

4. Coppied all data from old HD to new HD including hidden files and permissions (or cloned it with clonezilla),

5. Recreated valid RAID-linear array for second partition,

6. Confirmed everything is ready to boot and run on Zyxel NSA-210.

I hope I helped. :)

~~~~

Mr_KrzYch00 ( http://www.4my.eu )