Skip to content
/ fq Public
forked from wader/fq

Commit 49bd7c2

Browse files
fiatjaffiatjaf
committed
negentropy: add format.
1 parent f150085 commit 49bd7c2

File tree

11 files changed

+1661
-9
lines changed

11 files changed

+1661
-9
lines changed

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -124,6 +124,7 @@ mpeg_pes_packet,
124124
mpeg_spu,
125125
mpeg_ts,
126126
[msgpack](doc/formats.md#msgpack),
127+
[negentropy](doc/formats.md#negentropy),
127128
[nes](doc/formats.md#nes),
128129
ogg,
129130
ogg_page,

‎doc/formats.md‎

Lines changed: 41 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,7 @@
9696
|`mpeg_spu` |Sub&nbsp;Picture&nbsp;Unit&nbsp;(DVD&nbsp;subtitle) |<sub></sub>|
9797
|`mpeg_ts` |MPEG&nbsp;Transport&nbsp;Stream |<sub></sub>|
9898
|[`msgpack`](#msgpack) |MessagePack |<sub></sub>|
99+
|[`negentropy`](#negentropy) |Negentropy&nbsp;message |<sub></sub>|
99100
|[`nes`](#nes) |iNES/NES&nbsp;2.0&nbsp;cartridge&nbsp;ROM&nbsp;format |<sub></sub>|
100101
|`ogg` |OGG&nbsp;file |<sub>`ogg_page` `vorbis_packet` `opus_packet` `flac_metadatablock` `flac_frame`</sub>|
101102
|`ogg_page` |OGG&nbsp;page |<sub></sub>|
@@ -140,7 +141,7 @@
140141
|`ip_packet` |Group |<sub>`icmp` `icmpv6` `tcp_segment` `udp_datagram`</sub>|
141142
|`link_frame` |Group |<sub>`bsd_loopback_frame` `ether8023_frame` `ipv4_packet` `ipv6_packet` `sll2_packet` `sll_packet`</sub>|
142143
|`mp3_frame_tags` |Group |<sub>`mp3_frame_vbri` `mp3_frame_xing`</sub>|
143-
|`probe` |Group |<sub>`adts` `aiff` `apple_bookmark` `ar` `avi` `avro_ocf` `bitcoin_blkdat` `bplist` `bzip2` `caff` `elf` `fit` `flac` `gif` `gzip` `html` `jp2c` `jpeg` `json` `jsonl` `leveldb_table` `luajit` `macho` `macho_fat` `matroska` `moc3` `mp3` `mp4` `mpeg_ts` `nes` `ogg` `opentimestamps` `pcap` `pcapng` `png` `tar` `tiff` `toml` `tzif` `tzx` `wasm` `wav` `webp` `xml` `yaml` `zip`</sub>|
144+
|`probe` |Group |<sub>`adts` `aiff` `apple_bookmark` `ar` `avi` `avro_ocf` `bitcoin_blkdat` `bplist` `bzip2` `caff` `elf` `fit` `flac` `gif` `gzip` `html` `jp2c` `jpeg` `json` `jsonl` `leveldb_table` `luajit` `macho` `macho_fat` `matroska` `midi` `moc3` `mp3` `mp4` `mpeg_ts` `nes` `ogg` `opentimestamps` `pcap` `pcapng` `png` `tar` `tiff` `toml` `tzif` `tzx` `wasm` `wav` `webp` `xml` `yaml` `zip`</sub>|
144145
|`tcp_stream` |Group |<sub>`dns_tcp` `rtmp` `tls`</sub>|
145146
|`udp_payload` |Group |<sub>`dns`</sub>|
146147

@@ -873,37 +874,41 @@ Standard MIDI file.
873874

874875
### Notes
875876

876-
1. Only supports the MIDI 1.0 specification.
877-
2. Does only basic validation on the MIDI data.
877+
1. Only supports the MIDI 1.0 MIDI file specification.
878+
2. Only supports _MThd_ and _MTrk_ chunks.
879+
3. Does only basic validation on the MIDI data.
878880

879881
### Sample queries
880882

881883
1. Extract the track names from a MIDI file
882884
```
883-
fq -d midi -d midi '.. | select(.event=="Track Name")? | "\(.name)"' twinkle.mid
885+
fq -d midi -d midi '.. | select(.event=="track_name")? | "\(.track_name)"' midi/twinkle.mid
884886
```
885887

886888
2. Extract the tempo changes from a MIDI file
887889
```
888-
fq -d midi '.. | select(.event=="Tempo")?.tempo' twinkle.mid
890+
fq -d midi '.. | select(.event=="tempo")?.tempo' midi/twinkle.mid
889891
```
890892

891893
3. Extract the key changes from a MIDI file
892894
```
893-
fq -d midi '.. | select(.event=="Key Signature")?.key' key-signatures.mid
895+
fq -d midi '.. | select(.event=="key_signature")?.key_signature' midi/twinkle.mid
894896
```
895897

896-
4. Extract NoteOn and NoteOff events:
898+
4. Extract NoteOn events:
897899
```
898-
fq -d midi 'grep_by(.event=="Note On" or .event=="Note Off") | "\(.event) \(.time.tick) \(.note)"' twinkle.mid
900+
fq -d midi 'grep_by(.event=="note_on") | [.time.tick, .note_on.note] | join(" ")' midi/twinkle.mid
899901
```
900902

901903
### Authors
902-
- transcriptaze.development@gmail.com
904+
- [transcriptaze](https://github.com/transcriptaze)
903905

904906
### References
905907

906908
1. [The Complete MIDI 1.0 Detailed Specification](https://www.midi.org/specifications/item/the-midi-1-0-specification)
909+
2. [Standard MIDI Files](https://midi.org/standard-midi-files)
910+
3. [Standard MIDI File (SMF) Format](http://midi.teragonaudio.com/tech/midifile.htm)
911+
4. [MIDI Files Specification](http://www.somascape.org/midi/tech/mfile.html)
907912

908913
## moc3
909914
MOC3 file.
@@ -1012,6 +1017,33 @@ $ fq -d msgpack torepr file.msgpack
10121017
### References
10131018
- https://github.com/msgpack/msgpack/blob/master/spec.md
10141019

1020+
## negentropy
1021+
Negentropy message.
1022+
1023+
### View a full Negentropy message
1024+
1025+
```
1026+
$ fq -d negentropy dd file
1027+
```
1028+
1029+
### Check how many ranges the message has and how many of those are of 'fingerprint' mode
1030+
1031+
```
1032+
$ fq -d negentropy '.bounds | length as $total | map(select(.mode == "fingerprint")) | length | {$total, fingerprint: .}' message
1033+
```
1034+
1035+
### Check get all ids in all idlists
1036+
1037+
```
1038+
$ fq -d negentropy '.bounds | map(select(.mode == "idlist") | .idlist | .ids) | flatten' message
1039+
```
1040+
1041+
### Authors
1042+
- fiatjaf, https://fiatjaf.com
1043+
1044+
### References
1045+
- https://github.com/hoytech/negentropy
1046+
10151047
## nes
10161048
iNES/NES 2.0 cartridge ROM format.
10171049

‎format/all/all.fqtest‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -141,6 +141,7 @@ mpeg_pes_packet MPEG Packetized elementary stream packet
141141
mpeg_spu Sub Picture Unit (DVD subtitle)
142142
mpeg_ts MPEG Transport Stream
143143
msgpack MessagePack
144+
negentropy Negentropy message
144145
nes iNES/NES 2.0 cartridge ROM format
145146
ogg OGG file
146147
ogg_page OGG page

‎format/all/all.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@ import (
4242
_ "github.com/wader/fq/format/mp4"
4343
_ "github.com/wader/fq/format/mpeg"
4444
_ "github.com/wader/fq/format/msgpack"
45+
_ "github.com/wader/fq/format/negentropy"
4546
_ "github.com/wader/fq/format/nes"
4647
_ "github.com/wader/fq/format/ogg"
4748
_ "github.com/wader/fq/format/opentimestamps"

‎format/format.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -148,6 +148,7 @@ var (
148148
MPEG_TS = &decode.Group{Name: "mpeg_ts"}
149149
MPES_PES = &decode.Group{Name: "mpeg_pes"}
150150
MsgPack = &decode.Group{Name: "msgpack"}
151+
Negentropy = &decode.Group{Name: "negentropy"}
151152
NES = &decode.Group{Name: "nes"}
152153
Ogg = &decode.Group{Name: "ogg"}
153154
Ogg_Page = &decode.Group{Name: "ogg_page"}

‎format/negentropy/negentropy.go‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
package negentropy
2+
3+
// https://github.com/hoytech/negentropy
4+
5+
import (
6+
"embed"
7+
"math"
8+
"time"
9+
10+
"github.com/wader/fq/format"
11+
"github.com/wader/fq/pkg/decode"
12+
"github.com/wader/fq/pkg/interp"
13+
"github.com/wader/fq/pkg/scalar"
14+
)
15+
16+
//go:embed negentropy.md
17+
var negFS embed.FS
18+
19+
func init() {
20+
interp.RegisterFormat(
21+
format.Negentropy,
22+
&decode.Format{
23+
Description: "Negentropy message",
24+
DecodeFn: decodeNegentropyMessage,
25+
Groups: []*decode.Group{},
26+
})
27+
interp.RegisterFS(negFS)
28+
}
29+
30+
const (
31+
version = 0x61
32+
fingerprintSize = 16
33+
)
34+
35+
var modeMapper = scalar.SintMapSymStr{
36+
0: "skip",
37+
1: "fingerprint",
38+
2: "idlist",
39+
}
40+
41+
type timestampTranslator struct{ last time.Time }
42+
43+
func (tt *timestampTranslator) MapSint(s scalar.Sint) (scalar.Sint, error) {
44+
if s.Actual == 0 {
45+
s.Description = "infinity"
46+
s.Actual = -1
47+
tt.last = time.Unix(math.MaxInt64, 0)
48+
return s, nil
49+
} else {
50+
s.Actual--
51+
52+
timestamp := tt.last.Add(time.Second * time.Duration(s.Actual-1))
53+
tt.last = timestamp
54+
s.Description = timestamp.Format(time.DateTime)
55+
return s, nil
56+
}
57+
}
58+
59+
func decodeNegentropyMessage(d *decode.D) any {
60+
tt := &timestampTranslator{last: time.Unix(0, 0)}
61+
62+
d.Endian = decode.BigEndian
63+
64+
v := d.FieldU8("version")
65+
if v != version {
66+
d.Fatalf("unexpected version %d (expected %d), is this really a negentropy message?", v, version)
67+
}
68+
69+
d.FieldStructArrayLoop("bounds", "bound", d.NotEnd, func(d *decode.D) {
70+
d.FieldSintFn("timestamp_delta", decodeVarInt, tt)
71+
72+
size := d.FieldSintFn("id_prefix_size", decodeVarInt)
73+
if size > 32 {
74+
d.Fatalf("unexpected id prefix size bigger than 32: %d", size)
75+
}
76+
if size > 0 {
77+
d.FieldRawLen("id_prefix", size*8, scalar.RawHex)
78+
}
79+
80+
mode := d.FieldSintFn("mode", decodeVarInt, modeMapper)
81+
switch mode {
82+
case 0:
83+
return
84+
case 1:
85+
d.FieldRawLen("fingerprint", fingerprintSize*8, scalar.RawHex)
86+
return
87+
case 2:
88+
d.FieldStruct("idlist", func(d *decode.D) {
89+
num := d.FieldSintFn("num", decodeVarInt)
90+
d.FieldArray("ids", func(d *decode.D) {
91+
for i := 0; i < int(num); i++ {
92+
d.FieldRawLen("id", 32*8, scalar.RawHex)
93+
}
94+
})
95+
})
96+
default:
97+
d.Fatalf("unexpected mode %d", mode)
98+
}
99+
})
100+
101+
return nil
102+
}
103+
104+
func decodeVarInt(d *decode.D) int64 {
105+
var res int64 = 0
106+
107+
for {
108+
b := d.U8()
109+
res = (res << 7) | (int64(b) & 127)
110+
if (b & 128) == 0 {
111+
break
112+
}
113+
}
114+
115+
return res
116+
}

‎format/negentropy/negentropy.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
### View a full Negentropy message
2+
3+
```
4+
$ fq -d negentropy dd file
5+
```
6+
7+
### Check how many ranges the message has and how many of those are of 'fingerprint' mode
8+
9+
```
10+
$ fq -d negentropy '.bounds | length as $total | map(select(.mode == "fingerprint")) | length | {$total, fingerprint: .}' message
11+
```
12+
13+
### Check get all ids in all idlists
14+
15+
```
16+
$ fq -d negentropy '.bounds | map(select(.mode == "idlist") | .idlist | .ids) | flatten' message
17+
```
18+
19+
### Authors
20+
- fiatjaf, https://fiatjaf.com
21+
22+
### References
23+
- https://github.com/hoytech/negentropy

‎format/negentropy/testdata/bigmessagewitheverything‎

4.36 KB
Binary file not shown.

0 commit comments

Comments
 (0)