Support reading AppleDict binary format

[soshial]

There is python code I found on Gist. It almost always works (but I had small hiccups on custom dictionaries), but we may fix it. Can you add the code to pyglossary?

[vinniec]

Has the feature been implemented?
I tried the script and apart from a couple of errors that I just reported, it seems to work (I tried with the Italian-German dictionary downloaded from this page: http://clasquin-johnson.co.za/michel/mac-os-x-dictionaries/south-european-languages/italian.html )
Well, it doesn't really work with another dictionary that I was interested in, so I wanted to see if your software could already do something about it.

[ilius]

No, we haven't done any work on this.

[vinniec]

I did a test with that script to convert the mentioned dictionary first to csv and then with pyglossary to an ifo of stardict and it works.
This is the code, I'm not very good so it's quite ugly.

#! /usr/bin/env python3

from struct import unpack
from zlib import decompress
import re

filename = './Body.data'
f = open(filename, 'rb')

def gen_entry():
	f.seek(0x40)
	limit = 0x40 + unpack('i', f.read(4))[0]
	f.seek(0x60)
	while f.tell()<limit:
		byte = f.read(4)
		##alternative for buf, sz is calculated
		# ~ flag = f.tell()
		# ~ sz = 0
		# ~ while True:
			# ~ zipp = f.read(sz)
			# ~ try:
				# ~ ##print(zipp)
				# ~ input(zipp.decode('utf-8'))
				# ~ buf = decompress(zipp[8:])
				# ~ ##print(buf)
				# ~ break
			# ~ except:
				# ~ print(sz)
				# ~ f.seek(flag)
				# ~ sz = sz+1
		sz, = unpack('i', byte)	#valore  troppo grande
		buf = decompress(f.read(sz)[8:])
			
		pos = 0
		while pos < len(buf):
			chunksize, = unpack('i', buf[pos:pos+4])
			entry = buf[pos:pos+chunksize].decode('utf-8')
			title = re.search('d:title="(.*?)"', entry).group(1)
			yield title, entry, sz, pos, chunksize
			pos += chunksize

if __name__ == "__main__":
	
	diz = {}
	for word, definition, sz, pos, chunk in gen_entry():
		defin = re.findall('<pre>(.*?)</pre>', definition, re.S)
		diz.update({word: defin})

	# ~ from pprint import pprint
	# ~ pprint(diz)
	
	with open('dizpy.csv', 'w') as f:
		for k, v in diz.items():
			ls = [k]+v
			#substitute " with '
			ls = [c.translate({34:39}) for c in ls]
			ls = ",".join([f'"{c}"' for c in ls])
			f.write(f"{ls}\n")

[ilius]

Any ideas how can we get the number of entries at the beginning, without reading the whole file?
We need that for PyGlossary plugin.

[ilius]

Actually lacking the number of entries will only disable the progress bar, but it would work.

I have done it on this branch:
https://github.com/ilius/pyglossary/tree/appledict-bin

Please test.

[vinniec]

Ok, I try

[vinniec]

Okay, I wanted to test it, but I couldn't figure out how to do it.
I saw that you added the source in ..."/plugins/appledict-bin.py" but in the gtk interface I don't see the apple input format.
So I tried to do --help but it wasn't mentioned in the help either.
Then I thought that you had implemented it without documenting it yet and I tried with this command line: "./main.py ./Body.data /tmp/diz.csv" but I get this error:
Traceback (most recent call last): File "./main.py", line 402, in <module> convertOptions=convertOptions, File "/home/vinnie/software/pyglossary/pyglossary-master/ui/ui_cmd.py", line 283, in run **convertOptions File "/home/vinnie/software/pyglossary/pyglossary-master/pyglossary/glossary.py", line 1006, in convert **readOptions File "/home/vinnie/software/pyglossary/pyglossary-master/pyglossary/glossary.py", line 595, in read log.error("Unknown extension \"%s\" for read support!", ext) UnboundLocalError: local variable 'ext' referenced before assignment
Sorry if I'm not smart enough to understand it alone.

[ilius]

I think you are using the master branch.
Use this link to download the source code for that branch:
https://github.com/ilius/pyglossary/archive/appledict-bin.zip

[vinniec]

Okay, so far, I've done two tests.
Obviously with the file I linked before it works.
I searched the network for other files (not easy) and I found another ( https://github.com/nathanReitinger/Blacks8-Mac-Dictionary/blob/master/dictionary(download)/%E2%80%9CBlacks_8_%5Bbeta%5D%E2%80%9D.dictionary.zip )
With this does not work, at the test I think the problem is that the definition uses a <div> instead of a <pre>
Example of definition:
<d:entry xmlns:d="http://www.apple.com/DTDs/DictionaryService-1.0.rng" id="MILITARY BOUNTY LAND" d:title="MILITARY BOUNTY LAND"> <div d:priority="1"><h1>MILITARY BOUNTY LAND</h1></div> <div> military bounty land.Land offered to members of the military as a reward for services. See donation land, bounty land under LAND; BOUNTY-LAND WARRANT. [Cases: Bounties 1.C.J.S. Rewards and Bounties §§ 44–45.] </div> </d:entry>

[ilius]

I added div tag.
Please test again (same link)
Also let me know if there is any other tag.
Thanks