[ZEPPELIN-1787] Add an example of Flink Notebook

[AlexanderShoshin]

What is this PR for?

This PR will add an example of batch processing with Flink to Zeppelin tutorial notebooks. There are no any Flink notebooks in the tutorial at the moment.

What type of PR is it?

Improvement

What is the Jira issue?

ZEPPELIN-1787

How should this be tested?

You should open Using Flink for batch processing notebook from the Zeppelin Tutorial folder and run all paragraphs one by one

Questions:

• Does the licenses files need update? - no
• Is there breaking changes for older versions? - no
• Does this needs documentation? - no

[tae-jun]

Link for the note on ZeppelinHub

For convenience :)

@AlexanderShoshin Is this link right?

[AlexanderShoshin]

Yes, but ZeppelinHub does not show all the paragraphs. I will try to find out the reason.

[AhyoungRyu]

@AlexanderShoshin Thanks for your contribution!
While I was just quickly looking through this example note in my Zeppelin, couple of things were noticed.

• AFAIK, OSX doesn't have wget by default so the OSX users might need to install by themselves. So I would suggest you to use curl instead of wget to download datasets. (curl is built-in command I guess)

• I saw you set the location of dataset to tmp/. It can occur sth link "permission denied" error for that dir in Zeppelin like below.

Caused by: org.apache.flink.runtime.JobException: 
Creating the input splits caused an error: 
File /tmp/flights98.csv does not exist or the user running Flink ('ahyoungryu') has insufficient permissions to access it.

[zjffdu]

How about refer this note in flink.md ?

[AlexanderShoshin]

@AhyoungRyu, thank you for your notes!

• I've changed wget to curl as you suggested.
• As for the error message with "permission denied" I think it is because %sh paragraphs with data download instructions did not finish correct. Each paragraph downloads for about 70 MB of data and unpack it then. They might have finished by timeout. In this case you don't have the /tmp/flights98.csv file at all. A added a recomendation to the notebook to increase shell.command.timeout.millisecs setting. It helped in my case.
• We can't store data sets in "home" folder because %sh and %flink may be run by different users. So they may have different home folders. The /tmp/ folder is a common folder which normally can be accessed by each user. For the case of limited access I've added chmod 666 /tmp/flights<YY>.csv command for each csv set. Maybe there is a better solution for this issue. I will be glad to receive any suggestions :)

[AlexanderShoshin]

@tae-jun, I found that ZeppelinHub can't display new notebooks correctly. It is because the note.json structure was changed after ZEPPELIN-212 was merged (two weeks ago). Now notebook has a results attribute (instead of result) to store paragraph results and it seems that ZeppelinHub can't see it.

[AlexanderShoshin]

@zjffdu, should we do this in current PR or make a new issue for the documentation improving?

[AhyoungRyu]

Thank you for the such precise explanation. Tested again and it works well!
BTW some markdown paragraphs are not shown like below, so i didn't noticed that there are some description in here(I need to click "show editor" to check what it contains).

It would be better to show the other description paragraph's result by default like first paragraph does :)

[AlexanderShoshin]

Oh, it should not looks like this. It might be another problem of new notebook json structure. I will correct this.

[AlexanderShoshin]

I've converted notebook to 0.6.2 format. So it should be displayed correctly now.
ZeppelinHub vew also works.

[AhyoungRyu]

Thanks for your quick update!
Since #1780 is trying to update the all tutorial notes' format to the latest one, having older format of note can't be best solution I think. Not sure, but as you said, I assume there is another problem \w new note json format itself.(If so, we need to fix this. But in another PR :D ) #1780 also has same problem as I left a comment in there.

[AhyoungRyu]

@AlexanderShoshin Actually it was my bad. I checked again and this commit is perfectly working. Please ignore my last comment and sorry for the confusion.

[bzz]

Is there any other feedback, or was everything addressed and we shall merge it now?

Saw @zjffdu 's

How about refer this note in flink.md ?

@AlexanderShoshin do you think it's worth to update the interpreter documentation as well under ./docs, to mention this example notebook?

Other than that, looks great to me!

[AlexanderShoshin]

I am confused a bit :)
@AhyoungRyu, do I need to drop my last commit or not?

[AhyoungRyu]

@AlexanderShoshin Right that would be better. So sorry about that 😭

[AlexanderShoshin]

@bzz, I am not sure. We already have a Word Count example there to describe Flink usage. Where should we place a link to this example?

[rawkintrevo]

@AlexanderShoshin I think what @zjffdu and @bzz meant was that you could possibly talk about this notebook in some where like docs/interpreter/flink.md which would make it for new users to find it.

[AhyoungRyu]

@AlexanderShoshin Seems the below two things need to be done before merge

1. revert the last commit 9013620

You can simply do it in your local by

please make sure you are on branch ZEPPELIN-1787 first 
$ git checkout ZEPPELIN-1787

and check the last commit is "convert notebook to 0.6.2 format" or not
$ git log 

if so,
$ git reset HEAD^ --hard 

check again it's converted well, it should be pointing this commit: "add download instruction, change "wget" to "curl""
$ git log

push with forced updated! 
$ git push origin -f ZEPPELIN-1787

2. add Flink docs link to the tutorial as @rawkintrevo said.

Please feel free to ping me if you need any help!

[AlexanderShoshin]

Sorry, I was not able to work on this issue during last weeks.
@AhyoungRyu, I will make the corrections soon.
Thanks

[AlexanderShoshin]

I've added a new commit to convert the notebook to 0.7.0 format because I found that 9013620 commit has several conflicts in json-file.
I've also merged master to avoid conflicts in flink.md.

[AhyoungRyu]

Not sure adding link localhost:8080 is good idea, since there might be some ppl using another host & port. "You can try Flink usage example located under Zeppelin Tutorial folder." will be enough I guess :)

[AhyoungRyu]

@AlexanderShoshin Checked again and it looks nice. Except the minor suggestion, LGTM. Thanks for your effort and the awesome tutorial note!

[AlexanderShoshin]

@AhyoungRyu, thanks for the note. I've removed an url to the notebook.

[AhyoungRyu]

@AlexanderShoshin Thanks! Will Merge into both master and branch-0.7 if there are no more comments on this.