SCINET-GEOSPATIAL-RESEARCH-WG/2020-08-28_SCINet-Geospatial-WG_Workshop-Session4-Reproducibility-with-Git-Github-Conda-Docker-Singularity_CHAT.txt at master · kerriegeil/SCINET-GEOSPATIAL-RESEARCH-WG · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
00:14:07	John Humphreys:	Hello!
00:14:20	Claire Baffaut:	Hello!
00:15:25	Rowan Gaffney:	Hello Everyone, Todays tutorial is on the website at: https://kerriegeil.github.io/SCINET-GEOSPATIAL-RESEARCH-WG/content/4-Session4-reproducibility-tools.html
00:16:40	Rowan Gaffney:	Hello Everyone, Todays tutorial is on the website at: https://kerriegeil.github.io/SCINET-GEOSPATIAL-RESEARCH-WG/content/4-Session4-reproducibility-tools.html
00:17:16	Rowan Gaffney:	If you would like to login with SSH, you can follow the instructions from session2 here: https://kerriegeil.github.io/SCINET-GEOSPATIAL-RESEARCH-WG/content/2-Session2-intro-to-ceres.html
00:18:59	Lina Castano-Duque:	Please make the cmd font bigger
00:20:42	Rowan Gaffney:	Lina - hope that is workable for you
00:21:03	Lina Castano-Duque:	It is a little better, thks
00:26:46	Danielle Lemay:	Are there any USDA policies relevant to Github or are we allowed to upload our USDA data and scripts to public repositories?
00:27:36	Rowan Gaffney:	Danielle, I will ask Kerrie to address this question in a moment
00:33:48	Yasasvy Nanyam:	Both GitHub and GitLab now offer private repos for no extra charge.
00:35:01	Rowan Gaffney:	Yash - thanks for the update.
00:37:47	Jeremy Edwards:	Are you going to cover some philosophy on when/how often to do a git commit?  I was taught to do them often and make each "atomic" meaning not to combine changes for unrelated reasons on a single commit.  I know that there are other strategies though.
00:38:23	Hye-Seon Kim:	can you copy your repo name?
00:38:28	Danielle Lemay:	I’m not finding you. Is it public?
00:38:39	Jennifer Chang:	https://github.com/kerriegeil
00:39:35	Danielle Lemay:	Thank you!
00:40:12	Karla Franco Melendez:	Do we have to create a repository to participate?
00:40:49	Rowan Gaffney:	Karla - you need a GitHub account, and then need to fork Kerries repository
00:40:49	Zeya Xue:	I saw this is a forked repo already. When I fork from Kerrie’s repo, I am getting her version instead of the octacat’s version, correct?
00:41:05	Rowan Gaffney:	Zeya - that is correct I beleive
00:41:08	Hye-Seon Kim:	Thanks
00:41:50	Rowan Gaffney:	Jeremy - good points, if we have some time we will try to discuss those points in more detail
00:41:50	Claire Baffaut:	IT seems that I did not have to type my own user name.
00:42:02	David Horvath:	I can'tm unmute
00:42:11	Lina Castano-Duque:	Mine worked, it did not asked either for my user name selection
00:42:51	David Horvath:	I just logged into GitHub but can't find your repository'
00:43:23	Susan Stillman:	https://github.com/kerriegeil
00:43:25	Rowan Gaffney:	David, Kerries repo is here: https://github.com/kerriegeil/Spoon-Knife
00:43:45	Rowan Gaffney:	you can go their, and fork it to your own repository
00:45:50	Max Feldman:	I received the error:
00:45:51	Elizabeth Chin:	I got this error: Warning: Permanently added 'github.com' (RSA) to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.
00:45:51	Max Feldman:	Permission denied (publickey).
fatal: Could not read from remote repository.
00:46:12	Sean Kearney:	FYI on my machine you can't paste into Powershell with ctr+V, but right clicking works.
00:46:32	Melanie Kammerer:	It worked fine for me.
00:46:32	Mackenzie Tietjen:	yes
00:46:35	ElHamidi.Hay:	yes
00:46:36	Sean Kearney:	yes
00:46:36	Lina Castano-Duque:	mine worked
00:46:36	Yang Yang:	yes
00:46:43	John Humphreys:	worked here
00:46:45	Zhanyou Xu:	How can I see the http address? I missed
00:46:50	HUIHUI ZHANG:	how to paste the clone link to powershell?
00:47:00	Melanie Kammerer:	right click
00:47:11	Max Feldman:	I can verify I am cloning from my repo
00:47:35	Rowan Gaffney:	Max - hmm ok, not sure why you are getting that issue
00:47:58	HUIHUI ZHANG:	right click didn't work
00:48:00	Rowan Gaffney:	HUIHUI, depends on your OS and shell, with powershell, I think a right click works something
00:48:05	Rowan Gaffney:	*sometimes
00:48:44	Claire Baffaut:	what does untracked mean?
00:49:02	Rowan Gaffney:	Claire - it means git is not "tracking" this file
00:49:18	Rowan Gaffney:	"git add" will add the file to the "tracked files"
00:50:41	Elizabeth Chin:	I suppose since it’s just git stuff and not HPC we can follow along on our local computers
00:51:49	Rowan Gaffney:	Elizabeth - excellent suggestion!
00:51:52	Rowan Gaffney:	thanks
00:51:52	Yang Yang:	I got this message: is it right?Your name and email address were configured automatically based
on your username and hostname. Please check that they are accurate.
You can suppress this message by setting them explicitly:

    git config --global user.name "Your Name"
    git config --global user.email you@example.com

After doing this, you may fix the identity used for this commit with:

    git commit --amend --reset-author

 3 files changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 yangyang1.txt
 create mode 100644 yangyang2.txt
 create mode 100644 yangyang3.txt
00:52:39	Rowan Gaffney:	Yang, I *think* those messages are ok
00:52:52	Yang Yang:	Cool, thanks Rowan
00:53:15	Rowan Gaffney:	I think those messages are about setting up some additional configurations for git (but not 100%) sure
00:53:53	Jennifer Chang:	Yes, Rowan is correct
00:54:06	Rowan Gaffney:	Thanks Jennifer!
01:03:14	Zhanyou Xu:	[zhanyou.xu@ceres home]$ mkdir my_new_repomkdir: cannot create directory ‘my_new_repo’: Permission denied
01:04:51	Rowan Gaffney:	Zhanyou, make sure you are in your $HOME directory, you can type: cd $HOME
01:05:00	Rowan Gaffney:	to move to your home directory
01:06:46	Max Feldman:	Is anyone else having the
01:07:32	Max Feldman:	Is anyone else having the following error? Permission denied (publickey). fatal: Could not read from remote repository.
01:08:49	Elizabeth Chin:	I am getting that error. It sounds like the ssh public key needs to be set up
01:08:51	Rowan Gaffney:	Max, I think Elizabeth Chen also had that issue.
01:08:51	Elizabeth Chin:	??
01:09:26	Zeya Xue:	If I don’t push some changes but I committed, can I still go back to those versions?
01:09:57	Rowan Gaffney:	Max and Elizabeth, I do not have any idea why you guys are having that issue...
01:10:59	Rowan Gaffney:	Zeya, the commit history are on your local commit history, so yes, you could "roll back" locally (but you may need to use the command line unless you have some other git software installed locally)
01:11:31	Zeya Xue:	Thanks!
01:16:28	Lina Castano-Duque:	can you show again and slower or type the code here about how to move to a computing noe
01:16:31	Lina Castano-Duque:	node
01:16:43	Rowan Gaffney:	Lina, you type "salloc"
01:17:22	Rowan Gaffney:	Siva - that makes sense now!
01:18:16	Elizabeth Chin:	what’s the difference between ‘source’ and ‘conda’ activate?
01:18:23	Rowan Gaffney:	Lina, did that work for you
01:18:38	Rowan Gaffney:	Elizabeth - will bring up your question to Kerrie
01:19:18	Zhanyou Xu:	obqueue
01:20:07	Yang Yang:	Can we add more packages later?
01:21:24	Elizabeth Chin:	It’s helpful. Thanks!
01:21:35	Rowan Gaffney:	Yang - yes you can install packages into environments after you create them
01:21:59	David Horvath:	target prefix. Aborting???
01:22:01	Zhanyou Xu:	do we have one conda ENV that has mots of the Python ML librayies? Then we do not to create our wown Env to take space resources.
01:22:03	Lina Castano-Duque:	does it take a long time to install pip? Mine has been at 73% for 1min
01:22:42	David Horvath:	base prefix. Aborting?>??  What am I doing wrong?
01:22:49	Melanie Kammerer:	Mine is stuck at pip as well 1%
01:23:34	David Horvath:	 conda create session3_env python=3.7 is the command I gave that resulted in the error
01:23:36	Lina Castano-Duque:	Alright, it took a while but is moving along now
01:24:30	Danielle Lemay:	David, you need —name before session3_env
01:25:35	Zhanyou Xu:	 A common machine learning (ML) ENV will be great! then we do not need to install our own libraries. Some of the libraires are different to install for me. I will vote for the common ENV
01:26:58	Rowan Gaffney:	Zhanyou - will definitely keep this request in mind and try to pursue this idea
01:27:17	Rowan Gaffney:	David - did Danielle suggestion work for you
01:27:48	Yasasvy Nanyam:	conda prefix -p <path to the directory> package1,package2,...
01:27:52	Elizabeth Chin:	if you give the environment a prefix like the project directory, then would you have to tell the condarc file to look for the env in that prefix? or will conda know automatically?
01:28:20	Rowan Gaffney:	Elizabeth - I am not sure, perhaps Yash can help with that question!
01:28:22	David Horvath:	just syntax and command failures. got it now.
01:28:31	Rowan Gaffney:	David - excellent!
01:28:40	Yang Yang:	Does this slow progress also occur when many people, say 40, are running their code in ceres?
01:28:48	Yang Yang:	Not yet...
01:29:03	David Horvath:	Just did python 3.7 do I need to add the others?
01:29:17	Rowan Gaffney:	David - that should work fine for this example
01:29:22	Zhanyou Xu:	There is error for me. i try to recreate it now
01:29:56	Elizabeth Chin:	Got it, thanks!
01:31:01	Dave Fleisher:	Working for me, just super slow
01:31:21	Zhanyou Xu:	[Errno 122] Disk quota exceeded: '/home/zhanyou.xu/.conda/pkgs/sortedcontainers-2.2.2-py_0.conda'[Errno 122] Disk quota exceeded: '/home/zhanyou.xu/.conda/pkgs/distributed-2.24.0-py37_0.conda'[Errno 122] Disk quota exceeded: '/home/zhanyou.xu/.conda/pkgs/tblib-1.7.0-py_0.conda'[Errno 122] Disk quota exceeded: '/home/zhanyou.xu/.conda/pkgs/intel-openmp-2020.2-254.conda'[Errno 122] Disk quota exceeded: '/home/zhanyou.xu/.conda/pkgs/partd-1.1.0-py_0.conda'[Errno 122] Disk quota exceeded: '/home/zhanyou.xu/.conda/pkgs/bokeh-2.2.0-py37_0.conda'
01:31:33	Zhanyou Xu:	what does these error message mean?
01:31:58	Hye-Seon Kim:	It's sill downloading and extracting packages.. how to stop? I should let it go..
01:32:20	Rowan Gaffney:	Hi Zhanyou, looks like you have hit the disk quota in the home directory (I *think* if you email the vrsc, they will expand you quota
01:32:26	Sivanandan Chudalayandi:	Zhanyou Xu: your home folder is full.
01:32:44	Zhanyou Xu:	how to enter ENV one created before?
01:32:48	Sivanandan Chudalayandi:	You can also do this on your project folder
01:33:01	Rowan Gaffney:	Good point Siva
01:34:49	Rowan Gaffney:	Hye-Seon, cntl+c should cancel the operation, but sometimes that will trip it up if you are mid command fyi
01:35:31	Zhanyou Xu:	list
01:37:06	Karla Franco Melendez:	What are the differences between env?
01:38:10	Rowan Gaffney:	Karla, environments are isolated from one another, so you can different sets of software installed in each environment
01:39:11	Melanie Kammerer:	Could someone clarify again when we should use 'source activate'? Just once at the beginning of each ssh or JupyterHub session?
01:39:35	Rowan Gaffney:	Melanie - will bring this up after we finish this section
01:39:51	Melanie Kammerer:	Ok, thanks
01:44:46	Sivanandan Chudalayandi:	https://stackoverflow.com/questions/49600611/python-anaconda-should-i-use-conda-activate-or-source-activate-in-linux
01:44:56	Melanie Kammerer:	Sorry- question wasn't clear. Just wanted to confirm that we need to type 'source activate' every time we log in
01:46:27	Rowan Gaffney:	oh sorry Melanie, yes you need to use "source activate" the first time you load the  miniconda module
01:46:44	Melanie Kammerer:	Ok, got it. Thanks!
01:46:49	Rowan Gaffney:	Great!
01:49:10	Elizabeth Chin:	can you have a conda environment with both python and r libraries? is it recommended to have separate env for R and Python stuff?
01:49:39	Sivanandan Chudalayandi:	are we moving on to session 5?
01:50:44	Kerrie Geil:	yes you can have r and python libraries in the same environment!
01:51:05	Elizabeth Chin:	nice. thanks!
01:51:18	Kerrie Geil:	you can separate out r and python environments is you want, but not sure it's "recommended" or anything
01:52:16	Kerrie Geil:	yes melanie, every single time
01:56:25	Lina Castano-Duque:	If I load a container in CERES, will I have access to the CERES installed software plus the software in the container at the same time?
01:57:42	Gerard Lazo:	Are there any kubernetes-style implementations that can be used on CERES?
01:58:52	Yasasvy Nanyam:	At the moment, we do not have kubernetes like implentations. We are looking at OpenStack but no ETA
01:59:02	Zhanyou Xu:	Will we all use same container by copy/paste the container from this tutorial? Can we use different containers every time?
02:01:11	Zhanyou Xu:	where can we get the container paramters?
02:03:51	Alicia Foxx:	Sorry if I missed this - how can you stop the conda environment code execution from before?
02:04:33	Gerard Lazo:	out-of-space
02:04:41	Hye-Seon Kim:	I got an error below
02:04:43	Hye-Seon Kim:	FATAL:   Image file already exists: "alpine_latest.sif" - will not overwrite
02:04:46	Kerrie Geil:	Alicia I'm not sure that you can- I wasn't able to. I just opened a new terminal and logged in again
02:05:02	Alicia Foxx:	Thank you, Kerrie
02:08:46	Zhanyou Xu:	WARN[0000] "/run/user/1727025670" directory set by $XDG_RUNTIME_DIR does not exist. Either create the directory or unset $XDG_RUNTIME_DIR.: stat /run/user/1727025670: no such file or directory: Trying to pull image in the event that it is a public image.FATAL:   While making image from oci registry: error fetching image to cache: unable to check if 185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321 exists in cache: failed to create temporary file: open /home/zhanyou.xu/.singularity/cache/oci-tmp/tmp_881154868: disk quota exceeded(zhanyou_alfalfa) [zhanyou.xu@ceres18-compute-24 ~]$
02:08:53	Zhanyou Xu:	what soes this mean?
02:09:10	Yasasvy Nanyam:	Zhanyou - Your home directory is full.
02:09:15	Kerrie Geil:	your home directory doesn't have enough space to build this container
02:09:25	Zhanyou Xu:	WOW!
02:10:04	Zhanyou Xu:	then we should I do in order to create a container?
02:10:41	Gerard Lazo:	I've run into a lot of out-of-space issues. What's the best way to request help from the VRSC to resolve this. Will the container use take up valuable disk space, or will links to images be minimized.
02:13:46	Kerrie Geil:	https://scinet.usda.gov/guide/singularity
02:14:26	Yasasvy Nanyam:	https://e.arsnet.usda.gov/sites/OCIO/scinet/accounts/SitePages/Project_Allocation_Request.aspx
02:16:11	Danielle Lemay:	Quick comment: if you have a software release with a Zenodo-created DOI, you can submit a 115 to ARIS.
02:16:21	Steven Schroeder:	you can also make use of the new /90daydata for extra space - but need to make sure you back it up somewhere else
02:16:32	Jennifer Chang:	Can change the singularity install location using 'SINGULARITY_CACHEDIR'... instructions here: https://www.hpc.iastate.edu/guides/containers
02:17:12	Jennifer Chang:	Thanks Yash! :D
02:19:05	Danielle Lemay:	THANK YOU!
02:19:20	Sivanandan Chudalayandi:	Thanks All!!
02:19:21	Elizabeth Chin:	Thank you! :)
02:19:24	Yanghui Kang:	Thank you!!
02:19:27	Yun Yang:	Thank you!
02:19:34	Hye-Seon Kim:	Thank you all
02:19:35	Sean Kearney:	Thanks!
02:19:41	John Humphreys:	Thanks!