Skip to content

make tls default if security is set...#519

Closed
riedel wants to merge 5 commits into
dask:mainfrom
riedel:default-tls-496
Closed

make tls default if security is set...#519
riedel wants to merge 5 commits into
dask:mainfrom
riedel:default-tls-496

Conversation

@riedel

@riedel riedel commented Sep 11, 2021

Copy link
Copy Markdown
Member

… or add temporary security if none is given and tls is set. Further it adds support for in memory TLS keys for workers. Further unneccessary tests are skipped for cluster ci targets. that weren't marked.

fixes #496. #520 and #521

@riedel riedel force-pushed the default-tls-496 branch 4 times, most recently from 191d976 to b08ec51 Compare September 11, 2021 23:22
@riedel riedel changed the title make tls default if security is set... [WIP] make tls default if security is set... Sep 11, 2021
@riedel

riedel commented Sep 12, 2021

Copy link
Copy Markdown
Member Author

3d6a576 fixes #520

@riedel riedel changed the title [WIP] make tls default if security is set... make tls default if security is set... Sep 12, 2021
@riedel

riedel commented Sep 12, 2021

Copy link
Copy Markdown
Member Author

467c328 fixes #521 , this needs to be very carefully reviewed! It actually skips far more test than before. Those should be all duplicates, though.

Why is this with in this PR? It is a rather stupid reason. The htcondor ci image had problems installing cryptography as a test dependency.

IMHO it could make sense to split this PR.

@riedel riedel left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For reference I commented on the changes I made

Comment thread ci/environment.yml
Comment thread ci/htcondor.sh
Comment thread conftest.py
Comment thread conftest.py
Comment thread dask_jobqueue/core.py
Comment thread dask_jobqueue/core.py
Comment thread dask_jobqueue/core.py
Comment thread dask_jobqueue/core.py
if value is not None and "\n" in value:
f = tempfile.NamedTemporaryFile(mode="wt")
# make sure that tmpfile survives by keeping a reference
setattr(self, "_job_" + key, f)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is ugly, but I having to clean up manually seemed more ugly. Ideas are welcome

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain what must be done here? Maybe @jacobtomlinson understands better.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am setting a reference to keep to the temp file from being deconstructed (which triggers its removal)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So it's about the problem underlined in #520 again?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you at least do this part in a separated function?

And just to be sure, does this creates a file in /tmp directory (which would not been shared with workers)? Or is this in the job execution folder? I guess it needs to be created in a shared folder?

Comment thread dask_jobqueue/tests/test_jobqueue_core.py
Comment thread setup.py
@riedel

riedel commented Sep 24, 2021

Copy link
Copy Markdown
Member Author

I am happy to strip down some parts of the PR if anyone is too scared of merging it :) .

@guillaumeeb guillaumeeb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this looks good, I'm just asking for some details in order to decide if we should merge it as is or not.

Comment thread ci/htcondor.sh
Comment thread conftest.py
Comment thread dask_jobqueue/core.py
Comment thread dask_jobqueue/core.py
Comment thread dask_jobqueue/core.py
Comment thread dask_jobqueue/core.py
if value is not None and "\n" in value:
f = tempfile.NamedTemporaryFile(mode="wt")
# make sure that tmpfile survives by keeping a reference
setattr(self, "_job_" + key, f)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain what must be done here? Maybe @jacobtomlinson understands better.

Comment thread dask_jobqueue/tests/test_jobqueue_core.py
Comment thread setup.py
@riedel

riedel commented Oct 5, 2021

Copy link
Copy Markdown
Member Author

Like I said: I am happy to try moving the issues related to #521 out of this PR. It is true that the failing HTCondor CI dependency is not really a reason to include. The reason was that fixing the prebuild CI-Images cannot easily be done in the same PR and there seems not really a reason to fix it in a separate PR, because only this PR breaks it. So it was like a bit of a chicken and egg decision. Would fix this after this PR. However, if you think that #521 should not be fixed, I am happy to take another path.

@riedel

riedel commented Oct 20, 2021

Copy link
Copy Markdown
Member Author

I detangled this PR into #526 , #524 and #523 which should close the above mentioned issues atomically and added a new bug #525 to track the remains.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

protocol should default to tls if security is set

2 participants