[AIRFLOW-2896] Improve HdfsSensor()

[XD-DENG]

Make sure you have checked all steps below.

Jira

• My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
  • https://issues.apache.org/jira/browse/AIRFLOW-2896
  • In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.

Description

• Here are some details about my PR, including screenshots of any UI changes:

1. Make documentation clearer (format for ignored_ext should be extensions like 'py' rather than '.py')
2. Ensure upper/lower case would not affect the usage of ignored_ext feagure.
3. Improve test coverage by adding tests for methods filter_for_ignored_ext() and filter_for_filesize()

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

• My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.

Code Quality

• Passes git diff upstream/master -u -- "*.py" | flake8 --diff

[XD-DENG]

1 out of 9 tests failed due to

ERROR: InvocationError for command '/home/travis/build/apache/incubator-airflow/scripts/ci/setup_env.sh' (exited with code 1)

/home/travis/.travis_cache//cdh/hadoop.tar.gz: Cannot open: Permission denied

Not sure whether it's a Travis-CI issue.

[bolkedebruin]

Hdfs is being refactored in #3560 and I think this is then already outdated

[XD-DENG]

Thanks @bolkedebruin . Didn't know the refactoring going on. Will close this PR.