Fix issues with Underscore in the asset pipeline #11938
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -35,6 +35,15 @@ | |
| THEME_SASS_DIRECTORIES = [] | ||
| SASS_LOAD_PATHS = ['common/static', 'common/static/sass'] | ||
|
|
||
| # A list of NPM installed libraries that should be copied into the common | ||
| # static directory. | ||
| NPM_INSTALLED_LIBRARIES = [ | ||
| 'underscore/underscore.js' | ||
| ] | ||
|
|
||
| # Directory to install static vendor files | ||
| NPM_VENDOR_DIRECTORY = path("common/static/common/js/vendor") | ||
|
|
||
|
|
||
| def configure_paths(): | ||
| """Configure our paths based on settings. Called immediately.""" | ||
|
|
@@ -292,6 +301,26 @@ def compile_templated_sass(systems, settings): | |
| print("\t\tFinished preprocessing {} assets.".format(system)) | ||
|
|
||
|
|
||
| def process_npm_assets(): | ||
|
Contributor
There was a problem hiding this comment. Is this something that we need to do in production as well? Is there a corresponding configuration PR?
Contributor
Author
There was a problem hiding this comment. I wasn't aware of a difference in update_assets between sandboxes and production. What kind of configuration change are you thinking of? My naive thinking was that this would get called the same way in all environments. It is very similar to how a lot of the other static assets are handled, e.g. Sass compilation, XModule bundling etc. Do they require separate prod configuration?
Contributor
Author
There was a problem hiding this comment. I just realized that you might be referring to the fact that I'm calling this from prereqs.py which isn't used in production. I'm going to remove that call as it was just a hack to try to get the Jasmine tests to run on Jenkins, and it didn't work anyway. I'll find a cleaner way to ensure that the vendor files are copied before the Jasmine tests are run on Jenkins.
Contributor
Author
There was a problem hiding this comment. @feanil pointed me at this Ansible script which is different for production than for sandboxes: We'll have to look at this together tomorrow to see whether this is going to be an issue. There is a comment that says "Gather assets using paver if possible" so I'm hopeful that this will "just work". |
||
| """ | ||
| Process vendor libraries installed via NPM. | ||
|
There was a problem hiding this comment. I think you will need a unit test for this method. It does look like there is an existing test_assets file. |
||
| """ | ||
| # Skip processing of the libraries if this is just a dry run | ||
| if tasks.environment.dry_run: | ||
| tasks.environment.info("install npm_assets") | ||
| return | ||
|
|
||
| # Ensure that the vendor directory exists | ||
| NPM_VENDOR_DIRECTORY.mkdir_p() | ||
|
|
||
| # Copy each file to the vendor directory, overwriting any existing file. | ||
| for library in NPM_INSTALLED_LIBRARIES: | ||
| sh('/bin/cp -rf node_modules/{library} {vendor_dir}'.format( | ||
| library=library, | ||
| vendor_dir=NPM_VENDOR_DIRECTORY, | ||
| )) | ||
|
|
||
|
|
||
| def process_xmodule_assets(): | ||
| """ | ||
| Process XModule static assets. | ||
|
|
@@ -387,6 +416,7 @@ def update_assets(args): | |
|
|
||
| compile_templated_sass(args.system, args.settings) | ||
| process_xmodule_assets() | ||
| process_npm_assets() | ||
| compile_coffeescript() | ||
| call_task('pavelib.assets.compile_sass', options={'system': args.system, 'debug': args.debug}) | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,134 @@ | ||
| """Unit tests for the Paver JavaScript testing tasks.""" | ||
|
|
||
| import ddt | ||
| from mock import patch | ||
| from paver.easy import call_task | ||
|
|
||
| import pavelib.js_test | ||
| from .utils import PaverTestCase | ||
|
|
||
|
|
||
| @ddt.ddt | ||
| class TestPaverJavaScriptTestTasks(PaverTestCase): | ||
| """ | ||
| Test the Paver JavaScript testing tasks. | ||
| """ | ||
|
|
||
| EXPECTED_DELETE_JAVASCRIPT_REPORT_COMMAND = u'find {platform_root}/reports/javascript -type f -delete' | ||
| EXPECTED_INSTALL_NPM_ASSETS_COMMAND = u'install npm_assets' | ||
| EXPECTED_COFFEE_COMMAND = ( | ||
| u'node_modules/.bin/coffee --compile `find {platform_root}/lms {platform_root}/cms ' | ||
| u'{platform_root}/common -type f -name "*.coffee"`' | ||
| ) | ||
| EXPECTED_JS_TEST_TOOL_OPTIONS = ( | ||
| u"{platform_root}/lms/static/js_test.yml " | ||
| u"{platform_root}/lms/static/js_test_coffee.yml " | ||
| u"{platform_root}/cms/static/js_test.yml " | ||
| u"{platform_root}/cms/static/js_test_squire.yml " | ||
| u"{platform_root}/common/lib/xmodule/xmodule/js/js_test.yml " | ||
| u"{platform_root}/common/static/js_test.yml " | ||
| u"{platform_root}/common/static/js_test_requirejs.yml " | ||
| u"--use-firefox " | ||
| u"--timeout-sec 600 " | ||
| u"--xunit-report " | ||
| u"{platform_root}/reports/javascript/javascript_xunit.xml" | ||
| ) | ||
| EXPECTED_COVERAGE_OPTIONS = ( | ||
| u' --coverage-xml {platform_root}/reports/javascript/coverage.xml' | ||
| ) | ||
|
|
||
| EXPECTED_COMMANDS = [ | ||
| u"make report_dir", | ||
| u'git clean -fqdx test_root/logs test_root/data test_root/staticfiles test_root/uploads', | ||
| u"find . -name '.git' -prune -o -name '*.pyc' -exec rm {} \\;", | ||
| u'rm -rf test_root/log/auto_screenshots/*', | ||
| u"rm -rf /tmp/mako_[cl]ms", | ||
| ] | ||
|
|
||
| def setUp(self): | ||
| super(TestPaverJavaScriptTestTasks, self).setUp() | ||
|
|
||
| # Mock the paver @needs decorator | ||
| self._mock_paver_needs = patch.object(pavelib.js_test.test_js, 'needs').start() | ||
| self._mock_paver_needs.return_value = 0 | ||
|
|
||
| # Cleanup mocks | ||
| self.addCleanup(self._mock_paver_needs.stop) | ||
|
|
||
| @ddt.data( | ||
| [""], | ||
| ["--coverage"], | ||
| ["--suite=lms"], | ||
| ["--suite=lms --coverage"], | ||
| ) | ||
| @ddt.unpack | ||
| def test_test_js_run(self, options_string): | ||
| """ | ||
| Test the "test_js_run" task. | ||
| """ | ||
| options = self.parse_options_string(options_string) | ||
| self.reset_task_messages() | ||
| call_task("pavelib.js_test.test_js_run", options=options) | ||
| self.verify_messages(options=options, dev_mode=False) | ||
|
|
||
| @ddt.data( | ||
| [""], | ||
| ["--port=9999"], | ||
| ["--suite=lms"], | ||
| ["--suite=lms --port=9999"], | ||
| ) | ||
| @ddt.unpack | ||
| def test_test_js_dev(self, options_string): | ||
| """ | ||
| Test the "test_js_run" task. | ||
| """ | ||
| options = self.parse_options_string(options_string) | ||
| self.reset_task_messages() | ||
| call_task("pavelib.js_test.test_js_dev", options=options) | ||
| self.verify_messages(options=options, dev_mode=True) | ||
|
|
||
| def parse_options_string(self, options_string): | ||
| """ | ||
| Parse a string containing the options for a test run | ||
| """ | ||
| parameters = options_string.split(" ") | ||
| suite = "all" | ||
| if "--system=lms" in parameters: | ||
| suite = "lms" | ||
| elif "--system=common" in parameters: | ||
| suite = "common" | ||
| coverage = "--coverage" in parameters | ||
| port = None | ||
| if "--port=9999" in parameters: | ||
| port = 9999 | ||
| return { | ||
| "suite": suite, | ||
| "coverage": coverage, | ||
| "port": port, | ||
| } | ||
|
|
||
| def verify_messages(self, options, dev_mode): | ||
| """ | ||
| Verify that the messages generated when running tests are as expected | ||
| for the specified options and dev_mode. | ||
| """ | ||
| is_coverage = options['coverage'] | ||
| port = options['port'] | ||
| expected_messages = [] | ||
| expected_messages.extend(self.EXPECTED_COMMANDS) | ||
| if not dev_mode and not is_coverage: | ||
| expected_messages.append(self.EXPECTED_DELETE_JAVASCRIPT_REPORT_COMMAND.format( | ||
| platform_root=self.platform_root | ||
| )) | ||
| expected_messages.append(self.EXPECTED_INSTALL_NPM_ASSETS_COMMAND) | ||
| expected_messages.append(self.EXPECTED_COFFEE_COMMAND.format(platform_root=self.platform_root)) | ||
| expected_test_tool_command = u'js-test-tool {command} {options}'.format( | ||
| command='dev' if dev_mode else 'run', | ||
| options=self.EXPECTED_JS_TEST_TOOL_OPTIONS.format(platform_root=self.platform_root), | ||
| ) | ||
| if is_coverage: | ||
| expected_test_tool_command += self.EXPECTED_COVERAGE_OPTIONS.format(platform_root=self.platform_root) | ||
| if port: | ||
| expected_test_tool_command += u" -p {port}".format(port=port) | ||
| expected_messages.append(expected_test_tool_command) | ||
| self.assertEquals(self.task_messages, expected_messages) |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps rename STATIC_VENDOR_DIRECTORY to NPM_VENDOR_DIRECTORY since it is important that don't mix checked-in files and npm installed files?