A deep DOM makes the CSS matching with DOM elements difficult.
It also slows down Javascript modifications to the DOM because changing the dimensions of an element makes the browser re-calculate the dimensions of it's parents. Same thing for Javascript events, that bubble up to the document root.
iFrames are the most complex HTML elements. They are pages, just like the main page, and the browser needs to create a new page context, which has a cost.
DOM queries are like looking in a large catalog of items. Even if the browsers made progress on the performances of queries, websites often make hundreds of them.
Try to reduce the number of queries by refactoring your Javascript code.
Avoid also to have a read query between two write queries. To be able to reduce the number repaints and optimize performances, browsers buffer the DOM writing operations and treat them in bulk. But each time a DOM reading is asked, the browser needs to empty the buffer. This can be particularly slow inside a loop.
This is the number of queries that could be avoided by removing all duplicated queries.
Simply save the result of a query in a variable. Ok it is not always simple, especially with third-party scripts, but at least do it with your own code.
It can be avoided by using \"event delegation\". Instead of binding events on each element one by one, events delegation binds them on the top level document element and uses the bubbling principle. It will imperceptibly slow down the event when it occurs, but the loading of the page will speed-up.
It is a bad practice because they clutter up the global namespace. If two scripts use the same variable name in the global scope, it can cause conflicts and it is generally hard to debug.
Global variables also take a (very) little bit longer to be accessed than variables in the local scope of a function.
This metric counts the number of DOM queries, DOM inserts, binds, etc. made by the Javascript before the DOMContentLoaded event.
Wait for this event before manipulating the DOM. Do not execute Javascript in the middle of the BODY as it slows down the construction of the DOM and makes a poor maintainability. This is what i call spaghetti code.
The JS Timeline tab can help you identify what's happening.
Current latest versions of jQuery are 1.11 (with support for old IE versions) and 2.1 (without).
Each new version of jQuery optimizes performances. Do not keep an old version of jQuery. Updating can sometimes break a few things, but it is generally quite easy to fix them up. So don't hesitate.
Having a huge number of CSS rules hurts performances. If the number of CSS rules is higher than the number of DOM elements, there is clearly a problem.
Huge stylesheets generally occur when the different pages of a website load all the CSS, concatenated in a single stylesheet, even if a large part of the rules are page-specific. Solution is to create one main CSS file with global rules and one custom files per page.
It can be useful, but only as a last resort. It is a bad practice because it overrides the normal cascading logic. The more you use !important, the more you need it again to over-override. This conducts to a poor maintainability.
Many property prefixes such as -moz- or -webkit- are not needed anymore, or by very few people. You can remove them or replace them with the non-prefixed version. This will help reducing your stylesheets weight.
This is one way to remove complexity from a CSS rule. Generally, when \"body\" is specified in a rule it can be removed, because an element is necessarily inside the body.
Some tags included inside other tags are obvious. For example, when \"ul li\" is specified in a rule, \"ul\" can be removed because the \"li\" element is always inside a \"ul\". Same thing for \"tr td\", \"select option\", ...
Lowering compexity in CSS selectors can make the page load a little faster.
This counts the number of requests not keeping the connection alive (specifying \"Connection: close\" in the response headers). It is only counting a request if it is followed by another request on the same domain.
This is slowing down the next request, because the brower needs to open a new connection to the server, which means a additional round-trip.
Correct the problem by setting a Keep-Alive header on the guilty server.
A deep DOM makes the CSS matching with DOM elements difficult.
It also slows down Javascript modifications to the DOM because changing the dimensions of an element makes the browser re-calculate the dimensions of it's parents. Same thing for Javascript events, that bubble up to the document root.
",
+ "isOkThreshold": 10,
+ "isBadThreshold": 20,
+ "isAbnormalThreshold": 30
+ },
+ "value": 1,
+ "bad": false,
+ "abnormal": false,
+ "score": 100,
+ "abnormalityScore": 0,
+ "offenders": ["body > h1[1]"]
+ });
done();
}).fail(function(err) {
diff --git a/test/fixtures/rulesCheckerInput.json b/test/fixtures/rulesCheckerInput.json
index 96b949c..fd05a45 100644
--- a/test/fixtures/rulesCheckerInput.json
+++ b/test/fixtures/rulesCheckerInput.json
@@ -1,25 +1,27 @@
{
- "tool1": {
- "metrics": {
- "metric1": 1236,
- "metric2": 222,
- "metric3": 6666,
- "metric4": 1000,
- "metric5": 3000,
- "metric6": 0,
- "metric7": 5000
+ "toolsResults": {
+ "tool1": {
+ "metrics": {
+ "metric1": 1236,
+ "metric2": 222,
+ "metric3": 6666,
+ "metric4": 1000,
+ "metric5": 3000,
+ "metric6": 0,
+ "metric7": 5000
+ },
+ "offenders": {
+ "metric1": [],
+ "metric2": [],
+ "metric3": ["offender1", "offender2"],
+ "metric5": []
+ }
},
- "offenders": {
- "metric1": [],
- "metric2": [],
- "metric3": ["offender1", "offender2"],
- "metric5": []
- }
- },
- "tool2": {
- "metrics": {
- "metric1": 22,
- "metric10": 22
+ "tool2": {
+ "metrics": {
+ "metric1": 22,
+ "metric10": 22
+ }
}
}
}
\ No newline at end of file
From 6089e5fe20adfb8d7eca0458d4b891328c7ab034 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ga=C3=ABl=20M=C3=A9tais?=
Date: Fri, 28 Nov 2014 17:53:39 +0100
Subject: [PATCH 04/98] Introduce debug instead of console.log
---
bin/cli.js | 8 ++++----
lib/rulesChecker.js | 10 ++++++++++
lib/runner.js | 5 +++--
lib/tools/phantomasWrapper.js | 16 +++++++++-------
package.json | 9 +++++----
5 files changed, 31 insertions(+), 17 deletions(-)
diff --git a/bin/cli.js b/bin/cli.js
index 3937df3..88b615a 100644
--- a/bin/cli.js
+++ b/bin/cli.js
@@ -1,6 +1,6 @@
#!/usr/bin/env node
-//var color = require('colors');
+var debug = require('debug')('ylt:cli');
var YellowLabTools = require('../lib/yellowlabtools');
@@ -17,17 +17,17 @@ var url = process.argv[2];
'use strict';
var ylt = new YellowLabTools(url);
- console.log('Test launched...');
+ debug('Test launched...');
ylt.
then(function(data) {
- console.log('Success');
+ debug('Success');
console.log(JSON.stringify(data, null, 2));
}).fail(function(err) {
- console.error('Test failed for %s', url);
+ debug('Test failed for %s', url);
console.error(err);
});
diff --git a/lib/rulesChecker.js b/lib/rulesChecker.js
index 042d73e..c576c72 100644
--- a/lib/rulesChecker.js
+++ b/lib/rulesChecker.js
@@ -1,4 +1,5 @@
var Q = require('q');
+var debug = require('debug')('ylt:ruleschecker');
var RulesChecker = function() {
'use strict';
@@ -8,6 +9,8 @@ var RulesChecker = function() {
var results = {};
var err = null;
+ debug('Starting checking rules');
+
for (var metricName in policies) {
var policy = policies[metricName];
@@ -38,9 +41,16 @@ var RulesChecker = function() {
rule.abnormalityScore = Math.min(Math.round(abnormalityScore), 0);
results[metricName] = rule;
+
+ } else {
+
+ debug('Metric %s not found for tool %s', metricName, policy.tool);
+
}
}
+ debug('Rules checking finished');
+
return results;
};
};
diff --git a/lib/runner.js b/lib/runner.js
index 67e641c..c9083c1 100644
--- a/lib/runner.js
+++ b/lib/runner.js
@@ -1,4 +1,5 @@
var Q = require('q');
+var debug = require('debug')('ylt:yellowlabtools');
var phantomasWrapper = require('./tools/phantomasWrapper');
var rulesChecker = require('./rulesChecker');
@@ -29,8 +30,8 @@ var Runner = function(params) {
deferred.resolve(data);
}).fail(function(err) {
- console.log('Run failed');
- console.log(err);
+ debug('Run failed');
+ debug(err);
deferred.reject(err);
});
diff --git a/lib/tools/phantomasWrapper.js b/lib/tools/phantomasWrapper.js
index 7186941..e1df01d 100644
--- a/lib/tools/phantomasWrapper.js
+++ b/lib/tools/phantomasWrapper.js
@@ -1,7 +1,9 @@
var async = require('async');
var Q = require('q');
+var debug = require('debug')('ylt:phantomaswrapper');
var phantomas = require('phantomas');
+
var PhantomasWrapper = function() {
'use strict';
@@ -47,16 +49,16 @@ var PhantomasWrapper = function() {
};
// Output the command line for debugging purpose
- console.log('If you want to reproduce the phantomas task only, copy the following command line:');
+ debug('If you want to reproduce the phantomas task only, copy the following command line:');
var optionsString = '';
for (var opt in options) {
optionsString += ' ' + '--' + opt + '=' + options[opt];
}
- console.log('node node_modules/phantomas/bin/phantomas.js --url=' + task.url + optionsString + ' --verbose');
+ debug('node node_modules/phantomas/bin/phantomas.js --url=' + task.url + optionsString + ' --verbose');
// Kill the application if nothing happens for 10 minutes
var killer = setTimeout(function() {
- console.log('Killing the app because the test on ' + task.url + ' was launched 10 minutes ago');
+ debug('Killing the app because the test on ' + task.url + ' was launched 10 minutes ago');
// If in server mode, forever will restart the server
process.exit(1);
}, 600000);
@@ -66,7 +68,7 @@ var PhantomasWrapper = function() {
async.retry(triesNumber, function(cb) {
phantomas(task.url, options, function(err, json, results) {
- console.log('Returning from Phantomas');
+ debug('Returning from Phantomas');
// Adding some YellowLabTools errors here
if (json && json.metrics && !json.metrics.javascriptExecutionTree) {
@@ -79,12 +81,12 @@ var PhantomasWrapper = function() {
// Don't cancel test if it is a timeout and we've got some results
if (err === 252 && json) {
- console.log('Timeout after ' + options.timeout + ' seconds. But it\'s not a problem, the test is valid.');
+ debug('Timeout after ' + options.timeout + ' seconds. But it\'s not a problem, the test is valid.');
err = null;
}
if (err) {
- console.log('Attempt failed. Error code ' + err);
+ debug('Attempt failed. Error code ' + err);
}
cb(err, json);
@@ -94,7 +96,7 @@ var PhantomasWrapper = function() {
clearTimeout(killer);
if (err) {
- console.log('All ' + triesNumber + ' attemps failed for the test');
+ debug('All ' + triesNumber + ' attemps failed for the test');
deferred.reject(err);
} else {
diff --git a/package.json b/package.json
index 896a474..eed79c0 100644
--- a/package.json
+++ b/package.json
@@ -10,12 +10,13 @@
},
"main": "./lib/yellowlabtools.js",
"dependencies": {
- "phantomas": "1.7.0",
- "express": "~4.10.1",
"async": "~0.9.0",
- "socket.io": "~1.2.0",
"body-parser": "~1.9.2",
- "compression": "~1.2.0"
+ "compression": "~1.2.0",
+ "debug": "^2.1.0",
+ "express": "~4.10.1",
+ "phantomas": "1.7.0",
+ "socket.io": "~1.2.0"
},
"devDependencies": {
"chai": "^1.9.2",
From 35aed5fb552179de6c81ff95477e7eb3b9dd6e62 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ga=C3=ABl=20M=C3=A9tais?=
Date: Fri, 28 Nov 2014 18:31:14 +0100
Subject: [PATCH 05/98] Test that console.log is never called
---
package.json | 4 +++-
test/api/yellowlabtoolsTest.js | 12 +++++++++++-
2 files changed, 14 insertions(+), 2 deletions(-)
diff --git a/package.json b/package.json
index eed79c0..5bb8f24 100644
--- a/package.json
+++ b/package.json
@@ -32,7 +32,9 @@
"matchdep": "^0.3.0",
"mocha": "^2.0.1",
"phantomjs": "^1.9.10",
- "q": "^1.1.2"
+ "q": "^1.1.2",
+ "sinon": "^1.12.1",
+ "sinon-chai": "^2.6.0"
},
"scripts": {
"test": "grunt test"
diff --git a/test/api/yellowlabtoolsTest.js b/test/api/yellowlabtoolsTest.js
index ddc285e..89c36ab 100644
--- a/test/api/yellowlabtoolsTest.js
+++ b/test/api/yellowlabtoolsTest.js
@@ -1,6 +1,11 @@
-var should = require('chai').should();
+var chai = require('chai');
+var sinon = require('sinon');
+var sinonChai = require('sinon-chai');
+var should = chai.should();
var YellowLabTools = require('../../lib/yellowlabtools.js');
+chai.use(sinonChai);
+
describe('yellowlabtools', function() {
@@ -28,6 +33,9 @@ describe('yellowlabtools', function() {
it('succeeds on simple-page.html', function(done) {
this.timeout(15000);
+ // Check if console.log is called
+ sinon.spy(console, 'log')
+
var url = 'http://localhost:8388/simple-page.html';
var ylt = new YellowLabTools(url)
@@ -67,6 +75,8 @@ describe('yellowlabtools', function() {
"offenders": ["body > h1[1]"]
});
+ console.log.should.not.have.been.called;
+
done();
}).fail(function(err) {
done(err);
From e18cd87a07fd0a3f7a1bd3600e1b8bc3879a8ad3 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ga=C3=ABl=20M=C3=A9tais?=
Date: Tue, 2 Dec 2014 00:08:07 +0100
Subject: [PATCH 06/98] Writing the API (workin in progress)
---
Gruntfile.js | 7 +-
app/lib/testQueue.js | 66 -----------------
bin/server.js | 22 ++++++
lib/server/controllers/apiController.js | 86 +++++++++++++++++++++++
lib/server/controllers/uiController.js | 0
lib/server/datastores/resultsDatastore.js | 9 +++
lib/server/datastores/runsDatastore.js | 36 ++++++++++
lib/server/datastores/runsQueue.js | 73 +++++++++++++++++++
package.json | 2 +-
test/api/phantomasWrapperTest.js | 12 ++--
test/api/yellowlabtoolsTest.js | 3 +-
test/server/runsDatastoreTest.js | 41 +++++++++++
test/server/runsQueueTest.js | 64 +++++++++++++++++
test/server/testQueueTest.js | 23 ------
14 files changed, 344 insertions(+), 100 deletions(-)
delete mode 100644 app/lib/testQueue.js
create mode 100644 bin/server.js
create mode 100644 lib/server/controllers/apiController.js
create mode 100644 lib/server/controllers/uiController.js
create mode 100644 lib/server/datastores/resultsDatastore.js
create mode 100644 lib/server/datastores/runsDatastore.js
create mode 100644 lib/server/datastores/runsQueue.js
create mode 100644 test/server/runsDatastoreTest.js
create mode 100644 test/server/runsQueueTest.js
delete mode 100644 test/server/testQueueTest.js
diff --git a/Gruntfile.js b/Gruntfile.js
index 1644902..41a5eef 100644
--- a/Gruntfile.js
+++ b/Gruntfile.js
@@ -36,7 +36,8 @@ module.exports = function(grunt) {
'lib/**/*.js',
'app/nodeControllers/*.js',
'app/public/scripts/*.js',
- 'phantomas_custom/**/*.js'
+ 'phantomas_custom/**/*.js',
+ 'test/**/*.js'
]
},
clean: {
@@ -70,13 +71,13 @@ module.exports = function(grunt) {
options: {
reporter: 'spec',
},
- src: ['coverage/test/api/*.js']
+ src: ['coverage/test/api/*.js', 'coverage/test/server/*.js']
},
'test-current-work': {
options: {
reporter: 'spec',
},
- src: ['coverage/test/api/yellowlabtoolsTest.js']
+ src: ['coverage/test/server/runsQueueTest.js']
},
coverage: {
options: {
diff --git a/app/lib/testQueue.js b/app/lib/testQueue.js
deleted file mode 100644
index f271ccb..0000000
--- a/app/lib/testQueue.js
+++ /dev/null
@@ -1,66 +0,0 @@
-/**
- * Creation of a queue and it's worker function
- */
-
-var util = require('util');
-var EventEmitter = require('events').EventEmitter;
-var async = require('async');
-var phantomasWrapper = require('./phantomasWrapper');
-
-
-var testQueue = function() {
- 'use strict';
-
- var currentTask = null;
- var self = this;
-
- var queue = async.queue(function(task, callback) {
- currentTask = task;
-
- console.log('Starting test ' + task.testId);
-
- phantomasWrapper.execute(task, function(err, json, results) {
- console.log('Test ' + task.testId + ' complete');
- currentTask = null;
- callback(err, json, results);
- self.emit('queueMoving');
- });
- });
-
-
- // Use this method to add a test to the queue
- this.push = queue.push;
-
-
- // Gives the position of a task in the queue
- // Returns 0 if it is the current running task
- // Returns -1 if not found
- this.indexOf = function(testId) {
- if (currentTask && currentTask.testId === testId) {
- return 0;
- }
-
- var position = -1;
- if (queue.length() > 0) {
- queue.tasks.forEach(function(task, index) {
- if (task.data.testId === testId) {
- position = index + 1;
- }
- });
- }
- return position;
- };
-
- this.testComplete = function(testId) {
- self.emit('testComplete', testId);
- };
-
- this.testFailed = function(testId) {
- self.emit('testFailed', testId);
- };
-};
-
-// extend the EventEmitter class
-util.inherits(testQueue, EventEmitter);
-
-module.exports = new testQueue();
\ No newline at end of file
diff --git a/bin/server.js b/bin/server.js
new file mode 100644
index 0000000..3f72983
--- /dev/null
+++ b/bin/server.js
@@ -0,0 +1,22 @@
+// Config file
+var settings = require('../server_config/settings.json');
+
+var express = require('express');
+var app = express();
+var server = require('http').createServer(app);
+var bodyParser = require('body-parser');
+var compress = require('compression');
+
+app.use(compress());
+app.use(bodyParser.urlencoded({ extended: false }));
+
+
+// Initialize the controllers
+var apiController = require('../lib/server/controllers/apiController')(app);
+var uiController = require('../lib/server/controllers/uiController')(app);
+
+
+// Launch the server
+server.listen(settings.serverPort, function() {
+ console.log('Listening on port %d', server.address().port);
+});
\ No newline at end of file
diff --git a/lib/server/controllers/apiController.js b/lib/server/controllers/apiController.js
new file mode 100644
index 0000000..fb9d7c6
--- /dev/null
+++ b/lib/server/controllers/apiController.js
@@ -0,0 +1,86 @@
+var debug = require('debug')('ylt:server');
+
+var runsQueue = require('../datastores/runsQueue');
+var runsDatastore = require('../datastores/runsDatastore');
+
+
+function ApiController(app) {
+ 'use strict';
+
+ // Retrieve the list of all runs
+ /*app.get('/runs', function(req, res) {
+ // NOT YET
+ });*/
+
+ // Create a new run
+ app.post('/runs', function(req, res) {
+
+ // Grab the test parameters
+ var run = {
+ // Generate a random run ID
+ _id: (Date.now()*1000 + Math.round(Math.random()*1000)).toString(36),
+ params: {
+ url: req.body.url,
+ waitForResponse: req.body.waitForResponse || true
+ }
+ };
+
+ // Add test to the testQueue
+ debug('Adding test %s to the queue', run._id);
+ var queuing = runsQueue.push(run._id);
+
+
+ // Save the run to the datastore
+ var position = runsQueue.getPosition(run._id);
+ run.status = {
+ statusCode: (position === 0) ? STATUS_RUNNING : STATUS_AWAITING,
+ position: position
+ };
+ runsDatastore.add(run);
+
+
+ // Listening for position updates
+ queuing.progress(function(position) {
+ var savedRun = runsDatastore.get(run._id);
+ savedRun.status = {
+ statusCode: STATUS_AWAITING,
+ position: position
+ };
+ runsDatastore.update(savedRun);
+ });
+
+
+ queuing.then(function() {
+
+ });
+
+ // The user doesn't not want to wait for the response
+ if (!params.waitForResponse) {
+
+ // Sending just the test id
+ res.setHeader('Content-Type', 'application/javascript');
+ res.send(JSON.stringify({
+ testId: testId
+ }));
+ }
+ });
+
+ // Retrive one run by id
+ app.get('/run/:id', function(req, res) {
+
+ });
+
+ // Delete one run by id
+ /*app.delete('/run/:id', function(req, res) {
+ // NOT YET
+ });*/
+
+
+ var STATUS_AWAITING = 'awaiting';
+ var STATUS_RUNNING = 'running';
+ var STATUS_DONE = 'done';
+ var STATUS_FAILED = 'failed';
+
+}
+
+module.exports = ApiController;
\ No newline at end of file
diff --git a/lib/server/controllers/uiController.js b/lib/server/controllers/uiController.js
new file mode 100644
index 0000000..e69de29
diff --git a/lib/server/datastores/resultsDatastore.js b/lib/server/datastores/resultsDatastore.js
new file mode 100644
index 0000000..954f9b7
--- /dev/null
+++ b/lib/server/datastores/resultsDatastore.js
@@ -0,0 +1,9 @@
+
+
+function ResultsDatastore() {
+ 'use strict';
+
+
+}
+
+module.exports = ResultsDatastore;
\ No newline at end of file
diff --git a/lib/server/datastores/runsDatastore.js b/lib/server/datastores/runsDatastore.js
new file mode 100644
index 0000000..c84ac4e
--- /dev/null
+++ b/lib/server/datastores/runsDatastore.js
@@ -0,0 +1,36 @@
+
+
+function RunsDatastore() {
+ 'use strict';
+
+ // NOT PERSISTING RUNS
+ // For the moment, maybe one day
+ var runs = {};
+
+
+ this.add = function(run) {
+ runs[run._id] = run;
+ };
+
+ this.get = function(runId) {
+ return runs[runId];
+ };
+
+ this.update = function(run) {
+ runs[run._id] = run;
+ };
+
+ this.delete = function(runId) {
+ delete runs[runId];
+ };
+
+ this.list = function() {
+ var runsArray = [];
+ Object.keys(runs).forEach(function(key) {
+ runsArray.push(runs[key]);
+ });
+ return runsArray;
+ };
+}
+
+module.exports = RunsDatastore;
\ No newline at end of file
diff --git a/lib/server/datastores/runsQueue.js b/lib/server/datastores/runsQueue.js
new file mode 100644
index 0000000..ad1a6b8
--- /dev/null
+++ b/lib/server/datastores/runsQueue.js
@@ -0,0 +1,73 @@
+var Q = require('q');
+
+
+function RunsQueue() {
+ 'use strict';
+
+ var queue = [];
+
+
+ this.push = function(runId) {
+ var deferred = Q.defer();
+
+ if (queue.length === 0) {
+
+ // The queue is empty, let's run immediatly
+ queue.push({
+ runId: runId
+ });
+
+ deferred.resolve();
+
+ } else {
+
+ queue.push({
+ runId: runId,
+ positionChangedCallback: function(position) {
+ deferred.notify(position);
+ },
+ itIsTimeCallback: function() {
+ deferred.resolve();
+ }
+ });
+ }
+
+ return deferred.promise;
+ };
+
+
+ this.getPosition = function(runId) {
+ // Position 0 means it's a work in progress (a run is removed AFTER it is finished, not before)
+ var position = -1;
+
+ queue.some(function(run, index) {
+ if (run.runId === runId) {
+ position = index;
+ return true;
+ }
+ return false;
+ });
+
+ return position;
+ };
+
+
+ this.remove = function(runId) {
+ var position = this.getPosition(runId);
+ if (position >= 0) {
+ queue.splice(position, 1);
+ }
+
+ // Update other runs' positions
+ queue.forEach(function(run, index) {
+ if (index === 0 && run.itIsTimeCallback) {
+ run.itIsTimeCallback();
+ } else if (index > 0 && run.positionChangedCallback) {
+ run.positionChangedCallback(index);
+ }
+ });
+
+ };
+}
+
+module.exports = RunsQueue;
\ No newline at end of file
diff --git a/package.json b/package.json
index 5bb8f24..29bca24 100644
--- a/package.json
+++ b/package.json
@@ -14,7 +14,7 @@
"body-parser": "~1.9.2",
"compression": "~1.2.0",
"debug": "^2.1.0",
- "express": "~4.10.1",
+ "express": "~4.10.4",
"phantomas": "1.7.0",
"socket.io": "~1.2.0"
},
diff --git a/test/api/phantomasWrapperTest.js b/test/api/phantomasWrapperTest.js
index 785bdbe..904915e 100644
--- a/test/api/phantomasWrapperTest.js
+++ b/test/api/phantomasWrapperTest.js
@@ -23,9 +23,9 @@ describe('phantomasWrapper', function() {
data.should.have.a.property('generator');
data.generator.should.contain('phantomas');
data.should.have.a.property('url').that.equals(url);
- data.should.have.a.property('metrics').that.is.an('object').not.empty;
- data.should.have.a.property('offenders').that.is.an('object').not.empty;
- data.offenders.should.have.a.property('javascriptExecutionTree').that.is.a('array').not.empty;
+ data.should.have.a.property('metrics').that.is.an('object').not.empty();
+ data.should.have.a.property('offenders').that.is.an('object').not.empty();
+ data.offenders.should.have.a.property('javascriptExecutionTree').that.is.a('array').not.empty();
done();
}).fail(function(err) {
@@ -73,9 +73,9 @@ describe('phantomasWrapper', function() {
data.should.have.a.property('generator');
data.generator.should.contain('phantomas');
data.should.have.a.property('url').that.equals(url);
- data.should.have.a.property('metrics').that.is.an('object').not.empty;
- data.should.have.a.property('offenders').that.is.an('object').not.empty;
- data.offenders.should.have.a.property('javascriptExecutionTree').that.is.a('array').not.empty;
+ data.should.have.a.property('metrics').that.is.an('object').not.empty();
+ data.should.have.a.property('offenders').that.is.an('object').not.empty();
+ data.offenders.should.have.a.property('javascriptExecutionTree').that.is.a('array').not.empty();
done();
}).fail(function(err) {
diff --git a/test/api/yellowlabtoolsTest.js b/test/api/yellowlabtoolsTest.js
index 89c36ab..b1dd85e 100644
--- a/test/api/yellowlabtoolsTest.js
+++ b/test/api/yellowlabtoolsTest.js
@@ -34,7 +34,7 @@ describe('yellowlabtools', function() {
this.timeout(15000);
// Check if console.log is called
- sinon.spy(console, 'log')
+ sinon.spy(console, 'log');
var url = 'http://localhost:8388/simple-page.html';
@@ -75,6 +75,7 @@ describe('yellowlabtools', function() {
"offenders": ["body > h1[1]"]
});
+ /*jshint expr: true*/
console.log.should.not.have.been.called;
done();
diff --git a/test/server/runsDatastoreTest.js b/test/server/runsDatastoreTest.js
new file mode 100644
index 0000000..750956d
--- /dev/null
+++ b/test/server/runsDatastoreTest.js
@@ -0,0 +1,41 @@
+var should = require('chai').should();
+var runsDatastore = require('../../lib/server/datastores/runsDatastore');
+
+describe('runsDatastore', function() {
+
+ var datastore = new runsDatastore();
+
+ var randomId = Math.round(Math.random() * 100000);
+
+ it('should accept a new run', function() {
+ datastore.should.have.a.property('add').that.is.a('function');
+
+ datastore.add({
+ _id: randomId,
+ otherData: 123456789
+ });
+ });
+
+ it('should have stored the run', function() {
+ datastore.should.have.a.property('get').that.is.a('function');
+
+ var run = datastore.get(randomId);
+
+ run.should.have.a.property('_id').that.equals(randomId);
+ });
+
+ it('should have exactly 1 run in the store', function() {
+ var runs = datastore.list();
+ runs.should.be.a('array');
+ runs.should.have.length(1);
+ runs[0].should.have.a.property('_id').that.equals(randomId);
+ });
+
+ it('should delete the run', function() {
+ datastore.delete(randomId);
+
+ var runs = datastore.list();
+ runs.should.be.a('array');
+ runs.should.have.length(0);
+ });
+});
diff --git a/test/server/runsQueueTest.js b/test/server/runsQueueTest.js
new file mode 100644
index 0000000..45a2a23
--- /dev/null
+++ b/test/server/runsQueueTest.js
@@ -0,0 +1,64 @@
+var should = require('chai').should();
+var runsQueue = require('../../lib/server/datastores/runsQueue.js');
+
+describe('runsQueue', function() {
+
+ var queue = new runsQueue();
+ var cccRun = null;
+
+ it('should accept a new runId', function(done) {
+ queue.should.have.a.property('push').that.is.a('function');
+
+ var aaaRun = queue.push('aaa');
+ queue.push('bbb');
+
+ aaaRun.then(function() {
+ done();
+ });
+ });
+
+ it('should return the right positions', function() {
+ var aaaPosition = queue.getPosition('aaa');
+ aaaPosition.should.equal(0);
+
+ var bbbPosition = queue.getPosition('bbb');
+ bbbPosition.should.equal(1);
+
+ var cccPosition = queue.getPosition('ccc');
+ cccPosition.should.equal(-1);
+ });
+
+ it('should refresh runs\' positions', function(done) {
+ cccRun = queue.push('ccc');
+
+ cccRun.progress(function(position) {
+ position.should.equal(1);
+
+ var positionDoubleCheck = queue.getPosition('ccc');
+ positionDoubleCheck.should.equal(1);
+
+ done();
+ });
+
+ queue.remove('aaa');
+ });
+
+ it('should fulfill the promise when first in the line', function(done) {
+ cccRun.then(function() {
+ done();
+ });
+
+ queue.remove('bbb');
+ });
+
+ it('should not keep removed runs', function() {
+ var aaaPosition = queue.getPosition('aaa');
+ aaaPosition.should.equal(-1);
+
+ var bbbPosition = queue.getPosition('bbb');
+ bbbPosition.should.equal(-1);
+
+ var cccPosition = queue.getPosition('ccc');
+ cccPosition.should.equal(0);
+ });
+});
\ No newline at end of file
diff --git a/test/server/testQueueTest.js b/test/server/testQueueTest.js
deleted file mode 100644
index 6ae1be6..0000000
--- a/test/server/testQueueTest.js
+++ /dev/null
@@ -1,23 +0,0 @@
-var should = require('chai').should();
-var testQueue = require('../../app/lib/testQueue.js');
-
-describe('testQueue', function() {
-
- var url = 'http://www.not.existing';
-
- it('should accept a new test with method push', function() {
- testQueue.should.have.property('push').that.is.a('function');
-
- var task = {
- testId: 'aaaaa',
- url: url,
- options: {}
- };
-
- testQueue.push(task, function(err, json, results) {
- done();
- });
-
- });
-
-});
\ No newline at end of file
From 906f0876a038bf300bffd333871946baa544e7fa Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ga=C3=ABl=20M=C3=A9tais?=
Date: Fri, 5 Dec 2014 16:14:08 +0100
Subject: [PATCH 07/98] API is starting to run
---
Gruntfile.js | 4 +-
bin/server.js | 2 +-
lib/metadata/policies.json | 2 +-
lib/server/controllers/apiController.js | 154 +++++++++++++++------
lib/server/controllers/uiController.js | 13 ++
lib/server/datastores/resultsDatastore.js | 72 +++++++++-
lib/server/datastores/runsDatastore.js | 53 ++++++-
lib/server/datastores/runsQueue.js | 11 +-
lib/tools/phantomasWrapper.js | 2 +-
lib/yellowlabtools.js | 13 +-
package.json | 1 +
test/api/resultsDatastoreTest.js | 74 ++++++++++
test/api/runsDatastoreTest.js | 81 +++++++++++
test/{server => api}/runsQueueTest.js | 8 +-
test/{api => core}/phantomasWrapperTest.js | 0
test/{api => core}/rulesCheckerTest.js | 0
test/{api => core}/yellowlabtoolsTest.js | 0
test/server/runsDatastoreTest.js | 41 ------
18 files changed, 423 insertions(+), 108 deletions(-)
create mode 100644 test/api/resultsDatastoreTest.js
create mode 100644 test/api/runsDatastoreTest.js
rename test/{server => api}/runsQueueTest.js (88%)
rename test/{api => core}/phantomasWrapperTest.js (100%)
rename test/{api => core}/rulesCheckerTest.js (100%)
rename test/{api => core}/yellowlabtoolsTest.js (100%)
delete mode 100644 test/server/runsDatastoreTest.js
diff --git a/Gruntfile.js b/Gruntfile.js
index 41a5eef..240d1fe 100644
--- a/Gruntfile.js
+++ b/Gruntfile.js
@@ -71,13 +71,13 @@ module.exports = function(grunt) {
options: {
reporter: 'spec',
},
- src: ['coverage/test/api/*.js', 'coverage/test/server/*.js']
+ src: ['coverage/test/core/*.js', 'coverage/test/api/*.js']
},
'test-current-work': {
options: {
reporter: 'spec',
},
- src: ['coverage/test/server/runsQueueTest.js']
+ src: ['coverage/test/server/runsDatastoreTest.js']
},
coverage: {
options: {
diff --git a/bin/server.js b/bin/server.js
index 3f72983..3cd54f7 100644
--- a/bin/server.js
+++ b/bin/server.js
@@ -8,7 +8,7 @@ var bodyParser = require('body-parser');
var compress = require('compression');
app.use(compress());
-app.use(bodyParser.urlencoded({ extended: false }));
+app.use(bodyParser.json());
// Initialize the controllers
diff --git a/lib/metadata/policies.json b/lib/metadata/policies.json
index 30c3402..40df848 100644
--- a/lib/metadata/policies.json
+++ b/lib/metadata/policies.json
@@ -365,6 +365,6 @@
"message": "
For each domain met, the browser needs to make a DNS look-up, which is slow. Avoid having to many different domains and the page should render faster.
By the way, domain sharding is not a good practice anymore.
",
"isOkThreshold": 10,
"isBadThreshold": 25,
- "isAbnormalThreshold": 40
+ "isAbnormalThreshold": 50
}
}
\ No newline at end of file
diff --git a/lib/server/controllers/apiController.js b/lib/server/controllers/apiController.js
index fb9d7c6..ee4e0de 100644
--- a/lib/server/controllers/apiController.js
+++ b/lib/server/controllers/apiController.js
@@ -1,86 +1,152 @@
-var debug = require('debug')('ylt:server');
+var debug = require('debug')('ylt:server');
-var runsQueue = require('../datastores/runsQueue');
-var runsDatastore = require('../datastores/runsDatastore');
+var YellowLabTools = require('../../yellowlabtools');
+var RunsQueue = require('../datastores/runsQueue');
+var RunsDatastore = require('../datastores/runsDatastore');
+var ResultsDatastore = require('../datastores/resultsDatastore');
-function ApiController(app) {
+var ApiController = function(app) {
'use strict';
+ var queue = new RunsQueue();
+ var runsDatastore = new RunsDatastore();
+ var resultsDatastore = new ResultsDatastore();
+
// Retrieve the list of all runs
- /*app.get('/runs', function(req, res) {
+ /*app.get('/api/runs', function(req, res) {
// NOT YET
});*/
// Create a new run
- app.post('/runs', function(req, res) {
+ app.post('/api/runs', function(req, res) {
- // Grab the test parameters
+ // Grab the test parameters and generate a random run ID
var run = {
- // Generate a random run ID
- _id: (Date.now()*1000 + Math.round(Math.random()*1000)).toString(36),
+ runId: (Date.now()*1000 + Math.round(Math.random()*1000)).toString(36),
params: {
url: req.body.url,
- waitForResponse: req.body.waitForResponse || true
+ waitForResponse: req.body.waitForResponse !== false
}
};
// Add test to the testQueue
- debug('Adding test %s to the queue', run._id);
- var queuing = runsQueue.push(run._id);
+ debug('Adding test %s to the queue', run.runId);
+ var queuePromise = queue.push(run.runId);
+
-
// Save the run to the datastore
- var position = runsQueue.getPosition(run._id);
- run.status = {
- statusCode: (position === 0) ? STATUS_RUNNING : STATUS_AWAITING,
- position: position
- };
- runsDatastore.add(run);
+ runsDatastore.add(run, queuePromise.startingPosition);
// Listening for position updates
- queuing.progress(function(position) {
- var savedRun = runsDatastore.get(run._id);
- savedRun.status = {
- statusCode: STATUS_AWAITING,
- position: position
- };
- runsDatastore.update(savedRun);
+ queuePromise.progress(function(position) {
+ runsDatastore.updatePosition(run.runId, position);
});
+ // Let's start the run
+ queuePromise.then(function() {
- queuing.then(function() {
-
+ runsDatastore.updatePosition(run.runId, 0);
+
+ debug('Launching test %s on %s', run.runId, run.params.url);
+
+ new YellowLabTools(run.params.url)
+ .then(function(data) {
+
+ debug('Success');
+ runsDatastore.markAsComplete(run.runId);
+
+ // Save result in datastore
+ data.runId = run.runId;
+ resultsDatastore.saveResult(data);
+
+ // Send result if the user was waiting
+ if (run.params.waitForResponse) {
+
+ res.redirect(302, '/api/results/' + run.runId);
+ }
+
+ }).fail(function(err) {
+
+ console.error('Test failed for %s', run.params.url);
+ console.error(err);
+ console.error(err.stack);
+
+ runsDatastore.markAsFailed(run.runId);
+
+ }).finally(function() {
+ queue.remove(run.runId);
+ });
+
+ }).fail(function(err) {
+ console.error('Error or YLT\'s core instanciation');
+ console.error(err);
+ console.error(err.stack);
});
- // The user doesn't not want to wait for the response
- if (!params.waitForResponse) {
-
- // Sending just the test id
- res.setHeader('Content-Type', 'application/javascript');
- res.send(JSON.stringify({
- testId: testId
- }));
+ // The user doesn't not want to wait for the response, sending the run ID only
+ if (!run.params.waitForResponse) {
+ console.log('Sending response without waiting.');
+ res.setHeader('Content-Type', 'application/json');
+ res.send(JSON.stringify({runId: run.runId}));
}
});
// Retrive one run by id
- app.get('/run/:id', function(req, res) {
+ app.get('/api/runs/:id', function(req, res) {
+ var runId = req.params.id;
+ var run = runsDatastore.get(runId);
+
+ if (run) {
+ res.setHeader('Content-Type', 'application/json');
+ res.send(JSON.stringify(run, null, 2));
+ } else {
+ res.status(404).send('Not found');
+ }
});
// Delete one run by id
- /*app.delete('/run/:id', function(req, res) {
- // NOT YET
+ /*app.delete('/api/runs/:id', function(req, res) {
+ deleteRun()
});*/
+ // Delete all
+ /*app.delete('/api/runs', function(req, res) {
+ purgeRuns()
+ });
- var STATUS_AWAITING = 'awaiting';
- var STATUS_RUNNING = 'running';
- var STATUS_DONE = 'done';
- var STATUS_FAILED = 'failed';
+ // List all
+ app.get('/api/runs', function(req, res) {
+ listRuns()
+ });
-}
+ // Exists
+ app.head('/api/runs/:id', function(req, res) {
+ existsX();
+ // Retourne 200 si existe ou 404 si n'existe pas
+ });
+ */
+
+ // Retrive one result by id
+ app.get('/api/results/:id', function(req, res) {
+ var runId = req.params.id;
+
+ resultsDatastore.getResult(runId)
+ .then(function(data) {
+ // This is the pivot format, we might need to clean it first?
+
+ // Hide phantomas results
+ data.toolsResults.phantomas = {};
+
+ res.setHeader('Content-Type', 'application/json');
+ res.send(JSON.stringify(data, null, 2));
+ }).fail(function() {
+ res.status(404).send('Not found');
+ });
+ });
+
+};
module.exports = ApiController;
\ No newline at end of file
diff --git a/lib/server/controllers/uiController.js b/lib/server/controllers/uiController.js
index e69de29..a6d906d 100644
--- a/lib/server/controllers/uiController.js
+++ b/lib/server/controllers/uiController.js
@@ -0,0 +1,13 @@
+var UiController = function(app) {
+ 'use strict';
+
+
+ // Create a new run
+ app.get('/', function(req, res) {
+
+ res.setHeader('Content-Type', 'text/html');
+ res.send('Test');
+ });
+};
+
+module.exports = UiController;
\ No newline at end of file
diff --git a/lib/server/datastores/resultsDatastore.js b/lib/server/datastores/resultsDatastore.js
index 954f9b7..3d2ce8a 100644
--- a/lib/server/datastores/resultsDatastore.js
+++ b/lib/server/datastores/resultsDatastore.js
@@ -1,9 +1,79 @@
+var fs = require('fs');
+var rimraf = require('rimraf');
+var path = require('path');
+var Q = require('q');
function ResultsDatastore() {
'use strict';
-
+ var resultFileName = 'results.json';
+ var resultsFolderName = 'results';
+ var resultsDir = path.join(__dirname, '..', '..', '..', resultsFolderName);
+
+
+ this.saveResult = function(testResults) {
+
+ var promise = createResultFolder(testResults.runId);
+
+ promise.then(function() {
+
+ var resultFilePath = path.join(resultsDir, testResults.runId, resultFileName);
+
+ return Q.nfcall(fs.writeFile, resultFilePath, JSON.stringify(testResults, null, 2));
+ });
+
+ return promise;
+ };
+
+
+ this.getResult = function(runId) {
+
+ var resultFilePath = path.join(resultsDir, runId, resultFileName);
+
+ return Q.nfcall(fs.readFile, resultFilePath, {encoding: 'utf8'}).then(function(data) {
+ return JSON.parse(data);
+ });
+ };
+
+
+ this.deleteResult = function(runId) {
+ var folder = path.join(resultsDir, runId);
+
+ return Q.nfcall(rimraf, folder);
+ };
+
+
+ // The folder /results/folderName/
+ function createResultFolder(folderName) {
+ var folder = path.join(resultsDir, folderName);
+
+ return createGlobalFolder().then(function() {
+ return Q.nfcall(fs.mkdir, folder);
+ });
+ }
+
+ // The folder /results/
+ function createGlobalFolder() {
+ var deferred = Q.defer();
+
+ // Create the results folder if it doesn't exist
+ fs.exists(resultsDir, function(exists) {
+ if (exists) {
+ deferred.resolve();
+ } else {
+ fs.mkdir(resultsDir, function(err) {
+ if (err) {
+ deferred.reject(err);
+ } else {
+ deferred.resolve();
+ }
+ });
+ }
+ });
+
+ return deferred.promise;
+ }
}
module.exports = ResultsDatastore;
\ No newline at end of file
diff --git a/lib/server/datastores/runsDatastore.js b/lib/server/datastores/runsDatastore.js
index c84ac4e..cf0af10 100644
--- a/lib/server/datastores/runsDatastore.js
+++ b/lib/server/datastores/runsDatastore.js
@@ -7,23 +7,68 @@ function RunsDatastore() {
// For the moment, maybe one day
var runs = {};
+ var STATUS_AWAITING = 'awaiting';
+ var STATUS_RUNNING = 'running';
+ var STATUS_COMPLETE = 'complete';
+ var STATUS_FAILED = 'failed';
- this.add = function(run) {
- runs[run._id] = run;
+
+ this.add = function(run, position) {
+ runs[run.runId] = run;
+ this.updatePosition(run.runId, position);
};
+
this.get = function(runId) {
return runs[runId];
};
+
- this.update = function(run) {
- runs[run._id] = run;
+ this.updatePosition = function(runId, position) {
+ var run = runs[runId];
+
+ if (position > 0) {
+ run.status = {
+ statusCode: STATUS_AWAITING,
+ position: position
+ };
+ } else {
+ run.status = {
+ statusCode: STATUS_RUNNING
+ };
+ }
+
+ runs[runId] = run;
};
+
+ this.markAsComplete = function(runId) {
+ var run = runs[runId];
+
+ run.status = {
+ statusCode: STATUS_COMPLETE
+ };
+
+ runs[runId] = run;
+ };
+
+
+ this.markAsFailed = function(runId) {
+ var run = runs[runId];
+
+ run.status = {
+ statusCode: STATUS_FAILED
+ };
+
+ runs[runId] = run;
+ };
+
+
this.delete = function(runId) {
delete runs[runId];
};
+
this.list = function() {
var runsArray = [];
Object.keys(runs).forEach(function(key) {
diff --git a/lib/server/datastores/runsQueue.js b/lib/server/datastores/runsQueue.js
index ad1a6b8..26a6717 100644
--- a/lib/server/datastores/runsQueue.js
+++ b/lib/server/datastores/runsQueue.js
@@ -9,8 +9,9 @@ function RunsQueue() {
this.push = function(runId) {
var deferred = Q.defer();
+ var startingPosition = queue.length;
- if (queue.length === 0) {
+ if (startingPosition === 0) {
// The queue is empty, let's run immediatly
queue.push({
@@ -32,7 +33,9 @@ function RunsQueue() {
});
}
- return deferred.promise;
+ var promise = deferred.promise;
+ promise.startingPosition = startingPosition;
+ return promise;
};
@@ -68,6 +71,10 @@ function RunsQueue() {
});
};
+
+ this.length = function() {
+ return queue.length;
+ };
}
module.exports = RunsQueue;
\ No newline at end of file
diff --git a/lib/tools/phantomasWrapper.js b/lib/tools/phantomasWrapper.js
index e1df01d..adb3dd7 100644
--- a/lib/tools/phantomasWrapper.js
+++ b/lib/tools/phantomasWrapper.js
@@ -64,7 +64,7 @@ var PhantomasWrapper = function() {
}, 600000);
// It's time to launch the test!!!
- var triesNumber = 3;
+ var triesNumber = 2;
async.retry(triesNumber, function(cb) {
phantomas(task.url, options, function(err, json, results) {
diff --git a/lib/yellowlabtools.js b/lib/yellowlabtools.js
index a0f4e38..a56e49d 100644
--- a/lib/yellowlabtools.js
+++ b/lib/yellowlabtools.js
@@ -14,24 +14,19 @@ var YellowLabTools = function(url, options) {
} else {
- // Generate a test id
- var testId = (Date.now()*1000 + Math.round(Math.random()*1000)).toString(36);
-
if (url.indexOf('http://') !== 0 && url.indexOf('https://') !== 0) {
url = 'http://' + url;
}
var params = {
- testId: testId,
url: url,
options: options || {}
};
- var runner = new Runner(params);
-
- runner.then(function(data) {
- deferred.resolve(data);
- });
+ var runner = new Runner(params)
+ .then(function(data) {
+ deferred.resolve(data);
+ });
}
return deferred.promise;
diff --git a/package.json b/package.json
index 29bca24..9ceae90 100644
--- a/package.json
+++ b/package.json
@@ -16,6 +16,7 @@
"debug": "^2.1.0",
"express": "~4.10.4",
"phantomas": "1.7.0",
+ "rimraf": "^2.2.8",
"socket.io": "~1.2.0"
},
"devDependencies": {
diff --git a/test/api/resultsDatastoreTest.js b/test/api/resultsDatastoreTest.js
new file mode 100644
index 0000000..4e8bdaf
--- /dev/null
+++ b/test/api/resultsDatastoreTest.js
@@ -0,0 +1,74 @@
+var should = require('chai').should();
+var resultsDatastore = require('../../lib/server/datastores/resultsDatastore');
+
+describe('resultsDatastore', function() {
+
+ var datastore = new resultsDatastore();
+
+ var testId1 = '123456789';
+ var testData1 = {
+ runId: testId1,
+ other: {
+ foo: 'foo',
+ bar: 1
+ }
+ };
+
+
+ it('should store a result', function(done) {
+ datastore.should.have.a.property('saveResult').that.is.a('function');
+
+ datastore.saveResult(testData1).then(function() {
+ done();
+ }).fail(function(err) {
+ done(err);
+ });
+ });
+
+ it('should store another result', function(done) {
+ var testData2 = {
+ runId: '987654321',
+ other: {
+ foo: 'foo',
+ bar: 2
+ }
+ };
+
+ datastore.saveResult(testData2).then(function() {
+ done();
+ }).fail(function(err) {
+ done(err);
+ });
+ });
+
+ it('should retrieve a result', function(done) {
+ datastore.getResult(testId1)
+ .then(function(results) {
+
+ // Compare results with testData
+ results.should.deep.equal(testData1);
+
+ done();
+ }).fail(function(err) {
+ done(err);
+ });
+ });
+
+ it('should delete a result', function(done) {
+ datastore.deleteResult(testId1)
+ .then(function() {
+ done();
+ }).fail(function(err) {
+ done(err);
+ });
+ });
+
+ it('should not find the result anymore', function(done) {
+ datastore.getResult(testId1)
+ .then(function(results) {
+ done('Error, the result is still in the datastore');
+ }).fail(function(err) {
+ done();
+ });
+ });
+});
diff --git a/test/api/runsDatastoreTest.js b/test/api/runsDatastoreTest.js
new file mode 100644
index 0000000..1ac4d03
--- /dev/null
+++ b/test/api/runsDatastoreTest.js
@@ -0,0 +1,81 @@
+var should = require('chai').should();
+var runsDatastore = require('../../lib/server/datastores/runsDatastore');
+
+describe('runsDatastore', function() {
+
+ var datastore = new runsDatastore();
+
+ var firstRunId = 333;
+ var secondRunId = 999;
+
+ it('should accept new runs', function() {
+ datastore.should.have.a.property('add').that.is.a('function');
+
+ datastore.add({
+ runId: firstRunId,
+ otherData: 123456789
+ }, 0);
+
+ datastore.add({
+ runId: secondRunId,
+ otherData: 'whatever'
+ }, 1);
+ });
+
+ it('should have stored the runs with a status "runnung"', function() {
+ datastore.should.have.a.property('get').that.is.a('function');
+
+ var firstRun = datastore.get(firstRunId);
+ firstRun.should.have.a.property('runId').that.equals(firstRunId);
+ firstRun.should.have.a.property('status').that.deep.equals({
+ statusCode: 'running'
+ });
+
+ var secondRun = datastore.get(secondRunId);
+ secondRun.should.have.a.property('runId').that.equals(secondRunId);
+ secondRun.should.have.a.property('status').that.deep.equals({
+ statusCode: 'awaiting',
+ position: 1
+ });
+
+ });
+
+ it('should have exactly 2 runs in the store', function() {
+ var runs = datastore.list();
+ runs.should.be.a('array');
+ runs.should.have.length(2);
+ runs[0].should.have.a.property('runId').that.equals(firstRunId);
+ });
+
+ it('shoud update statuses correctly', function() {
+
+ datastore.markAsComplete(firstRunId);
+ var firstRun = datastore.get(firstRunId);
+ firstRun.should.have.a.property('status').that.deep.equals({
+ statusCode: 'complete'
+ });
+
+ datastore.updatePosition(secondRunId, 0);
+ var secondRun = datastore.get(secondRunId);
+ secondRun.should.have.a.property('status').that.deep.equals({
+ statusCode: 'running'
+ });
+
+ datastore.markAsFailed(secondRunId);
+ secondRun = datastore.get(secondRunId);
+ secondRun.should.have.a.property('status').that.deep.equals({
+ statusCode: 'failed'
+ });
+
+ });
+
+ it('should delete a run', function() {
+ datastore.delete(firstRunId);
+
+ var runs = datastore.list();
+ runs.should.be.a('array');
+ runs.should.have.length(1);
+
+ runs[0].should.have.a.property('runId').that.equals(secondRunId);
+ });
+});
diff --git a/test/server/runsQueueTest.js b/test/api/runsQueueTest.js
similarity index 88%
rename from test/server/runsQueueTest.js
rename to test/api/runsQueueTest.js
index 45a2a23..1e8798d 100644
--- a/test/server/runsQueueTest.js
+++ b/test/api/runsQueueTest.js
@@ -4,13 +4,15 @@ var runsQueue = require('../../lib/server/datastores/runsQueue.js');
describe('runsQueue', function() {
var queue = new runsQueue();
+ var aaaRun = null;
+ var bbbRun = null;
var cccRun = null;
it('should accept a new runId', function(done) {
queue.should.have.a.property('push').that.is.a('function');
- var aaaRun = queue.push('aaa');
- queue.push('bbb');
+ aaaRun = queue.push('aaa');
+ bbbRun = queue.push('bbb');
aaaRun.then(function() {
done();
@@ -20,9 +22,11 @@ describe('runsQueue', function() {
it('should return the right positions', function() {
var aaaPosition = queue.getPosition('aaa');
aaaPosition.should.equal(0);
+ aaaRun.startingPosition.should.equal(0);
var bbbPosition = queue.getPosition('bbb');
bbbPosition.should.equal(1);
+ bbbRun.startingPosition.should.equal(1);
var cccPosition = queue.getPosition('ccc');
cccPosition.should.equal(-1);
diff --git a/test/api/phantomasWrapperTest.js b/test/core/phantomasWrapperTest.js
similarity index 100%
rename from test/api/phantomasWrapperTest.js
rename to test/core/phantomasWrapperTest.js
diff --git a/test/api/rulesCheckerTest.js b/test/core/rulesCheckerTest.js
similarity index 100%
rename from test/api/rulesCheckerTest.js
rename to test/core/rulesCheckerTest.js
diff --git a/test/api/yellowlabtoolsTest.js b/test/core/yellowlabtoolsTest.js
similarity index 100%
rename from test/api/yellowlabtoolsTest.js
rename to test/core/yellowlabtoolsTest.js
diff --git a/test/server/runsDatastoreTest.js b/test/server/runsDatastoreTest.js
deleted file mode 100644
index 750956d..0000000
--- a/test/server/runsDatastoreTest.js
+++ /dev/null
@@ -1,41 +0,0 @@
-var should = require('chai').should();
-var runsDatastore = require('../../lib/server/datastores/runsDatastore');
-
-describe('runsDatastore', function() {
-
- var datastore = new runsDatastore();
-
- var randomId = Math.round(Math.random() * 100000);
-
- it('should accept a new run', function() {
- datastore.should.have.a.property('add').that.is.a('function');
-
- datastore.add({
- _id: randomId,
- otherData: 123456789
- });
- });
-
- it('should have stored the run', function() {
- datastore.should.have.a.property('get').that.is.a('function');
-
- var run = datastore.get(randomId);
-
- run.should.have.a.property('_id').that.equals(randomId);
- });
-
- it('should have exactly 1 run in the store', function() {
- var runs = datastore.list();
- runs.should.be.a('array');
- runs.should.have.length(1);
- runs[0].should.have.a.property('_id').that.equals(randomId);
- });
-
- it('should delete the run', function() {
- datastore.delete(randomId);
-
- var runs = datastore.list();
- runs.should.be.a('array');
- runs.should.have.length(0);
- });
-});
From 49015e3422723d2bd8f43c6c01799cc2c4ddd96d Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ga=C3=ABl=20M=C3=A9tais?=
Date: Tue, 9 Dec 2014 09:04:28 +0100
Subject: [PATCH 08/98] API authentification (tbc...)
---
Gruntfile.js | 36 ++++-
bin/server.js | 17 ++-
bower.json | 2 +-
lib/server/authMiddleware.js | 52 +++++++
lib/server/controllers/apiController.js | 12 +-
package.json | 12 +-
server_config/settings-prod.json | 10 +-
server_config/settings.json | 10 +-
test/api/apiTest.js | 171 ++++++++++++++++++++++++
9 files changed, 300 insertions(+), 22 deletions(-)
create mode 100644 lib/server/authMiddleware.js
create mode 100644 test/api/apiTest.js
diff --git a/Gruntfile.js b/Gruntfile.js
index 240d1fe..b9e7296 100644
--- a/Gruntfile.js
+++ b/Gruntfile.js
@@ -1,5 +1,11 @@
module.exports = function(grunt) {
+ var DEV_SERVER_PORT = 8383;
+ var TEST_SERVER_PORT = 8387;
+
+ // Tell our Express server that Grunt launched it
+ process.env.GRUNTED = true;
+
// Project configuration.
grunt.initConfig({
pkg: grunt.file.readJSON('package.json'),
@@ -77,7 +83,7 @@ module.exports = function(grunt) {
options: {
reporter: 'spec',
},
- src: ['coverage/test/server/runsDatastoreTest.js']
+ src: ['coverage/test/api/apiTest.js']
},
coverage: {
options: {
@@ -85,11 +91,25 @@ module.exports = function(grunt) {
quiet: true,
captureFile: 'coverage/coverage.html'
},
- src: ['coverage/test/api/*.js']
+ src: ['coverage/test/core/*.js', 'coverage/test/api/*.js']
}
},
express: {
- test: {
+ dev: {
+ options: {
+ port: 8383,
+ server: './bin/server.js',
+ serverreload: true,
+ showStack: true
+ }
+ },
+ testServer: {
+ options: {
+ port: 8387,
+ server: './bin/server.js'
+ }
+ },
+ testSuite: {
options: {
port: 8388,
bases: 'test/www'
@@ -114,10 +134,15 @@ module.exports = function(grunt) {
'jshint'
]);
+ grunt.registerTask('dev', [
+ 'express:dev'
+ ]);
+
grunt.registerTask('test', [
'build',
'jshint',
- 'express:test',
+ 'express:testServer',
+ 'express:testSuite',
'clean:coverage',
'blanket',
'copy:coverage',
@@ -128,7 +153,8 @@ module.exports = function(grunt) {
grunt.registerTask('test-current-work', [
'build',
'jshint',
- 'express:test',
+ 'express:testServer',
+ 'express:testSuite',
'clean:coverage',
'blanket',
'copy:coverage',
diff --git a/bin/server.js b/bin/server.js
index 3cd54f7..8fda8f9 100644
--- a/bin/server.js
+++ b/bin/server.js
@@ -7,8 +7,11 @@ var server = require('http').createServer(app);
var bodyParser = require('body-parser');
var compress = require('compression');
+var authMiddleware = require('../lib/server/authMiddleware');
+
app.use(compress());
app.use(bodyParser.json());
+app.use(authMiddleware);
// Initialize the controllers
@@ -16,7 +19,13 @@ var apiController = require('../lib/server/controllers/apiController')
var uiController = require('../lib/server/controllers/uiController')(app);
-// Launch the server
-server.listen(settings.serverPort, function() {
- console.log('Listening on port %d', server.address().port);
-});
\ No newline at end of file
+// Let's start the server!
+if (!process.env.GRUNTED) {
+ // The server is not launched by Grunt
+ server.listen(settings.serverPort, function() {
+ console.log('Listening on port %d', server.address().port);
+ });
+}
+
+// For Grunt
+module.exports = app;
\ No newline at end of file
diff --git a/bower.json b/bower.json
index f248d80..b308e91 100644
--- a/bower.json
+++ b/bower.json
@@ -1,7 +1,7 @@
{
"name": "yellowlabtools",
"dependencies": {
- "angular": "~1.3.1",
+ "angular": "~1.3.5",
"ngModal": "git://github.com/gmetais/ngModal.git#1.2.3"
}
}
diff --git a/lib/server/authMiddleware.js b/lib/server/authMiddleware.js
new file mode 100644
index 0000000..e9f68c7
--- /dev/null
+++ b/lib/server/authMiddleware.js
@@ -0,0 +1,52 @@
+var config = require('../../server_config/settings.json');
+
+var jwt = require('jwt-simple');
+var debug = require('debug')('authMiddleware');
+
+
+var authMiddleware = function(req, res, next) {
+ 'use strict';
+
+ if (req.path.indexOf('/api/') === 0) {
+
+ // Test if it's an authorized key
+ if (req.headers && req.headers['x-api-key'] && isApiKeyValid(req.headers['x-api-key'])) {
+ next();
+ return;
+ }
+
+ // Test if it's an authorized token
+ if (req.headers && req.headers['x-api-token'] && isTokenValid(req.headers['x-api-token'])) {
+ next();
+ return;
+ }
+
+ res.status(401).send('Unauthorized');
+ }
+};
+
+
+function isApiKeyValid(apiKey) {
+ return (config.authorizedKeys[apiKey]) ? true : false;
+}
+
+
+function isTokenValid(token) {
+
+ var data = null;
+
+ try {
+ jwt.decode(token, config.tokenSalt);
+ } catch(err) {
+ debug('Error while decoding token');
+ debug(err);
+ return false;
+ }
+
+ return data.expire &&
+ data.expire > Date.now() &&
+ data.application &&
+ config.authorizedApplications.indexOf(data.application) >= 0;
+}
+
+module.exports = authMiddleware;
\ No newline at end of file
diff --git a/lib/server/controllers/apiController.js b/lib/server/controllers/apiController.js
index ee4e0de..a1c0952 100644
--- a/lib/server/controllers/apiController.js
+++ b/lib/server/controllers/apiController.js
@@ -13,10 +13,7 @@ var ApiController = function(app) {
var runsDatastore = new RunsDatastore();
var resultsDatastore = new ResultsDatastore();
- // Retrieve the list of all runs
- /*app.get('/api/runs', function(req, res) {
- // NOT YET
- });*/
+
// Create a new run
app.post('/api/runs', function(req, res) {
@@ -107,6 +104,11 @@ var ApiController = function(app) {
}
});
+ // Retrieve the list of all runs
+ /*app.get('/api/runs', function(req, res) {
+ // NOT YET
+ });*/
+
// Delete one run by id
/*app.delete('/api/runs/:id', function(req, res) {
deleteRun()
@@ -125,7 +127,7 @@ var ApiController = function(app) {
// Exists
app.head('/api/runs/:id', function(req, res) {
existsX();
- // Retourne 200 si existe ou 404 si n'existe pas
+ // Returns 200 if the result exists or 404 if not
});
*/
diff --git a/package.json b/package.json
index 9ceae90..0e8fe6c 100644
--- a/package.json
+++ b/package.json
@@ -11,16 +11,17 @@
"main": "./lib/yellowlabtools.js",
"dependencies": {
"async": "~0.9.0",
- "body-parser": "~1.9.2",
- "compression": "~1.2.0",
+ "body-parser": "~1.10.0",
+ "compression": "~1.2.1",
"debug": "^2.1.0",
"express": "~4.10.4",
+ "jwt-simple": "^0.2.0",
"phantomas": "1.7.0",
"rimraf": "^2.2.8",
"socket.io": "~1.2.0"
},
"devDependencies": {
- "chai": "^1.9.2",
+ "chai": "^1.10.0",
"grunt": "^0.4.5",
"grunt-blanket": "^0.0.8",
"grunt-contrib-clean": "^0.6.0",
@@ -29,11 +30,12 @@
"grunt-contrib-less": "^0.12.0",
"grunt-express": "^1.4.1",
"grunt-fontsmith": "^0.9.1",
- "grunt-mocha-test": "^0.12.2",
+ "grunt-mocha-test": "^0.12.4",
"matchdep": "^0.3.0",
"mocha": "^2.0.1",
- "phantomjs": "^1.9.10",
+ "phantomjs": "^1.9.12",
"q": "^1.1.2",
+ "request": "^2.49.0",
"sinon": "^1.12.1",
"sinon-chai": "^2.6.0"
},
diff --git a/server_config/settings-prod.json b/server_config/settings-prod.json
index f7b4cb5..8d06c6a 100644
--- a/server_config/settings-prod.json
+++ b/server_config/settings-prod.json
@@ -1,4 +1,12 @@
{
"serverPort": 80,
- "googleAnalyticsId": "UA-54493828-1"
+ "googleAnalyticsId": "UA-54493828-1",
+
+ "authorized-keys": {
+
+ },
+ "tokenSalt": "",
+ "authorizedApplications": [
+
+ ]
}
\ No newline at end of file
diff --git a/server_config/settings.json b/server_config/settings.json
index fb400fe..70f03bf 100644
--- a/server_config/settings.json
+++ b/server_config/settings.json
@@ -1,4 +1,12 @@
{
"serverPort": 8383,
- "googleAnalyticsId": ""
+ "googleAnalyticsId": "",
+
+ "authorizedKeys": {
+ "1234567890": "contact@gaelmetais.com"
+ },
+ "tokenSalt": "lake-city",
+ "authorizedApplications": [
+ "frontend"
+ ]
}
\ No newline at end of file
diff --git a/test/api/apiTest.js b/test/api/apiTest.js
new file mode 100644
index 0000000..e2469a0
--- /dev/null
+++ b/test/api/apiTest.js
@@ -0,0 +1,171 @@
+var should = require('chai').should();
+var request = require('request');
+var jwt = require('jwt-simple');
+
+var config = {
+ "authorizedKeys": {
+ "1234567890": "test@test.com"
+ },
+ "tokenSalt": "test-salt",
+ "authorizedApplications": ["wooot"]
+};
+
+var apiUrl = 'http://localhost:8387/api';
+var wwwUrl = 'http://localhost:8388';
+
+describe('api', function() {
+
+ var runId;
+
+ it('should not accept a query if there is no key in headers', function(done) {
+ this.timeout(5000);
+
+ request({
+ method: 'POST',
+ url: apiUrl + '/runs',
+ body: {
+ url: wwwUrl + '/simple-page.html',
+ waitForResponse: false
+ },
+ json: true
+ }, function(error, response, body) {
+ if (!error && response.statusCode === 401) {
+ done();
+ } else {
+ done(error || response.statusCode);
+ }
+ });
+ });
+
+ it('should refuse a query with an invalid key', function(done) {
+ this.timeout(5000);
+
+ request({
+ method: 'POST',
+ url: apiUrl + '/runs',
+ body: {
+ url: wwwUrl + '/simple-page.html',
+ waitForResponse: false
+ },
+ json: true,
+ headers: {
+ 'X-Api-Key': 'invalid'
+ }
+ }, function(error, response, body) {
+ if (!error && response.statusCode === 401) {
+ done();
+ } else {
+ done(error || response.statusCode);
+ }
+ });
+ });
+
+ it('should accept a query with a valid key', function(done) {
+ this.timeout(5000);
+
+ request({
+ method: 'POST',
+ url: apiUrl + '/runs',
+ body: {
+ url: wwwUrl + '/simple-page.html',
+ waitForResponse: false
+ },
+ json: true,
+ headers: {
+ 'X-Api-Key': Object.keys(config.authorizedKeys)[0]
+ }
+ }, function(error, response, body) {
+ if (!error && response.statusCode === 200) {
+
+ runId = body.runId;
+ runId.should.be.a('string');
+ done();
+
+ } else {
+ done(error || response.statusCode);
+ }
+ });
+ });
+
+ it('should refuse an expired token', function(done) {
+ this.timeout(5000);
+
+ request({
+ method: 'POST',
+ url: apiUrl + '/runs',
+ body: {
+ url: wwwUrl + '/simple-page.html',
+ waitForResponse: false
+ },
+ json: true,
+ headers: {
+ 'X-Api-Token': jwt.encode({
+ application: config.authorizedApplications[0],
+ expire: Date.now() - 60000
+ }, config.tokenSalt)
+ }
+ }, function(error, response, body) {
+ if (!error && response.statusCode === 401) {
+ done();
+ } else {
+ done(error || response.statusCode);
+ }
+ });
+ });
+
+ it('should refuse a token from an unknown app', function(done) {
+ this.timeout(5000);
+
+ request({
+ method: 'POST',
+ url: apiUrl + '/runs',
+ body: {
+ url: wwwUrl + '/simple-page.html',
+ waitForResponse: false
+ },
+ json: true,
+ headers: {
+ 'X-Api-Token': jwt.encode({
+ application: 'unknown-app',
+ expire: Date.now() + 60000
+ }, config.tokenSalt)
+ }
+ }, function(error, response, body) {
+ if (!error && response.statusCode === 401) {
+ done();
+ } else {
+ done(error || response.statusCode);
+ }
+ });
+ });
+
+ it('should accept a good token', function(done) {
+ this.timeout(5000);
+
+ request({
+ method: 'POST',
+ url: apiUrl + '/runs',
+ body: {
+ url: wwwUrl + '/simple-page.html',
+ waitForResponse: false
+ },
+ json: true,
+ headers: {
+ 'X-Api-Token': jwt.encode({
+ application: config.authorizedApplications[0],
+ expire: Date.now() + 60000
+ }, config.tokenSalt)
+ }
+ }, function(error, response, body) {
+ if (!error && response.statusCode === 200) {
+
+ runId = body.runId;
+ runId.should.be.a('string');
+ done();
+
+ } else {
+ done(error || response.statusCode);
+ }
+ });
+ });
+});
From 84faabca868591fe497d1677145f11fc1ed00047 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ga=C3=ABl=20M=C3=A9tais?=
Date: Wed, 10 Dec 2014 08:16:11 +0100
Subject: [PATCH 09/98] Fix Auth token
---
lib/server/authMiddleware.js | 2 +-
lib/tools/phantomasWrapper.js | 4 ++--
test/api/apiTest.js | 6 +++---
3 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/lib/server/authMiddleware.js b/lib/server/authMiddleware.js
index e9f68c7..fe70125 100644
--- a/lib/server/authMiddleware.js
+++ b/lib/server/authMiddleware.js
@@ -36,7 +36,7 @@ function isTokenValid(token) {
var data = null;
try {
- jwt.decode(token, config.tokenSalt);
+ data = jwt.decode(token, config.tokenSalt);
} catch(err) {
debug('Error while decoding token');
debug(err);
diff --git a/lib/tools/phantomasWrapper.js b/lib/tools/phantomasWrapper.js
index adb3dd7..8dbd614 100644
--- a/lib/tools/phantomasWrapper.js
+++ b/lib/tools/phantomasWrapper.js
@@ -58,10 +58,10 @@ var PhantomasWrapper = function() {
// Kill the application if nothing happens for 10 minutes
var killer = setTimeout(function() {
- debug('Killing the app because the test on ' + task.url + ' was launched 10 minutes ago');
+ debug('Killing the app because the test on %s was launched %d seconds ago', task.url, 3*options.timeout);
// If in server mode, forever will restart the server
process.exit(1);
- }, 600000);
+ }, 3*options.timeout*60*1000);
// It's time to launch the test!!!
var triesNumber = 2;
diff --git a/test/api/apiTest.js b/test/api/apiTest.js
index e2469a0..dacc553 100644
--- a/test/api/apiTest.js
+++ b/test/api/apiTest.js
@@ -4,10 +4,10 @@ var jwt = require('jwt-simple');
var config = {
"authorizedKeys": {
- "1234567890": "test@test.com"
+ "1234567890": "contact@gaelmetais.com"
},
- "tokenSalt": "test-salt",
- "authorizedApplications": ["wooot"]
+ "tokenSalt": "lake-city",
+ "authorizedApplications": ["frontend"]
};
var apiUrl = 'http://localhost:8387/api';
From 0da2ad0c1044568ae78fe8ee2d81b42f64ccecb4 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ga=C3=ABl=20M=C3=A9tais?=
Date: Fri, 12 Dec 2014 20:26:19 +0100
Subject: [PATCH 10/98] Add API limits configurable in settings.json
---
bin/server.js | 5 +-
lib/server/authMiddleware.js | 52 ------
lib/server/middlewares/apiLimitsMiddleware.js | 85 +++++++++
lib/server/middlewares/authMiddleware.js | 42 +++++
package.json | 1 -
server_config/settings.json | 6 +-
test/api/apiTest.js | 163 ++++++++----------
7 files changed, 202 insertions(+), 152 deletions(-)
delete mode 100644 lib/server/authMiddleware.js
create mode 100644 lib/server/middlewares/apiLimitsMiddleware.js
create mode 100644 lib/server/middlewares/authMiddleware.js
diff --git a/bin/server.js b/bin/server.js
index 8fda8f9..f8b76aa 100644
--- a/bin/server.js
+++ b/bin/server.js
@@ -7,11 +7,14 @@ var server = require('http').createServer(app);
var bodyParser = require('body-parser');
var compress = require('compression');
-var authMiddleware = require('../lib/server/authMiddleware');
+var authMiddleware = require('../lib/server/middlewares/authMiddleware');
+var apiLimitsMiddleware = require('../lib/server/middlewares/apiLimitsMiddleware');
+
app.use(compress());
app.use(bodyParser.json());
app.use(authMiddleware);
+app.use(apiLimitsMiddleware);
// Initialize the controllers
diff --git a/lib/server/authMiddleware.js b/lib/server/authMiddleware.js
deleted file mode 100644
index fe70125..0000000
--- a/lib/server/authMiddleware.js
+++ /dev/null
@@ -1,52 +0,0 @@
-var config = require('../../server_config/settings.json');
-
-var jwt = require('jwt-simple');
-var debug = require('debug')('authMiddleware');
-
-
-var authMiddleware = function(req, res, next) {
- 'use strict';
-
- if (req.path.indexOf('/api/') === 0) {
-
- // Test if it's an authorized key
- if (req.headers && req.headers['x-api-key'] && isApiKeyValid(req.headers['x-api-key'])) {
- next();
- return;
- }
-
- // Test if it's an authorized token
- if (req.headers && req.headers['x-api-token'] && isTokenValid(req.headers['x-api-token'])) {
- next();
- return;
- }
-
- res.status(401).send('Unauthorized');
- }
-};
-
-
-function isApiKeyValid(apiKey) {
- return (config.authorizedKeys[apiKey]) ? true : false;
-}
-
-
-function isTokenValid(token) {
-
- var data = null;
-
- try {
- data = jwt.decode(token, config.tokenSalt);
- } catch(err) {
- debug('Error while decoding token');
- debug(err);
- return false;
- }
-
- return data.expire &&
- data.expire > Date.now() &&
- data.application &&
- config.authorizedApplications.indexOf(data.application) >= 0;
-}
-
-module.exports = authMiddleware;
\ No newline at end of file
diff --git a/lib/server/middlewares/apiLimitsMiddleware.js b/lib/server/middlewares/apiLimitsMiddleware.js
new file mode 100644
index 0000000..7c17015
--- /dev/null
+++ b/lib/server/middlewares/apiLimitsMiddleware.js
@@ -0,0 +1,85 @@
+var config = require('../../../server_config/settings.json');
+
+var debug = require('debug')('apiLimitsMiddleware');
+
+
+var apiLimitsMiddleware = function(req, res, next) {
+ 'use strict';
+
+ debug('Entering API Limits Middleware with IP address %s', req.connection.remoteAddress);
+
+ if (req.path.indexOf('/api/') === 0 && !res.locals.hasApiKey) {
+
+
+ if (req.path === '/api/runs') {
+
+ if (!runsTable.accepts(req.connection.remoteAddress)) {
+ // Sorry :/
+ debug('Too many tests launched from IP address %s', req.connection.remoteAddress);
+ res.status(429).send('Too Many Requests');
+ return;
+ }
+
+ }
+
+ if (!callsTable.accepts(req.connection.remoteAddress)) {
+ // Sorry :/
+ debug('Too many API requests from IP address %s', req.connection.remoteAddress);
+ res.status(429).send('Too Many Requests');
+ return;
+ }
+
+ debug('Not blocked by the API limits');
+ // It's ok for the moment
+ }
+
+ next();
+};
+
+
+var RecordTable = function(maxPerDay) {
+ var table = {};
+
+ // Check if the user overpassed the limit and save its visit
+ this.accepts = function(ipAddress) {
+ if (table[ipAddress]) {
+
+ this.cleanEntry(ipAddress);
+
+ debug('%d visits in the last 24 hours', table[ipAddress].length);
+
+ if (table[ipAddress].length >= maxPerDay) {
+ return false;
+ } else {
+ table[ipAddress].push(Date.now());
+ }
+
+ } else {
+ table[ipAddress] = [];
+ table[ipAddress].push(Date.now());
+ }
+
+ return true;
+ };
+
+ // Clean the table for this guy
+ this.cleanEntry = function(ipAddress) {
+ table[ipAddress] = table[ipAddress].filter(function(date) {
+ return date > Date.now() - 1000*60*60*24;
+ });
+ };
+
+ // Clean the entire table once in a while
+ this.removeOld = function() {
+ for (var ipAddress in table) {
+ this.cleanEntry(ipAddress);
+ }
+ };
+
+};
+
+// Init the records tables
+var runsTable = new RecordTable(config.maxAnonymousRunsPerDay);
+var callsTable = new RecordTable(config.maxAnonymousCallsPerDay);
+
+module.exports = apiLimitsMiddleware;
\ No newline at end of file
diff --git a/lib/server/middlewares/authMiddleware.js b/lib/server/middlewares/authMiddleware.js
new file mode 100644
index 0000000..e5335e6
--- /dev/null
+++ b/lib/server/middlewares/authMiddleware.js
@@ -0,0 +1,42 @@
+var config = require('../../../server_config/settings.json');
+
+var debug = require('debug')('authMiddleware');
+
+
+var authMiddleware = function(req, res, next) {
+ 'use strict';
+
+ if (req.path.indexOf('/api/') === 0) {
+
+
+ if (req.headers && req.headers['x-api-key']) {
+
+ // Test if it's an authorized key
+ if (isApiKeyValid(req.headers['x-api-key'])) {
+
+ // Come in!
+ debug('Authorized key: %s', req.headers['x-api-key']);
+ res.locals.hasApiKey = true;
+
+ } else {
+
+ // Sorry :/
+ debug('Unauthorized key %s', req.headers['x-api-key']);
+ res.status(401).send('Unauthorized');
+ return;
+ }
+ } else {
+ debug('No authorization key');
+ // It's ok for the moment but you might be blocked by the apiLimitsMiddleware, dude
+ }
+ }
+
+ next();
+};
+
+
+function isApiKeyValid(apiKey) {
+ return (config.authorizedKeys[apiKey]) ? true : false;
+}
+
+module.exports = authMiddleware;
\ No newline at end of file
diff --git a/package.json b/package.json
index 0e8fe6c..1c0eb59 100644
--- a/package.json
+++ b/package.json
@@ -15,7 +15,6 @@
"compression": "~1.2.1",
"debug": "^2.1.0",
"express": "~4.10.4",
- "jwt-simple": "^0.2.0",
"phantomas": "1.7.0",
"rimraf": "^2.2.8",
"socket.io": "~1.2.0"
diff --git a/server_config/settings.json b/server_config/settings.json
index 70f03bf..2b05df1 100644
--- a/server_config/settings.json
+++ b/server_config/settings.json
@@ -5,8 +5,6 @@
"authorizedKeys": {
"1234567890": "contact@gaelmetais.com"
},
- "tokenSalt": "lake-city",
- "authorizedApplications": [
- "frontend"
- ]
+ "maxAnonymousRunsPerDay": 24,
+ "maxAnonymousCallsPerDay": 1000
}
\ No newline at end of file
diff --git a/test/api/apiTest.js b/test/api/apiTest.js
index dacc553..3183bf2 100644
--- a/test/api/apiTest.js
+++ b/test/api/apiTest.js
@@ -1,13 +1,11 @@
var should = require('chai').should();
var request = require('request');
-var jwt = require('jwt-simple');
+var Q = require('q');
var config = {
"authorizedKeys": {
"1234567890": "contact@gaelmetais.com"
- },
- "tokenSalt": "lake-city",
- "authorizedApplications": ["frontend"]
+ }
};
var apiUrl = 'http://localhost:8387/api';
@@ -16,26 +14,7 @@ var wwwUrl = 'http://localhost:8388';
describe('api', function() {
var runId;
-
- it('should not accept a query if there is no key in headers', function(done) {
- this.timeout(5000);
- request({
- method: 'POST',
- url: apiUrl + '/runs',
- body: {
- url: wwwUrl + '/simple-page.html',
- waitForResponse: false
- },
- json: true
- }, function(error, response, body) {
- if (!error && response.statusCode === 401) {
- done();
- } else {
- done(error || response.statusCode);
- }
- });
- });
it('should refuse a query with an invalid key', function(done) {
this.timeout(5000);
@@ -87,85 +66,81 @@ describe('api', function() {
});
});
- it('should refuse an expired token', function(done) {
- this.timeout(5000);
+ it('should accept up to 24 anonymous runs to the API', function(done) {
+ this.timeout(15000);
- request({
- method: 'POST',
- url: apiUrl + '/runs',
- body: {
- url: wwwUrl + '/simple-page.html',
- waitForResponse: false
- },
- json: true,
- headers: {
- 'X-Api-Token': jwt.encode({
- application: config.authorizedApplications[0],
- expire: Date.now() - 60000
- }, config.tokenSalt)
- }
- }, function(error, response, body) {
- if (!error && response.statusCode === 401) {
- done();
- } else {
- done(error || response.statusCode);
- }
- });
- });
+ function launchRun() {
+ var deferred = Q.defer();
- it('should refuse a token from an unknown app', function(done) {
- this.timeout(5000);
+ request({
+ method: 'POST',
+ url: apiUrl + '/runs',
+ body: {
+ url: wwwUrl + '/simple-page.html',
+ waitForResponse: false
+ },
+ json: true
+ }, function(error, response, body) {
+ if (error) {
+ deferred.reject(error);
+ } else {
+ deferred.resolve(response, body);
+ }
+ });
- request({
- method: 'POST',
- url: apiUrl + '/runs',
- body: {
- url: wwwUrl + '/simple-page.html',
- waitForResponse: false
- },
- json: true,
- headers: {
- 'X-Api-Token': jwt.encode({
- application: 'unknown-app',
- expire: Date.now() + 60000
- }, config.tokenSalt)
- }
- }, function(error, response, body) {
- if (!error && response.statusCode === 401) {
- done();
- } else {
- done(error || response.statusCode);
- }
- });
- });
+ return deferred.promise;
+ }
- it('should accept a good token', function(done) {
- this.timeout(5000);
+ launchRun()
+ .then(launchRun)
+ .then(launchRun)
+ .then(launchRun)
+ .then(launchRun)
+ .then(launchRun)
+ .then(launchRun)
+ .then(launchRun)
+ .then(launchRun)
+ .then(launchRun)
+ .then(launchRun)
+ .then(launchRun)
- request({
- method: 'POST',
- url: apiUrl + '/runs',
- body: {
- url: wwwUrl + '/simple-page.html',
- waitForResponse: false
- },
- json: true,
- headers: {
- 'X-Api-Token': jwt.encode({
- application: config.authorizedApplications[0],
- expire: Date.now() + 60000
- }, config.tokenSalt)
- }
- }, function(error, response, body) {
- if (!error && response.statusCode === 200) {
+ .then(function(response, body) {
+
+ // Here should still be ok
+ response.statusCode.should.equal(200);
- runId = body.runId;
- runId.should.be.a('string');
+ launchRun()
+ .then(launchRun)
+ .then(launchRun)
+ .then(launchRun)
+ .then(launchRun)
+ .then(launchRun)
+ .then(launchRun)
+ .then(launchRun)
+ .then(launchRun)
+ .then(launchRun)
+ .then(launchRun)
+ .then(launchRun)
+ .then(launchRun)
+ .then(launchRun)
+ .then(launchRun)
+
+ .then(function(response, body) {
+
+ // It should fail now
+ response.statusCode.should.equal(429);
done();
- } else {
- done(error || response.statusCode);
- }
+ })
+ .fail(function(error) {
+ done(error);
+ });
+
+ }).fail(function(error) {
+ done(error);
});
+
});
+
+
});
From 3ad6cdef51b074e1a465483acb40f0a54b0d044d Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ga=C3=ABl=20M=C3=A9tais?=
Date: Sat, 13 Dec 2014 12:11:33 +0100
Subject: [PATCH 11/98] Launch the test server with coverage
---
Gruntfile.js | 42 +++++++++++++++++++++++++++++++++++--
bin/cli.js | 2 --
bin/server.js | 4 +---
server_config/settings.json | 2 +-
test/api/apiTest.js | 18 +---------------
test/fixtures/settings.json | 10 +++++++++
6 files changed, 53 insertions(+), 25 deletions(-)
create mode 100644 test/fixtures/settings.json
diff --git a/Gruntfile.js b/Gruntfile.js
index b9e7296..9cf19b1 100644
--- a/Gruntfile.js
+++ b/Gruntfile.js
@@ -58,7 +58,8 @@ module.exports = function(grunt) {
coverage: {
files: [
{src: ['test/**'], dest: 'coverage/'},
- {src: ['lib/metadata/**'], dest: 'coverage/'}
+ {src: ['lib/metadata/**'], dest: 'coverage/'},
+ {src: ['bin/**'], dest: 'coverage/'}
]
}
},
@@ -70,6 +71,10 @@ module.exports = function(grunt) {
coverageLib: {
src: ['lib/'],
dest: 'coverage/lib/'
+ },
+ coverageBin: {
+ src: ['bin/'],
+ dest: 'coverage/bin/'
}
},
mochaTest: {
@@ -106,7 +111,7 @@ module.exports = function(grunt) {
testServer: {
options: {
port: 8387,
- server: './bin/server.js'
+ server: './coverage/bin/server.js'
}
},
testSuite: {
@@ -118,6 +123,37 @@ module.exports = function(grunt) {
}
});
+
+ // Custom task: copies the test settings.json file to the coverage folder, and checks if there's no missing fields
+ grunt.registerTask('copy-test-server-settings', function() {
+ var mainSettingsFile = './server_config/settings.json';
+ var testSettingsFile = './test/fixtures/settings.json';
+
+ var mainSettings = grunt.file.readJSON(mainSettingsFile);
+ var testSettings = grunt.file.readJSON(testSettingsFile);
+
+ // Recursively compare keys of two objects (not the values)
+ function compareKeys(original, copy, context) {
+ for (var key in original) {
+ if (!copy[key] && copy[key] !== '' && copy[key] !== 0) {
+ grunt.fail.warn('Settings file ' + testSettingsFile + ' doesn\'t contain key ' + context + '.' + key);
+ }
+ if (original[key] !== null && typeof original[key] === 'object') {
+ compareKeys(original[key], copy[key], context + '.' + key);
+ }
+ }
+ }
+
+ compareKeys(mainSettings, testSettings, 'settings');
+
+ var outputFile = './coverage/server_config/settings.json';
+ grunt.file.write(outputFile, JSON.stringify(testSettings, null, 4));
+ grunt.verbose.ok('File ' + outputFile + ' created');
+ });
+
+
+
+
require('matchdep').filterDev('grunt-*').forEach(grunt.loadNpmTasks);
grunt.registerTask('icons', [
@@ -141,6 +177,7 @@ module.exports = function(grunt) {
grunt.registerTask('test', [
'build',
'jshint',
+ 'copy-test-server-settings',
'express:testServer',
'express:testSuite',
'clean:coverage',
@@ -153,6 +190,7 @@ module.exports = function(grunt) {
grunt.registerTask('test-current-work', [
'build',
'jshint',
+ 'copy-test-server-settings',
'express:testServer',
'express:testSuite',
'clean:coverage',
diff --git a/bin/cli.js b/bin/cli.js
index 88b615a..0524668 100644
--- a/bin/cli.js
+++ b/bin/cli.js
@@ -1,5 +1,3 @@
-#!/usr/bin/env node
-
var debug = require('debug')('ylt:cli');
var YellowLabTools = require('../lib/yellowlabtools');
diff --git a/bin/server.js b/bin/server.js
index f8b76aa..95bb8a5 100644
--- a/bin/server.js
+++ b/bin/server.js
@@ -1,6 +1,3 @@
-// Config file
-var settings = require('../server_config/settings.json');
-
var express = require('express');
var app = express();
var server = require('http').createServer(app);
@@ -25,6 +22,7 @@ var uiController = require('../lib/server/controllers/uiController')(
// Let's start the server!
if (!process.env.GRUNTED) {
// The server is not launched by Grunt
+ var settings = require('../server_config/settings.json');
server.listen(settings.serverPort, function() {
console.log('Listening on port %d', server.address().port);
});
diff --git a/server_config/settings.json b/server_config/settings.json
index 2b05df1..5c69980 100644
--- a/server_config/settings.json
+++ b/server_config/settings.json
@@ -5,6 +5,6 @@
"authorizedKeys": {
"1234567890": "contact@gaelmetais.com"
},
- "maxAnonymousRunsPerDay": 24,
+ "maxAnonymousRunsPerDay": 50,
"maxAnonymousCallsPerDay": 1000
}
\ No newline at end of file
diff --git a/test/api/apiTest.js b/test/api/apiTest.js
index 3183bf2..e23f3d5 100644
--- a/test/api/apiTest.js
+++ b/test/api/apiTest.js
@@ -66,7 +66,7 @@ describe('api', function() {
});
});
- it('should accept up to 24 anonymous runs to the API', function(done) {
+ it('should accept up to 10 anonymous runs to the API', function(done) {
this.timeout(15000);
function launchRun() {
@@ -96,13 +96,6 @@ describe('api', function() {
.then(launchRun)
.then(launchRun)
.then(launchRun)
- .then(launchRun)
- .then(launchRun)
- .then(launchRun)
- .then(launchRun)
- .then(launchRun)
- .then(launchRun)
- .then(launchRun)
.then(function(response, body) {
@@ -115,15 +108,6 @@ describe('api', function() {
.then(launchRun)
.then(launchRun)
.then(launchRun)
- .then(launchRun)
- .then(launchRun)
- .then(launchRun)
- .then(launchRun)
- .then(launchRun)
- .then(launchRun)
- .then(launchRun)
- .then(launchRun)
- .then(launchRun)
.then(function(response, body) {
diff --git a/test/fixtures/settings.json b/test/fixtures/settings.json
new file mode 100644
index 0000000..619190f
--- /dev/null
+++ b/test/fixtures/settings.json
@@ -0,0 +1,10 @@
+{
+ "serverPort": "auto",
+ "googleAnalyticsId": "",
+
+ "authorizedKeys": {
+ "1234567890": "contact@gaelmetais.com"
+ },
+ "maxAnonymousRunsPerDay": 10,
+ "maxAnonymousCallsPerDay": 1000
+}
\ No newline at end of file
From 8abfe93df74084b62be1a6ccdc53bc9b2c934894 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ga=C3=ABl=20M=C3=A9tais?=
Date: Sat, 13 Dec 2014 13:56:21 +0100
Subject: [PATCH 12/98] The test itself starts the server in a before hook
---
Gruntfile.js | 14 +++-----------
bin/server.js | 19 ++++++++++---------
test/api/apiTest.js | 12 ++++++++++--
test/fixtures/settings.json | 2 +-
4 files changed, 24 insertions(+), 23 deletions(-)
diff --git a/Gruntfile.js b/Gruntfile.js
index 9cf19b1..ac5dc6a 100644
--- a/Gruntfile.js
+++ b/Gruntfile.js
@@ -108,12 +108,6 @@ module.exports = function(grunt) {
showStack: true
}
},
- testServer: {
- options: {
- port: 8387,
- server: './coverage/bin/server.js'
- }
- },
testSuite: {
options: {
port: 8388,
@@ -148,7 +142,7 @@ module.exports = function(grunt) {
var outputFile = './coverage/server_config/settings.json';
grunt.file.write(outputFile, JSON.stringify(testSettings, null, 4));
- grunt.verbose.ok('File ' + outputFile + ' created');
+ grunt.log.ok('File ' + outputFile + ' created');
});
@@ -177,10 +171,9 @@ module.exports = function(grunt) {
grunt.registerTask('test', [
'build',
'jshint',
- 'copy-test-server-settings',
- 'express:testServer',
'express:testSuite',
'clean:coverage',
+ 'copy-test-server-settings',
'blanket',
'copy:coverage',
'mochaTest:test',
@@ -190,10 +183,9 @@ module.exports = function(grunt) {
grunt.registerTask('test-current-work', [
'build',
'jshint',
- 'copy-test-server-settings',
- 'express:testServer',
'express:testSuite',
'clean:coverage',
+ 'copy-test-server-settings',
'blanket',
'copy:coverage',
'mochaTest:test-current-work'
diff --git a/bin/server.js b/bin/server.js
index 95bb8a5..cc7199c 100644
--- a/bin/server.js
+++ b/bin/server.js
@@ -20,13 +20,14 @@ var uiController = require('../lib/server/controllers/uiController')(
// Let's start the server!
-if (!process.env.GRUNTED) {
- // The server is not launched by Grunt
- var settings = require('../server_config/settings.json');
- server.listen(settings.serverPort, function() {
- console.log('Listening on port %d', server.address().port);
- });
-}
+var settings = require('../server_config/settings.json');
+server.listen(settings.serverPort, function() {
+ console.log('Listening on port %d', server.address().port);
-// For Grunt
-module.exports = app;
\ No newline at end of file
+ // For the tests
+ if (server.startTests) {
+ server.startTests();
+ }
+});
+
+module.exports = server;
\ No newline at end of file
diff --git a/test/api/apiTest.js b/test/api/apiTest.js
index e23f3d5..da46653 100644
--- a/test/api/apiTest.js
+++ b/test/api/apiTest.js
@@ -14,7 +14,12 @@ var wwwUrl = 'http://localhost:8388';
describe('api', function() {
var runId;
+ var apiServer;
+ before(function(done) {
+ apiServer = require('../../bin/server.js');
+ apiServer.startTests = done;
+ });
it('should refuse a query with an invalid key', function(done) {
this.timeout(5000);
@@ -67,7 +72,7 @@ describe('api', function() {
});
it('should accept up to 10 anonymous runs to the API', function(done) {
- this.timeout(15000);
+ this.timeout(5000);
function launchRun() {
var deferred = Q.defer();
@@ -126,5 +131,8 @@ describe('api', function() {
});
-
+ after(function() {
+ console.log('Closing the server');
+ apiServer.close();
+ });
});
diff --git a/test/fixtures/settings.json b/test/fixtures/settings.json
index 619190f..75b2365 100644
--- a/test/fixtures/settings.json
+++ b/test/fixtures/settings.json
@@ -1,5 +1,5 @@
{
- "serverPort": "auto",
+ "serverPort": "8387",
"googleAnalyticsId": "",
"authorizedKeys": {
From 792f7d8e21ef31827fa5833a229683031046f110 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ga=C3=ABl=20M=C3=A9tais?=
Date: Sun, 14 Dec 2014 15:47:58 +0100
Subject: [PATCH 13/98] Implements a general score calculator
---
Gruntfile.js | 2 +-
lib/rulesChecker.js | 1 -
lib/runner.js | 28 ++-
lib/scoreCalculator.js | 83 +++++++
lib/server/middlewares/apiLimitsMiddleware.js | 4 +-
test/api/apiTest.js | 210 +++++++++++++++++-
test/core/scoreCalculatorTest.js | 18 ++
test/core/yellowlabtoolsTest.js | 15 +-
test/fixtures/scoreInput.json | 68 ++++++
test/fixtures/scoreOutput.json | 34 +++
test/fixtures/scoreProfile.json | 35 +++
11 files changed, 478 insertions(+), 20 deletions(-)
create mode 100644 lib/scoreCalculator.js
create mode 100644 test/core/scoreCalculatorTest.js
create mode 100644 test/fixtures/scoreInput.json
create mode 100644 test/fixtures/scoreOutput.json
create mode 100644 test/fixtures/scoreProfile.json
diff --git a/Gruntfile.js b/Gruntfile.js
index ac5dc6a..81702ea 100644
--- a/Gruntfile.js
+++ b/Gruntfile.js
@@ -88,7 +88,7 @@ module.exports = function(grunt) {
options: {
reporter: 'spec',
},
- src: ['coverage/test/api/apiTest.js']
+ src: ['coverage/test/core/scoreCalculatorTest.js']
},
coverage: {
options: {
diff --git a/lib/rulesChecker.js b/lib/rulesChecker.js
index c576c72..9033f0e 100644
--- a/lib/rulesChecker.js
+++ b/lib/rulesChecker.js
@@ -7,7 +7,6 @@ var RulesChecker = function() {
this.check = function(data, policies) {
var results = {};
- var err = null;
debug('Starting checking rules');
diff --git a/lib/runner.js b/lib/runner.js
index c9083c1..caa81a4 100644
--- a/lib/runner.js
+++ b/lib/runner.js
@@ -1,8 +1,10 @@
-var Q = require('q');
-var debug = require('debug')('ylt:yellowlabtools');
+var Q = require('q');
+var debug = require('debug')('ylt:runner');
+
+var phantomasWrapper = require('./tools/phantomasWrapper');
+var rulesChecker = require('./rulesChecker');
+var scoreCalculator = require('./scoreCalculator');
-var phantomasWrapper = require('./tools/phantomasWrapper');
-var rulesChecker = require('./rulesChecker');
var Runner = function(params) {
'use strict';
@@ -27,6 +29,24 @@ var Runner = function(params) {
data.rules = rulesChecker.check(data, policies);
+ // Scores calculator
+ var scoreProfileGeneric = require('./metadata/scoreProfileGeneric.json');
+ data.scoreProfiles = {
+ generic : scoreCalculator.calculate(data, scoreProfileGeneric)
+ };
+
+
+ // Get the JS Execution Tree from offenders and put in the main object
+ try {
+ data.javascriptExecutionTree = JSON.parse(data.toolsResults.phantomas.offenders.javascriptExecutionTree[0]);
+ } catch(e) {
+ debug('Could not find nor parse phantomas.offenders.javascriptExecutionTree');
+ }
+
+ delete data.toolsResults.phantomas.metrics.javascriptExecutionTree;
+ delete data.toolsResults.phantomas.offenders.javascriptExecutionTree;
+
+ //Finished!
deferred.resolve(data);
}).fail(function(err) {
diff --git a/lib/scoreCalculator.js b/lib/scoreCalculator.js
new file mode 100644
index 0000000..8a0a399
--- /dev/null
+++ b/lib/scoreCalculator.js
@@ -0,0 +1,83 @@
+var Q = require('q');
+var debug = require('debug')('ylt:scoreCalculator');
+
+var ScoreCalculator = function() {
+ 'use strict';
+
+ this.calculate = function(data, profile) {
+
+ var results = {
+ categories: {}
+ };
+ var weight;
+ var categoryName;
+
+ debug('Starting calculating scores');
+
+ // Calculate categories
+ for (categoryName in profile.categories) {
+ var categoryResult = {
+ label: profile.categories[categoryName].label
+ };
+
+ var sum = 0;
+ var totalWeight = 0;
+ var rules = [];
+
+ for (var policyName in profile.categories[categoryName].policies) {
+ weight = profile.categories[categoryName].policies[policyName];
+
+ if (data.rules[policyName]) {
+ sum += data.rules[policyName].score * weight;
+ } else {
+ // Max value if rule is not here
+ sum += 100 * weight;
+ debug('Warning: could not find rule %s', policyName);
+ }
+
+ totalWeight += weight;
+ rules.push(policyName);
+ }
+
+ if (totalWeight === 0) {
+ categoryResult.categoryScore = 100;
+ } else {
+ categoryResult.categoryScore = Math.round(sum / totalWeight);
+ }
+
+ categoryResult.rules = rules;
+ results.categories[categoryName] = categoryResult;
+ }
+
+
+ // Calculate general score
+ var globalSum = 0;
+ var globalTotalWeight = 0;
+
+ for (categoryName in profile.globalScore) {
+ weight = profile.globalScore[categoryName];
+
+ if (results.categories[categoryName]) {
+ globalSum += results.categories[categoryName].categoryScore * weight;
+ } else {
+ globalSum += 100 * weight;
+ }
+ globalTotalWeight += profile.globalScore[categoryName];
+ }
+
+ if (globalTotalWeight === 0) {
+ results.globalScore = 100;
+ } else {
+ results.globalScore = Math.round(globalSum / globalTotalWeight);
+ }
+
+
+
+ debug('Score calculation finished:');
+ debug(results);
+
+ return results;
+ };
+};
+
+module.exports = new ScoreCalculator();
\ No newline at end of file
diff --git a/lib/server/middlewares/apiLimitsMiddleware.js b/lib/server/middlewares/apiLimitsMiddleware.js
index 7c17015..cce5df2 100644
--- a/lib/server/middlewares/apiLimitsMiddleware.js
+++ b/lib/server/middlewares/apiLimitsMiddleware.js
@@ -16,7 +16,7 @@ var apiLimitsMiddleware = function(req, res, next) {
if (!runsTable.accepts(req.connection.remoteAddress)) {
// Sorry :/
debug('Too many tests launched from IP address %s', req.connection.remoteAddress);
- res.status(429).send('Too Many Requests');
+ res.status(429).send('Too many requests');
return;
}
@@ -25,7 +25,7 @@ var apiLimitsMiddleware = function(req, res, next) {
if (!callsTable.accepts(req.connection.remoteAddress)) {
// Sorry :/
debug('Too many API requests from IP address %s', req.connection.remoteAddress);
- res.status(429).send('Too Many Requests');
+ res.status(429).send('Too many requests');
return;
}
diff --git a/test/api/apiTest.js b/test/api/apiTest.js
index da46653..44c599f 100644
--- a/test/api/apiTest.js
+++ b/test/api/apiTest.js
@@ -8,25 +8,30 @@ var config = {
}
};
-var apiUrl = 'http://localhost:8387/api';
+var serverUrl = 'http://localhost:8387';
var wwwUrl = 'http://localhost:8388';
describe('api', function() {
- var runId;
+
+ var syncRunResultUrl;
+ var asyncRunId;
var apiServer;
+
+ // Start the server
before(function(done) {
apiServer = require('../../bin/server.js');
apiServer.startTests = done;
});
+
it('should refuse a query with an invalid key', function(done) {
this.timeout(5000);
request({
method: 'POST',
- url: apiUrl + '/runs',
+ url: serverUrl + '/api/runs',
body: {
url: wwwUrl + '/simple-page.html',
waitForResponse: false
@@ -44,12 +49,67 @@ describe('api', function() {
});
});
- it('should accept a query with a valid key', function(done) {
+
+ it('should launch a synchronous run', function(done) {
+ this.timeout(15000);
+
+ request({
+ method: 'POST',
+ url: serverUrl + '/api/runs',
+ body: {
+ url: wwwUrl + '/simple-page.html',
+ waitForResponse: true
+ },
+ json: true,
+ headers: {
+ 'X-Api-Key': Object.keys(config.authorizedKeys)[0]
+ }
+ }, function(error, response, body) {
+ if (!error && response.statusCode === 302) {
+
+ response.headers.should.have.a.property('location').that.is.a('string');
+ syncRunResultUrl = response.headers.location;
+
+ done();
+ } else {
+ done(error || response.statusCode);
+ }
+ });
+ });
+
+
+ it('should retrieve the results for the synchronous run', function(done) {
+ this.timeout(15000);
+
+ request({
+ method: 'GET',
+ url: serverUrl + syncRunResultUrl,
+ json: true,
+ }, function(error, response, body) {
+ if (!error && response.statusCode === 200) {
+
+ body.should.have.a.property('runId').that.is.a('string');
+ body.should.have.a.property('params').that.is.an('object');
+ body.should.have.a.property('scoreProfiles').that.is.an('object');
+ body.should.have.a.property('rules').that.is.an('object');
+ body.should.have.a.property('toolsResults').that.is.an('object');
+ body.should.have.a.property('javascriptExecutionTree').that.is.an('object');
+
+ done();
+
+ } else {
+ done(error || response.statusCode);
+ }
+ });
+ });
+
+
+ it('should launch a run without waiting for the response', function(done) {
this.timeout(5000);
request({
method: 'POST',
- url: apiUrl + '/runs',
+ url: serverUrl + '/api/runs',
body: {
url: wwwUrl + '/simple-page.html',
waitForResponse: false
@@ -61,8 +121,35 @@ describe('api', function() {
}, function(error, response, body) {
if (!error && response.statusCode === 200) {
- runId = body.runId;
- runId.should.be.a('string');
+ asyncRunId = body.runId;
+ asyncRunId.should.be.a('string');
+ done();
+
+ } else {
+ done(error || response.statusCode);
+ }
+ });
+ });
+
+
+ it('should respond run status: running', function(done) {
+ this.timeout(5000);
+
+ request({
+ method: 'GET',
+ url: serverUrl + '/api/runs/' + asyncRunId,
+ json: true,
+ headers: {
+ 'X-Api-Key': Object.keys(config.authorizedKeys)[0]
+ }
+ }, function(error, response, body) {
+ if (!error && response.statusCode === 200) {
+
+ body.runId.should.equal(asyncRunId);
+ body.status.should.deep.equal({
+ statusCode: 'running'
+ });
+
done();
} else {
@@ -79,13 +166,16 @@ describe('api', function() {
request({
method: 'POST',
- url: apiUrl + '/runs',
+ url: serverUrl + '/api/runs',
body: {
url: wwwUrl + '/simple-page.html',
waitForResponse: false
},
json: true
}, function(error, response, body) {
+
+ lastRunId = body.runId;
+
if (error) {
deferred.reject(error);
} else {
@@ -131,6 +221,110 @@ describe('api', function() {
});
+
+ it('should respond 404 to unknown runId', function(done) {
+ this.timeout(5000);
+
+ request({
+ method: 'GET',
+ url: serverUrl + '/api/runs/unknown',
+ json: true
+ }, function(error, response, body) {
+ if (!error && response.statusCode === 404) {
+ done();
+ } else {
+ done(error || response.statusCode);
+ }
+ });
+ });
+
+
+ it('should respond 404 to unknown result', function(done) {
+ this.timeout(5000);
+
+ request({
+ method: 'GET',
+ url: serverUrl + '/api/results/unknown',
+ json: true
+ }, function(error, response, body) {
+ if (!error && response.statusCode === 404) {
+ done();
+ } else {
+ done(error || response.statusCode);
+ }
+ });
+ });
+
+
+ it('should respond status complete to the first run', function(done) {
+ this.timeout(12000);
+
+ function checkStatus() {
+ request({
+ method: 'GET',
+ url: serverUrl + '/api/runs/' + asyncRunId,
+ json: true
+ }, function(error, response, body) {
+ if (!error && response.statusCode === 200) {
+
+ body.runId.should.equal(asyncRunId);
+
+ if (body.status.statusCode === 'running') {
+ setTimeout(checkStatus, 250);
+ } else if (body.status.statusCode === 'complete') {
+ done();
+ } else {
+ done(body.status.statusCode);
+ }
+
+ } else {
+ done(error || response.statusCode);
+ }
+ });
+ }
+
+ checkStatus();
+ });
+
+
+ it('should find the result of the async run', function(done) {
+ this.timeout(5000);
+
+ request({
+ method: 'GET',
+ url: serverUrl + '/api/results/' + asyncRunId,
+ json: true,
+ }, function(error, response, body) {
+ if (!error && response.statusCode === 200) {
+
+ body.should.have.a.property('runId').that.equals(asyncRunId);
+ body.should.have.a.property('params').that.is.an('object');
+ body.params.url.should.equal(wwwUrl + '/simple-page.html');
+
+ body.should.have.a.property('scoreProfiles').that.is.an('object');
+ body.scoreProfiles.should.have.a.property('generic').that.is.an('object');
+ body.scoreProfiles.generic.should.have.a.property('globalScore').that.is.a('number');
+ body.scoreProfiles.generic.should.have.a.property('categories').that.is.an('object');
+
+ body.should.have.a.property('rules').that.is.an('object');
+
+ body.should.have.a.property('toolsResults').that.is.an('object');
+ body.toolsResults.should.have.a.property('phantomas').that.is.an('object');
+
+ body.should.have.a.property('javascriptExecutionTree').that.is.an('object');
+ body.javascriptExecutionTree.should.have.a.property('data').that.is.an('object');
+ body.javascriptExecutionTree.data.should.have.a.property('type').that.equals('main');
+
+ done();
+
+ } else {
+ done(error || response.statusCode);
+ }
+ });
+ });
+
+
+ // Stop the server
after(function() {
console.log('Closing the server');
apiServer.close();
diff --git a/test/core/scoreCalculatorTest.js b/test/core/scoreCalculatorTest.js
new file mode 100644
index 0000000..171fe93
--- /dev/null
+++ b/test/core/scoreCalculatorTest.js
@@ -0,0 +1,18 @@
+var should = require('chai').should();
+var scoreCalculator = require('../../lib/scoreCalculator');
+
+describe('scoreCalculator', function() {
+
+ it('should have a method calculate', function() {
+ scoreCalculator.should.have.property('calculate').that.is.a('function');
+ });
+
+ it('should produce a nice rules object', function() {
+ var data = require('../fixtures/scoreInput.json');
+ var profile = require('../fixtures/scoreProfile.json');
+ var expected = require('../fixtures/scoreOutput.json');
+
+ var results = scoreCalculator.calculate(data, profile);
+ results.should.deep.equals(expected);
+ });
+});
diff --git a/test/core/yellowlabtoolsTest.js b/test/core/yellowlabtoolsTest.js
index b1dd85e..eea43e5 100644
--- a/test/core/yellowlabtoolsTest.js
+++ b/test/core/yellowlabtoolsTest.js
@@ -9,28 +9,28 @@ chai.use(sinonChai);
describe('yellowlabtools', function() {
- it('returns a promise', function() {
+ it('should return a promise', function() {
var ylt = new YellowLabTools();
ylt.should.have.property('then').that.is.a('function');
ylt.should.have.property('fail').that.is.a('function');
});
- it('fails an undefined url', function(done) {
+ it('should fail an undefined url', function(done) {
var ylt = new YellowLabTools().fail(function(err) {
err.should.be.a('string').that.equals('URL missing');
done();
});
});
- it('fails with an empty url string', function(done) {
+ it('should fail with an empty url string', function(done) {
var ylt = new YellowLabTools('').fail(function(err) {
err.should.be.a('string').that.equals('URL missing');
done();
});
});
- it('succeeds on simple-page.html', function(done) {
+ it('should succeeds on simple-page.html', function(done) {
this.timeout(15000);
// Check if console.log is called
@@ -75,6 +75,13 @@ describe('yellowlabtools', function() {
"offenders": ["body > h1[1]"]
});
+ // Test javascriptExecutionTree
+ data.toolsResults.phantomas.metrics.should.not.have.a.property('javascriptExecutionTree');
+ data.toolsResults.phantomas.offenders.should.not.have.a.property('javascriptExecutionTree');
+ data.should.have.a.property('javascriptExecutionTree').that.is.an('object');
+ data.javascriptExecutionTree.should.have.a.property('data');
+ data.javascriptExecutionTree.data.should.have.a.property('type').that.equals('main');
+
/*jshint expr: true*/
console.log.should.not.have.been.called;
diff --git a/test/fixtures/scoreInput.json b/test/fixtures/scoreInput.json
new file mode 100644
index 0000000..81e6a4f
--- /dev/null
+++ b/test/fixtures/scoreInput.json
@@ -0,0 +1,68 @@
+{
+ "rules": {
+ "metric1": {
+ "policy": {
+ "tool": "tool1",
+ "label": "The metric 1",
+ "message": "A great message",
+ "isOkThreshold": 1000,
+ "isBadThreshold": 3000,
+ "isAbnormalThreshold": 5000
+ },
+ "value": 1236,
+ "bad": true,
+ "abnormal": false,
+ "score": 88,
+ "abnormalityScore": 0
+ },
+ "metric2": {
+ "value": 222,
+ "bad": false,
+ "abnormal": false,
+ "score": 100,
+ "abnormalityScore": 0
+ },
+ "metric3": {
+ "value": 6666,
+ "bad": true,
+ "abnormal": true,
+ "score": 0,
+ "abnormalityScore": -42
+ },
+ "metric4": {
+ "value": 1000,
+ "bad": false,
+ "abnormal": false,
+ "score": 100,
+ "abnormalityScore": 0
+ },
+ "metric5": {
+ "value": 3000,
+ "bad": true,
+ "abnormal": false,
+ "score": 0,
+ "abnormalityScore": 0
+ },
+ "metric6": {
+ "value": 0,
+ "bad": false,
+ "abnormal": false,
+ "score": 100,
+ "abnormalityScore": 0
+ },
+ "metric7": {
+ "value": 5000,
+ "bad": true,
+ "abnormal": true,
+ "score": 0,
+ "abnormalityScore": 0
+ },
+ "metric8": {
+ "value": 22,
+ "bad": true,
+ "abnormal": true,
+ "score": 0,
+ "abnormalityScore": -100
+ }
+ }
+}
\ No newline at end of file
diff --git a/test/fixtures/scoreOutput.json b/test/fixtures/scoreOutput.json
new file mode 100644
index 0000000..4630bbf
--- /dev/null
+++ b/test/fixtures/scoreOutput.json
@@ -0,0 +1,34 @@
+{
+ "globalScore": 69,
+ "categories": {
+ "category1": {
+ "label": "Category 1",
+ "categoryScore": 87,
+ "rules": [
+ "metric1",
+ "metric2",
+ "metric3",
+ "metric4"
+ ]
+ },
+ "category2": {
+ "label": "Category 2",
+ "categoryScore": 31,
+ "rules": [
+ "metric5",
+ "metric6",
+ "metric7",
+ "metric8",
+ "unexistantMetric1"
+ ]
+ },
+ "category3": {
+ "label": "Category 3",
+ "categoryScore": 100,
+ "rules": [
+ "unexistantMetric1",
+ "unexistantMetric2"
+ ]
+ }
+ }
+}
\ No newline at end of file
diff --git a/test/fixtures/scoreProfile.json b/test/fixtures/scoreProfile.json
new file mode 100644
index 0000000..8856126
--- /dev/null
+++ b/test/fixtures/scoreProfile.json
@@ -0,0 +1,35 @@
+{
+ "categories": {
+ "category1": {
+ "label": "Category 1",
+ "policies": {
+ "metric1": 2,
+ "metric2": 1,
+ "metric3": 0.5,
+ "metric4": 2
+ }
+ },
+ "category2": {
+ "label": "Category 2",
+ "policies": {
+ "metric5": 2,
+ "metric6": 1,
+ "metric7": 0.5,
+ "metric8": 2,
+ "unexistantMetric1": 1
+ }
+ },
+ "category3": {
+ "label": "Category 3",
+ "policies": {
+ "unexistantMetric1": 2,
+ "unexistantMetric2": 1
+ }
+ }
+ },
+ "globalScore": {
+ "category1": 2,
+ "category2": 1,
+ "category3": 0.1
+ }
+}
\ No newline at end of file
From 90dac7dd19efef1d04335bafad90385577e6dcb0 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ga=C3=ABl=20M=C3=A9tais?=
Date: Sun, 14 Dec 2014 21:35:46 +0100
Subject: [PATCH 14/98] Created the generic score profile
---
lib/metadata/policies.json | 28 +++----
lib/metadata/scoreProfileGeneric.json | 105 +++++++++++++++++++++++++-
2 files changed, 118 insertions(+), 15 deletions(-)
diff --git a/lib/metadata/policies.json b/lib/metadata/policies.json
index 40df848..c858342 100644
--- a/lib/metadata/policies.json
+++ b/lib/metadata/policies.json
@@ -2,7 +2,7 @@
"DOMelementsCount": {
"tool": "phantomas",
"label": "DOM elements count",
- "message": "
A high number of DOM elements means a lot of work for the browser to render the page.
It also slows down Javascript DOM queries, as there are more elements to search through.
",
+ "message": "
A high number of DOM elements means a lot of work for the browser to render the page.
It also slows down JavaScript DOM queries, as there are more elements to search through.
A deep DOM makes the CSS matching with DOM elements difficult.
It also slows down Javascript modifications to the DOM because changing the dimensions of an element makes the browser re-calculate the dimensions of it's parents. Same thing for Javascript events, that bubble up to the document root.
",
+ "message": "
A deep DOM makes the CSS matching with DOM elements difficult.
It also slows down JavaScript modifications to the DOM because changing the dimensions of an element makes the browser re-calculate the dimensions of it's parents. Same thing for JavaScript events, that bubble up to the document root.
DOM queries are like looking in a large catalog of items. Even if the browsers made progress on the performances of queries, websites often make hundreds of them.
Try to reduce the number of queries by refactoring your Javascript code.
Avoid also to have a read query between two write queries. To be able to reduce the number repaints and optimize performances, browsers buffer the DOM writing operations and treat them in bulk. But each time a DOM reading is asked, the browser needs to empty the buffer. This can be particularly slow inside a loop.
",
+ "message": "
DOM queries are like looking in a large catalog of items. Even if the browsers made progress on the performances of queries, websites often make hundreds of them.
Try to reduce the number of queries by refactoring your JavaScript code.
Avoid also to have a read query between two write queries. To be able to reduce the number repaints and optimize performances, browsers buffer the DOM writing operations and treat them in bulk. But each time a DOM reading is asked, the browser needs to empty the buffer. This can be particularly slow inside a loop.
This metric counts the number of DOM queries, DOM inserts, binds, etc. made by the Javascript before the DOMContentLoaded event.
Wait for this event before manipulating the DOM. Do not execute Javascript in the middle of the BODY as it slows down the construction of the DOM and makes a poor maintainability. This is what i call spaghetti code.
The JS Timeline tab can help you identify what's happening.
",
+ "message": "
This metric counts the number of DOM queries, DOM inserts, binds, etc. made by the JavaScript before the DOMContentLoaded event.
Wait for this event before manipulating the DOM. Do not execute JavaScript in the middle of the BODY as it slows down the construction of the DOM and makes a poor maintainability. This is what i call spaghetti code.
The JS Timeline tab can help you identify what's happening.
A deep DOM makes the CSS matching with DOM elements difficult.
It also slows down Javascript modifications to the DOM because changing the dimensions of an element makes the browser re-calculate the dimensions of it's parents. Same thing for Javascript events, that bubble up to the document root.
",
+ "message": "
A deep DOM makes the CSS matching with DOM elements difficult.
It also slows down JavaScript modifications to the DOM because changing the dimensions of an element makes the browser re-calculate the dimensions of it's parents. Same thing for JavaScript events, that bubble up to the document root.
+
+
+
\ No newline at end of file
diff --git a/front/src/views/about.html b/front/src/views/about.html
new file mode 100644
index 0000000..d0f7199
--- /dev/null
+++ b/front/src/views/about.html
@@ -0,0 +1 @@
+{{about}}
\ No newline at end of file
diff --git a/front/src/views/dashboard.html b/front/src/views/dashboard.html
new file mode 100644
index 0000000..29fb6e9
--- /dev/null
+++ b/front/src/views/dashboard.html
@@ -0,0 +1,2 @@
+
+
+
-
\ No newline at end of file
+
+
+
+
diff --git a/front/src/views/about.html b/front/src/views/about.html
index d0f7199..3016791 100644
--- a/front/src/views/about.html
+++ b/front/src/views/about.html
@@ -1 +1,8 @@
-{{about}}
\ No newline at end of file
+
+
+
+
+
+
\ No newline at end of file
diff --git a/front/src/views/dashboard.html b/front/src/views/dashboard.html
index 29fb6e9..c128f64 100644
--- a/front/src/views/dashboard.html
+++ b/front/src/views/dashboard.html
@@ -1,2 +1,19 @@
-
{{dashboard}}
-
{{runId}}
\ No newline at end of file
+
+
Grades
+
+
+
+
+
{{category.label}}
+
+
+
+
+
{{rule.policy.label}}
+
{{rule.value}}
+
+
+
+
+
+
\ No newline at end of file
diff --git a/front/src/views/index.html b/front/src/views/index.html
index b475959..4a6df0c 100644
--- a/front/src/views/index.html
+++ b/front/src/views/index.html
@@ -1 +1,4 @@
-this is a html index by {{toto}}
\ No newline at end of file
+
Free online test to help speeding up heavy web pages
+
+
+
(This is a BETA, your feedback is more than welcome)
This metric counts the number of DOM queries, DOM inserts, binds, etc. made by the JavaScript before the DOMContentLoaded event.
Wait for this event before manipulating the DOM. Do not execute JavaScript in the middle of the BODY as it slows down the construction of the DOM and makes a poor maintainability. This is what i call spaghetti code.
The JS Timeline tab can help you identify what's happening.
Current latest versions of jQuery are 1.11 (with support for old IE versions) and 2.1 (without).
Each new version of jQuery optimizes performances. Do not keep an old version of jQuery. Updating can sometimes break a few things, but it is generally quite easy to fix them up. So don't hesitate.
A deep DOM makes the CSS matching with DOM elements difficult.
-
It also slows down Javascript modifications to the DOM because changing the dimensions of an element makes the browser re-calculate the dimensions of it's parents. Same thing for Javascript events, that bubble up to the document root.
-
-
-
-
-
Number of iframes
-
- {{phantomasResults.metrics.iframesCount}}
-
-
-
-
-
iFrames are the most complex HTML elements. They are pages, just like the main page, and the browser needs to create a new page context, which has a cost.
IDs of HTML elements must be document-wide unique. This can cause problems with getElementById returning the wrong element.
-
-
-
-
-
-
-
-
{{notations.jsDomManipulations}}
-
DOM manipulations
-
-
-
-
DOM inserts
-
- {{phantomasResults.metrics.DOMinserts}}
-
-
-
-
-
-
Working with the DOM in Javascript triggers layout calculations and slows down the page.
-
Try, as much as possible, to have an HTML page fully generated by the server instead of making changes with JS.
-
-
-
-
-
DOM queries
-
- {{phantomasResults.metrics.DOMqueries}}
-
-
-
-
-
DOM queries are like looking in a large catalog of items. Even if the browsers made progress on the performances of queries, websites often make hundreds of them.
-
Try to reduce the number of queries by refactoring your Javascript code.
-
Avoid also to have a read query between two write queries. To be able to reduce the number repaints and optimize performances, browsers buffer the DOM writing operations and treat them in bulk. But each time a DOM reading is asked, the browser needs to empty the buffer. This can be particularly slow inside a loop.
This is the number of queries that could be avoided by removing all duplicated queries.
-
Simply save the result of a query in a variable. Ok it is not always simple, especially with third-party scripts, but at least do it with your own code.
-
-
-
-
-
Events bound
-
- {{phantomasResults.metrics.eventsBound}}
-
-
-
-
-
-
Binding too many events has a cost.
-
It can be avoided by using "event delegation". Instead of binding events on each element one by one, events delegation binds them on the top level document element and uses the bubbling principle. It will imperceptibly slow down the event when it occurs, but the loading of the page will speed-up.
-
-
-
-
-
-
-
-
{{notations.jsBadPractices}}
-
Bad Javascript
-
-
-
-
Javascript errors
-
- {{phantomasResults.metrics.jsErrors}}
-
-
-
-
-
-
Just to let you know there are some errors on the page.
-
Please note that some errors only occur in the PhantomJS browser, so you might need to double check on other browsers.
-
-
-
-
-
eval calls
-
{{phantomasResults.metrics.evalCalls}}
-
-
-
-
The 'eval' function is slow and a bad coding practice. Try to get rid of it.
-
-
-
-
-
document.write calls
-
{{phantomasResults.metrics.documentWriteCalls}}
-
-
-
-
They slow down the page construction, especially if they are used to insert scripts in the page. Remove them ASAP.
-
If you cannot remove them because they come from a third-party script (such as ads), have a look at PostScribe.
-
-
-
-
-
Console messages
-
{{phantomasResults.metrics.consoleMessages}}
-
-
-
-
Try to keep your console clean when in production. Debugging is good for development only.
-
Writing in the console has a cost, especially when dumping large object variables.
-
There is also a problem with Internet Explorer 8, not knowing the console object.
It is a bad practice because they clutter up the global namespace. If two scripts use the same variable name in the global scope, it can cause conflicts and it is generally hard to debug.
-
Global variables also take a (very) little bit longer to be accessed than variables in the local scope of a function.
-
-
-
-
-
DOM manipulations in body
-
- {{inBodyDomManipulations}}
-
-
-
-
-
This metric counts the number of DOM queries, DOM inserts, binds, etc. made by the Javascript before the DOMContentLoaded event.
-
Wait for this event before manipulating the DOM. Do not execute Javascript in the middle of the BODY as it slows down the construction of the DOM and makes a poor maintainability. This is what i call spaghetti code.
-
The JS Timeline tab can help you identify what's happening.
-
-
-
-
-
-
-
-
{{notations.jQueryLoading}}
-
jQuery version
-
-
-
-
jQuery version
-
{{phantomasResults.metrics.jQueryVersion}}
-
-
-
-
Current latest versions of jQuery are 1.11 (with support for old IE versions) and 2.1 (without).
-
Each new version of jQuery optimizes performances. Do not keep an old version of jQuery. Updating can sometimes break a few things, but it is generally quite easy to fix them up. So don't hesitate.
jQuery is a heavy library. You should **never** load jQuery more than one on the same page.
-
-
-
-
-
-
-
-
{{notations.cssComplexity}}
-
CSS complexity
-
-
-
-
Rules count
-
- {{phantomasResults.metrics.cssRules}}
-
-
-
-
-
Having a huge number of CSS rules hurts performances. If the number of CSS rules is higher than the number of DOM elements, there is clearly a problem.
-
Huge stylesheets generally occur when the different pages of a website load all the CSS, concatenated in a single stylesheet, even if a large part of the rules are page-specific. Solution is to create one main CSS file with global rules and one custom files per page.
This is a bad practice as it slows down browsers. There are some simpler CSS3 methods for doing this.
-
-
-
-
-
Uses of !important
-
- {{phantomasResults.metrics.cssImportants}}
-
-
-
-
-
-
It can be useful, but only as a last resort. It is a bad practice because it overrides the normal cascading logic. The more you use !important, the more you need it again to over-override. This conducts to a poor maintainability.
-
-
-
-
-
Old IE fixes
-
- {{phantomasResults.metrics.cssOldIEFixes}}
-
-
-
-
-
-
What browser do you need to support? Once you've got the answer, take a look at these old rules that pollute your CSS code and remove them.
Many property prefixes such as -moz- or -webkit- are not needed anymore, or by very few people. You can remove them or replace them with the non-prefixed version. This will help reducing your stylesheets weight.
This is one way to remove complexity from a CSS rule. Generally, when "body" is specified in a rule it can be removed, because an element is necessarily inside the body.
Some tags included inside other tags are obvious. For example, when "ul li" is specified in a rule, "ul" can be removed because the "li" element is always inside a "ul". Same thing for "tr td", "select option", ...
-
Lowering compexity in CSS selectors can make the page load a little faster.
-
-
-
-
-
-
-
-
{{notations.requests}}
-
Requests number
-
-
-
-
Total requests
-
- {{phantomasResults.metrics.requests}}
-
-
-
-
-
This is one of the most important performance rule. Every request is slowing down the page loading.
-
There are several technics to reduce their number:
-
-
Concatenate JS files
-
Concatenate CSS files
-
Embed or inline small JS or CSS files in the HTML
-
Create sprites or icon fonts
-
Base64 encode small images in HTML or stylesheets
-
Use lazyloading for images
-
-
-
-
-
-
-
Documents
-
- {{phantomasResults.metrics.htmlCount}}
-
-
-
-
-
-
Scripts
-
- {{phantomasResults.metrics.jsCount}}
-
-
-
-
-
-
Stylesheets
-
- {{phantomasResults.metrics.cssCount}}
-
-
-
-
-
-
Images
-
- {{phantomasResults.metrics.imageCount}}
-
-
-
-
-
-
Fonts
-
- {{phantomasResults.metrics.webfontCount}}
-
-
-
-
-
-
Videos
-
- {{phantomasResults.metrics.videoCount}}
-
-
-
-
-
-
JSON
-
- {{phantomasResults.metrics.jsonCount}}
-
-
-
-
-
-
Other
-
- {{phantomasResults.metrics.otherCount}}
-
-
-
-
-
-
-
-
-
{{notations.network}}
-
Network
-
-
-
-
404 not found
-
- {{phantomasResults.metrics.notFound}}
-
-
-
-
-
-
404 errors are never cached, so each time a page ask for it, it hits se server. Even if it is behind a CDN or a reverse-proxy cache.
This counts the number of requests not keeping the connection alive (specifying "Connection: close" in the response headers). It is only counting a request if it is followed by another request on the same domain.
-
This is slowing down the next request, because the brower needs to open a new connection to the server, which means a additional round-trip.
-
Correct the problem by setting a Keep-Alive header on the guilty server.
This is one way to remove complexity from a CSS rule. Generally, when \"body\" is specified in a rule it can be removed, because an element is necessarily inside the body.
Some tags included inside other tags are obvious. For example, when \"ul li\" is specified in a rule, \"ul\" can be removed because the \"li\" element is always inside a \"ul\". Same thing for \"tr td\", \"select option\", ...
Lowering compexity in CSS selectors can make the page load a little faster.
\ No newline at end of file
diff --git a/front/src/views/rule.html b/front/src/views/rule.html
index e69de29..06c5f4b 100644
--- a/front/src/views/rule.html
+++ b/front/src/views/rule.html
@@ -0,0 +1,39 @@
+
+
\ No newline at end of file
diff --git a/front/src/views/timeline.html b/front/src/views/timeline.html
index f3fcb25..de7894f 100644
--- a/front/src/views/timeline.html
+++ b/front/src/views/timeline.html
@@ -1,5 +1,5 @@
-
+
Javascript Timeline
This graph gives a quick view of when the Javascript interactions with the DOM occur during the loading of the page.
\ No newline at end of file
diff --git a/front/src/views/about.html b/front/src/views/about.html
index 3016791..7824c61 100644
--- a/front/src/views/about.html
+++ b/front/src/views/about.html
@@ -1,8 +1,9 @@
-
-
-
-
-
-
\ No newline at end of file
+
This is done by loading the webpage via PhantomJS and collecting various metrics and statistics with the help of Phantomas. These metrics are categorized and transformed into scores. It also gives in-depth details so developpers can correct the detected issues.
+
+
By the way, it's free because we are geeks, not businessmen. All we want is a ★ on GitHub. It will boost our motivation to add more awesome features!!!
\ No newline at end of file
diff --git a/front/src/views/index.html b/front/src/views/index.html
index f676eee..7bc52c3 100644
--- a/front/src/views/index.html
+++ b/front/src/views/index.html
@@ -1,4 +1,5 @@
Free online test to help speeding up heavy web pages
+
+
+
+
+
+
+
+
+
Yellow Lab Tools is a free online tool that analyzes performance and front-end quality.
See the full report for this page
From 2ddf2e8341a5b04f2adb7adffd8e1629d2f28ecf Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ga=C3=ABl=20M=C3=A9tais?=
Date: Tue, 23 Dec 2014 16:54:24 +0100
Subject: [PATCH 55/98] Fix Test again button on Rule page
---
front/src/js/controllers/ruleCtrl.js | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/front/src/js/controllers/ruleCtrl.js b/front/src/js/controllers/ruleCtrl.js
index ae7d58c..63fa2ef 100644
--- a/front/src/js/controllers/ruleCtrl.js
+++ b/front/src/js/controllers/ruleCtrl.js
@@ -1,6 +1,6 @@
var ruleCtrl = angular.module('ruleCtrl', []);
-ruleCtrl.controller('RuleCtrl', ['$scope', '$rootScope', '$routeParams', '$location', '$sce', 'Menu', 'Results', 'Runs', function($scope, $rootScope, $routeParams, $location, $sce, Menu, Results, Run) {
+ruleCtrl.controller('RuleCtrl', ['$scope', '$rootScope', '$routeParams', '$location', '$sce', 'Menu', 'Results', 'Runs', function($scope, $rootScope, $routeParams, $location, $sce, Menu, Results, Runs) {
$scope.runId = $routeParams.runId;
$scope.policyName = $routeParams.policy;
$scope.Menu = Menu.setCurrentPage(null, $scope.runId);
From c0e02726be4058dd349d54d26ce4fd609de96ba1 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ga=C3=ABl=20M=C3=A9tais?=
Date: Tue, 23 Dec 2014 17:33:13 +0100
Subject: [PATCH 56/98] Fix categories order on dashboard
---
front/src/js/controllers/dashboardCtrl.js | 3 +++
front/src/views/dashboard.html | 2 +-
2 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/front/src/js/controllers/dashboardCtrl.js b/front/src/js/controllers/dashboardCtrl.js
index db1c4dd..12c269c 100644
--- a/front/src/js/controllers/dashboardCtrl.js
+++ b/front/src/js/controllers/dashboardCtrl.js
@@ -20,6 +20,9 @@ dashboardCtrl.controller('DashboardCtrl', ['$scope', '$rootScope', '$routeParams
}
function init() {
+ // By default, Angular sorts object's attributes alphabetically. Countering this problem by retrieving the keys order here.
+ $scope.categoriesOrder = Object.keys($scope.result.scoreProfiles.generic.categories);
+
$scope.globalScore = Math.max($scope.result.scoreProfiles.generic.globalScore, 0);
$scope.tweetText = 'My website\'s score is ' + $scope.globalScore + '/100 on #YellowLabTools!';
}
diff --git a/front/src/views/dashboard.html b/front/src/views/dashboard.html
index 5f504d0..896ee77 100644
--- a/front/src/views/dashboard.html
+++ b/front/src/views/dashboard.html
@@ -11,7 +11,7 @@
This is the number of queries that could be avoided by removing all duplicated queries.
Simply save the result of a query in a variable. Ok it is not always simple, especially with third-party scripts, but at least do it with your own code.
+ Did you know? Yellow Lab Tools now has an API! Here is the JSON output for this run. Checkout the API doc.
+
+
From e9505f31f76c8a67054704cb70c51a3dfa2b7b17 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ga=C3=ABl=20M=C3=A9tais?=
Date: Sun, 28 Dec 2014 16:57:45 +0100
Subject: [PATCH 60/98] Update phantomas and other dependencies
---
bower.json | 6 +++---
package.json | 18 +++++++++---------
2 files changed, 12 insertions(+), 12 deletions(-)
diff --git a/bower.json b/bower.json
index 9d7da0f..8609ca8 100644
--- a/bower.json
+++ b/bower.json
@@ -1,8 +1,8 @@
{
"name": "yellowlabtools",
"dependencies": {
- "angular": "~1.3.5",
- "angular-route": "~1.3.6",
- "angular-resource": "~1.3.6"
+ "angular": "~1.3.8",
+ "angular-route": "~1.3.8",
+ "angular-resource": "~1.3.7"
}
}
diff --git a/package.json b/package.json
index c7cfe8c..f77e320 100644
--- a/package.json
+++ b/package.json
@@ -12,12 +12,12 @@
"dependencies": {
"async": "~0.9.0",
"body-parser": "~1.10.0",
- "compression": "~1.2.1",
- "debug": "^2.1.0",
- "express": "~4.10.4",
- "phantomas": "1.7.0",
- "rimraf": "^2.2.8",
- "q": "^1.1.2"
+ "compression": "~1.2.2",
+ "debug": "~2.1.0",
+ "express": "~4.10.6",
+ "phantomas": "1.8.0",
+ "rimraf": "~2.2.8",
+ "q": "~1.1.2"
},
"devDependencies": {
"chai": "^1.10.0",
@@ -31,9 +31,9 @@
"grunt-fontsmith": "^0.9.1",
"grunt-mocha-test": "^0.12.4",
"matchdep": "^0.3.0",
- "mocha": "^2.0.1",
- "phantomjs": "^1.9.12",
- "request": "^2.49.0",
+ "mocha": "^2.1.0",
+ "phantomjs": "^1.9.13",
+ "request": "^2.51.0",
"sinon": "^1.12.1",
"sinon-chai": "^2.6.0"
},
From dccb9bb49cea5ada648b0bcac0ac2ed6976e2150 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ga=C3=ABl=20M=C3=A9tais?=
Date: Sun, 28 Dec 2014 18:05:10 +0100
Subject: [PATCH 61/98] Move phantomas_custom to the phantomas tool directory
---
lib/runner.js | 2 +-
.../tools/phantomas/custom_modules}/core/scopeYLT/scopeYLT.js | 0
.../custom_modules}/modules/domComplexYLT/domComplexYLT.js | 2 +-
.../phantomas/custom_modules}/modules/domQYLT/domQYLT.js | 2 +-
.../custom_modules}/modules/eventListYLT/eventListYLT.js | 0
.../tools/phantomas/custom_modules}/modules/jQYLT/jQYLT.js | 2 +-
.../phantomas/custom_modules}/modules/jsErrYLT/jsErrYLT.js | 0
.../custom_modules}/modules/jsFileLoadYLT/jsFileLoadYLT.js | 0
.../phantomas/custom_modules}/modules/jsTreeYLT/jsTreeYLT.js | 0
.../phantomas/custom_modules}/modules/keepAlive/keepAlive.js | 0
.../custom_modules}/modules/windowPerfYLT/windowPerfYLT.js | 0
lib/tools/{ => phantomas}/phantomasWrapper.js | 4 ++--
test/core/phantomasWrapperTest.js | 2 +-
13 files changed, 7 insertions(+), 7 deletions(-)
rename {phantomas_custom => lib/tools/phantomas/custom_modules}/core/scopeYLT/scopeYLT.js (100%)
rename {phantomas_custom => lib/tools/phantomas/custom_modules}/modules/domComplexYLT/domComplexYLT.js (98%)
rename {phantomas_custom => lib/tools/phantomas/custom_modules}/modules/domQYLT/domQYLT.js (99%)
rename {phantomas_custom => lib/tools/phantomas/custom_modules}/modules/eventListYLT/eventListYLT.js (100%)
rename {phantomas_custom => lib/tools/phantomas/custom_modules}/modules/jQYLT/jQYLT.js (99%)
rename {phantomas_custom => lib/tools/phantomas/custom_modules}/modules/jsErrYLT/jsErrYLT.js (100%)
rename {phantomas_custom => lib/tools/phantomas/custom_modules}/modules/jsFileLoadYLT/jsFileLoadYLT.js (100%)
rename {phantomas_custom => lib/tools/phantomas/custom_modules}/modules/jsTreeYLT/jsTreeYLT.js (100%)
rename {phantomas_custom => lib/tools/phantomas/custom_modules}/modules/keepAlive/keepAlive.js (100%)
rename {phantomas_custom => lib/tools/phantomas/custom_modules}/modules/windowPerfYLT/windowPerfYLT.js (100%)
rename lib/tools/{ => phantomas}/phantomasWrapper.js (95%)
diff --git a/lib/runner.js b/lib/runner.js
index 0cd4dcb..ab6ea6d 100644
--- a/lib/runner.js
+++ b/lib/runner.js
@@ -1,7 +1,7 @@
var Q = require('q');
var debug = require('debug')('ylt:runner');
-var phantomasWrapper = require('./tools/phantomasWrapper');
+var phantomasWrapper = require('./tools/phantomas/phantomasWrapper');
var jsExecutionTransformer = require('./tools/jsExecutionTransformer');
var rulesChecker = require('./rulesChecker');
var scoreCalculator = require('./scoreCalculator');
diff --git a/phantomas_custom/core/scopeYLT/scopeYLT.js b/lib/tools/phantomas/custom_modules/core/scopeYLT/scopeYLT.js
similarity index 100%
rename from phantomas_custom/core/scopeYLT/scopeYLT.js
rename to lib/tools/phantomas/custom_modules/core/scopeYLT/scopeYLT.js
diff --git a/phantomas_custom/modules/domComplexYLT/domComplexYLT.js b/lib/tools/phantomas/custom_modules/modules/domComplexYLT/domComplexYLT.js
similarity index 98%
rename from phantomas_custom/modules/domComplexYLT/domComplexYLT.js
rename to lib/tools/phantomas/custom_modules/modules/domComplexYLT/domComplexYLT.js
index a64f3c6..f03bf84 100644
--- a/phantomas_custom/modules/domComplexYLT/domComplexYLT.js
+++ b/lib/tools/phantomas/custom_modules/modules/domComplexYLT/domComplexYLT.js
@@ -28,7 +28,7 @@ exports.module = function(phantomas) {
// duplicated ID (issue #392)
phantomas.setMetric('DOMidDuplicated'); // @desc number of duplicated IDs found in DOM
- var Collection = require('../../../node_modules/phantomas/lib/collection'),
+ var Collection = require('../../../../../../node_modules/phantomas/lib/collection'),
DOMids = new Collection();
phantomas.on('domId', function(id) {
diff --git a/phantomas_custom/modules/domQYLT/domQYLT.js b/lib/tools/phantomas/custom_modules/modules/domQYLT/domQYLT.js
similarity index 99%
rename from phantomas_custom/modules/domQYLT/domQYLT.js
rename to lib/tools/phantomas/custom_modules/modules/domQYLT/domQYLT.js
index 8f0e4c9..78260d2 100644
--- a/phantomas_custom/modules/domQYLT/domQYLT.js
+++ b/lib/tools/phantomas/custom_modules/modules/domQYLT/domQYLT.js
@@ -246,7 +246,7 @@ exports.module = function(phantomas) {
// count DOM queries by either ID, tag name, class name and selector query
// @see https://dvcs.w3.org/hg/domcore/raw-file/tip/Overview.html#dom-document-doctype
- var Collection = require('../../../node_modules/phantomas/lib/collection'),
+ var Collection = require('../../../../../../node_modules/phantomas/lib/collection'),
DOMqueries = new Collection();
phantomas.on('domQuery', function(type, query, fnName, context) {
diff --git a/phantomas_custom/modules/eventListYLT/eventListYLT.js b/lib/tools/phantomas/custom_modules/modules/eventListYLT/eventListYLT.js
similarity index 100%
rename from phantomas_custom/modules/eventListYLT/eventListYLT.js
rename to lib/tools/phantomas/custom_modules/modules/eventListYLT/eventListYLT.js
diff --git a/phantomas_custom/modules/jQYLT/jQYLT.js b/lib/tools/phantomas/custom_modules/modules/jQYLT/jQYLT.js
similarity index 99%
rename from phantomas_custom/modules/jQYLT/jQYLT.js
rename to lib/tools/phantomas/custom_modules/modules/jQYLT/jQYLT.js
index 149fce0..b1014c3 100644
--- a/phantomas_custom/modules/jQYLT/jQYLT.js
+++ b/lib/tools/phantomas/custom_modules/modules/jQYLT/jQYLT.js
@@ -298,7 +298,7 @@ exports.module = function(phantomas) {
// count Sizzle calls to detect duplicated queries
- var Collection = require('../../../node_modules/phantomas/lib/collection'),
+ var Collection = require('../../../../../../node_modules/phantomas/lib/collection'),
sizzleCalls = new Collection(),
jQueryLoading = new Collection();
diff --git a/phantomas_custom/modules/jsErrYLT/jsErrYLT.js b/lib/tools/phantomas/custom_modules/modules/jsErrYLT/jsErrYLT.js
similarity index 100%
rename from phantomas_custom/modules/jsErrYLT/jsErrYLT.js
rename to lib/tools/phantomas/custom_modules/modules/jsErrYLT/jsErrYLT.js
diff --git a/phantomas_custom/modules/jsFileLoadYLT/jsFileLoadYLT.js b/lib/tools/phantomas/custom_modules/modules/jsFileLoadYLT/jsFileLoadYLT.js
similarity index 100%
rename from phantomas_custom/modules/jsFileLoadYLT/jsFileLoadYLT.js
rename to lib/tools/phantomas/custom_modules/modules/jsFileLoadYLT/jsFileLoadYLT.js
diff --git a/phantomas_custom/modules/jsTreeYLT/jsTreeYLT.js b/lib/tools/phantomas/custom_modules/modules/jsTreeYLT/jsTreeYLT.js
similarity index 100%
rename from phantomas_custom/modules/jsTreeYLT/jsTreeYLT.js
rename to lib/tools/phantomas/custom_modules/modules/jsTreeYLT/jsTreeYLT.js
diff --git a/phantomas_custom/modules/keepAlive/keepAlive.js b/lib/tools/phantomas/custom_modules/modules/keepAlive/keepAlive.js
similarity index 100%
rename from phantomas_custom/modules/keepAlive/keepAlive.js
rename to lib/tools/phantomas/custom_modules/modules/keepAlive/keepAlive.js
diff --git a/phantomas_custom/modules/windowPerfYLT/windowPerfYLT.js b/lib/tools/phantomas/custom_modules/modules/windowPerfYLT/windowPerfYLT.js
similarity index 100%
rename from phantomas_custom/modules/windowPerfYLT/windowPerfYLT.js
rename to lib/tools/phantomas/custom_modules/modules/windowPerfYLT/windowPerfYLT.js
diff --git a/lib/tools/phantomasWrapper.js b/lib/tools/phantomas/phantomasWrapper.js
similarity index 95%
rename from lib/tools/phantomasWrapper.js
rename to lib/tools/phantomas/phantomasWrapper.js
index 8dbd614..803fb00 100644
--- a/lib/tools/phantomasWrapper.js
+++ b/lib/tools/phantomas/phantomasWrapper.js
@@ -43,8 +43,8 @@ var PhantomasWrapper = function() {
'windowPerformance' // overriden
].join(','),
'include-dirs': [
- 'phantomas_custom/core',
- 'phantomas_custom/modules'
+ 'lib/tools/phantomas/custom_modules/core',
+ 'lib/tools/phantomas/custom_modules/modules'
].join(',')
};
diff --git a/test/core/phantomasWrapperTest.js b/test/core/phantomasWrapperTest.js
index 904915e..62ddd08 100644
--- a/test/core/phantomasWrapperTest.js
+++ b/test/core/phantomasWrapperTest.js
@@ -1,5 +1,5 @@
var should = require('chai').should();
-var phantomasWrapper = require('../../lib/tools/phantomasWrapper');
+var phantomasWrapper = require('../../lib/tools/phantomas/phantomasWrapper');
describe('phantomasWrapper', function() {
From fb323c2086d400dcba2b2cf05cb4e5b6828a49b6 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ga=C3=ABl=20M=C3=A9tais?=
Date: Mon, 29 Dec 2014 08:20:46 +0100
Subject: [PATCH 62/98] Import DOMqueriesWithoutResults fonctionnality
---
.../custom_modules/core/scopeYLT/scopeYLT.js | 4 +-
.../custom_modules/modules/domQYLT/domQYLT.js | 69 +++++++++++++++----
2 files changed, 58 insertions(+), 15 deletions(-)
diff --git a/lib/tools/phantomas/custom_modules/core/scopeYLT/scopeYLT.js b/lib/tools/phantomas/custom_modules/core/scopeYLT/scopeYLT.js
index 9ed515a..c7478bb 100644
--- a/lib/tools/phantomas/custom_modules/core/scopeYLT/scopeYLT.js
+++ b/lib/tools/phantomas/custom_modules/core/scopeYLT/scopeYLT.js
@@ -7,7 +7,7 @@
*/
/* global document: true, window: true */
-exports.version = '0.1';
+exports.version = '0.2';
exports.module = function(phantomas) {
'use strict';
@@ -62,7 +62,7 @@ exports.module = function(phantomas) {
// After
if (enabled && callbackAfter) {
- callbackAfter.call(this, result);
+ callbackAfter.call(this, result, arguments);
}
}
diff --git a/lib/tools/phantomas/custom_modules/modules/domQYLT/domQYLT.js b/lib/tools/phantomas/custom_modules/modules/domQYLT/domQYLT.js
index 78260d2..81985f4 100644
--- a/lib/tools/phantomas/custom_modules/modules/domQYLT/domQYLT.js
+++ b/lib/tools/phantomas/custom_modules/modules/domQYLT/domQYLT.js
@@ -3,12 +3,13 @@
*/
/* global Element: true, Document: true, Node: true, window: true */
-exports.version = '0.9.a';
+exports.version = '0.10.a';
exports.module = function(phantomas) {
'use strict';
phantomas.setMetric('DOMqueries'); // @desc number of all DOM queries @offenders
+ phantomas.setMetric('DOMqueriesWithoutResults'); // @desc number of DOM queries that retutned nothing @offenders
phantomas.setMetric('DOMqueriesById'); // @desc number of document.getElementById calls
phantomas.setMetric('DOMqueriesByClassName'); // @desc number of document.getElementsByClassName calls
phantomas.setMetric('DOMqueriesByTagName'); // @desc number of document.getElementsByTagName calls
@@ -21,14 +22,13 @@ exports.module = function(phantomas) {
phantomas.once('init', function() {
phantomas.evaluate(function() {
(function(phantomas) {
- function querySpy(type, query, fnName, context) {
- phantomas.emit('domQuery', type, query, fnName, context); // @desc DOM query has been made
+ function querySpy(type, query, fnName, context, hasNoResults) {
+ phantomas.emit('domQuery', type, query, fnName, context, hasNoResults); // @desc DOM query has been made
}
phantomas.spy(Document.prototype, 'getElementById', function(id) {
phantomas.incrMetric('DOMqueriesById');
phantomas.addOffender('DOMqueriesById', '#%s (in %s)', id, '#document');
- querySpy('id', '#' + id, 'getElementById', '#document');
phantomas.enterContext({
type: 'getElementById',
@@ -41,9 +41,13 @@ exports.module = function(phantomas) {
backtrace: phantomas.getBacktrace()
});
- }, function(result) {
+ }, function(result, args) {
+ var id = args[0];
+
+ querySpy('id', '#' + id, 'getElementById', '#document', (result === null));
+
var moreData = {
- resultsNumber : result ? 1 : 0
+ resultsNumber : (result === null) ? 0 : 1
};
phantomas.leaveContext(moreData);
});
@@ -56,7 +60,6 @@ exports.module = function(phantomas) {
phantomas.incrMetric('DOMqueriesByClassName');
phantomas.addOffender('DOMqueriesByClassName', '.%s (in %s)', className, context);
- querySpy('class', '.' + className, 'getElementsByClassName', context);
phantomas.enterContext({
type: 'getElementsByClassName',
@@ -70,7 +73,14 @@ exports.module = function(phantomas) {
});
}
- function selectorClassNameAfter(result) {
+ function selectorClassNameAfter(result, args) {
+ /*jshint validthis: true */
+
+ var className = args[0];
+ var context = phantomas.getDOMPath(this);
+
+ querySpy('class', '.' + className, 'getElementsByClassName', context, (result.length === 0));
+
var moreData = {
resultsNumber : (result && result.length > 0) ? result.length : 0
};
@@ -88,7 +98,6 @@ exports.module = function(phantomas) {
phantomas.incrMetric('DOMqueriesByTagName');
phantomas.addOffender('DOMqueriesByTagName', '%s (in %s)', tagName, context);
- querySpy('tag name', tagName.toLowerCase(), 'getElementsByTagName', context);
phantomas.enterContext({
type: 'getElementsByTagName',
@@ -102,7 +111,14 @@ exports.module = function(phantomas) {
});
}
- function selectorTagNameSpyAfter(result) {
+ function selectorTagNameSpyAfter(result, args) {
+ /*jshint validthis: true */
+
+ var tagName = args[0];
+ var context = phantomas.getDOMPath(this);
+
+ querySpy('tag name', tagName.toLowerCase(), 'getElementsByTagName', context, (result.length === 0));
+
var moreData = {
resultsNumber : (result && result.length > 0) ? result.length : 0
};
@@ -112,11 +128,11 @@ exports.module = function(phantomas) {
phantomas.spy(Document.prototype, 'getElementsByTagName', selectorTagNameSpyBefore, selectorTagNameSpyAfter);
phantomas.spy(Element.prototype, 'getElementsByTagName', selectorTagNameSpyBefore, selectorTagNameSpyAfter);
+
// selector queries
function selectorQuerySpy(selector, context) {
phantomas.incrMetric('DOMqueriesByQuerySelectorAll');
phantomas.addOffender('DOMqueriesByQuerySelectorAll', '%s (in %s)', selector, context);
- querySpy('selector', selector, 'querySelectorAll', context);
}
function selectorQuerySpyBefore(selector) {
@@ -137,7 +153,14 @@ exports.module = function(phantomas) {
});
}
- function selectorQuerySpyAfter(result) {
+ function selectorQuerySpyAfter(result, args) {
+ /*jshint validthis: true */
+
+ var selector = args[0];
+ var context = phantomas.getDOMPath(this);
+
+ querySpy('selector', selector, 'querySelectorAll', context, (!result || result.length === 0));
+
var moreData = {
resultsNumber : result ? 1 : 0
};
@@ -162,7 +185,14 @@ exports.module = function(phantomas) {
});
}
- function selectorAllQuerySpryAfter(result) {
+ function selectorAllQuerySpryAfter(result, args) {
+ /*jshint validthis: true */
+
+ var selector = args[0];
+ var context = phantomas.getDOMPath(this);
+
+ querySpy('selector', selector, 'querySelectorAll', context, (!result || result.length === 0));
+
var moreData = {
resultsNumber : (result && result.length > 0) ? result.length : 0
};
@@ -244,6 +274,19 @@ exports.module = function(phantomas) {
});
});
+ // report DOM queries that return no results (issue #420)
+ phantomas.on('domQuery', function(type, query, fnName, context, hasNoResults) {
+ // ignore DOM queries within DOM fragments (used internally by jQuery)
+ if (context.indexOf('body') !== 0 && context.indexOf('#document') !== 0) {
+ return;
+ }
+
+ if (hasNoResults === true) {
+ phantomas.incrMetric('DOMqueriesWithoutResults');
+ phantomas.addOffender('DOMqueriesWithoutResults', '%s (in %s) using %s', query, context, fnName);
+ }
+ });
+
// count DOM queries by either ID, tag name, class name and selector query
// @see https://dvcs.w3.org/hg/domcore/raw-file/tip/Overview.html#dom-document-doctype
var Collection = require('../../../../../../node_modules/phantomas/lib/collection'),
From 6fe61a041eb94bad8dd073aa772012a28e2d33ac Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ga=C3=ABl=20M=C3=A9tais?=
Date: Mon, 29 Dec 2014 10:26:08 +0100
Subject: [PATCH 63/98] Add DOMqueriesWithoutResults as a rule
---
lib/metadata/policies.js | 8 ++++++++
lib/metadata/scoreProfileGeneric.json | 1 +
2 files changed, 9 insertions(+)
diff --git a/lib/metadata/policies.js b/lib/metadata/policies.js
index b9cb910..c01566d 100644
--- a/lib/metadata/policies.js
+++ b/lib/metadata/policies.js
@@ -58,6 +58,14 @@ var policies = {
"isAbnormalThreshold": 500,
"takeOffendersFrom": "DOMqueriesDuplicated"
},
+ "DOMqueriesWithoutResults": {
+ "tool": "phantomas",
+ "label": "DOM queries without result",
+ "message": "
Number of queries that return no result.
It suggests the query is not used on the page, probably because it is some dead code.
Or maybe the code is trying to find an HTML block that is not always here. Look at the JS Timeline to see if the scripts correctly figures out the HTML block is not here and immediatly stops interacting further with the DOM.
",
+ "isOkThreshold": 0,
+ "isBadThreshold": 100,
+ "isAbnormalThreshold": 200
+ },
"eventsBound": {
"tool": "phantomas",
"label": "Events bound",
diff --git a/lib/metadata/scoreProfileGeneric.json b/lib/metadata/scoreProfileGeneric.json
index 8feb3ff..d104e51 100644
--- a/lib/metadata/scoreProfileGeneric.json
+++ b/lib/metadata/scoreProfileGeneric.json
@@ -14,6 +14,7 @@
"policies": {
"DOMinserts": 2,
"DOMqueries": 1,
+ "DOMqueriesWithoutResults": 2,
"DOMqueriesAvoidable": 2,
"eventsBound": 1
}
From 0b23fad9093ab33f31f3a03ea605a8dba5c7849f Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ga=C3=ABl=20M=C3=A9tais?=
Date: Mon, 29 Dec 2014 10:35:18 +0100
Subject: [PATCH 64/98] Remove custom closedConnections module implemented in
Phantomas
---
.../modules/keepAlive/keepAlive.js | 37 -------------------
1 file changed, 37 deletions(-)
delete mode 100644 lib/tools/phantomas/custom_modules/modules/keepAlive/keepAlive.js
diff --git a/lib/tools/phantomas/custom_modules/modules/keepAlive/keepAlive.js b/lib/tools/phantomas/custom_modules/modules/keepAlive/keepAlive.js
deleted file mode 100644
index 4550e77..0000000
--- a/lib/tools/phantomas/custom_modules/modules/keepAlive/keepAlive.js
+++ /dev/null
@@ -1,37 +0,0 @@
-/**
- * Analyzes if HTTP responses keep the connections alive.
- */
-
-exports.version = '0.1';
-
-exports.module = function(phantomas) {
- 'use strict';
-
- phantomas.setMetric('closedConnections'); // @desc requests not keeping the connection alive and slowing down the next request @offenders
-
- var closedConnectionHosts = {};
-
- phantomas.on('recv', function(entry, res) {
- var connectionHeader = (entry.headers.Connection || '').toLowerCase();
- // Taking the protocol in account, in case the same domain is called with two different protocols.
- var host = entry.protocol + '://' + entry.domain;
-
- if (connectionHeader.indexOf('close') >= 0) {
- // Don't blame it immediatly, wait to see if the connection is needed a second time.
- closedConnectionHosts[host] = entry.url;
- }
- });
-
- phantomas.on('send', function(entry, res) {
- var host = entry.protocol + '://' + entry.domain;
- var previousClosedConnection = closedConnectionHosts[host];
-
- if (previousClosedConnection) {
- // There was a closed connection. We can blame it safely now!
- phantomas.incrMetric('closedConnections');
- phantomas.addOffender('closedConnections', previousClosedConnection);
-
- closedConnectionHosts[host] = null;
- }
- });
-};
From 519f2ba61739f7767cb0a0cf2b29f0485ffcdd5a Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ga=C3=ABl=20M=C3=A9tais?=
Date: Mon, 29 Dec 2014 12:17:52 +0100
Subject: [PATCH 65/98] Remove custom domComplexity module, maxDepth offenders
added to phantomas
---
.../modules/domComplexYLT/domComplexYLT.js | 135 ------------------
lib/tools/phantomas/phantomasWrapper.js | 1 -
2 files changed, 136 deletions(-)
delete mode 100644 lib/tools/phantomas/custom_modules/modules/domComplexYLT/domComplexYLT.js
diff --git a/lib/tools/phantomas/custom_modules/modules/domComplexYLT/domComplexYLT.js b/lib/tools/phantomas/custom_modules/modules/domComplexYLT/domComplexYLT.js
deleted file mode 100644
index f03bf84..0000000
--- a/lib/tools/phantomas/custom_modules/modules/domComplexYLT/domComplexYLT.js
+++ /dev/null
@@ -1,135 +0,0 @@
-/**
- * Analyzes DOM complexity
- */
-/* global document: true, Node: true, window: true */
-
-exports.version = '1.0.a';
-
-exports.module = function(phantomas) {
- 'use strict';
-
- // total length of HTML comments (including brackets)
- phantomas.setMetric('commentsSize'); // @desc the size of HTML comments on the page @offenders
-
- // total length of text nodes with whitespaces only (i.e. pretty formatting of HTML)
- phantomas.setMetric('whiteSpacesSize'); // @desc the size of text nodes with whitespaces only
-
- // count all tags
- phantomas.setMetric('DOMelementsCount'); // @desc total number of HTML element nodes
- phantomas.setMetric('DOMelementMaxDepth'); // @desc maximum level on nesting of HTML element node
-
- // nodes with inlines CSS (style attribute)
- phantomas.setMetric('nodesWithInlineCSS'); // @desc number of nodes with inline CSS styling (with style attribute) @offenders
-
- // images
- phantomas.setMetric('imagesScaledDown'); // @desc number of nodes that have images scaled down in HTML @offenders
- phantomas.setMetric('imagesWithoutDimensions'); // @desc number of nodes without both width and height attribute @offenders
-
- // duplicated ID (issue #392)
- phantomas.setMetric('DOMidDuplicated'); // @desc number of duplicated IDs found in DOM
-
- var Collection = require('../../../../../../node_modules/phantomas/lib/collection'),
- DOMids = new Collection();
-
- phantomas.on('domId', function(id) {
- DOMids.push(id);
- });
-
- // HTML size
- phantomas.on('report', function() {
- phantomas.setMetricEvaluate('bodyHTMLSize', function() { // @desc the size of body tag content (document.body.innerHTML.length)
- return document.body && document.body.innerHTML.length || 0;
- });
-
- phantomas.evaluate(function() {
- (function(phantomas) {
- var runner = new phantomas.nodeRunner(),
- whitespacesRegExp = /^\s+$/,
- DOMelementMaxDepth = 0,
- DOMelementMaxDepthElts = [],
- size = 0;
-
- runner.walk(document.body, function(node, depth) {
- switch (node.nodeType) {
- case Node.COMMENT_NODE:
- size = node.textContent.length + 7; // ''.length
- phantomas.incrMetric('commentsSize', size);
-
- // log HTML comments bigger than 64 characters
- if (size > 64) {
- phantomas.addOffender('commentsSize', phantomas.getDOMPath(node) + ' (' + size + ' characters)');
- }
- break;
-
- case Node.ELEMENT_NODE:
- phantomas.incrMetric('DOMelementsCount');
-
- if (depth > DOMelementMaxDepth) {
- DOMelementMaxDepth = depth;
- DOMelementMaxDepthElts = [phantomas.getDOMPath(node)];
- } else if (depth === DOMelementMaxDepth) {
- DOMelementMaxDepthElts.push(phantomas.getDOMPath(node));
- }
-
- // report duplicated ID (issue #392)
- if (node.id) {
- phantomas.emit('domId', node.id);
- }
-
- // ignore inline
\ No newline at end of file
diff --git a/package.json b/package.json
index 6b34d28..80f6fd9 100644
--- a/package.json
+++ b/package.json
@@ -32,6 +32,7 @@
"grunt-express": "^1.4.1",
"grunt-fontsmith": "^0.9.1",
"grunt-mocha-test": "^0.12.4",
+ "grunt-replace": "^0.8.0",
"matchdep": "^0.3.0",
"mocha": "^2.1.0",
"phantomjs": "^1.9.13",
diff --git a/server_config/settings-prod.json b/server_config/settings-prod.json
index 8d06c6a..6b383c4 100644
--- a/server_config/settings-prod.json
+++ b/server_config/settings-prod.json
@@ -1,12 +1,10 @@
{
"serverPort": 80,
- "googleAnalyticsId": "UA-54493828-1",
+ "googleAnalyticsId": "",
- "authorized-keys": {
+ "authorizedKeys": {
},
- "tokenSalt": "",
- "authorizedApplications": [
-
- ]
+ "maxAnonymousRunsPerDay": 50,
+ "maxAnonymousCallsPerDay": 2000
}
\ No newline at end of file
From 5c80f3197a6610d14d5c3ffd178ae6d6a7ad3952 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ga=C3=ABl=20M=C3=A9tais?=
Date: Tue, 30 Dec 2014 09:20:05 +0100
Subject: [PATCH 74/98] Add version number in the footer
---
Gruntfile.js | 4 ++++
front/src/css/main.css | 3 +++
front/src/less/main.less | 3 +++
front/src/main.html | 3 ++-
4 files changed, 12 insertions(+), 1 deletion(-)
diff --git a/Gruntfile.js b/Gruntfile.js
index f2c3ae6..2ef5149 100644
--- a/Gruntfile.js
+++ b/Gruntfile.js
@@ -42,6 +42,10 @@ module.exports = function(grunt) {
{
match: 'googleAnalyticsId',
replacement: '<%= settings.googleAnalyticsId %>'
+ },
+ {
+ match: 'version',
+ replacement: 'v<%= pkg.version %>'
}
]
},
diff --git a/front/src/css/main.css b/front/src/css/main.css
index 6022ca3..9ff35ad 100644
--- a/front/src/css/main.css
+++ b/front/src/css/main.css
@@ -126,6 +126,9 @@ h1 span {
.footer a {
color: inherit;
}
+.footer .version {
+ font-size: 0.7em;
+}
/* Icons */
.icon-lab {
font-family: "fontsmith-icons";
diff --git a/front/src/less/main.less b/front/src/less/main.less
index a2c6c0f..ae2cc21 100644
--- a/front/src/less/main.less
+++ b/front/src/less/main.less
@@ -126,6 +126,9 @@ h1 span {
a {
color: inherit;
}
+ .version {
+ font-size: 0.7em;
+ }
}
/* Icons */
diff --git a/front/src/main.html b/front/src/main.html
index 8522dcd..53ccfc5 100644
--- a/front/src/main.html
+++ b/front/src/main.html
@@ -33,7 +33,8 @@
\ No newline at end of file
diff --git a/front/src/views/dashboard.html b/front/src/views/dashboard.html
index 0fba7bc..e95421d 100644
--- a/front/src/views/dashboard.html
+++ b/front/src/views/dashboard.html
@@ -1,4 +1,4 @@
-
+