Snyk Code for Static Application Security Testing (SAST)

SAST tools speed comparison: Snyk Code vs SonarQube and LGTM

We’ve been asked to provide a comparison of scan times between Snyk Code and two common SAST tools: LGTM and SonarQube. For our research, we made several assumptions, but we’ve shared the details in order to be transparent.

TL;DR  

Static Application Security Testing (SAST) can only be developer-friendly when it provides near real-time feedback and does not delay your development processes. Snyk Code is up to 106 times faster than LGTM. On average, Snyk Code is 5x times faster than SonarQube or 14x times faster than LGTM. In summary, Snyk Code proves to be one of the fastest semantic scanning engines on the market.

Test field

We have selected 48 JavaScript open source repositories (listed below). The idea was to mimic typically modern developer code sets and JavaScript seemed a good common delimiter. We have chosen a random sample from top-rated repositories on GitHub to represent real-world challenges.

As scanners, we have the Community Edition of SonarQube which is a broadly used open source static analysis tool. It runs locally, so we needed to provide a quite decent PC. As previous tests using the free SonarCloud edition showed: SonarQube on a good PC is faster than free SonarCloud, so it is not unfair to use the local engine instead of the cloud version.  We have to take one of the existing developer machines (details below). The second contestant is LGTM which originates from a company called Semmle which was acquired by GitHub. LGTM uses a deep semantic code search based on CodeQL. We used the LGTM SaaS offering. 

Finally, Snyk Code is Snyk’s SAST solution. It is based on the former DeepCode scan engine, now with several months of additional development time within Snyk under its belt. Aside from being developer-friendly and highly accurate, one of Snyk Code’s design goals is to be extremely fast. It uses a proprietary constraint engine to achieve this. We selected this field as (1) the licenses allow us to run and compare, (2) these are semantic engines and not linters (like ESlint), (3) these are common and widely used engines, and (4) we have both a locally running and a SaaS in the field. 

Speed is essential when a SAST solution wants to be developer-friendly. It’s widely known that developers are most efficient when security issues are identified during the development process so they can be addressed before the code gets checked in. The code is fresh in their minds. Snyk Code provides IDE plugins that embed seamlessly into the developer workflow. On top of that, Snyk Code provides easy to understand data flow diagrams and extensive explanations, including examples of fixes used in open source libraries with the same context. But for the purposes of this test, we focused on speed. 

The way we tested is we ran a scan for each of the repositories — SonarQube locally, LGTM and Snyk Code as SaaS. As mentioned above, this led to our repo selection as we did not want to make scan times dependent on network bandwidth. 

We also selected real-life open source repositories and not benchmarks as the quest here is to simulate what real developers do. Using benchmarks to compare SAST tools has its own problems, but in this case it would not help our cause.

Test results

Below are the general statistical values (rounded) in seconds of scan time. (n=48, rounded to integer):

AverageMaxMin5-percentile95-percentileMediansd
SonarQube110895636325777127
LGTM312183521411046189352
Snyk Code2216255721228

Here are the results as a scatterplot (lower is better, logarithmic scale, done with R and ggplot2):

The logarithmic axis over-emphasizes the spread in the lower field. The standard deviation of Snyk Code is actually much smaller than the one of SonarQube for example. 

Below is a box plot of the values (lower is better, we used a logarithmic scale to fit all values in the graph, done with R and ggplot2, median and average are shown):

The plot clearly shows that in the majority of cases, Snyk is dramatically faster than either of the other two. The median of Snyk is 6.7x (SonarQube) up to 16.4x (LGTM) times faster, which shows that the results do not rely on some extremely good outliers but instead are general ones.

In the above diagram, the spread of values in the LGTM column is noteworthy. It roughly lays between 2 minutes (somehow ok) and more than 17 minutes (not acceptable). Developers will not wait for the results taking several minutes during their workday. Luckily, the median of LGTM is around 3 minutes.

In comparison to the first speed test we did (that time as DeepCode, now Snyk Code) we noticed a lot of performance improvement because we optimized loading to the cloud service significantly in the past months. Also, this time, SonarQube was faster, probably because we used a stronger PC with higher grades of parallelism compared to the first test.

Another interesting observation is that SaaS versus locally-installed engines are not actually that different from a speed perspective. 

Assumptions and limitations

  • We have chosen SonarQube (Community Edition 8.9.1) and LGTM as the license allows us these comparisons and they are broadly used. 
  • We have chosen 48 medium JavaScript repositories as this test field seems to reflect what typical developers work on. We sampled randomly from the top ~200k GitHub repos by stars. Code size was not a criterion.
  • We forced a full scan and did not use a differential scanning mechanism. For example, in the case of Snyk Code in IDE plugins, after an initial scan the IDE plugin does differential scans and saves bandwidth.
  • We used an Intel Xeon @ 2GHz with 16 cores and 64GB RAM

Use Snyk Code for free

Scan your code for security vulnerabilities within repos or directly from your IDE.

Raw data

We also want to provide the raw data so you can apply your own statistics. (Times in seconds)

RepositorySonarQubeLGTMSnyk Code
neumino/chateau1012599
ecomfe/etpl8023724
ustwo/ustwo.com-frontend7727820
node-inspector/node-inspector15737321
q-nick/npm-gui712278
dojo/dojox31766253
majimboo/node-benchmarks631415
prettier/prettier8427745
d3/d3.github.com895110467
adobe/node-smb-server7919413
noble/bleno661415
wagtail/wagtail115146256
meteor/blaze8319412
dwyl/hapi-socketio-redis-chat-example681415
chancancode/hn-reader6416520
shipshapecode/ember-shepherd8215230
draptik/angulardemorestful761636
icebob/vue-express-mongo-boilerplate774229
ngryman/ribs641937
rjrodger/seneca-examples7017612
telepat-io/telepat-api681738
sematext/logagent-js681628
cookpad/elasticfox-ec2tag8717311
polonel/trudesk170183577
saintedlama/passport-local-mongoose721737
cholalabs/passport-localapikey6335
mozilla/openbadger7918420
inbasic/turbo-download-manager9420631
pgherveou/gulp-awspublish651525
swagger-api/swagger-socket937467
tactivos/jquery-sew692255
johansatge/jpeg-autorotate631526
mqttjs/mqtt-packet6517312
fulcrum-agile/fulcrum8015218
microweber/microweber300937162
adamhalasz/uniqid681515
componentjs/builder2.js711428
frankyghost/projekktor921959
nielsutrecht/jwt-angular-spring882888
petereigenschink/steganography.js691415
mixteam/mixsln7520548
borismus/webvr-boilerplate7021342
arrix/node-readability7715219
iamcal/js-emoji724126
jbavari/ionic-socket.io-redis-chat14938419
cheeaun/steepless691425
joewalker/devtools.html177274
gaearon/react-hot-loader8016215