The NN b-tagger

Algorithm overview

MC tuning / variable selection

Measuring performance in p14 data

Caveats

Future plans / p17 / schedule

Conclusions

Andy Haas*

Columbia University

*NN ideas and work are entriely the work of:

Tim Scanlon

Imperial College, London

and

Miruna Anastasoaie
Radboud Univerisity, Nijmegen

The NN algorithm - motivation

For quite a while, we've had several “lifetime” taggers at D0...

CSIP

Based on track IP significances. Relatively simple combination of tracks / jet. Carefully tuned selection of tracks / Chi2 / pT / nSMT, etc.

JLIP

Based on track IP significances. More intricate combination of track probabilities. Carefully tuned true IP sig in data based on categories of Chi2 / Nhits, etc.

SVT

Attempts to reconstruct consistent secondary decay vertices of b/c hadrons inside jets. Uses information of how well the high-IPsig tracks fit together in ways physically consistent with secondary decays.

And for some time, we've realized that these taggers are not 100% correlated

They pick out different quantities within jets and some algorithms are good for some types of jets and others are better for other types of jets

Solutions:

Use ANDs and/or ORs of taggers / operating points

Use a fancier multivariate technique

Likelihood approach

Throw them in a NN

J. Levêque
likelihood combination

circa Oct. '04

The NN strategy

Optimize a NN in Monte Carlo for distinguishing b-jets from non-b jets

Apply this NN to data and measure its performance, using muon-jets

The NN implementation is the one in ROOT, a multi-layer feed-forward back-error-propogating sigmoid network

The MC samples were tt, Z->q,c,b, and QCD q,c,b.

Care was taken to merge (weight) the samples for similar pT distributions...

The input variables considered were taken from CSIP, JLIP, and SVT “meta-outputs”, not final tag decisions

SLT was not used in the NN, to be able to measure performance in data

CSIP output was combined into a super-variable containing info about all the different number of track categories

5 variables from SVT were considered
“Super-loose” track cuts used to increase the available information

3 variables from JLIP were considered

Several aspects of MC optimization:

How many and which variables to use for input

Number of hidden nodes (depends on the inputs!)

Training algorithm, training length / epochs

Jets used for training

Choosing variables / hidden nodes

Start with some minimal NN (a few variables you “know” are good)

Add each of the remaining variables, and see which one improves performance most – keep that one

Repeat 2 until you're out of variables or performance stops improving

NNs were tested with different number of hidden nodes (but always 1 hidden layer)

24 hidden nodes was found to be optimal

Minimization algorithm

The mean-squared-error of the NN must be minimized by adjusting the neuron and synapse sigmoid weights

This is a problem in non-linear optimization

Several dozen algorithms exist and 6 are implemented in ROOT

These 6 were all tested, and the one that tended to give the smallest error in the least time was chosen

BFGS was best, which chooses the direction and gradient of steepest descent at each step using the weight matrices themselves (not just the outputs)

1000 epochs was determined to be sufficient for finding a stable solution

Input training jets

This is the smartest thing Tim ever did

I would not have thought of doing this!

It's not a good idea to throw every jet in all the MC samples at the NN

It takes too long to train, since the number of jets is large

The NN will waste its power figuring out very easy cases, instead of focusing on the more difficult separations

Simple cuts should be used to do the easy work of removing jets that are almost certainly not b's

But the cuts should not be too tight since this will limit the maximal efficiency of the NN

The jets were “pre-selected” using cuts on the most powerful variable of each of the 3 algorithms

These cuts were then tuned by training different NNs using each of the cuts and measuring their performances in MC

Sumamry of MC-tuned NN

MC results

MC results

Applying the NN to data

We first want to check that the inputs for the NN variables look similar in data and MC

A b-enriched data sample (96% b) was used for the comparison

Applying the NN to data

And check the same on light jets

data / MC comparison

The NN output also looks similar in data and MC for b-jets and light-jets

Measuring efficiency in data

As with other taggers, System8 is used to measure efficiency

I won't go into details, since it's an hour-long talk of its own... but the basics are:

Take two data samples, b-enriched and normal

Take two nearly uncorrelated taggers

NN

SLT – muon

Use a system of equations to solve for the efficiencies of both taggers in data

I'll point out some of the novel methods and results that relate to the NN tagger

K_b

This is the correlation between the SLT and NN tagger

Usually it is taken to be a constant

Here is it measured as a function of pT

System8 error parameterizations

This is a more general feature of the NN error parameterizations...

The errors are measured individually in each pT / |eta| bin

Note the higher errors at pT>60 GeV (lack of statistics) and larger |eta| (worse fits)

Total efficiency errors

Light jet efficiency

Mesured in the standard way from the QCD and EM1TRK skims using negative tag rates and scale factors from MC

Binned in 3 |eta| regions

Light jet tag rate uncertainty

Total:

From b-content

uncertainty in MC:

New feature!

From differences in
EMF<.8 jets:

Parameterization
uncertainties:

Data results

Performance is similarly better in data, just like it was in MC

We've finally reached the 60% eff, 1% fake rate that was used in the RunII SUSY/Higgs study!*

*For taggable jets only... other jets may not apply.

Some caveats

In this certification, as with all others in the past, several items have been swept under the rug, either because we hadn't thought of them yet, or because there was not the personpower to study them!

In addition to taggability being a strong function of PV |z| position, we also expect the b-tagging efficiency and fake rates to be dependent on this quantity. Not studied!

So expect some extra error if you apply tagging to multiple jets in the same event. The tag rates are correlated since they share the same PV |z|. Overall error, probably a few % - but comparable to the total systematic b-tagging error!

Same goes for effects due to:

the number of min-bias interactions

the detector performance during which the data was taken (SMT HDIs on/off, etc.)

There are no light-jet data/MC scale factors

You can't apply the data/MC scale factor for b/c jets to light-jets! In fact, the correction is probably > 1.

No tau-jet TRFs or data/MC scale factors

No systematic is assigned to the b->mu / b->inclusive efficiency scale factor, derived from MC

This could be as big as 5%, we really don't know

All of these issues (and more) will be studied for p17 certification

p17 Plans

Of course, we've already started working on a NN for p17

We are very lucky that Miruna will be continuing with the NN effort

We desperately need help from others to work on the many other tasks below!

Things to do for an initial p17 certification:

General p17 / CAF infrastructure – largely done, you just heard about it...

Making of data and MC samples – coming along nicely, but we're still late...

Retuning of JLIP inputs – mostly done

Optimization of SVT inputs – needs some minor work

Retraining of NN on MC – Miruna is making rapid progress
(No changes to the variables, network topolgy, or training method are forseen...)

Re-measuring the data performances – Miruna has agreed to do this as well, with help
(with some improvements in method)

Things to do for an optimal / improved p17 certification:

Optimization of taggability requirements – nobody working on it

Study all the new systematic effects from the last slide – nobody, nobody, and... nobody

PV |z|, number of min bias vertices, time dependence

light jet and tau data/MC scale factors, tau TRF

Uncertainty due to b->mu/b->incl tagging efficiency scale factor

Conclusions

The basic strategy: take inputs from previous taggers, combine in NN tuned on MC, apply to data, measure performance: seems to work.

Tim and Miruna have been clever, careful, and thourough, resulting in an algorithm which is an excellent combination of our current b-tagging abilities.

The performance has been measured in data and a few new techniques have been used to do so more accurately. Lower / more realistic systematics.

The result gives us b-tagging which is in line with initial Run IIa predictions.

There are still a significant number of items which were not included in b-ID certification, possibly resulting in underestimated systematic uncertainties in some cases. Caveat emptor!

Progress is already being made on an initial p17 certification of the NN algorithm.

There's significant work left to do to have an optimal p17 algorithm with fully realistic systematic errors.

Andy Haas

2004-07-07T15:59:13