CaseMatters.com
Solutions for IP Attorneys & Expert Witnesses

Source Code Tokenizing    Samples
Sample 1

This sample illustrates the output from tokenizing a single file: CPlusCopy.cpp.

With Language Syntax

TokenCount
!1
#include1
&2
(17
)17
*30
+8
-6
/6
01
21
20031
<1
=8
>1
All1
C1
CPlusCopy1
CPlusLib1
CTORCALL_COMPLETEconstructor1
Call1
ConstructorDestructor1
Copy1
Copyright1
Good1
Guys1
If1
Inc1
Return1
Routine1
[2
]2
an1
anything1
are1
array2
bytes1
char4
constructObjects1
constructObjectsvoid1
constructor3
copy3
copychar1
cpp1
destination2
do1
don1
each1
for2
from5
h1
if3
in1
is1
length6
length_t5
long1
number1
numberOfObjects4
objects2
of4
or1
p9
p01
p11
ptr2
reserved1
returnptr1
returnto1
rights1
set1
sizeOfObjects4
sizeoflength_t1
t1
that1
the3
to5
up1
void2
while1
zero1
{6
|0
}6

Language Syntax Removed

TokenCount
01
21
20031
All1
C1
CPlusCopy1
CPlusLib1
CTORCALL_COMPLETEconstructor1
Call1
ConstructorDestructor1
Copy1
Copyright1
Good1
Guys1
If1
Inc1
Return1
Routine1
an1
anything1
are1
array2
bytes1
constructObjects1
constructObjectsvoid1
constructor3
copy3
copychar1
cpp1
destination2
don1
each1
from5
h1
in1
is1
length6
length_t5
number1
numberOfObjects4
objects2
of4
or1
p9
p01
p11
ptr2
reserved1
returnptr1
returnto1
rights1
set1
sizeOfObjects4
sizeoflength_t1
t1
that1
the3
to5
up1
zero1

Sample 2

This sample illustrates the output from tokenizing a single file: objectUtils.cpp.

With Language Syntax

TokenCount
!1
#include1
&2
(17
)17
*29
+6
-6
/6
01
21
20031
<1
=8
>1
All1
Bad1
CPlusLib1
CTORCALL_COMPLETEconstructor1
Call1
ConstructorDestructor1
Copyright1
Guys1
If1
Inc1
Return1
Routines1
Utiltity1
[2
]2
anything1
array2
bytes1
callConstructors1
callConstructorsvoid1
char4
construct1
constructor3
cpp1
dest5
destination2
do1
don1
duplicate3
duplicatechar1
each1
for2
h1
howMany3
if3
in1
is1
length6
length_t5
number1
object1
objectSize3
objectUtils1
objects1
of2
or1
p9
p01
p11
ptr2
reserved1
returndest1
returnptr1
rights1
sizeoflength_t1
source5
t1
the4
void2
while1
zero1
{6
|0
}6

Language Syntax Removed

TokenCount
01
21
20031
All1
Bad1
CPlusLib1
CTORCALL_COMPLETEconstructor1
Call1
ConstructorDestructor1
Copyright1
Guys1
If1
Inc1
Return1
Routines1
Utiltity1
anything1
array2
bytes1
callConstructors1
callConstructorsvoid1
construct1
constructor3
cpp1
dest5
destination2
don1
duplicate3
duplicatechar1
each1
h1
howMany3
in1
is1
length6
length_t5
number1
object1
objectSize3
objectUtils1
objects1
of2
or1
p9
p01
p11
ptr2
reserved1
returndest1
returnptr1
rights1
sizeoflength_t1
source5
t1
the4
zero1

Sample 3

This sample illustrates the output from tokenizing both files listed above (CPlusCopy.cpp and objectUtils.cpp) as if they were included in the same code base.

This table illustrates the tokens and token count for all (both in this case) files combined.

This table illustrates the tokens and token count qualified by the file extension for all (both in this case) files combined. Please note that in this case both files have the same extension so both tables will contain the same number of tokens with identical counts.

TokenCount
04
12
22
20032
All2
Bad1
C1
CPlusCopy1
CPlusLib2
CTORCALL_COMPLETE2
Call2
ConstructorDestructor2
Copy1
Copyright2
Good1
Guys2
If2
Inc2
Return2
Routine1
Routines1
Utiltity1
an1
anything2
are1
array4
bytes2
callConstructors2
char10
construct1
constructObjects2
constructor8
copy4
cpp2
dest6
destination4
do2
don2
duplicate4
each2
for4
from5
h2
howMany3
if6
in2
include2
is2
length12
length_t12
long1
number2
numberOfObjects4
object1
objectSize3
objectUtils1
objects3
of6
or2
p22
ptr6
reserved2
return4
rights2
set1
sizeOfObjects4
sizeof2
source5
t2
that1
the7
to6
up1
void6
while2
zero2
TokenCountExtension
04.CPP
12.CPP
22.CPP
20032.CPP
All2.CPP
Bad1.CPP
C1.CPP
CPlusCopy1.CPP
CPlusLib2.CPP
CTORCALL_COMPLETE2.CPP
Call2.CPP
ConstructorDestructor2.CPP
Copy1.CPP
Copyright2.CPP
Good1.CPP
Guys2.CPP
If2.CPP
Inc2.CPP
Return2.CPP
Routine1.CPP
Routines1.CPP
Utiltity1.CPP
an1.CPP
anything2.CPP
are1.CPP
array4.CPP
bytes2.CPP
callConstructors2.CPP
char10.CPP
construct1.CPP
constructObjects2.CPP
constructor8.CPP
copy4.CPP
cpp2.CPP
dest6.CPP
destination4.CPP
do2.CPP
don2.CPP
duplicate4.CPP
each2.CPP
for4.CPP
from5.CPP
h2.CPP
howMany3.CPP
if6.CPP
in2.CPP
include2.CPP
is2.CPP
length12.CPP
length_t12.CPP
long1.CPP
number2.CPP
numberOfObjects4.CPP
object1.CPP
objectSize3.CPP
objectUtils1.CPP
objects3.CPP
of6.CPP
or2.CPP
p22.CPP
ptr6.CPP
reserved2.CPP
return4.CPP
rights2.CPP
set1.CPP
sizeOfObjects4.CPP
sizeof2.CPP
source5.CPP
t2.CPP
that1.CPP
the7.CPP
to6.CPP
up1.CPP
void6.CPP
while2.CPP
zero2.CPP
Copyright 1999-, CaseMatters.com. All rights reserved. US Patents 8,433,708; 8,744,135; and patent pending.