Simulation 2- EPV

[WeiFoo]

Reference

• A simulation of the number of events per variable in logistic regression analysis

[WeiFoo]

EXP1

Description

• KStest(small data set-> small data set): sample original datasets with N in {50,100,150,200} with EPV = 15 (if total defective instances< 15, then use all of the defective instances in that data set) as source,randomly sample corresponding number of instances in each data set as target, then get the matched features.
• Train(small data set): Using the source data set used in KS test as the source to train Logistic Regression learner, N in {50,100,150,200}
• Predict: Using the original data set as the target, to predict defects. Divide each data set into two halves, and use each half as testing data, and repeat 10 times
• Repeat the above procedure 20 times

Flow Chart

Result

Table

Target       | WPDP  | HDP-JC | HDP-Scipy | HDP-Scipy-IQR | N-50  | N-50-IQR | N-100 | N-100-IQR | N-150 | N-150-IQR | N-200 | N-200-IQR
ar3          | 0.574 | 0.823  | 0.842     | 0.0           | 0.758 | 0.086    | 0.808 | 0.066     | 0.823 | 0.065     | 0.825 | 0.119    
skarbonka    | 0.569 | 0.694  | 0.669     | 0.0           | 0.637 | 0.077    | 0.653 | 0.056     | 0.676 | 0.014     | 0.659 | 0.087    
poi-1.5      | 0.707 | 0.701  | 0.712     | 0.0           | 0.661 | 0.064    | 0.631 | 0.068     | 0.645 | 0.067     | 0.603 | 0.094    
arc          | 0.67  | 0.701  | 0.67      | 0.0           | 0.611 | 0.072    | 0.651 | 0.025     | 0.667 | 0.02      | 0.651 | 0.046    
velocity-1.4 | 0.725 | 0.391  | 0.432     | 0.0           | 0.452 | 0.054    | 0.451 | 0.026     | 0.451 | 0.035     | 0.451 | 0.017    
JDT          | 0.795 | 0.767  | 0.702     | 0.0           | 0.702 | 0.061    | 0.69  | 0.039     | 0.7   | 0.033     | 0.692 | 0.037    
LC           | 0.575 | 0.655  | 0.647     | 0.0           | 0.635 | 0.04     | 0.636 | 0.028     | 0.649 | 0.013     | 0.644 | 0.014    
zxing        | 0.605 | 0.65   | 0.624     | 0.0           | 0.622 | 0.026    | 0.623 | 0.013     | 0.623 | 0.007     | 0.622 | 0.009    
xalan-2.4    | 0.755 | 0.751  | 0.59      | 0.0           | 0.651 | 0.048    | 0.659 | 0.069     | 0.634 | 0.07      | 0.638 | 0.068    
redaktor     | 0.744 | 0.537  | 0.508     | 0.0           | 0.522 | 0.026    | 0.519 | 0.032     | 0.51  | 0.026     | 0.516 | 0.01     
cm1          | 0.653 | 0.717  | 0.633     | 0.0           | 0.617 | 0.024    | 0.633 | 0.008     | 0.633 | 0.002     | 0.633 | 0.002    
ar1          | 0.582 | 0.734  | 0.704     | 0.0           | 0.653 | 0.081    | 0.683 | 0.056     | 0.681 | 0.059     | 0.69  | 0.046    
camel-1.0    | 0.55  | 0.639  | 0.655     | 0.0           | 0.624 | 0.063    | 0.617 | 0.043     | 0.633 | 0.037     | 0.638 | 0.038    
ar4          | 0.657 | 0.816  | 0.798     | 0.0           | 0.683 | 0.178    | 0.784 | 0.065     | 0.797 | 0.028     | 0.794 | 0.012    
ar5          | 0.804 | 0.911  | 0.897     | 0.0           | 0.799 | 0.226    | 0.72  | 0.229     | 0.763 | 0.184     | 0.659 | 0.214    
ar6          | 0.654 | 0.64   | 0.626     | 0.0           | 0.578 | 0.066    | 0.534 | 0.034     | 0.552 | 0.078     | 0.555 | 0.069    
safe         | 0.706 | 0.818  | 0.772     | 0.0           | 0.715 | 0.067    | 0.769 | 0.047     | 0.738 | 0.082     | 0.728 | 0.083    
apache       | 0.714 | 0.717  | 0.71      | 0.0           | 0.663 | 0.039    | 0.71  | 0.034     | 0.718 | 0.012     | 0.71  | 0.017    
PDE          | 0.684 | 0.717  | 0.665     | 0.0           | 0.644 | 0.077    | 0.627 | 0.07      | 0.659 | 0.031     | 0.67  | 0.006    
EQ           | 0.583 | 0.783  | 0.716     | 0.0           | 0.742 | 0.045    | 0.744 | 0.015     | 0.738 | 0.013     | 0.737 | 0.03     
mw1          | 0.612 | 0.727  | 0.71      | 0.0           | 0.573 | 0.122    | 0.589 | 0.15      | 0.654 | 0.072     | 0.675 | 0.041    
tomcat       | 0.778 | 0.818  | 0.716     | 0.0           | 0.653 | 0.073    | 0.746 | 0.019     | 0.73  | 0.066     | 0.744 | 0.041    
ML           | 0.734 | 0.692  | 0.635     | 0.0           | 0.635 | 0.045    | 0.633 | 0.023     | 0.637 | 0.028     | 0.629 | 0.012    
pc1          | 0.787 | 0.752  | 0.671     | 0.0           | 0.655 | 0.03     | 0.671 | 0.01      | 0.671 | 0.011     | 0.671 | 0.0      
pc4          | 0.9   | 0.682  | 0.694     | 0.0           | 0.65  | 0.073    | 0.639 | 0.058     | 0.682 | 0.059     | 0.684 | 0.03     
ant-1.3      | 0.609 | 0.835  | 0.726     | 0.0           | 0.689 | 0.032    | 0.708 | 0.037     | 0.707 | 0.034     | 0.71  | 0.026    
pc3          | 0.794 | 0.738  | 0.496     | 0.0           | 0.638 | 0.031    | 0.643 | 0.025     | 0.646 | 0.02      | 0.643 | 0.031    
xerces-1.2   | 0.624 | 0.489  | 0.489     | 0.0           | 0.485 | 0.01     | 0.486 | 0.007     | 0.487 | 0.01      | 0.489 | 0.01  

Comapre HDP-Scipy with N-50

Comapre HDP-Scipy with N-100

Comapre HDP-Scipy with N-150

###Comapre HDP-Scipy with N-200

[WeiFoo]

EXP2: pretty much the same as EXP1, except for EPV = 10

Result

Target       | WPDP  | HDP-JC | HDP-Scipy | HDP-Scipy-IQR | N-50  | N-50-IQR | N-100 | N-100-IQR | N-150 | N-150-IQR | N-200 | N-200-IQR
ar3          | 0.574 | 0.823  | 0.842     | 0.0           | 0.746 | 0.133    | 0.752 | 0.093     | 0.813 | 0.102     | 0.828 | 0.04     
skarbonka    | 0.569 | 0.694  | 0.669     | 0.0           | 0.617 | 0.064    | 0.66  | 0.041     | 0.644 | 0.032     | 0.664 | 0.04     
poi-1.5      | 0.707 | 0.701  | 0.712     | 0.0           | 0.614 | 0.08     | 0.612 | 0.078     | 0.627 | 0.112     | 0.552 | 0.065    
arc          | 0.67  | 0.701  | 0.67      | 0.0           | 0.627 | 0.083    | 0.666 | 0.01      | 0.648 | 0.045     | 0.659 | 0.053    
velocity-1.4 | 0.725 | 0.391  | 0.432     | 0.0           | 0.458 | 0.042    | 0.464 | 0.041     | 0.451 | 0.009     | 0.451 | 0.017    
JDT          | 0.795 | 0.767  | 0.702     | 0.0           | 0.691 | 0.052    | 0.691 | 0.076     | 0.696 | 0.041     | 0.692 | 0.047    
LC           | 0.575 | 0.655  | 0.647     | 0.0           | 0.619 | 0.056    | 0.638 | 0.07      | 0.642 | 0.016     | 0.643 | 0.014    
zxing        | 0.605 | 0.65   | 0.624     | 0.0           | 0.607 | 0.032    | 0.607 | 0.031     | 0.609 | 0.021     | 0.609 | 0.026    
xalan-2.4    | 0.755 | 0.751  | 0.59      | 0.0           | 0.616 | 0.067    | 0.661 | 0.022     | 0.664 | 0.036     | 0.659 | 0.053    
redaktor     | 0.744 | 0.537  | 0.508     | 0.0           | 0.508 | 0.041    | 0.512 | 0.032     | 0.509 | 0.027     | 0.508 | 0.017    
cm1          | 0.653 | 0.717  | 0.633     | 0.0           | 0.619 | 0.023    | 0.627 | 0.015     | 0.626 | 0.012     | 0.633 | 0.003    
ar1          | 0.582 | 0.734  | 0.704     | 0.0           | 0.659 | 0.079    | 0.67  | 0.075     | 0.681 | 0.067     | 0.697 | 0.032    
camel-1.0    | 0.55  | 0.639  | 0.655     | 0.0           | 0.611 | 0.061    | 0.645 | 0.022     | 0.645 | 0.04      | 0.655 | 0.022    
ar4          | 0.657 | 0.816  | 0.798     | 0.0           | 0.667 | 0.139    | 0.776 | 0.117     | 0.769 | 0.048     | 0.78  | 0.024    
ar5          | 0.804 | 0.911  | 0.897     | 0.0           | 0.731 | 0.22     | 0.663 | 0.25      | 0.67  | 0.19      | 0.686 | 0.229    
ar6          | 0.654 | 0.64   | 0.626     | 0.0           | 0.537 | 0.025    | 0.555 | 0.033     | 0.553 | 0.05      | 0.547 | 0.069    
safe         | 0.706 | 0.818  | 0.772     | 0.0           | 0.707 | 0.104    | 0.734 | 0.056     | 0.726 | 0.044     | 0.738 | 0.047    
apache       | 0.714 | 0.717  | 0.71      | 0.0           | 0.663 | 0.02     | 0.71  | 0.027     | 0.708 | 0.004     | 0.696 | 0.021    
PDE          | 0.684 | 0.717  | 0.665     | 0.0           | 0.607 | 0.101    | 0.639 | 0.04      | 0.658 | 0.077     | 0.643 | 0.062    
EQ           | 0.583 | 0.783  | 0.716     | 0.0           | 0.735 | 0.067    | 0.732 | 0.074     | 0.742 | 0.036     | 0.692 | 0.079    
mw1          | 0.612 | 0.727  | 0.71      | 0.0           | 0.629 | 0.114    | 0.684 | 0.05      | 0.665 | 0.086     | 0.609 | 0.106    
tomcat       | 0.778 | 0.818  | 0.716     | 0.0           | 0.635 | 0.119    | 0.717 | 0.056     | 0.726 | 0.082     | 0.734 | 0.063    
ML           | 0.734 | 0.692  | 0.635     | 0.0           | 0.635 | 0.042    | 0.653 | 0.019     | 0.634 | 0.024     | 0.629 | 0.038    
pc1          | 0.787 | 0.752  | 0.671     | 0.0           | 0.656 | 0.028    | 0.669 | 0.014     | 0.671 | 0.0       | 0.671 | 0.0      
pc4          | 0.9   | 0.682  | 0.694     | 0.0           | 0.611 | 0.111    | 0.6   | 0.121     | 0.603 | 0.131     | 0.666 | 0.093    
ant-1.3      | 0.609 | 0.835  | 0.726     | 0.0           | 0.671 | 0.032    | 0.708 | 0.036     | 0.707 | 0.041     | 0.716 | 0.019    
pc3          | 0.794 | 0.738  | 0.496     | 0.0           | 0.631 | 0.051    | 0.657 | 0.066     | 0.655 | 0.047     | 0.677 | 0.078    
xerces-1.2   | 0.624 | 0.489  | 0.489     | 0.0           | 0.489 | 0.009    | 0.483 | 0.007     | 0.485 | 0.008     | 0.489 | 0.002  

Comapre HDP-Scipy with N-50

Comapre HDP-Scipy with N-100

Comapre HDP-Scipy with N-150

Comapre HDP-Scipy with N-200

[timm]

Need the above chart as a pretty diagram for paper

[WeiFoo]

UPDATES: When reviewing EPV paper, I noticed that the above result, especially EPV= 10, actually is the worst case, because I only used at most 10 defective rows in each data set, according to matched features we got as follows, we know that for some EPV data sets, there should be 20,30 or 40 defects in the training data sets if possible as that 1998 paper suggested.

Small source--Small target in KS test, Small source-- Large target in prediction

(dim =1 : number of matched features used in each source-target, 120 is the counts)
dim=1 : 120
dim=2 : 249
dim=3 : 171
dim=4 : 60
dim=5 : 14

Large source--Small target in KS test, Large source-- Large target in prediction

dim=1 : 220
dim=2 : 136
dim=3 : 64
dim=4 : 21

BUT, here is the tricky situation:

• we first want to sample the original data set based on EPV theory to prepare sampled source and target data sets for KS test, where we should know the exact dims to determine number of defective rows in the supervised sampled source data, but this dim information is obtained based those KS tests, which use those sampled data set.

Right now, I have two tentative solutions for this paper we're working on:

• option 1: report the results in above EXP2, where only 10 defects in each data set, which is the worst case in our study, because only 120/ 614 and 220/441 data sets has 1 dimension; or report results in the EXP3, where each data set has at 20 defects if possible, which is EVP = 10, and dim = 2, the median dim in our all experiments.
• option 2: first, we use 10 defects in all supervised sampling data sets( the worst case as mentioned), then we do KS test based on both supervised and unsupervised sampling data sets. Now we know the actual dims for each source-target data set. Then, we use this information to do supervised EPV=10 sample, here, the number of defects in each sampling data set will be determined by the actually real dims we just get from the feedback, then we update the suppressed sampling data set. We could do KS test again, and everything afterwards will be the same as what we discussed. and report this result.

Comments:
intuitively, option 2 will be more accurate. but we need to decide when to stop feedback. also, such feedback system will look complicated, which looks like a drawback.

EXP3:

[WeiFoo]

EXP3: at most 20 defects in each supervised sampling data set.

assume dims = 2 for every data set and EVP=10 as well

Target       | WPDP  | HDP-JC | HDP-Scipy | HDP-Scipy-IQR | N-50  | N-50-IQR | N-100 | N-100-IQR | N-150 | N-150-IQR | N-200 | N-200-IQR
ar3          | 0.574 | 0.823  | 0.842     | 0.0           | 0.708 | 0.21     | 0.821 | 0.029     | 0.825 | 0.048     | 0.82  | 0.15     
skarbonka    | 0.569 | 0.694  | 0.669     | 0.0           | 0.629 | 0.046    | 0.645 | 0.056     | 0.665 | 0.044     | 0.652 | 0.035    
poi-1.5      | 0.707 | 0.701  | 0.712     | 0.0           | 0.615 | 0.101    | 0.645 | 0.034     | 0.645 | 0.05      | 0.639 | 0.029    
arc          | 0.67  | 0.701  | 0.67      | 0.0           | 0.643 | 0.06     | 0.649 | 0.049     | 0.663 | 0.02      | 0.658 | 0.019    
velocity-1.4 | 0.725 | 0.391  | 0.432     | 0.0           | 0.459 | 0.057    | 0.451 | 0.042     | 0.451 | 0.029     | 0.451 | 0.026    
JDT          | 0.795 | 0.767  | 0.702     | 0.0           | 0.696 | 0.051    | 0.702 | 0.045     | 0.691 | 0.063     | 0.681 | 0.037    
LC           | 0.575 | 0.655  | 0.647     | 0.0           | 0.618 | 0.064    | 0.64  | 0.008     | 0.647 | 0.01      | 0.639 | 0.01     
zxing        | 0.605 | 0.65   | 0.624     | 0.0           | 0.61  | 0.036    | 0.623 | 0.014     | 0.627 | 0.008     | 0.621 | 0.021    
xalan-2.4    | 0.755 | 0.751  | 0.59      | 0.0           | 0.628 | 0.118    | 0.65  | 0.061     | 0.645 | 0.051     | 0.649 | 0.055    
redaktor     | 0.744 | 0.537  | 0.508     | 0.0           | 0.525 | 0.032    | 0.524 | 0.027     | 0.511 | 0.012     | 0.508 | 0.015    
cm1          | 0.653 | 0.717  | 0.633     | 0.0           | 0.621 | 0.029    | 0.633 | 0.001     | 0.633 | 0.0       | 0.633 | 0.0      
ar1          | 0.582 | 0.734  | 0.704     | 0.0           | 0.645 | 0.094    | 0.687 | 0.054     | 0.691 | 0.042     | 0.698 | 0.031    
camel-1.0    | 0.55  | 0.639  | 0.655     | 0.0           | 0.605 | 0.047    | 0.617 | 0.066     | 0.653 | 0.025     | 0.647 | 0.037    
ar4          | 0.657 | 0.816  | 0.798     | 0.0           | 0.667 | 0.114    | 0.782 | 0.049     | 0.798 | 0.007     | 0.792 | 0.016    
ar5          | 0.804 | 0.911  | 0.897     | 0.0           | 0.542 | 0.387    | 0.788 | 0.135     | 0.833 | 0.213     | 0.754 | 0.176    
ar6          | 0.654 | 0.64   | 0.626     | 0.0           | 0.548 | 0.038    | 0.537 | 0.024     | 0.546 | 0.043     | 0.546 | 0.051    
safe         | 0.706 | 0.818  | 0.772     | 0.0           | 0.685 | 0.065    | 0.762 | 0.059     | 0.74  | 0.066     | 0.718 | 0.055    
apache       | 0.714 | 0.717  | 0.71      | 0.0           | 0.661 | 0.038    | 0.699 | 0.03      | 0.724 | 0.016     | 0.711 | 0.023    
PDE          | 0.684 | 0.717  | 0.665     | 0.0           | 0.637 | 0.079    | 0.593 | 0.122     | 0.664 | 0.044     | 0.659 | 0.02     
EQ           | 0.583 | 0.783  | 0.716     | 0.0           | 0.748 | 0.07     | 0.738 | 0.027     | 0.738 | 0.057     | 0.734 | 0.04     
mw1          | 0.612 | 0.727  | 0.71      | 0.0           | 0.612 | 0.1      | 0.67  | 0.09      | 0.679 | 0.046     | 0.691 | 0.03     
tomcat       | 0.778 | 0.818  | 0.716     | 0.0           | 0.655 | 0.139    | 0.673 | 0.083     | 0.742 | 0.075     | 0.74  | 0.063    
ML           | 0.734 | 0.692  | 0.635     | 0.0           | 0.621 | 0.033    | 0.626 | 0.016     | 0.626 | 0.021     | 0.625 | 0.022    
pc1          | 0.787 | 0.752  | 0.671     | 0.0           | 0.655 | 0.037    | 0.671 | 0.004     | 0.671 | 0.0       | 0.671 | 0.0      
pc4          | 0.9   | 0.682  | 0.694     | 0.0           | 0.607 | 0.095    | 0.665 | 0.044     | 0.677 | 0.069     | 0.689 | 0.01     
ant-1.3      | 0.609 | 0.835  | 0.726     | 0.0           | 0.675 | 0.073    | 0.698 | 0.049     | 0.715 | 0.048     | 0.715 | 0.027    
pc3          | 0.794 | 0.738  | 0.496     | 0.0           | 0.639 | 0.087    | 0.653 | 0.021     | 0.643 | 0.016     | 0.643 | 0.015    
xerces-1.2   | 0.624 | 0.489  | 0.489     | 0.0           | 0.486 | 0.01     | 0.486 | 0.007     | 0.489 | 0.012     | 0.489 | 0.003   

Comapre HDP-Scipy with N-50

Comapre HDP-Scipy with N-100

Comapre HDP-Scipy with N-150

Comapre HDP-Scipy with N-200

[timm]

enough great results! time to start writing!

[WeiFoo]

Sure, I've started!