docs

txt · Aug 18, 2024 · a04827f · a04827f
1 parent 591f672
commit a04827f
Show file tree

Hide file tree

Showing 6 changed files with 337 additions and 4 deletions.
diff --git a/docs/img/lit.png b/docs/img/lit.png
diff --git a/docs/img/table2.png b/docs/img/table2.png
diff --git a/docs/img/venn2.png b/docs/img/venn2.png
diff --git a/docs/proj0lit.html b/docs/proj0lit.html
@@ -0,0 +1,223 @@
+<!DOCTYPE html>
+<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
+<head>
+  <meta charset="utf-8" />
+  <meta name="generator" content="pandoc" />
+  <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
+  <title>Project0: Motivation, Lit Review, Methods</title>
+  <style>
+    code{white-space: pre-wrap;}
+    span.smallcaps{font-variant: small-caps;}
+    div.columns{display: flex; gap: min(4vw, 1.5em);}
+    div.column{flex: auto; overflow-x: auto;}
+    div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
+    /* The extra [class] is a hack that increases specificity enough to
+       override a similar rule in reveal.js */
+    ul.task-list[class]{list-style: none;}
+    ul.task-list li input[type="checkbox"] {
+      font-size: inherit;
+      width: 0.8em;
+      margin: 0 0.8em 0.2em -1.6em;
+      vertical-align: middle;
+    }
+    /* CSS for syntax highlighting */
+    pre > code.sourceCode { white-space: pre; position: relative; }
+    pre > code.sourceCode > span { line-height: 1.25; }
+    pre > code.sourceCode > span:empty { height: 1.2em; }
+    .sourceCode { overflow: visible; }
+    code.sourceCode > span { color: inherit; text-decoration: inherit; }
+    div.sourceCode { margin: 1em 0; }
+    pre.sourceCode { margin: 0; }
+    @media screen {
+    div.sourceCode { overflow: auto; }
+    }
+    @media print {
+    pre > code.sourceCode { white-space: pre-wrap; }
+    pre > code.sourceCode > span { display: inline-block; text-indent: -5em; padding-left: 5em; }
+    }
+    pre.numberSource code
+      { counter-reset: source-line 0; }
+    pre.numberSource code > span
+      { position: relative; left: -4em; counter-increment: source-line; }
+    pre.numberSource code > span > a:first-child::before
+      { content: counter(source-line);
+        position: relative; left: -1em; text-align: right; vertical-align: baseline;
+        border: none; display: inline-block;
+        -webkit-touch-callout: none; -webkit-user-select: none;
+        -khtml-user-select: none; -moz-user-select: none;
+        -ms-user-select: none; user-select: none;
+        padding: 0 4px; width: 4em;
+        background-color: #ffffff;
+        color: #a0a0a0;
+      }
+    pre.numberSource { margin-left: 3em; border-left: 1px solid #a0a0a0;  padding-left: 4px; }
+    div.sourceCode
+      { color: #1f1c1b; background-color: #ffffff; }
+    @media screen {
+    pre > code.sourceCode > span > a:first-child::before { text-decoration: underline; }
+    }
+    code span { color: #1f1c1b; } /* Normal */
+    code span.al { color: #bf0303; background-color: #f7e6e6; font-weight: bold; } /* Alert */
+    code span.an { color: #ca60ca; } /* Annotation */
+    code span.at { color: #0057ae; } /* Attribute */
+    code span.bn { color: #b08000; } /* BaseN */
+    code span.bu { color: #644a9b; font-weight: bold; } /* BuiltIn */
+    code span.cf { color: #1f1c1b; font-weight: bold; } /* ControlFlow */
+    code span.ch { color: #924c9d; } /* Char */
+    code span.cn { color: #aa5500; } /* Constant */
+    code span.co { color: #898887; } /* Comment */
+    code span.cv { color: #0095ff; } /* CommentVar */
+    code span.do { color: #607880; } /* Documentation */
+    code span.dt { color: #0057ae; } /* DataType */
+    code span.dv { color: #b08000; } /* DecVal */
+    code span.er { color: #bf0303; text-decoration: underline; } /* Error */
+    code span.ex { color: #0095ff; font-weight: bold; } /* Extension */
+    code span.fl { color: #b08000; } /* Float */
+    code span.fu { color: #644a9b; } /* Function */
+    code span.im { color: #ff5500; } /* Import */
+    code span.in { color: #b08000; } /* Information */
+    code span.kw { color: #1f1c1b; font-weight: bold; } /* Keyword */
+    code span.op { color: #1f1c1b; } /* Operator */
+    code span.ot { color: #006e28; } /* Other */
+    code span.pp { color: #006e28; } /* Preprocessor */
+    code span.re { color: #0057ae; background-color: #e0e9f8; } /* RegionMarker */
+    code span.sc { color: #3daee9; } /* SpecialChar */
+    code span.ss { color: #ff5500; } /* SpecialString */
+    code span.st { color: #bf0303; } /* String */
+    code span.va { color: #0057ae; } /* Variable */
+    code span.vs { color: #bf0303; } /* VerbatimString */
+    code span.wa { color: #bf0303; } /* Warning */
+  </style>
+  <link rel="stylesheet" href="style.css" />
+
+  <link rel="icon" type="image/x-icon" href="favicon.ico">
+
+</head>
+<body>
+<div class=wrapper>
+<p>
+csc 591-024, (8290)<br>
+csc 791-024, (8291)<br>
+fall 2024, special topics in computer science<br>
+Tim Menzies, timm@ieee.org, com sci, nc state
+<hr>
+<a href="index.html">home</a>
+:: <a href="timetable.html">timetable</a>
+:: <a href="syllabus.html">syllabus</a>
+:: <a href="groups.html">groups</a> 
+:: <a href="https://moodle-courses2425.wolfware.ncsu.edu/course/view.php?id=4181&bp=s">moodle</a>
+:: <a href="https://github.com/txt/se4ai24/blob/main/LICENSE">license</a>  </p>
+<img src="img/brain.png" align=right width=280
+style="padding: 10px; -webkit-filter: drop-shadow(10px 10px 10px #222); filter: drop-shadow(10px 10px 10px #222); ">
+
+
+<header id="title-block-header">
+<h1 class="title">Project0: Motivation, Lit Review, Methods</h1>
+</header>
+<p>To Deliver: the first 3-4 pages of a paper</p>
+<center>
+<img src="https://ause-journal.github.io/img/venn.png" width=400>
+</center>
+<p>Example: <a
+href="https://arxiv.org/pdf/1705.03697">https://arxiv.org/pdf/1705.03697</a>.</p>
+<ul>
+<li>The “Stanford model” introduction
+<ul>
+<li>para1: Everyone does X</li>
+<li>para2: But X has a problem</li>
+<li>para3: We have this new idea/insight based on Y</li>
+<li>para4: So for the rest of this paper we will try Z</li>
+<li>then, some lists
+<ul>
+<li>resaerch questions (things your check)</li>
+<li>contributions (here, you will have to make stuff up)</li>
+</ul></li>
+</ul></li>
+<li>Related Work</li>
+<li>Descriptions of algorithms, success measures, statistical
+methods</li>
+<li>Your proposed new idea.</li>
+</ul>
+<p>A constant problem in defect prediction is what classifier should be
+applied to build the defect predictors? To address this problem, many
+researchers run ranking studies where performance scores are collected
+from many classifiers executed on many software defect data sets.</p>
+<p>This section assesses those ranking studies. We will say a ranking
+study is “good” if it compares multiple learners using multiple data
+sets and multiple evaluation criteria while at the same time doing
+something to address the data imbalance problem.</p>
+<p>In July 2017, we searched scholar.google.com for the conjunction of
+“software” and “defect prediction” and “OO” and “CK” published in the
+last decade. This returned 231 results. We only selected OO and CK
+keywords since CK metrics are more popular and better than process
+metrics for software defect prediction [ 53]. From that list, we
+selected “highly-cited” papers, which we defined as having more than 10
+citations per year. This reduced our population of papers down to 107.
+After reading the titles and abstracts of those papers, and skimming the
+contents of the potentially interesting papers, we found 22 papers of
+Table 4 that either performed ranking studies (as defined above) or
+studied the effects of class imbalance on defect prediction. In the
+column “evaluated using multiple criteria”, papers scored more than “1”
+if they used multiple performance scores of the kind listed at the end
+of Section 2.2.</p>
+<p>We find that, in those 22 papers from Table 4, numerous classifiers
+have used AUC as the measure to evaluate the software defect predictor
+studies. We also found that majority of papers (from last column of
+Table 4, 6/7=85%) in SE community has used SMOTE to fix the data
+imbalance. This also made us to propose SMOTUNED. As noted in, no single
+classification technique always dominates. That said, Table IX of a
+recent study by Ghotra et al. ranks numerous classifiers using data
+similar to what we use here (i.e., OO JAVA systems described using CK
+metrics). Using their work, we can select a range of classifiers for
+this study ranking from “best” to “worst’: see Table 3.</p>
+<p>The key observation to be made from this survey is that, as shown in
+Figure 2, the overwhelming majority of prior papers in our sample do not
+satisfy our definition of a “good” project (the sole exception is the
+recent Bennin et al. [ 4] which we explore in RQ4). Accordingly, the
+rest of this paper defines and executes a “good” ranking study, with an
+additional unique feature of an auto-tuning version of SMOTE.</p>
+<p>A literaeture review should respect and disrespect the past</p>
+<ul>
+<li>Show the reader you know the past;</li>
+<li>Show the reader, why you need to do something new.</li>
+<li>And there should be some throw to the rest of this paper; e.g.</li>
+</ul>
+<blockquote>
+<p>In summary, prior work suffered from: (1) Some methods only find
+bias, without trying to fix it; (2) Some methods for fixing bias have an
+undesired side effect: leaner performance was degraded. For the rest of
+this paper, we explore a solution that finds root causes of bias, and
+directly implement mitigation of those causes (resulting in less bias
+and better performance than seen in prior work).</p>
+</blockquote>
+<p>Run a query in <a href="https://scholar.google.com">Google
+Scholar</a></p>
+<ul>
+<li>e.g “active learning” “software engineering”</li>
+</ul>
+<p>Sort down the citation counts on the first 10 pages (100 numbers).
+Find the “elbow” (the inflection point is the point furthest from the
+line drawn min to max, e.g. see “X”, below).</p>
+<div class="sourceCode" id="cb1"><pre
+class="sourceCode numberSource python numberLines"><code class="sourceCode python"><span id="cb1-1"><a href="#cb1-1"></a>  <span class="op">|</span> o.</span>
+<span id="cb1-2"><a href="#cb1-2"></a>  <span class="op">|</span>  o  . </span>
+<span id="cb1-3"><a href="#cb1-3"></a>  <span class="op">|</span>        .</span>
+<span id="cb1-4"><a href="#cb1-4"></a>  <span class="op">|</span>   <span class="dv">0</span>       .</span>
+<span id="cb1-5"><a href="#cb1-5"></a>  <span class="op">|</span>             <span class="op">/</span> <span class="st">&#39;  </span></span>
+<span id="cb1-6"><a href="#cb1-6"></a><span class="er">  |      o     /       .</span></span>
+<span id="cb1-7"><a href="#cb1-7"></a>  <span class="op">|</span>           X            .</span>
+<span id="cb1-8"><a href="#cb1-8"></a>  <span class="op">|</span>                 o            .      </span>
+<span id="cb1-9"><a href="#cb1-9"></a>  <span class="op">|</span>                        o     o</span>
+<span id="cb1-10"><a href="#cb1-10"></a>  <span class="op">|</span>                                </span>
+<span id="cb1-11"><a href="#cb1-11"></a>  .<span class="op">-------------------------------------</span></span></code></pre></div>
+<p>Skim all papers above the inflection point. Look for 3 to 6 common
+themes. This takes us to</p>
+<p><img src="img/table2.png"></p>
+<p><img src="img/venn2.png"></p>
+
+
+
+
+</div>
+</body>
+</html>
diff --git a/docs/proj0lit.md b/docs/proj0lit.md
@@ -0,0 +1,110 @@
+% Project0: Motivation,  Lit Review, Methods
+
+To Deliver: the first 3-4 pages of a paper
+
+<center><img src="https://ause-journal.github.io/img/venn.png" width=400></center>
+
+Example: [https://arxiv.org/pdf/1705.03697](https://arxiv.org/pdf/1705.03697).
+
+- The "Stanford model" introduction
+  - para1: Everyone does X
+  - para2: But X has a problem
+  - para3: We have this new idea/insight based on Y
+  - para4: So for the rest of this paper we will try Z
+  - then, some lists
+    - resaerch questions (things your check)
+    - contributions (here, you will have to make stuff up)
+- Related Work
+- Descriptions of algorithms, success measures, statistical methods
+- Your proposed new idea.
+
+
+A constant problem in defect prediction is what classifier should
+be applied to build the defect predictors? To address this problem,
+many researchers run ranking studies where performance scores
+are collected from many classifiers executed on many software
+defect data sets.
+
+This section assesses those ranking studies. We will say a ranking
+study is “good” if it compares multiple learners using multiple data
+sets and multiple evaluation criteria while at the same time doing
+something to address the data imbalance problem.
+
+In July 2017, we searched
+scholar.google.com for
+the conjunction of “software” and “defect prediction” and “OO” and “CK”
+published in the last
+decade. This returned
+231 results. We only selected OO and CK keywords since CK metrics
+are more popular and
+better than process metrics for software defect prediction [ 53]. From that list, we selected
+“highly-cited” papers, which we defined as having more than 10
+citations per year. This reduced our population of papers down
+to 107. After reading the titles and abstracts of those papers, and
+skimming the contents of the potentially interesting papers, we
+found 22 papers of Table 4 that either performed ranking studies
+(as defined above) or studied the effects of class imbalance on defect
+prediction. In the column “evaluated using multiple criteria”, papers
+scored more than “1” if they used multiple performance scores of
+the kind listed at the end of Section 2.2.
+
+We find that, in those 22 papers from Table 4, numerous classifiers have used AUC as the measure to evaluate the software defect
+predictor studies. We also found that majority of papers (from last
+column of Table 4, 6/7=85%) in SE community has used SMOTE to
+fix the data imbalance. This also made us to
+propose SMOTUNED. As noted in, no single classification
+technique always dominates. That said, Table IX of a recent study
+by Ghotra et al.  ranks numerous classifiers using data similar
+to what we use here (i.e., OO JAVA systems described using CK
+metrics). Using their work, we can select a range of classifiers for
+this study ranking from “best” to “worst’: see Table 3.
+
+The key observation to be made from this survey is that, as
+shown in Figure 2, the overwhelming majority of prior papers in
+our sample do not satisfy our definition of a “good” project (the sole
+exception is the recent Bennin et al. [ 4] which we explore in RQ4).
+Accordingly, the rest of this paper defines and executes a “good”
+ranking study, with an additional unique feature of an auto-tuning
+version of SMOTE.
+
+A literaeture review should respect and disrespect the past
+
+- Show the reader you know the past;
+- Show the reader, why you need to do something new.
+- And there should be some  throw to the rest of this paper; e.g.
+
+> In summary, prior work suffered from: (1) Some methods only find
+bias, without trying to fix it; (2) Some methods for fixing bias
+have an undesired side effect: leaner performance was degraded.
+For the rest of this paper, we explore a solution that finds root
+causes of bias, and directly implement mitigation of those causes
+(resulting in less bias and better performance than seen in prior
+work).
+
+
+Run a query in [Google Scholar](https://scholar.google.com)
+
+- e.g "active learning" "software engineering"
+
+Sort down the citation counts on the first 10 pages (100 numbers). Find the "elbow" (the inflection point is the point
+furthest from the line drawn min to max, e.g. see "X", below).
+
+      | o.
+      |  o  . 
+      |        .
+      |   0       .
+      |             / '  
+      |      o     /       .
+      |           X            .
+      |                 o            .      
+      |                        o     o
+      |                                
+      .-------------------------------------
+
+Skim all papers above the inflection point. Look for 3 to 6 common themes. This takes us to
+
+<img src="img/table2.png">
+
+<img src="img/venn2.png">
+
+
diff --git a/docs/syllabus.md b/docs/syllabus.md
@@ -72,9 +72,9 @@ This  policy is absolute and total (i.e. there no, zero, zip, nada, none, appeal
 |----------------|------------|--------|------------|---------------|-------------|
 |homeworks (five)|group       | 5 \* 0 |            | yes           | replaced if you resubmit in a week<br> or replaced if you do final exam |
 |mid-term        |solo        | 25     | mid-term   | yes           | replaced if you do final exam |
-|lit review      |group       | 25     | end-Sept   | yes           | optionally, replaced in the final reports |
-|project report1 |group       | 25     | end-Oct. submit to get feedback on your work, so you can improve the  report2  | yes | optionally, replaced by the final paper|
-|project report2 |group       | 25     | end-Nov    | no            | when you submit, students must must specify if this is to be marked out of 25 of 75 |
+|report0: motivation, related work, methods | lit review      |group       | 25     | end-Sept   | yes           | optionally, replaced in the final reports |
+|report1: results |group    | 25     | end-Oct. submit to get feedback on your work, so you can improve the  report2  | yes | optionally, replaced by the final paper|
+|report2: report1 (improved)  |group       | 25     | end-Nov    | no            | when you submit, students must must specify if this is to be marked out of 25 of 75 |
 |final exam      |solo        | 30     |            | no            | if students take this exam, then it replaces the homework and mid-term grade |            
 
 With the final grades, the following grade scale will be used: 
@@ -91,7 +91,7 @@ Groups must post the homeworks each  week, even if it is incomplete, OR THEY WIL
 
 Otherwise, students are marked -1 for "not on time" or "try again". If you get a "-1" mark and resubmit, they are replaced with a "0".
 
-For the lit reviews and papers, students will lose 2 marks per day for late submissions (weekend = 2 day).
+For the rport, students will lose 2 marks per day for late submissions (weekend = 2 day).
 
 
 ## Policies