From 673a624d58adc15566d31646eb4dee073fb5994b Mon Sep 17 00:00:00 2001 From: timm Date: Mon, 9 Sep 2024 22:15:35 -0400 Subject: [PATCH] doco --- docs/hw3.html | 186 ++++++++++++++++++++++++++++++++++++++++++++-- docs/hw3.md | 137 +++++++++++++++++++++++++++++++++- docs/timetable.md | 2 +- 3 files changed, 314 insertions(+), 11 deletions(-) diff --git a/docs/hw3.html b/docs/hw3.html index 3a65402..2ddccab 100644 --- a/docs/hw3.html +++ b/docs/hw3.html @@ -20,6 +20,73 @@ margin: 0 0.8em 0.2em -1.6em; vertical-align: middle; } + /* CSS for syntax highlighting */ + pre > code.sourceCode { white-space: pre; position: relative; } + pre > code.sourceCode > span { line-height: 1.25; } + pre > code.sourceCode > span:empty { height: 1.2em; } + .sourceCode { overflow: visible; } + code.sourceCode > span { color: inherit; text-decoration: inherit; } + div.sourceCode { margin: 1em 0; } + pre.sourceCode { margin: 0; } + @media screen { + div.sourceCode { overflow: auto; } + } + @media print { + pre > code.sourceCode { white-space: pre-wrap; } + pre > code.sourceCode > span { display: inline-block; text-indent: -5em; padding-left: 5em; } + } + pre.numberSource code + { counter-reset: source-line 0; } + pre.numberSource code > span + { position: relative; left: -4em; counter-increment: source-line; } + pre.numberSource code > span > a:first-child::before + { content: counter(source-line); + position: relative; left: -1em; text-align: right; vertical-align: baseline; + border: none; display: inline-block; + -webkit-touch-callout: none; -webkit-user-select: none; + -khtml-user-select: none; -moz-user-select: none; + -ms-user-select: none; user-select: none; + padding: 0 4px; width: 4em; + background-color: #ffffff; + color: #a0a0a0; + } + pre.numberSource { margin-left: 3em; border-left: 1px solid #a0a0a0; padding-left: 4px; } + div.sourceCode + { color: #1f1c1b; background-color: #ffffff; } + @media screen { + pre > code.sourceCode > span > a:first-child::before { text-decoration: underline; } + } + code span { color: #1f1c1b; } /* Normal */ + code span.al { color: #bf0303; background-color: #f7e6e6; font-weight: bold; } /* Alert */ + code span.an { color: #ca60ca; } /* Annotation */ + code span.at { color: #0057ae; } /* Attribute */ + code span.bn { color: #b08000; } /* BaseN */ + code span.bu { color: #644a9b; font-weight: bold; } /* BuiltIn */ + code span.cf { color: #1f1c1b; font-weight: bold; } /* ControlFlow */ + code span.ch { color: #924c9d; } /* Char */ + code span.cn { color: #aa5500; } /* Constant */ + code span.co { color: #898887; } /* Comment */ + code span.cv { color: #0095ff; } /* CommentVar */ + code span.do { color: #607880; } /* Documentation */ + code span.dt { color: #0057ae; } /* DataType */ + code span.dv { color: #b08000; } /* DecVal */ + code span.er { color: #bf0303; text-decoration: underline; } /* Error */ + code span.ex { color: #0095ff; font-weight: bold; } /* Extension */ + code span.fl { color: #b08000; } /* Float */ + code span.fu { color: #644a9b; } /* Function */ + code span.im { color: #ff5500; } /* Import */ + code span.in { color: #b08000; } /* Information */ + code span.kw { color: #1f1c1b; font-weight: bold; } /* Keyword */ + code span.op { color: #1f1c1b; } /* Operator */ + code span.ot { color: #006e28; } /* Other */ + code span.pp { color: #006e28; } /* Preprocessor */ + code span.re { color: #0057ae; background-color: #e0e9f8; } /* RegionMarker */ + code span.sc { color: #3daee9; } /* SpecialChar */ + code span.ss { color: #ff5500; } /* SpecialString */ + code span.st { color: #bf0303; } /* String */ + code span.va { color: #0057ae; } /* Variable */ + code span.vs { color: #bf0303; } /* VerbatimString */ + code span.wa { color: #bf0303; } /* Warning */ @@ -62,7 +129,7 @@

HW3 : Testing an Research Hypotheses

independent values

Run the following twice (once for the low dimensional data sets and once for the other). See what conclusions are found.

-

Noe that the following is quickly written pseudo code. May have +

Now that the following is quickly written pseudo code. May have mistakes. You fix them. Have fun!

-

Experimental Scripts -Must be “Commissioned”

+

Experiments, +part1: comission for one data set

+

First you must write an experiment function

+ +

That experiment file needs to loop through some options and write to +a list of SOME instances (one per option). Note that one treatment must +be asIs that runs over the data and collects all the +distances to heaven. This is the baseline result against which +everything else will be compared.

+
d = DATA().adds(csv(the.train))
+b4 = [d.chebyshev(row) for row in d.rows]
+somes = [stats.SOME(b4,f"asIs,{len(d.rows)}")]
+

Then you need to loop through some options to collect some numbers +into a list. This gets added to SOME with a name that +identiges the treatment. In the following ,see some +=:

+
rnd = lambda z: z
+scoring_policies = [
+  ('exploit', lambda B, R,: B - R),
+  ('explore', lambda B, R :  (exp(B) + exp(R))/ (1E-30 + abs(exp(B) - exp(R))))]
+
+for what,how in scoring_policies:
+  for the.Last in [0,20, 30, 40]:
+    for the.branch in [False, True]:
+      start = time()
+      result = []
+      runs = 0
+      for _ in range(repeats):
+         tmp=d.shuffle().activeLearning(score=how)
+         runs += len(tmp)
+         result += [rnd(d.chebyshev(tmp[0]))]
+
+      pre=f"{what}/b={the.branch}" if the.Last >0 else "rrp"
+      tag = f"{pre},{int(runs/repeats)}"
+      print(tag, f": {(time() - start) /repeats:.2f} secs")
+      somes +=   [stats.SOME(result,    tag)]
+
pre=f"{what}/b={the.branch}" if the.Last >0 else "rrp"
+tag = f"{pre},{int(runs/repeats)}"
+somes +=   [stats.SOME(result,    tag)]
+

When all the looping is done, you have to print the result:

+
stats.report(somes, 0.01)
+

(In the above, “0.01” controls the size of the smallest difference we +can print in the output.)

+

Experimental Scripts +Must be “Commissioned”

The scripts you write for these experiments are always quirky and complex. It is very easy to make mistakes and have to throw out days of compute. So test experimental scripts have to be commissioned.

@@ -123,9 +239,67 @@

Experimental Scripts statistical validity?
  • Does d.shuffle() really jiggle the order of the data?
    • -

    How to run a long experiment

    -

    How to summarize a long -experiments

    +

    Experiments, part2: +run it over many datasets

    +

    Makefile has a tool for generating a todo file for +running multiple experiments

    +

    Lets say your experument can be called from the command line +-e branch.

    +

    For example:

    +
    make Act=branch actb4 # this outputs
+
+mkdir -p .../tmp/branch
+rm .../tmp/branch/*
+python3 .../ezr.py  -D -t .../Apache_AllMeasurements.csv -e branch | tee .../tmp/branch/Apache_AllMeasurements.csv &
+python3 .../ezr.py  -D -t .../HSMGP_num.csv -e branch              | tee .../tmp/branch/HSMGP_num.csv &
+python3 .../ezr.py  -D -t ../SQL_AllMeasurements.csv -e branch     | tee .../tmp/branch/SQL_AllMeasurements.csv &
+...
    +

    You can catch the output of actb4 into a +todo file:

    +
    make Act=branch actb4 > ~/tmp/branch.sh
    +

    See here for a full example branch.sh.

    +

    You can now run all this to generate lots of output files. See here for a sample.

    +

    All those outputs can be summarizes with the rq.sh +script:

    +
    cd ~/tmp/branch ; bash ~/gits/timm/ezr/etc/rq.sh 
+
+RANK                0            1            2             3            
+exploi/b=True      92            4            4                         
+explore/b=True     80           16            2             2          
+exploi/b=False     71           24            4                       
+explore/b=False    59           27           10             4        
+rrp                10           16           14            37       
+asIs                2            8           12            12      
+#
+#EVALS
+RANK                      0            1            2             3 
+exploi/b=True      29 (  8)     35 (  0)     20 (  0)       0 (  0)
+explore/b=True     29 (  8)     29 (  0)     20 (  0)      30 (  0)   
+exploi/b=False     28 (  8)     26 (  4)     20 (  0)       0 (  0)  
+explore/b=False    28 (  4)     31 (  8)     30 (  0)      30 (  0) 
+rrp                 4 (  0)      4 (  0)      4 (  0)       5 (  0)
+asIs              3840 (  0)   6581 (  0)   12835 (  0)   16307 (  0)  
+#
+#DELTAS
+RANK                      0            1            2             3   
+exploi/b=True      73 ( 23)     48 (  0)     41 (  0)       0 (  0)  
+explore/b=True     74 ( 21)     61 (  0)     24 (  0)      24 (  0)   
+exploi/b=False     73 ( 26)     59 ( 19)     46 (  0)       0 (  0)  
+explore/b=False    71 ( 22)     58 ( 15)     52 (  0)      54 (  0) 
+rrp                61 (  0)     50 (  0)     22 (  0)      22 ( 11)
    +

    RANKS: how often treatments are in rank 0,1,2,…

    +

    EVALS: is the budgets used to achieve those ranks.

    +

    DELTAS: are the 100*(asIs - now)/asIs change.

    +

    If rq.sh fails:

    +

    What to hand in

    Submit a url link to moodle with a repo link that has a /hw3 subdirectory

    diff --git a/docs/hw3.md b/docs/hw3.md index 4fa9e90..fe5ccdc 100644 --- a/docs/hw3.md +++ b/docs/hw3.md @@ -12,7 +12,7 @@ Use the extension from hw1 to find which data sets have less than 6 independent Run the following twice (once for the low dimensional data sets and once for the other). See what conclusions are found. -Noe that the following is quickly written pseudo code. May have mistakes. You fix them. Have fun! +Now that the following is quickly written pseudo code. May have mistakes. You fix them. Have fun! - for N in (20,30,40,50) - d = DATA.new().csv(data) @@ -40,7 +40,67 @@ Noe that the following is quickly written pseudo code. May have mistakes. You fi - hint: d.clone().adds(some).chebyshevs().rows - return the rows of some, sorted on chebyshev. -## Experimental Scripts Must be “Commissioned” +## Experiments, part1: comission for one data set + +First you must write an experiment function + +- Best not to edit ezr.py +- Better to write an extesions, like in hw1, where you write a seperate file + that includes my code. + +That experiment file needs to loop through some options and write +to a list of SOME instances (one per option). Note that one treatment +must be `asIs` that runs over the data and collects all the distances +to heaven. This is the baseline result against which everything +else will be compared. + +```python +d = DATA().adds(csv(the.train)) +b4 = [d.chebyshev(row) for row in d.rows] +somes = [stats.SOME(b4,f"asIs,{len(d.rows)}")] +``` +Then you need to loop through some options to collect +some numbers into a list. +This gets added to `SOME` +with a name that identiges the treatment. In the following ,see `some +=`: + + +```python +rnd = lambda z: z +scoring_policies = [ + ('exploit', lambda B, R,: B - R), + ('explore', lambda B, R : (exp(B) + exp(R))/ (1E-30 + abs(exp(B) - exp(R))))] + +for what,how in scoring_policies: + for the.Last in [0,20, 30, 40]: + for the.branch in [False, True]: + start = time() + result = [] + runs = 0 + for _ in range(repeats): + tmp=d.shuffle().activeLearning(score=how) + runs += len(tmp) + result += [rnd(d.chebyshev(tmp[0]))] + + pre=f"{what}/b={the.branch}" if the.Last >0 else "rrp" + tag = f"{pre},{int(runs/repeats)}" + print(tag, f": {(time() - start) /repeats:.2f} secs") + somes += [stats.SOME(result, tag)] +``` + +```python +pre=f"{what}/b={the.branch}" if the.Last >0 else "rrp" +tag = f"{pre},{int(runs/repeats)}" +somes += [stats.SOME(result, tag)] +``` +When all the looping is done, you have to print the result: +``` +stats.report(somes, 0.01) +``` +(In the above, "0.01" controls the size of the smallest difference +we can print in the output.) + +### Experimental Scripts Must be “Commissioned” The scripts you write for these experiments are always quirky and complex. It is very easy to make mistakes and have to throw out days of compute. So test experimental scripts have to be commissioned. @@ -54,9 +114,78 @@ Also: add in tests to check that the expected stuff is actually happening. e.g. - Does you code really run some experimental treatment 20 times for statistical validity? - Does d.shuffle() really jiggle the order of the data? -## How to run a long experiment +## Experiments, part2: run it over many datasets + +`Makefile` has a tool for generating a todo file for running multiple experiments + +Lets say your experument can be called from the command line `-e branch`. + +For example: + +```bash +make Act=branch actb4 # this outputs + +mkdir -p .../tmp/branch +rm .../tmp/branch/* +python3 .../ezr.py -D -t .../Apache_AllMeasurements.csv -e branch | tee .../tmp/branch/Apache_AllMeasurements.csv & +python3 .../ezr.py -D -t .../HSMGP_num.csv -e branch | tee .../tmp/branch/HSMGP_num.csv & +python3 .../ezr.py -D -t ../SQL_AllMeasurements.csv -e branch | tee .../tmp/branch/SQL_AllMeasurements.csv & +... +``` + +You can catch the output of `actb4` into a `todo` file: + +```bash +make Act=branch actb4 > ~/tmp/branch.sh +``` +See here for a full example [branch.sh](branch.sh). + +You can now run all this to generate lots of output files. See [here](branch.zip) for a sample. + +All those outputs can be summarizes with the [rq.sh](https://github.com/timm/ezr/blob/main/etc/rq.sh) script: + + +``` +cd ~/tmp/branch ; bash ~/gits/timm/ezr/etc/rq.sh + +RANK 0 1 2 3 +exploi/b=True 92 4 4 +explore/b=True 80 16 2 2 +exploi/b=False 71 24 4 +explore/b=False 59 27 10 4 +rrp 10 16 14 37 +asIs 2 8 12 12 +# +#EVALS +RANK 0 1 2 3 +exploi/b=True 29 ( 8) 35 ( 0) 20 ( 0) 0 ( 0) +explore/b=True 29 ( 8) 29 ( 0) 20 ( 0) 30 ( 0) +exploi/b=False 28 ( 8) 26 ( 4) 20 ( 0) 0 ( 0) +explore/b=False 28 ( 4) 31 ( 8) 30 ( 0) 30 ( 0) +rrp 4 ( 0) 4 ( 0) 4 ( 0) 5 ( 0) +asIs 3840 ( 0) 6581 ( 0) 12835 ( 0) 16307 ( 0) +# +#DELTAS +RANK 0 1 2 3 +exploi/b=True 73 ( 23) 48 ( 0) 41 ( 0) 0 ( 0) +explore/b=True 74 ( 21) 61 ( 0) 24 ( 0) 24 ( 0) +exploi/b=False 73 ( 26) 59 ( 19) 46 ( 0) 0 ( 0) +explore/b=False 71 ( 22) 58 ( 15) 52 ( 0) 54 ( 0) +rrp 61 ( 0) 50 ( 0) 22 ( 0) 22 ( 11) +``` + +RANKS: how often treatments are in rank 0,1,2,... + +EVALS: is the budgets used to achieve those ranks. + +DELTAS: are the `100*(asIs - now)/asIs` change. + +### If rq.sh fails: + + +- Your experiment script does not mention an `asIs` output +- When you name your treatments, you add in more than one "," -## How to summarize a long experiments ## What to hand in diff --git a/docs/timetable.md b/docs/timetable.md index f08cf29..58980fb 100644 --- a/docs/timetable.md +++ b/docs/timetable.md @@ -7,7 +7,7 @@ |:-----|:----------------|---------| |sept3 | [Tut1](hw1.html) | | |Sep10 | [Tut2](hw2.html) | | -|Sep19 | Tut3 | | +|Sep19 | [Tut3](hw3.html) | | |Sep17 | Wellness day | no class| |Sep26 | Tut4 | | |Oct1 | Lit review due | [About lit Review](litreview.html) |