0.7.156 -- resolved mess with zsh and bash

README.md

@@ -1,4 +1,4 @@
-# Daylily AWS Ephemeral Cluster Setup (0.7.155)
+# Daylily AWS Ephemeral Cluster Setup (0.7.156)
 _(0.7.155)_
 
 **beta release**
@@ -87,7 +87,6 @@ I will be assessing:
 > Would you like to lend a hand?  [contact me](mailto:john@daylilyinformatics.com)
 
 
-
 # What's It All About?
 
 ## BFAIR: Bioinformatics [FAIR](https://www.go-fair.org/fair-principles/) Principles 
@@ -251,22 +250,25 @@ You may run in any region or AZ you wish to try. This said, the majority of test
 
 
 ## Prerequisites (On Your Local Machine) 
-Local machine development has been exclusively on a mac, using the `zsh` shell (I am moving everything to bash, so there is a mix of bash and zsh at the moment. All `.` and `sourcing` that happens on your local machine expects `zsh`).
+Local machine development has been carried out exclusively on a mac using the `zsh` shell. `bash` should work as well (if you have issues with conda w/mac+bash, confirm that after miniconda install and conda init, the correct `.bashrc` and `.bash_profile` files were updated by `conda init`).
  
 _suggestion: run things in tmux or screen_
 
 Very good odds this will work on any mac and most Linux distros (ubuntu 22.04 are what the cluster nodes run). Windows, I can't say.
 
 ### System Packages
-Install with `brew` or `apt-get` ('):
-- `python3`, tested with `3.12.8`
+Install with `brew` or `apt-get`:
+- `python3`, tested with `3.11.0
 - `git`, tested with `2.46.0`
-- `jq`, tested with `jq-1.7.1`
 - `wget`, tested with `1.25.0`
 - `awscli`, tested with `2.22.4` [AWS CLI docs](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html)
 - `tmux` (optional, but suggested)
 - `emacs` (optional, I guess, but I'm not sure how to live without it)
 
+#### Check if your prereq's meet the min versions required
+```bash
+./bin/check_prereq_sw.sh 
+```
 
 ### AWS CLI Configuration
 #### Opt 2
@@ -323,29 +325,32 @@ git clone https://github.com/Daylily-Informatics/daylily.git  # or, if you have
 cd daylily
 ```
 
-### Miniconda
+### Install Miniconda (homebrew is not advised)
 _tested with conda version **`24.11.1`**_
 
-```bash
 Install with:
 ```bash
-source bin/install_miniconda
+./bin/install_miniconda
 ```
-- This will leave you in a terminal with conda activated, indicated by `(base)` in the terminal prompt.
+- open a new terminal/shell, and conda should be available: `conda -v`.
 
-### DAYCLI Environment
+### Install DAYCLI Environment
 _from `daylily` root dir_
 
 ```bash
-#!/bin/zsh
 
-source bin/init_daycli 
+./bin/init_daycli 
+
+conda activate DAYCLI
 
 # DAYCLI should now be active... did it work?
 colr  'did it work?' 0,100,255 255,100,0
 
 ```
-- This will leave you in a terminal with the conda DAYCLI activated, indicated by `(DAYCLI)` in the terminal prompt.
+
+- You should see:
+ 
+  > ![](docs/images/diw.png)
 
 
 <p valign="middle"><img src="docs/images/000000.png" valign="bottom" ></p>
@@ -377,17 +382,17 @@ _running in a tmux/screen session is advised as the copy may take 1-many hours_
 ```bash
 conda activate DAYCLI
 # help
-bash ./bin/create_daylily_omics_analysis_s3.sh -h
+./bin/create_daylily_omics_analysis_s3.sh -h
 
-AWS_PROFILE=<your_profile>
+export AWS_PROFILE=<your_profile>
 BUCKET_PREFIX=<your_prefix>
 REGION=us-west-2
 
 # dryrun
-bash ./bin/create_daylily_omics_analysis_s3.sh  --disable-warn --region $REGION --profile $AWS_PROFILE --bucket-prefix $BUCKET_PREFIX
+./bin/create_daylily_omics_analysis_s3.sh  --disable-warn --region $REGION --profile $AWS_PROFILE --bucket-prefix $BUCKET_PREFIX
 
 # run for real
-bash ./bin/create_daylily_omics_analysis_s3.sh  --disable-warn --region $REGION --profile $AWS_PROFILE --bucket-prefix $BUCKET_PREFIX --disable-dryrun
+./bin/create_daylily_omics_analysis_s3.sh  --disable-warn --region $REGION --profile $AWS_PROFILE --bucket-prefix $BUCKET_PREFIX --disable-dryrun
 
 ```
 
@@ -406,7 +411,7 @@ _this command will take ~5min to complete, and much longer if you expand to all
 ```bash
 
 conda activate DAYCLI
-AWS_PROFILE=daylily-service
+export AWS_PROFILE=daylily-service
 REGION=us-west-2          
 OUT_TSV=./init_daylily_cluster.tsv
 
@@ -474,14 +479,13 @@ Once you have selected an AZ && have a reference bucket ready in the region this
 
 The following script will check a variety of required resources, attempt to create some if missing and then prompt you to select various options which will all be used to create a new parallel cluster yaml config, which in turn is used to create the cluster via `StackFormation`. [The template yaml file can be checked out here](config/day_cluster/prod_cluster.yaml).
 
-```zsh
-#!/bin/zsh
-AWS_PROFILE=daylily-service
+```bash
+export AWS_PROFILE=daylily-service
 REGION_AZ=us-west-2c
-source bin/daylily-create-ephemeral-cluster --region-az $REGION_AZ --profile $AWS_PROFILE
+./bin/daylily-create-ephemeral-cluster --region-az $REGION_AZ --profile $AWS_PROFILE
 
 # And to bypass the non-critical warnings (which is fine, not all can be resolved )
-source bin/daylily-create-ephemeral-cluster --region-az $REGION_AZ --profile $AWS_PROFILE --pass-on-warn # If you created an inline policy with a name other than daylily-service-cluster-policy, you will need to acknowledge the warning to proceed (assuming the policy permissions were granted other ways)
+./bin/daylily-create-ephemeral-cluster --region-az $REGION_AZ --profile $AWS_PROFILE --pass-on-warn # If you created an inline policy with a name other than daylily-service-cluster-policy, you will need to acknowledge the warning to proceed (assuming the policy permissions were granted other ways)
 
 ```    
 
@@ -558,8 +562,7 @@ Once logged in, as the 'ubuntu' user, run the following commands:
 ### Run Remote Slurm Tests On Headnode
 
 ```bash
-#!/bin/zsh
-source ./bin/daylily-run-ephemeral-cluster-remote-tests $pem_file $region $AWS_PROFILE
+./bin/daylily-run-ephemeral-cluster-remote-tests $pem_file $region $AWS_PROFILE
 ```
 
 A successful test will look like this:
@@ -579,7 +582,6 @@ pcluster list-clusters --region $REGION
 [See the instructions here](#first-time-logging-into-head-node) to confirm the headnode is configured and ready to run the daylily pipeline.
 
 
-
 <p valign="middle"><img src="docs/images/000000.png" valign="bottom" ></p>
 
 # Costs
@@ -777,7 +779,7 @@ ssh -i $pem_file ubuntu@$cluster_ip_address
 ##### Facilitated
 
 ```bash
-AWS_PROFILE=<profile_name>
+export AWS_PROFILE=<profile_name>
 bin/daylily-ssh-into-headnode 
 ```
 
@@ -813,7 +815,7 @@ From your remote terminal that you created the cluster with, run the following c
 ```bash
 conda activate DAYCLI
 
-source ./bin/daylily-cfg-headnode $PATH_TO_PEM $CLUSTER_AWS_REGION $AWS_PROFILE
+./bin/daylily-cfg-headnode $PATH_TO_PEM $CLUSTER_AWS_REGION $AWS_PROFILE
 ```
 
 > If the problem persists, ssh into the headnode, and attempt to run the commands as the ubuntu user which are being attempted by the `daylily-cfg-headnode` script.

bin/calc_daylily_aws_cost_estimates.py

@@ -1,3 +1,5 @@
+#!/bin/env python
+
 from daylib.day_cost_ec2 import AWSGenomicsAnalyzer
 
 def main():

bin/calcuate_spotprice_for_cluster_yaml.py

@@ -1,4 +1,5 @@
 #!/usr/bin/env python
+
 import subprocess
 import statistics
 import argparse

bin/cat_input_files.sh

@@ -1,10 +1,5 @@
 #!/bin/bash
 
-# Ensure compatibility with zsh and bash
-if [ -n "$ZSH_VERSION" ]; then
-  emulate -LR bash  # Ensure bash-like behavior in zsh
-fi
-
 # Display usage information
 usage() {
   echo "Usage: $0 -s SAMPLENAME -p SAMPLE_MATCH_PATTERN -i INPUT_DIR -o OUTPUT_DIR"
@@ -42,7 +37,7 @@ if [[ ! -d "$OUTPUT_DIR" ]]; then
 fi
 
 echo ""
-echo ">>>  >> > THIS IS A SIMPLE UTILITY AND NOT PRODUCTION TESTED FOR YOUR USE CASE < <<  <<<"
+echo ">>>  >> > THIS IS A SIMPLE UTILITY AND NOT PRODUCTION TESTED < <<  <<<"
 echo ""
 sleep 2
 

bin/check_current_spot_market_by_zones.py

@@ -466,6 +466,17 @@ def main():
     if args.profile == "SETME":
         print(f"\n\nERROR:: Your AWS_PROFILE is set to >{os.getenv('AWS_PROFILE')}<, Please set your AWS_PROFILE environment variable and specify the same string with --profile.\n")
         raise SystemExit
+    if os.environ.get('AWS_PROFILE','') == '':
+        os.environ['AWS_PROFILE'] = args.profile
+    elif args.profile != os.environ['AWS_PROFILE']:
+        print(f"\n\nERROR:: Your AWS_PROFILE is set to >{os.getenv('AWS_PROFILE')}<, Please set your AWS_PROFILE environment variable and specify the same string with --profile.\n")
+        raise SystemExit
+    elif args.profile == os.environ['AWS_PROFILE']:
+        print(f"\n\nAWS_PROFILE is set to >{os.getenv('AWS_PROFILE')}< and profile passed is >{args.profile}<, all good.\n")
+    else:
+        print(f"\n\nAWS_PROFILE is set to >{os.getenv('AWS_PROFILE')}< and profile passed is >{args.profile}<, there is a problem.\n")
+        raise SystemExit
+
 
     zones = args.zones.split(',')
     spot_data = collect_spot_prices(instance_types, zones, args.profile)

bin/check_system_pkgs.sh → bin/check_prereq_sw.sh

@@ -2,9 +2,8 @@
 
 # Set required versions
 required_versions=(
-    "python3:3.12.8"
+    "python3:3.11.0"
     "git:2.46.0"
-    "jq:1.7.1"
     "wget:1.25.0"
     "aws:2.22.4"
 )
@@ -14,9 +13,8 @@ pass_on_warn=${pass_on_warn:-0}
 
 # Function to compare versions
 check_version() {
-    local tool=$1
-    local required=$2
-    local current=$3
+    local required=$1
+    local current=$2
 
     if [[ "$(printf '%s\n' "$required" "$current" | sort -V | head -n1)" != "$required" ]]; then
         return 1 # Current version is less than required
@@ -38,12 +36,13 @@ check_tool() {
     fi
 
     # Check version
-    if ! check_version "$tool" "$required_version" "$current_version"; then
+    if ! check_version "$required_version" "$current_version"; then
         echo "Error: $tool version $current_version does not meet the required version $required_version."
         [[ "$pass_on_warn" -eq 1 ]] && return 0 || exit 1
     fi
 
-    echo "$tool version $current_version meets the requirement of $required_version or higher."
+    # Print success message if version is sufficient
+    echo "$tool version $current_version meets the required version $required_version or higher."
 }
 
 # Loop through required tools and check
@@ -58,9 +57,6 @@ for entry in "${required_versions[@]}"; do
         git)
             check_tool "git" "$required_version" "git --version | awk '{print \$3}'"
             ;;
-        jq)
-            check_tool "jq" "$required_version" "jq --version | sed 's/jq-//g'"
-            ;;
         wget)
             check_tool "wget" "$required_version" "wget --version | head -n1 | awk '{print \$3}'"
             ;;

bin/create_budget.sh

@@ -78,9 +78,9 @@ write_or_append_tags_to_s3() {
 
   echo "Writing or appending tags to S3 file: $S3_BUCKET_PATH" url: $S3_BUCKET_URL
   # Check if the file exists in S3
-  if aws s3 ls "$S3_BUCKET_PATH" > /dev/null 2>&1; then
+  if aws s3 ls "$S3_BUCKET_PATH" --profile $AWS_PROFILE > /dev/null 2>&1; then
     echo "File exists. Downloading the existing file."
-    aws s3 cp "$S3_BUCKET_PATH" "$TEMP_LOCAL_FILE" --region "$REGION"
+    aws s3 cp "$S3_BUCKET_PATH" "$TEMP_LOCAL_FILE" --region "$REGION" --profile $AWS_PROFILE
 
     if [[ $? -ne 0 ]]; then
       echo "ERROR: Failed to download existing file from S3."
@@ -97,7 +97,7 @@ write_or_append_tags_to_s3() {
 
   # Upload the updated file back to S3
   echo "Uploading updated file to S3..."
-  aws s3 cp "$TEMP_LOCAL_FILE" "$S3_BUCKET_PATH" --region "$REGION"
+  aws s3 cp "$TEMP_LOCAL_FILE" "$S3_BUCKET_PATH" --region "$REGION" --profile $AWS_PROFILE
 
   if [[ $? -eq 0 ]]; then
     echo "Successfully updated S3 file: $S3_BUCKET_PATH"
@@ -153,12 +153,12 @@ BUDGET_JSON=$(echo "$BUDGET_TEMPLATE" | \
 TEMP_BUDGET_JSON=$(mktemp)
 echo "$BUDGET_JSON" > "$TEMP_BUDGET_JSON"
 
-ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
+ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text --profile $AWS_PROFILE)
 
 # Create the budget using AWS CLI
 echo "Creating budget for project: $PROJECT_NAME with amount: $AMOUNT USD in region: $REGION"
 aws budgets create-budget --account-id "$ACCOUNT_ID" \
-    --budget file://"$TEMP_BUDGET_JSON" --region "$REGION"
+    --budget file://"$TEMP_BUDGET_JSON" --region "$REGION" --profile $AWS_PROFILE
 
 # Check if the budget creation succeeded
 if [[ $? -eq 0 ]]; then
@@ -169,7 +169,7 @@ if [[ $? -eq 0 ]]; then
   IFS=',' read -ra THRESHOLD_ARRAY <<< "$THRESHOLDS"
   for threshold in "${THRESHOLD_ARRAY[@]}"; do
     echo P:$PROJECT_NAME T:$threshold E:$EMAIL R:$REGION A:$AMOUNT AI:$ACCOUNT_ID 
-    aws budgets create-notification --account-id "$ACCOUNT_ID" \
+    aws budgets create-notification --account-id "$ACCOUNT_ID" --profile $AWS_PROFILE \
       --budget-name "$PROJECT_NAME" \
       --notification '{"ComparisonOperator":"GREATER_THAN","NotificationType":"ACTUAL","Threshold":'"$threshold"',"ThresholdType":"PERCENTAGE"}' \
       --region "$REGION"   --subscribers '[{"Address":"'"$EMAIL"'","SubscriptionType":"EMAIL"}]'