Merge branch 'master' into 'Elapsed-AllocCPUs-ReqMem_ClusterAnalysis'

# Conflicts: # Cluster_Analysis.ipynb

Merge branch 'master' into 'Elapsed-AllocCPUs-ReqMem_ClusterAnalysis'
# Conflicts: # Cluster_Analysis.ipynb
e4fbdb8b · Ryan Randles Jones · d16fd96e · aaf31d04 · e4fbdb8b · e4fbdb8b
Commit e4fbdb8b authored 4 years ago by Ryan Randles Jones
--- a/Cluster_Analysis.ipynb
+++ b/Cluster_Analysis.ipynb
@@ -11,7 +11,7 @@
   "cell_type": "markdown",
   "metadata": {},
   "source": [
-    "This notebook is for the clustering analysis of ReqMemCPU, AllocCPUS, and Elapsed.\n",
+    "This notebook is for clustering analysis of ReqMemCPU, AllocCPUS, and Elapsed.\n",
    "ReqMemCPU is the amount of RAM in gigs for each job as requested by the user.\n",
    "AllocCPUS is the amount of cores that were used for each job.\n",
    "Elapsed is the amount of time, in hours, that job took to run."
@@ -81,8 +81,8 @@
   "metadata": {},
   "outputs": [],
   "source": [
-    "start_date = '2021-01-01'\n",
+    "start_date = 'yyyy-mm-dd'\n",
-    "end_date = '2021-01-08'"
+    "end_date = 'yyyy-mm-dd'"
   ]
  },
  {

 %% Cell type:markdown id: tags:
 # Purpose
 %% Cell type:markdown id: tags:
-This notebook is for the clustering analysis of ReqMemCPU, AllocCPUS, and Elapsed.
+This notebook is for clustering analysis of ReqMemCPU, AllocCPUS, and Elapsed.
 ReqMemCPU is the amount of RAM in gigs for each job as requested by the user.
 AllocCPUS is the amount of cores that were used for each job.
 Elapsed is the amount of time, in hours, that job took to run.
 %% Cell type:markdown id: tags:
 # Assumptions and Restrictions
 %% Cell type:markdown id: tags:
 Based on extensiive data and clustering exploration, this Notebook is set to graph up to 4 clusters (n_clusters = 4 in kmeans). In order to raise the number of clusters, more code will have to be added to add more 2d histograms of those extra cluster groups. And in order to lower the number of clusters, the code would have to be modified to expect fewer than 4 clusters as an input.
 %% Cell type:markdown id: tags:
 # Data Setup Options
 %% Cell type:markdown id: tags:
 There are 6 decisions to make in the set up of the data.
 Date Range: Choose a start date and an end date of data that you want to see cluster analysis of.
            The format is yyyy-mm-dd
 Bracketing Values: Choose a minimum and maximum value for ReqMemCPU, AllocCPUS, and Elapsed.
                   These values will allow you to "zoom in" or "zoom out" on your data.
 1. Upper/LowerGB - min/max ReqMemCPU: most of the data lies between 1 and 150 gigs.
    Most of the ReqMemCPU above 150 are outliers
 2. Upper/LowerAllocCPU - min/max AllocCPUS: most of the data lies between 1 and 260 cores.
    Most of the AllocCPUS above 260 are outliers
 3. Upper/LowerElapsed - min/max Elapsed: 150.02 hours is the highest Elapsed goes to.
 Data Normalization: There are three choices for normalization of the data - 'none', '0-1', or 'log'
 1. 'none' - no data normalization. Data is clustered and graphed as is.
 2. '0-1'- all data in the date range and bracketing ranges chosen will be scaled to have values between 0 and 1.
    This would be useful if your bracketing ranges differ greatly from each other.
 3. 'log' - all data in the date range and bracketing ranges chosen will be scaled to have log values.
    This would be useful if your bracketing ranges create data that is very large and would be easier to
    visualize with log values.
 2D Histogram X and Y Axes: This will set a min and max for the x and y axes in the 2D histograms of each of the four clusters. All the x and y axes are the same across the 2d histograms. This allows the user to "zoom" in or out of the data.
 %% Cell type:markdown id: tags:
 ## Date Range
 %% Cell type:code id: tags:
 ``` 
-start_date = '2021-01-01'
+start_date = 'yyyy-mm-dd'
-end_date = '2021-01-08'
+end_date = 'yyyy-mm-dd'
 ```
 %% Cell type:markdown id: tags:
 ## Bracketing Values
 %% Cell type:code id: tags:
 ``` 
 # sets min and max parameters for ReqMemCPU - user requested
 LowerlimitGB = 0
 UpperlimitGB = 150
 ```
 %% Cell type:code id: tags:
 ``` 
 # sets min and max parameters for AllocCPUS - allocated by slurm
 LowerlimitAllocCPU = 0
 UpperlimitAllocCPU = 260
 ```
 %% Cell type:code id: tags:
 ``` 
 # sets min and max parameters for Elapsed
 LowerlimitElapsed = 0
 UpperlimitElapsed = 150.02 # = 6.25 days
 ```
 %% Cell type:markdown id: tags:
 ## Data Normalization
 %% Cell type:code id: tags:
 ``` 
 # Enter 'none', '0-1', or 'log' as a choice for data nomralization
 Data_Normalization_Choice = 'none'
 ```
 %% Cell type:markdown id: tags:
 # 2D Histogram X and Y Axes
 %% Cell type:code id: tags:
 ``` 
 xaxis_min = 0
 xaxis_max = 80
 yaxis_min = 0
 yaxis_max = 20
 ```
 %% Cell type:markdown id: tags:
 # Imports
 %% Cell type:code id: tags:
 ``` 
 # must run
 import sqlite3
 import slurm2sql
 import pandas as pd
 import matplotlib.pyplot as plt
 %matplotlib inline
 import seaborn as sns
 import seaborn as sb
 import plotly.express as px
 import matplotlib.ticker as ticker
 import numpy as np
 from mpl_toolkits.mplot3d import Axes3D
 import os
 from RC_styles import rc_styles as style
 from sklearn.cluster import KMeans
 ```
 %% Cell type:markdown id: tags:
 # Database Creation
 %% Cell type:code id: tags:
 ``` 
 #connecting to database
 db = sqlite3.connect('cluster_analysis_from_'+str(start_date)+'to'+str(end_date)+'.db')
 ```
 %% Cell type:code id: tags:
 ``` 
 #creating a database based on the start date
 slurm2sql.slurm2sql(db, ['-S',start_date, '-E', end_date,'-X', '-a']) #-X is allocations, -a is all users
 ```
 %% Cell type:code id: tags:
 ``` 
 df = pd.read_sql('SELECT * FROM slurm', db)
 ```
 %% Cell type:code id: tags:
 ``` 
 #Deleting the database
 os.remove('cluster_analysis_from_'+str(start_date)+'to'+str(end_date)+'.db')
 os.remove('cluster_analysis_from_'+str(start_date)+'to'+str(end_date)+'.db-shm')
 os.remove('cluster_analysis_from_'+str(start_date)+'to'+str(end_date)+'.db-wal')
 ```
 %% Cell type:markdown id: tags:
 # Dataset Creation
 %% Cell type:code id: tags:
 ``` 
 # df_1 is dataframe of all completed jobs
 df_1 = df[df.State.str.contains('COMPLETED')]
 df_1.head(20)
 ```
 %% Cell type:code id: tags:
 ``` 
 # dataset of needed columns for all graphs below
 df_completed = df_1.loc[:,['ReqMemCPU', 'Elapsed', 'AllocCPUS']]
 #df_1.head(5)
 ```
 %% Cell type:code id: tags:
 ``` 
 # converts units in ReqMemCPU column from bytes to gigs and rounds up to nearest whole number
 df_completed['ReqMemCPU'] = df_completed['ReqMemCPU'].div(1024**3).apply(np.ceil).apply(int)
 #df_completed.head()
 ```
 %% Cell type:code id: tags:
 ``` 
 # converts Elapsed time to hours (from seconds) and rounds up to nearest 2 decimal places
 df_completed['Elapsed'] = df_completed['Elapsed'].div(3600).round(2)
 ```
 %% Cell type:code id: tags:
 ``` 
 # creates dataset of ReqMemCPU, Elapsed, and AllocCPUS for completed jobs using the min and max parameters created above
 df_clustering = df_completed[(df_completed['ReqMemCPU'] <= UpperlimitGB) &
                       (df_completed['ReqMemCPU'] >= LowerlimitGB) &
                       (df_completed['AllocCPUS'] <= UpperlimitAllocCPU) &
                       (df_completed['AllocCPUS'] >= LowerlimitAllocCPU)
                       &
                       (df_completed['Elapsed'] <= UpperlimitElapsed) &
                       (df_completed['Elapsed'] >= LowerlimitElapsed)]
 ```
 %% Cell type:markdown id: tags:
 # Elbow Method to Determine Number of Clusters
 %% Cell type:code id: tags:
 ``` 
 # sets up info for plotting the optimal number of clusters - uses df_runtime_cluster datasaet
 Sum_of_squared_distances = []
 K = range(1,10)
 for k in K:
    km = KMeans(n_clusters=k)
    km = km.fit(df_clustering)
    Sum_of_squared_distances.append(km.inertia_)
 ```
 %% Cell type:code id: tags:
 ``` 
 # the bend in the graph is the optimal number of clusters for graphs using the df_runtime_cluster dataset
 plt.plot(K, Sum_of_squared_distances, 'bx-')
 plt.xlabel('k')
 plt.ylabel('Sum_of_squared_distances')
 plt.title('Elbow Method For Optimal k')
 plt.show()
 ```
 %% Cell type:markdown id: tags:
 # Normalizing the Data for ReqMem/Elapsed
 %% Cell type:code id: tags:
 ``` 
 if Data_Normalization_Choice == '0-1':
    column_max = df_clustering.max()
    df_clustering_max = column_max.max()
    fit = df_clustering / df_clustering_max
    print("0-1")
 elif Data_Normalization_Choice == 'log':
    fit = np.log10(df_clustering+1)
    print("log")
 else:
    fit = df_clustering
    print("none")
 ```
 %% Cell type:markdown id: tags:
 # kmeans Clustering
 %% Cell type:code id: tags:
 ``` 
 # sets to clusters and returns the cluster points
 kmeans_cluster = KMeans(n_clusters=4, random_state=111)
 kmeans_cluster.fit(fit)
 print(kmeans_cluster.cluster_centers_)
 ```
 %% Cell type:markdown id: tags:
 # Reverting Cluster Points Back to align with UnNormalized data
 %% Cell type:code id: tags:
 ``` 
 if Data_Normalization_Choice == '0-1':
    clusterpoints = kmeans_cluster.cluster_centers_ * df_clustering_max
    print("0-1")
 elif Data_Normalization_Choice == 'log':
    clusterpoints = 10 ** (kmeans_cluster.cluster_centers_) - 1
    print("log")
 else:
    clusterpoints = kmeans_cluster.cluster_centers_
    print("none")
    print(clusterpoints[:,0],clusterpoints[:,1])
 ```
 %% Cell type:markdown id: tags:
 # Separating the Clusters for 2d Histograms
 %% Cell type:code id: tags:
 ``` 
 # creates dataset of ReqMemCPU, Elapsed, and AllocCPUS using the parameters in the labels shown above
 #Purple
 cluster_0 = df_clustering[kmeans_cluster.labels_ == 0]
 #Green
 cluster_1 = df_clustering[kmeans_cluster.labels_ == 1]
 #Yellow
 cluster_2 = df_clustering[kmeans_cluster.labels_ == 2]
 #Red
 cluster_3 = df_clustering[kmeans_cluster.labels_ == 3]
 ```
 %% Cell type:code id: tags:
 ``` 
 # returns the min and max ReqMemCPU, Elapsed, and AllocCPUS for each cluster using the datasets created above.
 # These are the parameters for the scatter plots of each cluster
 print("Purple Cluster")
 print("ReqMemCPU:", "min =",cluster_0.ReqMemCPU.min()," ","max =",cluster_0.ReqMemCPU.max())
 print("Elapsed:", "min =",cluster_0.Elapsed.min()," ","max =",cluster_0.Elapsed.max())
 print("AllocCPUS:", "min =",cluster_0.AllocCPUS.min()," ","max =",cluster_0.AllocCPUS.max())
 print("\nBlue Cluster")
 print("ReqMemCPU:", "min =",cluster_1.ReqMemCPU.min()," ","max =",cluster_1.ReqMemCPU.max())
 print("Elapsed:", "min =",cluster_1.Elapsed.min()," ","max =",cluster_1.Elapsed.max())
 print("AllocCPUS:", "min =",cluster_1.AllocCPUS.min()," ","max =",cluster_1.AllocCPUS.max())
 print("\nYellow Cluster")
 print("ReqMemCPU:", "min =",cluster_2.ReqMemCPU.min()," ","max =",cluster_2.ReqMemCPU.max())
 print("Elapsed:", "min =",cluster_2.Elapsed.min()," ","max =",cluster_2.Elapsed.max())
 print("AllocCPUS:", "min =",cluster_2.AllocCPUS.min()," ","max =",cluster_2.AllocCPUS.max())
 print("\nRed Cluster")
 print("ReqMemCPU:", "min =",cluster_3.ReqMemCPU.min()," ","max =",cluster_3.ReqMemCPU.max())
 print("Elapsed:", "min =",cluster_3.Elapsed.min()," ","max =",cluster_3.Elapsed.max())
 print("AllocCPUS:", "min =",cluster_3.AllocCPUS.min()," ","max =",cluster_3.AllocCPUS.max())
 ```
 %% Cell type:code id: tags:
 ``` 
 # Creates datasets used to make the 2d histograms that correspond to each cluster scatter plot.
 # The groupby does not change the data, but it does make a small enough dataset
 # for purple cluster
 df_0_2d1 = cluster_0.groupby(['ReqMemCPU','Elapsed']).sum().reset_index()
 df_0_2d2 = cluster_0.groupby(['AllocCPUS','Elapsed']).sum().reset_index()
 df_0_2d3 = cluster_0.groupby(['ReqMemCPU','AllocCPUS']).sum().reset_index()
 # for blue cluster
 df_1_2d1 = cluster_1.groupby(['ReqMemCPU','Elapsed']).sum().reset_index()
 df_1_2d2 = cluster_1.groupby(['AllocCPUS','Elapsed']).sum().reset_index()
 df_1_2d3 = cluster_1.groupby(['ReqMemCPU','AllocCPUS']).sum().reset_index()
 # for yellow cluster
 df_2_2d1 = cluster_2.groupby(['ReqMemCPU','Elapsed']).sum().reset_index()
 df_2_2d2 = cluster_2.groupby(['AllocCPUS','Elapsed']).sum().reset_index()
 df_2_2d3 = cluster_2.groupby(['ReqMemCPU','AllocCPUS']).sum().reset_index()
 # for red cluster
 df_3_2d1 = cluster_3.groupby(['ReqMemCPU','Elapsed']).sum().reset_index()
 df_3_2d2 = cluster_3.groupby(['AllocCPUS','Elapsed']).sum().reset_index()
 df_3_2d3 = cluster_3.groupby(['ReqMemCPU','AllocCPUS']).sum().reset_index()
 ```
 %% Cell type:code id: tags:
 ``` 
 # Creating bins
 ####Purple
 purple_rqmem_min = np.min(cluster_0.ReqMemCPU.min())
 purple_rqmem_max = np.max(cluster_0.ReqMemCPU.max())
 purple_elapsed_min = np.min(cluster_0.Elapsed.min())
 purple_elapsed_max = np.max(cluster_0.Elapsed.max())
 purple_alloc_min = np.min(cluster_0.AllocCPUS.min())
 purple_alloc_max = np.max(cluster_0.AllocCPUS.max())
 x_purple_rqmem_elapsed_bins = list(range(purple_rqmem_max))
 y_purple_rqmem_elapsed_bins = list(range(int(purple_elapsed_max)))
 x_purple_alloc_elapsed_bins = list(range(purple_alloc_max))
 y_purple_alloc_elapsed_bins = list(range(int(purple_elapsed_max)))
 x_purple_reqmem_alloc_bins = list(range(purple_rqmem_max))
 y_purple_reqmem_alloc_bins = list(range(int(purple_alloc_max)))
 ####Blue
 blue_rqmem_min = np.min(cluster_1.ReqMemCPU.min())
 blue_rqmem_max = np.max(cluster_1.ReqMemCPU.max())
 blue_elapsed_min = np.min(cluster_1.Elapsed.min())
 blue_elapsed_max = np.max(cluster_1.Elapsed.max())
 blue_alloc_min = np.min(cluster_1.AllocCPUS.min())
 blue_alloc_max = np.max(cluster_1.AllocCPUS.max())
 x_blue_rqmem_elapsed_bins = list(range(blue_rqmem_max))
 y_blue_rqmem_elapsed_bins = list(range(int(blue_elapsed_max)))
 x_blue_alloc_elapsed_bins = list(range(blue_alloc_max))
 y_blue_alloc_elapsed_bins = list(range(int(blue_elapsed_max)))
 x_blue_reqmem_alloc_bins = list(range(blue_rqmem_max))
 y_blue_reqmem_alloc_bins = list(range(int(blue_alloc_max)))
 ####Yellow
 yellow_rqmem_min = np.min(cluster_2.ReqMemCPU.min())
 yellow_rqmem_max = np.max(cluster_2.ReqMemCPU.max())
 yellow_elapsed_min = np.min(cluster_2.Elapsed.min())
 yellow_elapsed_max = np.max(cluster_2.Elapsed.max())
 yellow_alloc_min = np.min(cluster_2.AllocCPUS.min())
 yellow_alloc_max = np.max(cluster_2.AllocCPUS.max())
 x_yellow_rqmem_elapsed_bins = list(range(yellow_rqmem_max))
 y_yellow_rqmem_elapsed_bins = list(range(int(yellow_elapsed_max)))
 x_yellow_alloc_elapsed_bins = list(range(yellow_alloc_max))
 y_yellow_alloc_elapsed_bins = list(range(int(yellow_elapsed_max)))
 x_yellow_reqmem_alloc_bins = list(range(yellow_rqmem_max)) # list range gives one bin per gig
 y_yellow_reqmem_alloc_bins = list(range(yellow_alloc_max)) # list range gives one bin per cpu
 ####Red
 red_rqmem_min = np.min(cluster_3.ReqMemCPU.min())
 red_rqmem_max = np.max(cluster_3.ReqMemCPU.max())
 red_elapsed_min = np.min(cluster_3.Elapsed.min())
 red_elapsed_max = np.max(cluster_3.Elapsed.max())
 red_alloc_min = np.min(cluster_3.AllocCPUS.min())
 red_alloc_max = np.max(cluster_3.AllocCPUS.max())
 x_red_rqmem_elapsed_bins = list(range(red_rqmem_max))
 y_red_rqmem_elapsed_bins = list(range(int(red_elapsed_max)))
 x_red_alloc_elapsed_bins = list(range(int(red_alloc_max)))
 y_red_alloc_elapsed_bins = list(range(int(red_elapsed_max)))
 x_red_reqmem_alloc_bins = list(range(red_rqmem_max)) # list range gives one bin per gig
 y_red_reqmem_alloc_bins = list(range(red_alloc_max)) # list range gives one bin per cpu
 ```
 %% Cell type:markdown id: tags:
 # Summary Stats
 %% Cell type:code id: tags:
 ``` 
 # number of purple cluster jobs and users
 cluster_0_jobs = cluster_0.shape[0]
 users_0 = cluster_0.loc[:,['ReqMemCPU', 'Elapsed', 'AllocCPUS']]
 users_0['user'] = pd.Series(df_1['User'])
 cluster_0_users = users_0.drop_duplicates(subset=['user'])
 # number of green cluster jobs and users
 cluster_1_jobs = cluster_1.shape[0]
 users_1 = cluster_1.loc[:,['ReqMemCPU', 'Elapsed', 'AllocCPUS']]
 users_1['user'] = pd.Series(df_1['User'])
 cluster_1_users = users_1.drop_duplicates(subset=['user'])
 # number of yellow cluster jobs and users
 cluster_2_jobs = cluster_2.shape[0]
 users_2 = cluster_2.loc[:,['ReqMemCPU', 'Elapsed', 'AllocCPUS']]
 users_2['user'] = pd.Series(df_1['User'])
 cluster_2_users = users_2.drop_duplicates(subset=['user'])
 # number of red cluster jobs and users
 cluster_3_jobs = cluster_3.shape[0]
 users_3 = cluster_3.loc[:,['ReqMemCPU', 'Elapsed', 'AllocCPUS']]
 users_3['user'] = pd.Series(df_1['User'])
 cluster_3_users = users_3.drop_duplicates(subset=['user'])
 summary_stats = pd.DataFrame({'Job Count': [cluster_0_jobs, cluster_1_jobs, cluster_2_jobs, cluster_3_jobs],
                              'User Count': [cluster_0_users.shape[0], cluster_1_users.shape[0], cluster_2_users.shape[0], cluster_3_users.shape[0]]},
                               index=['Purple Cluster','Blue Cluster', 'Yellow Cluster', 'Red Cluster'])
 summary_stats.head()
 ```
 %% Cell type:markdown id: tags:
 # Plotting of the Clusters in 1d and 3d Graphs
 %% Cell type:code id: tags:
 ``` 
 print(summary_stats)
 figure = plt.figure()
 figure.set_size_inches(20,15)
 # ReqMem/Elapsed 2d Graph
 rqmem_elapsed_clustergraph = figure.add_subplot(2,3,1)
 rqmem_elapsed_clustergraph.scatter(df_clustering['ReqMemCPU'],df_clustering['Elapsed'],
                                   c=kmeans_cluster.labels_, cmap='rainbow')
 rqmem_elapsed_clustergraph.scatter(clusterpoints[:,0] ,clusterpoints[:,1], color='black')
 plt.xlabel('ReqMemCPU(gigs)')
 plt.ylabel('Elapsed(hours)')
 plt.title('Runtime/Requested Gigs RAM')
 # Alloc/Elapsed 2d Graph
 alloc_elapsed_clustergraph = figure.add_subplot(2,3,2)
 alloc_elapsed_clustergraph.scatter(df_clustering['AllocCPUS'],df_clustering['Elapsed'],
                                   c=kmeans_cluster.labels_, cmap='rainbow')
 alloc_elapsed_clustergraph.scatter(clusterpoints[:,2] ,clusterpoints[:,1], color='black')
 plt.xlabel('AllocCPUS')
 plt.ylabel('Elapsed(hours)')
 plt.title('Runtime/Core')
 # ReqMem/Alloc 2d Graph
 rqmem_alloc_clustergraph = figure.add_subplot(2,3,3)
 rqmem_alloc_clustergraph.scatter(df_clustering['ReqMemCPU'],df_clustering['AllocCPUS'],
                                   c=kmeans_cluster.labels_, cmap='rainbow')
 rqmem_alloc_clustergraph.scatter(clusterpoints[:,0] ,clusterpoints[:,2], color='black')
 plt.xlabel('ReqMemCPU(gigs)')
 plt.ylabel('AllocCPUS')
 plt.title('Cores/Requested Gigs RAM')
 ########### 3d Graphs
 # ReqMem/Alloc 3d Graph
 rqmem_alloc_clustergraph_3d = figure.add_subplot(2,3,4, projection='3d')
 rqmem_alloc_clustergraph_3d.scatter(df_clustering['ReqMemCPU'], df_clustering['AllocCPUS'], df_clustering['Elapsed'],
                                     c=kmeans_cluster.labels_ ,cmap='rainbow')
 rqmem_alloc_clustergraph_3d.scatter(clusterpoints[:,0] ,clusterpoints[:,2], color='black')
 rqmem_alloc_clustergraph_3d.set_xlabel('ReqMemCPU(gigs')
 rqmem_alloc_clustergraph_3d.set_ylabel('AllocCPUS')
 rqmem_alloc_clustergraph_3d.set_zlabel('Elapsed(hours)')
 # sets size and color for gridlines by axis
 rqmem_alloc_clustergraph_3d.xaxis._axinfo["grid"].update({"linewidth":.5, "color" : "black"})
 rqmem_alloc_clustergraph_3d.yaxis._axinfo["grid"].update({"linewidth":.5, "color" : "black"})
 rqmem_alloc_clustergraph_3d.zaxis._axinfo["grid"].update({"linewidth":.5, "color" : "black"})
 # Alloc/Elapsed 3d Graph
 alloc_elapsed_clustergraph_3d = figure.add_subplot(2,3,5, projection='3d')
 alloc_elapsed_clustergraph_3d.scatter(df_clustering['AllocCPUS'], df_clustering['ReqMemCPU'], df_clustering['Elapsed'],
                                      c=kmeans_cluster.labels_ ,cmap='rainbow')
 alloc_elapsed_clustergraph_3d.scatter(clusterpoints[:,2] ,clusterpoints[:,1], color='black')
 alloc_elapsed_clustergraph_3d.set_xlabel('AllocCPUS')
 alloc_elapsed_clustergraph_3d.set_ylabel('ReqMemCPU(gigs)')
 alloc_elapsed_clustergraph_3d.set_zlabel('Elapsed(hours)')
 alloc_elapsed_clustergraph_3d.xaxis._axinfo["grid"].update({"linewidth":.5, "color" : "black"})
 alloc_elapsed_clustergraph_3d.yaxis._axinfo["grid"].update({"linewidth":.5, "color" : "black"})
 alloc_elapsed_clustergraph_3d.zaxis._axinfo["grid"].update({"linewidth":.5, "color" : "black"})
 # ReqMem/Elapsed 3d Graph
 rqmem_elapsed_clustergraph_3d = figure.add_subplot(2,3,6, projection='3d')
 rqmem_elapsed_clustergraph_3d.scatter(df_clustering['ReqMemCPU'], df_clustering['Elapsed'], df_clustering['AllocCPUS'],
                                      c=kmeans_cluster.labels_ ,cmap='rainbow')
 rqmem_elapsed_clustergraph_3d.scatter(clusterpoints[:,0] ,clusterpoints[:,1], color='black')
 rqmem_elapsed_clustergraph_3d.set_xlabel('ReqMemCPU(gigs)')
 rqmem_elapsed_clustergraph_3d.set_ylabel('Elapsed(hours)')
 rqmem_elapsed_clustergraph_3d.set_zlabel('AllocCPUS')
 rqmem_elapsed_clustergraph_3d.xaxis._axinfo["grid"].update({"linewidth":.5, "color" : "black"})
 rqmem_elapsed_clustergraph_3d.yaxis._axinfo["grid"].update({"linewidth":.5, "color" : "black"})
 rqmem_elapsed_clustergraph_3d.zaxis._axinfo["grid"].update({"linewidth":.5, "color" : "black"})
 plt.show()
 ```
 %% Cell type:markdown id: tags:
 # Plotting of Each Cluster Group in 2d Histograms
 %% Cell type:code id: tags:
 ``` 
 fig = plt.figure()
 fig.set_size_inches(20,10)
 #fig.tight_layout()
 #####Blue
 ax = fig.add_subplot(431)
 rqmem_elapsed_blue_hist = ax.hist2d(cluster_1['ReqMemCPU'],cluster_1['Elapsed'],
                                bins =[x_blue_rqmem_elapsed_bins, y_blue_rqmem_elapsed_bins],
                                      cmap = plt.cm.Blues)
 ax.set_xlabel('ReqMemCPU(gigs)')
 ax.set_ylabel('Elapsed(hours)')
 ax.set_title('Blue Cluster')
 ax.set_xlim(xaxis_min,xaxis_max)
 ax.set_ylim(yaxis_min,yaxis_max)
 ax2 = fig.add_subplot(432)
 alloc_elapsed_blue_hist = ax2.hist2d(cluster_1['AllocCPUS'],cluster_1['Elapsed'],
                                bins =[x_blue_alloc_elapsed_bins, y_blue_alloc_elapsed_bins],
                                      cmap = plt.cm.Blues)
 ax2.set_xlabel('AllocCPUS')
 ax2.set_ylabel('Elapsed(hours)')
 ax2.set_title('Blue Cluster')
 ax2.set_xlim(xaxis_min,xaxis_max)
 ax2.set_ylim(yaxis_min,yaxis_max)
 ax3 = fig.add_subplot(433)
 reqmem_alloc_blue_hist = ax3.hist2d(cluster_1['ReqMemCPU'],cluster_1['AllocCPUS'],
                                bins =[x_blue_reqmem_alloc_bins, y_blue_reqmem_alloc_bins],
                                     cmap = plt.cm.Blues)
 ax3.set_xlabel('ReqMemCPU(gigs)')
 ax3.set_ylabel('AllocCPUS')
 ax3.set_title('Blue Cluster')
 ax3.set_xlim(xaxis_min,xaxis_max)
 ax3.set_ylim(yaxis_min,yaxis_max)
 ####Purple
 ax4 = fig.add_subplot(434) # This represents a (3x3) grid (row x col) and we are plotting the (1) subplot. The last number increments row-wise.
 rqmem_elapsed_purple_hist = ax4.hist2d(cluster_0['ReqMemCPU'],cluster_0['Elapsed'],
                                bins =[x_purple_rqmem_elapsed_bins, y_purple_rqmem_elapsed_bins],
                                      cmap = plt.cm.Blues)
 ax4.set_xlabel('ReqMemCPU(gigs)')
 ax4.set_ylabel('Elapsed(hours)')
 ax4.set_title('Purple Cluster')
 ax4.set_xlim(xaxis_min,xaxis_max)
 ax4.set_ylim(yaxis_min,yaxis_max)
 ax5 = fig.add_subplot(435) # Second subplot
 alloc_elapsed_purple_hist = ax5.hist2d(cluster_0['AllocCPUS'],cluster_0['Elapsed'],
                                bins =[x_purple_alloc_elapsed_bins, y_purple_alloc_elapsed_bins],
                                       cmap = plt.cm.Blues)
 ax5.set_xlabel('AllocCPUS')
 ax5.set_ylabel('Elapsed(hours)')
 ax5.set_title('Purple Cluster')
 ax5.set_xlim(xaxis_min,xaxis_max)
 ax5.set_ylim(yaxis_min,yaxis_max)
 ax6 = fig.add_subplot(436)
 reqmem_alloc_purple_hist = ax6.hist2d(cluster_0['ReqMemCPU'],cluster_0['AllocCPUS'],
                                bins =[x_purple_reqmem_alloc_bins, y_purple_reqmem_alloc_bins],
                                      cmap = plt.cm.Blues) # use magma or
 ax6.set_xlabel('ReqMemCPU(gigs)')
 ax6.set_ylabel('AllocCPUS')
 ax6.set_title('Purple Cluster')
 ax6.set_xlim(xaxis_min,xaxis_max)
 ax6.set_ylim(yaxis_min,yaxis_max)
 #####Red
 ax7 = fig.add_subplot(437)
 rqmem_elapsed_red_hist = ax7.hist2d(cluster_3['ReqMemCPU'],cluster_3['Elapsed'],
                                bins =[x_red_rqmem_elapsed_bins, y_red_rqmem_elapsed_bins],
                                    cmap = plt.cm.Blues)
 ax7.set_xlabel('ReqMemCPU(gigs)')
 ax7.set_ylabel('Elapsed(hours)')
 ax7.set_title('Red Cluster')
 ax7.set_xlim(xaxis_min,xaxis_max)
 ax7.set_ylim(yaxis_min,yaxis_max)
 ax8 = fig.add_subplot(438)
 alloc_elapsed_red_hist = ax8.hist2d(cluster_3['AllocCPUS'],cluster_3['Elapsed'],
                                bins =[x_red_reqmem_alloc_bins, y_red_reqmem_alloc_bins],
                                    cmap = plt.cm.Blues)
 ax8.set_xlabel('AllocCPUS')
 ax8.set_ylabel('Elapsed(hours)')
 ax8.set_title('Red Cluster')
 ax8.set_xlim(xaxis_min,xaxis_max)
 ax8.set_ylim(yaxis_min,yaxis_max)
 ax9 = fig.add_subplot(439)
 reqmem_alloc_red_hist = ax9.hist2d(cluster_3['ReqMemCPU'],cluster_3['AllocCPUS'],
                                bins =[x_red_reqmem_alloc_bins, y_red_reqmem_alloc_bins],
                                   cmap = plt.cm.Blues)
 ax9.set_xlabel('ReqMemCPU(gigs)')
 ax9.set_ylabel('AllocCPUS')
 ax9.set_title('Red Cluster')
 ax9.set_xlim(xaxis_min,xaxis_max)
 ax9.set_ylim(yaxis_min,yaxis_max)
 #####Yellow
 ax10 = fig.add_subplot(4,3,10)
 rqmem_elapsed_yellow_hist = ax10.hist2d(cluster_2['ReqMemCPU'],cluster_2['Elapsed'],
                                bins =[x_yellow_rqmem_elapsed_bins, y_yellow_rqmem_elapsed_bins],
                                    cmap = plt.cm.Blues)
 ax10.set_xlabel('ReqMemCPU(gigs)')
 ax10.set_ylabel('Elapsed(hours)')
 ax10.set_title('Yellow Cluster')
 ax10.set_xlim(xaxis_min,xaxis_max)
 ax10.set_ylim(yaxis_min,yaxis_max)
 ax11 = fig.add_subplot(4,3,11)
 alloc_elapsed_yellow_hist = ax11.hist2d(cluster_2['AllocCPUS'],cluster_2['Elapsed'],
                                bins =[x_yellow_reqmem_alloc_bins, y_yellow_reqmem_alloc_bins],
                                    cmap = plt.cm.Blues)
 ax11.set_xlabel('AllocCPUS')
 ax11.set_ylabel('Elapsed(hours)')
 ax11.set_title('Yellow Cluster')
 ax11.set_xlim(xaxis_min,xaxis_max)
 ax11.set_ylim(yaxis_min,yaxis_max)
 ax12 = fig.add_subplot(4,3,12)
 reqmem_alloc_yellow_hist = ax12.hist2d(cluster_2['ReqMemCPU'],cluster_2['AllocCPUS'],
                                bins =[x_yellow_reqmem_alloc_bins, y_yellow_reqmem_alloc_bins],
                                   cmap = plt.cm.Blues)
 ax12.set_xlabel('ReqMemCPU(gigs)')
 ax12.set_ylabel('AllocCPUS')
 ax12.set_title('Yellow Cluster')
 ax12.set_xlim(xaxis_min,xaxis_max)
 ax12.set_ylim(yaxis_min,yaxis_max)
 # sets the spacing between plots
 # top = space between title and graphs - increase number to bring title down and decrease to bring title up
 # left = space to the left
 # wspace = padding on both sides of graphs
 # hspace = padding on top and bottom of graphs
 #bottom = the bottom of the subplots of the figure
 #top = the top of the subplots of the figure
 fig.subplots_adjust(left=0.0, wspace=0.2, top=3, hspace=.2)
 figure.suptitle('Clusters', fontsize=20)
 plt.show()
 ```
 %% Cell type:code id: tags:
 ``` 
 xaxis_min = 0
 xaxis_max = 10
 yaxis_min = 0
 yaxis_max = 5
 ```
 %% Cell type:code id: tags:
 ``` 
 # only the darkest spots
 # blue reqmem/elapsed denstiy spot: reqmem = 16-17 ,elapsed = 0-1
 # blue alloc/elapsed denstiy spot: alloc = 4-5 ,elapsed = 0-1
 # blue reqmem/alloc denstiy spot: reqmem = 16-17 ,alloc = 4-5
 # purple reqmem/elapsed denstiy spot: reqmem = 8-9 ,elapsed = 0-1
 # purple alloc/elapsed denstiy spot: alloc = 1-2 ,elapsed = 0-1
 # purple reqmem/alloc denstiy spot: reqmem = 8-9 ,alloc = 1-2
 # red reqmem/elapsed denstiy spot: reqmem = ,elapsed =
 # red alloc/elapsed denstiy spot: alloc = ,elapsed =
 # red reqmem/alloc denstiy spot: reqmem = ,alloc =
 # yellow reqmem/elapsed denstiy spot: reqmem = ,elapsed =
 # yellow alloc/elapsed denstiy spot: alloc = ,elapsed =
 # yellow reqmem/alloc density spot: reqmem = 0-10, alloc = 1-2
 ```

--- a/Release_Notes.md
+++ b/Release_Notes.md
+# Release notes version - 1.0 (12/08/2020)
+# Cluster Analysis
+This initial version creates the dataset, kmeans clustering, and resulting graphs to analize how our users are utilizing the cluster.
+Features included:
+- User input to choose date range of data to analyze
+- User input to choose min and max values for ReqMemCPU, AllocCPUS, and Elapsed
+- User input to choose how data is normalized: 0-1, log, or no normalization
+- User input to choose min and max x and y axes for 2D histogram graphs
+    # Next Release Planned Features (coming December 2020)
+    - data on job counts for each density spot in 2d histograms
+    - summary statistics for each cluster in the form of the count of jobs and the count of users per cluster
+# Release Notes version - 1.1 Bug Fix (12/15/2020)
+Dataset for completed jobs orginally had all jobs and each of their job steps. This skewed the clustering graphs, as there were more data points than individual jobs ran. The data is now being pulled into the dataset using only allocated jobs (done with -X in the slurm2sql.slurm2sql command), which results in each row of the dataset being a different job.
+# Release Notes verion - 2.0 (12/22/2020)
+Added summary stats for each cluster. This includes the count for both jobs ran and users running those jobs for each of the four clusters.
+- summary statistics in the form of a table showing the job and user count for each cluster
+* Data on stats for each density spot in the 2d histograms will come in another notebook. This notebook will be a deeper analysis of each 2d histogram for each cluster. This notebook should be released by end of January 2021.