Names of the Collaborators¶

  • Kaamala Lalith Sai Reddy
  • Kaushik Mantha
  • Preeth Nath Manjunath
  • Vineel Rayapati
  • Harshitha Konduru
  • Saketh Saridena

Here is the link to our github repository: https://github.com/saketh-saridena/HealthCare-DSAF-Project¶

About the data set¶

This is a table of data about patients with cancer. Each row is a record of one patient.

  1. BCRPATIENTBARCODE: This is a unique code for each patient.
  2. AgeAtDiagnosis (yrs): This shows the age of the patient when they were diagnosed with cancer.
  3. VITALSTATUS: This tells whether the patient is alive or not.
  4. TUMORSTAGE: This tells how advanced the cancer is.
  5. TUMORGRADE: This shows how abnormal the cancer cells are.
  6. TUMORRESIDUALDISEASE: This tells if some cancer remains after treatment.
  7. PRIMARYTHERAPYOUTCOMESUCCESS: This shows the result of the primary treatment.
  8. PERSONNEOPLASMCANCERSTATUS: This tells the current cancer status of the patient.
  9. OverallSurvival(mos): This is the survival time in months since diagnosis.
  10. ProgressionFreeStatus: This tells if the cancer has progressed or not.
  11. ProgressionFreeSurvival (mos)#: This is the time in months the patient lived without cancer progressing.
  12. PlatinumFreeInterval (mos)*: This is the time in months the patient lived without needing a specific treatment.
  13. PlatinumStatus: This gives information about the patient's response to a specific treatment.

The rest of the table has similar data for many patients, and some data like PlatinumFreeInterval (mos)* and PlatinumStatus have values like "Tooearly" which likely means it's too early to measure or evaluate those factors.

The information here can help doctors and researchers understand how different factors like age or treatment affect cancer patients' survival and disease progression.

Statement of the Question of Interest:¶

The data provided enumerates comprehensive patient profiles from a cancer registry, including their diagnosis, treatment, and follow-up statistics. Our primary intent is to discern meaningful patterns and associations that might elucidate the underpinnings of patient outcomes and response to therapy. Key points of investigation will include, but not be limited to:

  1. Understanding the distribution and impact of demographic variables such as age at diagnosis on treatment outcomes and survival metrics.
  2. Examining the relationship between tumor stage and grade at diagnosis with primary therapy outcome success, and discerning any substantial correlations.
  3. Investigating the association between tumor residual disease and progression-free survival, overall survival as well as platinum free interval.
  4. Analyzing the variance in therapy response and survival metrics among patients with different statuses of primary neoplasm cancer and vital status.
  5. Delving into the correlation, if any, between platinum status and other patient and tumor characteristics, including tumor grade, tumor stage, and age at diagnosis.
  6. Exploring the implications of the aforementioned associations and their potential in guiding clinical decisions and prognostication.

Incorporating a statistical analysis, the aim is to unravel potentially significant insights from the dataset that could further our comprehension of cancer biology, treatment efficacies, and patient prognoses, thereby contributing to more informed and tailored clinical interventions.

In [49]:
import pandas as pd
import numpy as np

# Assume df is your DataFrame
df = pd.read_csv('augmented_clinical_data.csv')

# Handle Missing Data: Impute with median
for column in df.columns:
    if df[column].dtype == object:  # if the column is categorical
        df[column].fillna(df[column].mode()[0], inplace=True)  # fill NA with mode
    else:
        df[column].fillna(df[column].median(), inplace=True)  # fill NA with median


# Outlier Detection: Assume negative values in PlatinumFreeInterval (mos)* are erroneous
df['PlatinumFreeInterval (mos)*'] = df['PlatinumFreeInterval (mos)*'].apply(lambda x: np.nan if x < 0 else x)


# Encoding Categorical Variables: Label encoding
from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()

for column in df.select_dtypes(include=[object]).columns:
    df[column] = le.fit_transform(df[column])

# Save the cleaned data
df.to_csv('cleaned_data.csv', index=False)
In [34]:
import pandas as pd

# Assuming df is your DataFrame
print(df.describe())

       BCRPATIENTBARCODE  AgeAtDiagnosis (yrs)  VITALSTATUS  TUMORSTAGE  \
count         988.000000            988.000000   988.000000  988.000000   
mean          493.500000             57.618512     1.470648    4.701417   
std           285.355334             13.013369     1.133513    1.058051   
min             0.000000             27.210000     0.000000    0.000000   
25%           246.750000             47.852500     0.000000    4.000000   
50%           493.500000             57.850000     2.000000    5.000000   
75%           740.250000             67.965000     2.000000    5.000000   
max           987.000000             87.470000     3.000000    6.000000   

       TUMORGRADE  TUMORRESIDUALDISEASE  PRIMARYTHERAPYOUTCOMESUCCESS  \
count  988.000000            988.000000                    988.000000   
mean     0.663968              1.274291                      0.943320   
std      0.472589              1.192117                      1.124844   
min      0.000000              0.000000                      0.000000   
25%      0.000000              0.000000                      0.000000   
50%      1.000000              1.000000                      0.000000   
75%      1.000000              2.000000                      2.000000   
max      1.000000              3.000000                      3.000000   

       PERSONNEOPLASMCANCERSTATUS  OverallSurvival(mos)  \
count                  988.000000            988.000000   
mean                     0.624494             47.985466   
std                      0.484499             36.238455   
min                      0.000000              0.030000   
25%                      0.000000             17.885000   
50%                      1.000000             39.380000   
75%                      1.000000             72.250000   
max                      1.000000            179.180000   

       ProgressionFreeStatus  ProgressionFreeSurvival (mos)#  \
count             988.000000                      988.000000   
mean                0.608300                      445.931174   
std                 0.488378                      252.905594   
min                 0.000000                        0.000000   
25%                 0.000000                      226.750000   
50%                 1.000000                      450.500000   
75%                 1.000000                      676.250000   
max                 1.000000                      814.000000   

       PlatinumFreeInterval (mos)*  PlatinumStatus  
count                   923.000000      988.000000  
mean                     41.578527        1.405870  
std                      36.876989        1.051966  
min                       0.000000        0.000000  
25%                      11.265000        0.000000  
50%                      25.690000        2.000000  
75%                      67.475000        2.000000  
max                     129.710000        3.000000
In [35]:
import seaborn as sns
import matplotlib.pyplot as plt

correlation_matrix = df.corr()
sns.heatmap(correlation_matrix, annot=True, cmap='coolwarm')
plt.show()
In [36]:
import seaborn as sns
import matplotlib.pyplot as plt
import pandas as pd
import warnings
warnings.filterwarnings("ignore")


# Histogram
sns.histplot(df['AgeAtDiagnosis (yrs)'])
plt.title('Distribution of Ages at Diagnosis')
plt.show()
In [50]:
import pandas as pd
import warnings
warnings.filterwarnings("ignore")

# Assuming df is your DataFrame
categorical_cols = df.select_dtypes(include=['object', 'category']).columns.tolist()
numerical_cols = df.select_dtypes(include=['number']).columns.tolist()

print(f'Categorical Columns: {categorical_cols}')
print(f'Numerical Columns: {numerical_cols}')

for column in df.columns:  # Replace with your categorical column list
    sns.countplot(data=df, x=column)
    plt.show()

Categorical Columns: []
Numerical Columns: ['BCRPATIENTBARCODE', 'AgeAtDiagnosis (yrs)', 'VITALSTATUS', 'TUMORSTAGE', 'TUMORGRADE', 'TUMORRESIDUALDISEASE', 'PRIMARYTHERAPYOUTCOMESUCCESS', 'PERSONNEOPLASMCANCERSTATUS', 'OverallSurvival(mos)', 'ProgressionFreeStatus', 'ProgressionFreeSurvival (mos)#', 'PlatinumFreeInterval (mos)*', 'PlatinumStatus']
In [38]:
from lifelines import KaplanMeierFitter

# Define the Kaplan-Meier fitter object
kmf = KaplanMeierFitter()

# Fit the data to the Kaplan-Meier model
kmf.fit(durations = df['OverallSurvival(mos)'], event_observed = df['PlatinumStatus'])

# Plot the survival function estimate
kmf.plot_survival_function()
plt.title('Kaplan-Meier Survival Curve')
plt.show()
In [39]:
df.head()
Out[39]:
BCRPATIENTBARCODE AgeAtDiagnosis (yrs) VITALSTATUS TUMORSTAGE TUMORGRADE TUMORRESIDUALDISEASE PRIMARYTHERAPYOUTCOMESUCCESS PERSONNEOPLASMCANCERSTATUS OverallSurvival(mos) ProgressionFreeStatus ProgressionFreeSurvival (mos)# PlatinumFreeInterval (mos)* PlatinumStatus
0 732 52.21 1 4 1 3 3 0 1.54 0 43 NaN 3
1 179 62.21 1 5 1 3 0 0 2.56 0 293 NaN 3
2 273 63.32 1 2 1 0 0 1 0.52 0 35 NaN 3
3 226 46.47 1 6 1 3 0 0 2.62 0 294 NaN 3
4 147 48.72 1 5 1 0 0 0 6.59 0 589 NaN 3
In [40]:
# Random Forest Regressor

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error

# Load data
data = pd.read_csv('augmented_clinical_data.csv')

# Convert non-numeric entries to NaN
data = data.apply(pd.to_numeric, errors='coerce')

# Now you can use median imputation
data = data.fillna(data.median())

# Drop the columns with string values or that won't be used for this model 
data = data.drop(columns=['BCRPATIENTBARCODE', 'VITALSTATUS', 'TUMORRESIDUALDISEASE', 'PRIMARYTHERAPYOUTCOMESUCCESS', 'PERSONNEOPLASMCANCERSTATUS', 'ProgressionFreeStatus', 'PlatinumFreeInterval (mos)*', 'PlatinumStatus'])

# Encode categorical columns
le = LabelEncoder()
data['TUMORSTAGE'] = le.fit_transform(data['TUMORSTAGE'])
data['TUMORGRADE'] = le.fit_transform(data['TUMORGRADE'])

# Handle missing values (simple imputation with median)
data = data.fillna(data.median())

# Split data into features and target variable
X = data.drop(columns=['OverallSurvival(mos)', 'ProgressionFreeSurvival (mos)#'])
y = data['OverallSurvival(mos)']  # Or choose other target variable

# Split data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Create a Random Forest Regressor model and fit it to the training data
reg = RandomForestRegressor(random_state=42)
reg.fit(X_train, y_train)

# Make predictions on the test data
y_pred = reg.predict(X_test)

print(y_pred)

[ 76.4641      39.37958     69.6767      77.6621      42.45473333
  97.6503      17.72482679  52.4983      16.2536      78.6419
  46.47813333  25.1094      22.3427      71.3822      89.7052
  36.1384      33.3624      31.64022     52.8079      28.2219
  76.7516      19.3574      38.8744      46.6859      71.1786
  29.5172      21.6157      28.0747      45.9202      44.33004667
  50.58426     97.27506667  31.6329      33.83175     62.4976
  13.8088      72.9414      79.52885417  78.3628      57.3036
  28.5486      55.4764      21.6157      43.6837      40.0109
  81.6795      98.699675    65.0312      31.6626      30.3835
  27.91753333  57.58184548  63.8801      19.4511793   68.38899333
  19.4515      28.4643      23.9944      38.1643      49.9481
  37.5494      41.1508      79.40296667  39.9554      38.2536
  29.3254      56.32282667  71.9273      14.1429       8.3567
  19.4511793   14.6095      59.4615      18.5897      52.8599
  32.38879731  46.838       60.8788      51.9082      37.1555
  38.3365      30.4396928   45.8648      70.7376      73.61047714
  42.41256667  80.1573      27.8924      43.0411      33.3624
  24.61728333  27.4916      85.6587      28.86236     39.182
  67.41459333  35.3634      80.1573      59.14638571  46.6892
  34.7208      56.1583      71.95423833  95.8607      10.7933
  27.0032      39.6629      53.7221      23.3286      30.8992
  21.6157      19.4511793   95.429385    48.9596      15.9053
  23.7706      34.59352286  21.7045      71.78726667  20.47416
  26.915       62.39715167  68.1354      45.5237      90.0314
  29.046       49.101      114.7439      34.1519      31.7703
  26.7705      31.447       54.1101      51.49950333  98.2064
  41.6946      24.31916     26.83749462  59.3508      21.07959048
  59.0937      79.5996      95.9853      71.711       40.5197
  14.5055      15.9739      29.69        40.669       51.4624
  15.5874      90.1324      67.3041      18.0613      64.8609
  19.4511793   88.6615      31.28602778  85.333       28.5882
  83.5064      27.1889      21.6008      58.40857      9.0392
  39.1587      36.6053      37.38338     59.7624      20.44277333
  85.0358      77.3303      42.8167      38.6831      39.6397
  48.3148       8.4213      20.44277333  14.9326       8.3567
  81.0133      74.60632667  54.4589      63.5965      18.7469
   5.7942      46.6815      19.4511793   49.9481      31.64022
 117.6849      26.4913      19.5032     110.2846      48.7186
  36.5494      95.9853      34.7208    ]
In [41]:
from sklearn.metrics import mean_absolute_error
from math import sqrt
from sklearn.metrics import r2_score

# Evaluate the model using R^2 score
r2 = r2_score(y_test, y_pred)
print(f'R^2 Score: {r2:.2f}')

# Evaluate the model using Mean Squared Error
mse = mean_squared_error(y_test, y_pred)
print(f'Mean Squared Error: {mse:.2f}')

# Evaluate the model using Mean Absolute Error
mae = mean_absolute_error(y_test, y_pred)
print(f'Mean Absolute Error: {mae:.2f}')

# Evaluate the model using Root Mean Squared Error
rmse = sqrt(mean_squared_error(y_test, y_pred))
print(f'Root Mean Squared Error: {rmse:.2f}')

R^2 Score: -0.45
Mean Squared Error: 2160.57
Mean Absolute Error: 36.56
Root Mean Squared Error: 46.48
In [42]:
# Gradient Boosting Regressor.

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

# Load data
data = pd.read_csv('augmented_clinical_data.csv')

# Convert non-numeric entries to NaN
data = data.apply(pd.to_numeric, errors='coerce')

# Now you can use median imputation
data = data.fillna(data.median())

# Drop the columns with string values or that won't be used for this model (e.g., barcode, vital status)
data = data.drop(columns=['BCRPATIENTBARCODE', 'VITALSTATUS', 'TUMORRESIDUALDISEASE', 'PRIMARYTHERAPYOUTCOMESUCCESS', 'PERSONNEOPLASMCANCERSTATUS', 'ProgressionFreeStatus', 'PlatinumFreeInterval (mos)*', 'PlatinumStatus'])

# Encode categorical columns
le = LabelEncoder()
data['TUMORSTAGE'] = le.fit_transform(data['TUMORSTAGE'])
data['TUMORGRADE'] = le.fit_transform(data['TUMORGRADE'])

# Handle missing values (simple imputation with median)
data = data.fillna(data.median())

# Split data into features and target variable
X = data.drop(columns=['OverallSurvival(mos)', 'ProgressionFreeSurvival (mos)#'])
y = data['OverallSurvival(mos)']  # Or choose other target variable

# Split data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Create a Gradient Boosting Regressor model and fit it to the training data
gb = GradientBoostingRegressor(random_state=42)
gb.fit(X_train, y_train)

# Make predictions on the test data
y_pred = gb.predict(X_test)
print(y_pred)

[46.77102302 42.33772269 43.22593716 59.94734329 43.22593716 75.52771823
 42.33772269 46.42296377 31.93693311 69.10750489 49.26079878 39.36622843
 42.33772269 66.09708292 65.11087114 43.22593716 35.06030185 37.81259947
 52.01455268 36.7047276  49.20700488 46.19646766 38.77949009 44.8600168
 56.67706224 43.22593716 46.42296377 38.89841061 51.54455783 42.33772269
 43.22593716 63.76104413 43.22593716 39.43335203 56.11182579 31.93693311
 46.77102302 65.13571804 49.20700488 51.54455783 49.30523231 54.43767445
 46.42296377 52.01455268 46.19646766 46.42296377 59.3165326  53.82202081
 37.02471753 45.4548933  46.84130893 46.0914547  47.76214276 34.7359016
 50.00531881 46.0914547  43.22593716 45.26901208 45.05442993 45.26901208
 46.77102302 46.0914547  43.22593716 42.33772269 46.42296377 41.32844397
 45.26901208 73.7798961  38.89841061 45.60109018 34.7359016  34.15821015
 58.24475061 38.77949009 51.11717916 47.76214276 46.84130893 47.27043924
 58.5059097  43.9152725  45.26901208 36.54359348 46.84130893 45.05442993
 51.10643074 49.26079878 60.52759743 46.0914547  47.18655783 35.06030185
 46.84130893 46.84130893 59.94734329 43.02543453 43.22593716 50.00531881
 38.77949009 60.52759743 47.76214276 42.6103992  40.58467458 46.84130893
 67.72688762 47.12550084 38.61163374 43.22593716 38.21867745 46.84130893
 38.21867745 28.59056383 46.42296377 34.7359016  52.01455268 58.24475061
 41.01219667 46.42296377 49.09096197 46.84130893 49.26079878 46.0914547
 36.7047276  59.3165326  66.09708292 50.61521515 53.51111616 49.71128004
 58.24475061 79.33163841 46.42296377 47.18655783 42.92818556 35.06030185
 52.2752113  45.60109018 56.31648045 46.42296377 43.41395649 36.54359348
 47.27043924 42.55581881 45.05442993 45.60109018 47.12550084 58.24475061
 38.21867745 43.8749093  31.93693311 31.93693311 49.71128004 46.42296377
 32.87242689 59.94734329 46.84130893 43.22593716 46.84130893 34.7359016
 73.7798961  42.93126801 46.0119996  48.20368759 46.84130893 46.84130893
 46.84130893 51.10643074 34.23140617 45.4548933  51.10643074 43.22593716
 52.28572506 38.77949009 65.11087114 49.20700488 45.4548933  42.33772269
 50.61521515 56.05286985 43.8749093  38.77949009 34.81731151 45.60109018
 49.20700488 50.00531881 53.11180484 47.27043924 46.19646766 34.23140617
 47.12550084 34.7359016  45.26901208 37.81259947 79.33163841 42.92818556
 38.89841061 56.67866977 46.42296377 46.84130893 47.12550084 40.58467458]
In [43]:
# Evaluate the model using R^2 score
r2 = r2_score(y_test, y_pred)
print(f'R^2 Score: {r2:.2f}')

# Evaluate the model using Mean Absolute Error
mae = mean_absolute_error(y_test, y_pred)
print(f'Mean Absolute Error: {mae:.2f}')

# Evaluate the model using Root Mean Squared Error
rmse = np.sqrt(mean_squared_error(y_test, y_pred))
print(f'Root Mean Squared Error: {rmse:.2f}')

R^2 Score: -0.08
Mean Absolute Error: 32.18
Root Mean Squared Error: 40.09
In [44]:
# Support Vector Regression or SVR
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder, StandardScaler
from sklearn.svm import SVR
from sklearn.metrics import r2_score, mean_absolute_error, mean_squared_error

# Load data
data = pd.read_csv('augmented_clinical_data.csv')

# Convert non-numeric entries to NaN
data = data.apply(pd.to_numeric, errors='coerce')

# Now you can use median imputation
data = data.fillna(data.median())

# Drop the columns with string values or that won't be used for this model (e.g., barcode, vital status)
data = data.drop(columns=['BCRPATIENTBARCODE', 'VITALSTATUS', 'TUMORRESIDUALDISEASE', 'PRIMARYTHERAPYOUTCOMESUCCESS', 'PERSONNEOPLASMCANCERSTATUS', 'ProgressionFreeStatus', 'PlatinumFreeInterval (mos)*', 'PlatinumStatus'])

# Encode categorical columns
le = LabelEncoder()
data['TUMORSTAGE'] = le.fit_transform(data['TUMORSTAGE'])
data['TUMORGRADE'] = le.fit_transform(data['TUMORGRADE'])

# Handle missing values (simple imputation with median)
data = data.fillna(data.median())

# Split data into features and target variable
X = data.drop(columns=['OverallSurvival(mos)', 'ProgressionFreeSurvival (mos)#'])
y = data['OverallSurvival(mos)']  # Or choose other target variable

# Standardize the data (important for SVR)
scaler = StandardScaler()
X = scaler.fit_transform(X)

# Split data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Create a Support Vector Regression model and fit it to the training data
svr = SVR(kernel='rbf', C=1.0, epsilon=0.1)  # You can adjust the parameters
svr.fit(X_train, y_train)

# Make predictions on the test data
y_pred = svr.predict(X_test)
print(y_pred)

[38.4821187  38.73443575 35.20192054 47.33390264 35.0530975  38.45937453
 38.7000558  39.52395708 45.29349395 47.55467895 38.5461431  38.48148638
 38.67005985 46.05793565 47.86634596 35.73014844 36.61057819 45.55893715
 42.38418792 37.47646873 39.77287936 48.14085804 39.06959118 37.96424985
 47.63103587 35.56787366 39.59713469 36.08750988 38.17735038 38.73936346
 35.38802976 38.58403016 35.28298504 47.29838011 48.39335007 45.27285014
 38.48378249 48.40844769 39.82875889 38.12942675 48.13492575 38.48806057
 39.58539731 42.36351327 47.96043056 39.35479008 36.9857332  44.96685045
 36.52375117 44.70553769 40.37190449 38.49655916 38.91977231 38.85043591
 38.47563753 38.48821338 35.70673472 38.48260509 38.46097792 38.47963313
 38.48012386 38.50625584 35.6215215  38.66198326 39.5089024  34.60332063
 38.47683903 48.3114045  35.91960181 41.6999548  38.85043591 42.75382886
 48.59893811 38.9382397  48.25391964 38.85249414 40.05067896 36.73543229
 37.76209341 43.5110385  38.48002194 38.82825125 40.72781474 38.46914979
 38.42100274 38.54195786 34.64192724 38.48569758 38.01312864 36.61057819
 40.35127766 40.09813371 47.25946892 48.65060258 35.76932022 38.47513663
 39.0777757  34.64192724 38.91070996 42.76792422 37.54516255 40.68870914
 48.64443566 43.75654921 48.34330003 35.11292923 38.65412222 40.9396257
 38.6490002  34.3348495  39.59321461 38.85043591 42.09079925 48.58382023
 38.48796687 39.29775935 43.02314578 40.75030111 38.54902679 38.50426288
 37.47011583 37.03729911 46.03235547 48.64597573 38.28349355 38.37507176
 48.58833814 46.67418625 39.67713362 38.0323707  44.59876376 36.61450744
 48.47900229 41.36869634 38.48489007 39.39286842 42.56453392 38.81844201
 36.7894147  38.47356308 38.46850808 41.55818229 43.72041755 48.56546301
 38.65283285 37.25785755 45.32783243 45.25907088 38.31280524 39.52773985
 34.72417273 47.10013601 40.0178935  35.44348812 40.85848524 38.84227222
 48.28179445 46.44335677 37.33271988 38.21508454 40.52471882 40.76723243
 39.88146247 38.41104619 37.4541562  44.77647071 38.42501651 35.40274748
 46.60645542 39.01160543 47.88561541 39.83310978 44.88245479 38.70912824
 48.64655756 47.04713532 37.26474138 39.02447083 37.67290038 41.69346112
 39.81575112 38.47612678 47.60200107 36.75861968 48.15652062 37.44130948
 44.1898508  38.85043591 38.4784017  45.55893715 46.68576305 44.5559391
 35.90766812 38.42824727 39.63675416 39.95339927 43.70596647 37.54516255]
In [45]:
# Evaluate the model using R^2 score
r2 = r2_score(y_test, y_pred)
print(f'R^2 Score: {r2:.2f}')

# Evaluate the model using Mean Absolute Error
mae = mean_absolute_error(y_test, y_pred)
print(f'Mean Absolute Error: {mae:.2f}')

# Evaluate the model using Root Mean Squared Error
rmse = np.sqrt(mean_squared_error(y_test, y_pred))
print(f'Root Mean Squared Error: {rmse:.2f}')

R^2 Score: -0.08
Mean Absolute Error: 31.20
Root Mean Squared Error: 40.00
In [46]:
# XGBOOST
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
from xgboost import XGBRegressor
from sklearn.metrics import r2_score, mean_absolute_error, mean_squared_error

# Load data
data = pd.read_csv('augmented_clinical_data.csv')

# Convert non-numeric entries to NaN
data = data.apply(pd.to_numeric, errors='coerce')

# Now you can use median imputation
data = data.fillna(data.median())

# Drop the columns with string values or that won't be used for this model (e.g., barcode, vital status)
data = data.drop(columns=['BCRPATIENTBARCODE', 'VITALSTATUS', 'TUMORRESIDUALDISEASE', 'PRIMARYTHERAPYOUTCOMESUCCESS', 'PERSONNEOPLASMCANCERSTATUS', 'ProgressionFreeStatus', 'PlatinumFreeInterval (mos)*', 'PlatinumStatus'])

# Encode categorical columns
le = LabelEncoder()
data['TUMORSTAGE'] = le.fit_transform(data['TUMORSTAGE'])
data['TUMORGRADE'] = le.fit_transform(data['TUMORGRADE'])

# Handle missing values (simple imputation with median)
data = data.fillna(data.median())

# Split data into features and target variable
X = data.drop(columns=['OverallSurvival(mos)', 'ProgressionFreeSurvival (mos)#'])
y = data['OverallSurvival(mos)']  # Or choose other target variable

# Split data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Create an XGBoost Regressor model and fit it to the training data
xgb = XGBRegressor(objective ='reg:squarederror', random_state=42)
xgb.fit(X_train, y_train)

# Make predictions on the test data
y_pred = xgb.predict(X_test)

print(y_pred)

[72.81677   40.344833  25.001568  75.106575  59.83178   92.410904
 34.925518  43.470512  23.637596  75.018074  60.964397  37.146423
 40.53371   66.47034   55.12716   28.445349  28.039612  28.506954
 58.436367  26.940414  59.989452  28.12701   27.532608  26.751383
 44.190018  64.135086  32.855614  41.546432  51.76315   40.344833
 40.061127  84.13286   25.001568  43.884556  52.95763   23.637596
 28.940378  81.15814   54.470833  71.42955   48.172764  66.96872
 32.855614  58.436367  55.12716   57.438892  66.68569   34.60141
 34.784378  32.404816  25.832361  44.602726  58.45159   23.323845
 32.71289   28.940378  28.445349  32.71289   92.410904  32.71289
 59.611217  50.215736  43.457024  45.469574  43.470512  26.582335
 32.71289   58.72604    8.029582  12.536561  23.323845  26.922785
 36.68636   36.53539   53.702965  57.360996  33.70054   61.322693
 61.22324   34.890102  32.71289   23.323845  47.134903  41.059036
 74.29407   64.84373   85.65584   28.940378  52.066765  28.039612
 25.832361  33.70054   69.126305  37.41851   55.786793  32.71289
 27.532608  85.65584   58.45159   26.922785  76.83346   47.134903
 50.231377  56.048016  18.801517  35.19564   45.469574  60.16158
 45.469574  29.923569  32.855614  23.323845  56.176712  36.68636
 27.759983  45.082283  50.01071   47.134903  60.964397  50.215736
 26.940414  55.60436   74.84497   54.463314  62.00533   39.703728
 36.68636   73.66445   29.204742  78.8994    32.404816  28.039612
 52.95763   46.34404   74.50199   44.51732   31.328955  23.323845
 39.6777    41.059036  42.74645   36.65752   56.048016  43.516705
 45.469574  24.15611   23.637596  23.637596  45.76646   43.470512
 22.128225  91.05649   33.70054   54.248512  40.671856  23.323845
 98.78825   31.53687   51.3903    33.911926  71.53948   47.134903
 59.00466   74.29407    7.6880054 55.295235  74.29407   34.128002
 46.58572   15.87913   55.12716   54.470833  34.60141   40.344833
 54.463314  51.461143  24.15611   53.94891   38.37828   12.536561
 54.470833  32.71289   59.317482  61.322693  28.12701    7.6880054
 56.0197    23.323845  32.71289   28.506954  73.66445   32.404816
  8.029582  74.29407   29.204742  45.43078   56.048016  76.83346  ]
In [47]:
# Evaluate the model using R^2 score
r2 = r2_score(y_test, y_pred)
print(f'R^2 Score: {r2:.2f}')

# Evaluate the model using Mean Absolute Error
mae = mean_absolute_error(y_test, y_pred)
print(f'Mean Absolute Error: {mae:.2f}')

# Evaluate the model using Root Mean Squared Error
rmse = np.sqrt(mean_squared_error(y_test, y_pred))
print(f'Root Mean Squared Error: {rmse:.2f}')

R^2 Score: -0.34
Mean Absolute Error: 35.48
Root Mean Squared Error: 44.56
In [48]:
# Accuracy Comparision
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
from sklearn.svm import SVR
from xgboost import XGBRegressor
from sklearn.metrics import r2_score, mean_absolute_error, mean_squared_error

# Load and preprocess data
# ... (your data loading and preprocessing code here) ...

# Split data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Initialize models
models = {
    "Random Forest": RandomForestRegressor(random_state=42),
    "Gradient Boosting": GradientBoostingRegressor(random_state=42),
    "Support Vector": SVR(),
    "XGBoost": XGBRegressor(objective ='reg:squarederror', random_state=42)
}

# Train models, make predictions, and evaluate
for model_name, model in models.items():
    # Fit model
    model.fit(X_train, y_train)
    
    # Make predictions
    y_pred = model.predict(X_test)
    
    # Evaluate model
    r2 = r2_score(y_test, y_pred)
    mae = mean_absolute_error(y_test, y_pred)
    rmse = np.sqrt(mean_squared_error(y_test, y_pred))
    
    print(f"{model_name} - R² Score: {r2:.2f}, Mean Absolute Error: {mae:.2f}, Root Mean Squared Error: {rmse:.2f}")

Random Forest - R² Score: -0.45, Mean Absolute Error: 36.56, Root Mean Squared Error: 46.48
Gradient Boosting - R² Score: -0.08, Mean Absolute Error: 32.18, Root Mean Squared Error: 40.09
Support Vector - R² Score: -0.07, Mean Absolute Error: 31.22, Root Mean Squared Error: 39.93
XGBoost - R² Score: -0.34, Mean Absolute Error: 35.48, Root Mean Squared Error: 44.56

Based on the given values of R² Score, Mean Absolute Error (MAE), and Root Mean Squared Error (RMSE), the Support Vector Machine (SVM) model seems to perform the best among the four models tested.

  1. R² Score (Coefficient of Determination):

    • The R² score is a statistical measure of how well the regression predictions approximate the real data points. A higher R² score indicates a better fit for the data. Among the four models, SVM has the highest R² score of -0.07, though it's still a negative value which isn't ideal.

  2. Mean Absolute Error (MAE):

    • MAE is a measure of errors between paired observations expressing the same phenomenon. Lower values of MAE indicate better model performance. Among the four models, SVM has the lowest MAE of 31.22.

  3. Root Mean Squared Error (RMSE):

    • RMSE is the square root of the average of square differences between prediction and actual observation. Lower values of RMSE indicate better model performance. Among the four models, SVM also has the lowest RMSE of 39.93.

The SVM model appears to be the most accurate. However, it's important to note that the R² score is negative for all models, which indicates that all models are performing poorly. Usually, R² score ranges from 0 to 1, where a higher value (closer to 1) suggests a good model fit to the data.

Conclusion:¶

The analysis of the given dataset of cancer patients reveals an intricate relationship between various clinical and demographic variables, such as age at diagnosis, tumor stage, tumor grade, therapy outcome, and the status of neoplasm cancer, on the overall survival and progression-free survival of individuals. It is observed that the majority of the patients are living, with varying degrees of tumor progression and therapeutic response. However, a considerable number of patients are in the 'Too early' category for the Platinum status, which suggests a need for further longitudinal follow-ups to better understand the platinum free interval and its implications.

The categorization of Tumor Residual Disease, although not uniform across the dataset, provides a significant insight into the extent of disease post-therapeutic interventions. The varying outcomes, ranging from complete response to stable disease, underscore the heterogeneity in therapeutic response among the patients.

Furthermore, the Progression-Free Survival (PFS) and Overall Survival (OS) columns, although populated with different scales of measurement (months), indicate the variance in survival rates among patients, which could potentially be linked to the tumor stage, grade, and the efficacy of the primary therapy administered.

In summary, the dataset provides a rich foundation for conducting more granular analysis to decipher patterns and correlations, which could be instrumental in formulating personalized treatment strategies for better patient outcomes. Further statistical analysis and perhaps the employment of machine learning algorithms could unearth patterns and predictive markers for therapy response and survival rates among cancer patients. This could be invaluable in advancing the field of oncology towards more personalized and effective therapeutic interventions.

In [ ]: