Botensilimab, Balstilimab and Regorafenib or Botensilimab and Balstilimab for the Treatment of Advanced or Metastatic Microsatellite Stable Colorectal Cancer

Inclusion Criteria: * Documented informed consent of the participant * In the phase II randomized portion of the study only, until two biopsies are obtained on ten patients receiving BBR and ten patients receiving BB: Agreement to biopsy of the same tumor at baseline and at 4 weeks * Agreement to biopsy of the same tumor at baseline and at 4 weeks * This applies only to patients with easily accessible tumors, where the risk of a biopsy is deemed acceptable. If a biopsy is determined to be unsafe patients will be exempt from this requirement * Agreement to allow the collection of blood for correlatives at baseline, 1 week, 2 weeks, 4 weeks, 8 weeks, and at the time progressive disease * Age: ≥ 18 years * Eastern Cooperative Oncology Group (ECOG) ≤ 1 * Histologically or cytologically confirmed advanced or metastatic progressive proficient mismatch repair (pMMR)/MSS adenocarcinoma of the colon or rectum * No active brain metastases or leptomeningeal metastases, except for patients who underwent definitive surgery or radiation without progression following repeat imaging (at least 4 weeks after the intervention) and were systemic steroids have been discontinued at least 2 weeks prior study treatment * Known extended RAS and BRAF status, as per local standard of practice. Tumor mutation burden (TMB) and PD-L1 status will be collected when available but are not mandated for enrollment * Patients must have progressed following exposure to all of the following agents in the advanced/metastatic setting OR in the neoadjuvant/adjuvant setting if disease recurred within 6 months of last treatment. Patients who were intolerant to prior systemic chemotherapy regimens are eligible if there is documented evidence of clinically significant intolerance despite adequate supportive measures * Fluoropyrimidines (capecitabine or 5-FU) * Irinotecan * Oxaliplatin * Anti-EGFR therapy if RAS and BRAF wild type with left colon primary * Patients must have evidence of progression on or after the last treatment received and within 6 months prior to study enrollment * Measurable disease by Response Evaluation Criteria in Solid Tumors (RECIST) version (v)1.1 * Fully recovered from the acute toxic effects (except alopecia) to ≤ grade 1 to prior anti-cancer therapy, excluding endocrinopathies stable on medication, stable neuropathy that is grade 2 or less, and alopecia * Patients may not have metastatic liver disease defined as: No history of liver metastatic disease OR history of resected or ablated liver metastases without evidence of disease recurrence in the liver for at least 4 months before enrollment * Total bilirubin ≤ 1.5 X upper limit of normal (ULN) * Aspartate aminotransferase (AST) ≤ 2.5 x ULN * Alanine aminotransferase (ALT) ≤ 2.5 x ULN * Serum creatinine ≤ 1.5 x ULN or creatinine clearance ≥ 40 mL/min (measured or calculated using the Cockroft-Gault formula) * Hemoglobin ≥ 9 g/dl * Absolute neutrophil count (ANC) ≥ 1,500/µl * Platelets ≥ 75,000/mm\^3 * Albumin ≥ 3.0 g/dl * Women of childbearing potential (WOCBP): Negative urine or serum pregnancy test * If the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required * Agreement by females and males of childbearing potential to use an effective method of birth control or abstain from heterosexual activity for the course of the study through at least 120 days after the last dose of protocol therapy * Childbearing potential defined as not being surgically sterilized (men) OR have not been free from menses for \> 1 year and ≥ 50 years of age (women only) OR amenorrheic for ≥ 2 years without a hysterectomy and bilateral oophorectomy and a follicle-stimulating hormone value in the postmenopausal range upon pre-study (screening) evaluation (women only) OR status is post-hysterectomy, bilateral oophorectomy, or tubal ligation (women only) Exclusion Criteria: * Prior allogeneic organ transplantation * Prior immunotherapy with PD-1 or PD-L1 or CTLA-4 targeting agents * Prior regorafenib * Surgical intervention within 4 weeks prior to study treatment, except for minor procedures such as port placement * Patients with a condition requiring systemic treatment with corticosteroids (\> 10 mg daily prednisone equivalent) within 14 days or another immunosuppressive medication within 30 days of the first dose of study treatment. Inhaled or topical steroids, and adrenal replacement steroid doses ≤ 10 mg daily prednisone equivalent, are permitted in the absence of active autoimmune disease * Chemotherapy, radiation therapy, biological therapy, immunotherapy within 14 days or five half-lives (whichever is shorter for non-radiation therapy) prior to day 1 of protocol therapy * Strong CYP3A4 inducers or CYP3A4 inhibitors within 14 days prior to day 1 of protocol therapy * Patients unwilling to refrain from drinking grapefruit juice and taking St. John's Wort while on study * Herbal medications other than cannabidiol (CBD) unless reviewed by the principal investigator (PI) and deemed to unlikely interact with study drugs * Clinically significant (i.e., active) cardiovascular disease: cerebral vascular accident/stroke or myocardial infarction within 12 months of enrollment, unstable angina, congestive heart failure (New York Heart Association class ≥ III), or serious uncontrolled cardiac arrhythmia requiring medication * History of allergic reactions attributed to compounds of similar chemical or biologic composition to study agent * Patients must not have uncontrolled hypertension as defined by systolic blood pressure (SBP) \> 150mmHg or diastolic blood pressure (DBP) \> 90mmHg despite optimal medical management. Patients whose blood pressure can be controlled medically are allowed to be rescreened once blood pressure (BP) is under control * Active autoimmune disease or history of autoimmune disease that required systemic treatment within 2 years before starting treatment, i.e., with use of disease-modifying agents or immunosuppressive drugs * History of acute thrombotic venous events in the last 30 days before enrollment. If within 30 days, the patient should be on anticoagulants and without symptoms * Obstructive bowel symptoms related to unresected primary or carcinomatosis * Inability to swallow and absorb oral tablets (such as but not limited to inability to swallow pills, malabsorption issues, ongoing nausea or vomiting) * Non-healing wounds * Symptomatic active bleeding * Any evidence of current interstitial lung disease (ILD) or pneumonitis or a prior history of ILD or non-infectious pneumonitis requiring high-dose glucocorticoids * History or current evidence of any condition, co-morbidity, therapy, any active infections, or laboratory abnormality that might confound the results of the study, interfere with the patient's participation for the full duration of the study, or is not in the best interest of the patient to participate, in the opinion of the treating Investigator * Known previous severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection within 10 days for mild or asymptomatic infections or 20 days for severe/critical illness prior to cycle 1 day 1 (C1D1) * Uncontrolled infection with human Immunodeficiency virus (HIV). Patients on stable highly active antiretroviral therapy (HAART) with undetectable viral load and normal CD4 counts for at least 6 months prior to study entry are eligible. Serological testing for HIV at screening is not required * Known to be positive for hepatitis B virus (HBV) surface antigen, or any other positive test for HBV indicating acute or chronic infection. Patients who are receiving or who have received anti-HBV therapy and have undetectable HBV deoxyribonucleic acid (DNA) for at least 6 months prior to study entry are eligible. Serological testing for HBV at screening is not required * Known active hepatitis C virus (HCV) as determined by positive serology and confirmed by polymerase chain reaction (PCR). Patients on or who have received antiretroviral therapy are eligible provided they are virus-free by PCR for at least 6 months prior to study entry. Serological testing for HCV at screening is not required * Dependence on total parenteral nutrition or intravenous hydration * Clinically significant uncontrolled illness * Concurrent non-colorectal second malignancy (present during screening) requiring treatment or history of a non-colorectal second primary metastatic malignancy within 2 years prior to the first dose of study treatment. Patients with history of prior early-stage basal/squamous cell skin cancer, low-risk prostate cancer eligible for active surveillance or noninvasive or in situ cancers who have undergone definitive treatment at any time are also eligible. Patients with a prior or concurrent malignancy whose natural history or treatment does not have the potential to interfere with the safety or efficacy assessment of the investigational regimen are eligible for this trial after PI approval * Females only: Pregnant or breastfeeding * Psychiatric or substance abuse disorders that would interfere with cooperation with the requirements of the study * Any other condition that would, in the investigator's judgment, contraindicate the patient's participation in the clinical study due to safety concerns with clinical study procedures * Prospective participants who, in the opinion of the investigator, may not be able to comply with all study procedures (including compliance issues related to feasibility/logistics)