Regorafenib, Ipilimumab and Nivolumab for the Treatment of Chemotherapy Resistant Microsatellite Stable Metastatic Colorectal Cancer

Inclusion Criteria: * A signed informed consent must be obtained prior to conducting any study-specific procedures * Histological or cytological confirmed advanced, metastatic, or progressive mismatch repair protein proficient (pMMR)/microsatellite stable (MSS) adenocarcinoma of colon or rectum * Microsatellite status should be performed per local standard of practice (e.g., immunohistochemistry \[IHC\] and/or polymerase chain reaction \[PCR\], or next-generation sequencing). Only participants with pMMR/MSS metastatic colorectal cancer (mCRC) are eligible * Known extended RAS and BRAF status as per local standard of practice * Participant must have progressed following exposure of all the following agents or below: * Prior exposure to the following: * Fluoropyrimidines (capecitabine or fluorouracil \[5-FU\]) * Irinotecan * Oxaliplatin * Anti-EGFR therapy if RAS and BRAF wild type with left colon primary * Patient must have evidence of progression on or after the last treatment regimen received and within 6 months prior to study enrollment * Patients who were intolerant to prior systemic chemotherapy regimens are eligible if there is documented evidence of clinically significant intolerance despite adequate supportive measures * Adjuvant/neoadjuvant chemotherapy can be considered as one line of chemotherapy for advanced/metastatic disease if the participant had disease recurrence within 6 months of completion * Eastern Cooperative Oncology Group (ECOG) performance status of 0 to 1 * Total bilirubin =\< 1.5 x the upper limit of normal (ULN) (performed within 7 days before treatment initiation) * Alanine transaminase (ALT) and aspartate aminotransferase (AST) =\< 3 x ULN if no liver metastases; ALT or AST =\< 5 x ULN allowed for patients with liver involvement (performed within 7 days before treatment initiation) * Platelet count \>= 100,000 /mm\^3 (performed within 7 days before treatment initiation) * Hemoglobin (Hb) \>= 9 g/dL (performed within 7 days before treatment initiation) * White blood cells (WBC) \>= 2000/uL (performed within 7 days before treatment initiation) * Absolute neutrophil count (ANC) \>= 1500/mm\^3 (performed within 7 days before treatment initiation) * Serum creatinine =\< 1.5 x ULN or creatinine clearance \>= 40 mL/min (measured or calculated using the Cockcroft-Gault formula) (performed within 7 days before treatment initiation) * Measurable disease as determined by RECISTv1.1 * Provision of recent tumor tissue (as defined below) is mandatory for all participants at screening (Formalin-fixed paraffin-embedded block or minimum of 20 slides) * Tumor tissue obtained within 180 days of enrollment and after the last dose of most recent anti-cancer therapy * Or a new biopsy Exceptions for patients with no recent baseline tumor tissues or biopsies may be considered after documented discussion and approval with the principal investigator (PI) of the study * Anticipated life expectancy greater than 3 months * Be able to swallow and absorb oral tablets * Women of childbearing potential (WOCBP) must agree to follow instructions for method(s) of contraception for the duration of study intervention and 120 days after last dose of regorafenib and 5 months after the last dose of nivolumab. Males who are sexually active with WOCBP must agree to follow instructions for method(s) of contraception for the duration of study intervention and 120 days after last dose of regorafenib and 7 months after the last dose of nivolumab. In addition, male participants must be willing to refrain from sperm donation during this time. Contraceptive use by men or women should be consistent with local regulations regarding the methods of contraception for those participating in clinical studies Exclusion Criteria: * Participants with microsatellite instability high (MSI-H) colorectal cancer * Prior therapy with regorafenib, anti-PD-1, PD-L1, or CTLA-4 inhibitors * Systemic anti-cancer treatment within 14 days or less than 5 half-lives (whichever is shorter) of the first dose of study treatment * Has unresolved clinically significant toxicity of greater than or equal to National Cancer Institute Common Terminology Criteria for Adverse Events (AEs) (NCI-CTCAE, v5.0) grade 2 attributed to any prior therapies (excluding anemia, lymphopenia, alopecia, skin pigmentation, and platinum-induced neurotoxicity) * Arterial or venous thrombotic or embolic events such as cerebrovascular accident (including transient ischemic attacks), deep vein thrombosis or pulmonary embolism within 3 months before the start of study medication (except for adequately treated catheter-related venous thrombosis occurring more than one month before the start of study medication) * Congestive heart failure \>= New York Heart Association (NYHA) class 2 * Unstable angina (angina symptoms at rest), new-onset angina (begun within the last 3 months), myocardial infarction less than 6 months before start of study drug * Uncontrolled cardiac arrhythmias * Poorly controlled hypertension, defined as a blood pressure consistently above 150/90 mmHg despite optimal medical management * Persistent proteinuria of NCI-CTCAE grade 3. Urine dipstick result of 3+ or abnormal, based on type of urine test strip used, is allowed if protein excretion (estimated by urine protein/creatinine ratio on a random urine sample) is \< 3.5 g/24 hr * Major surgical procedure or significant traumatic injury within 28 days before start of study medication. Note: If participants received major surgery, they must have recovered adequately from the toxicity and/or complications from the intervention prior to starting therapy * Non-healing wound, non-healing ulcer, or non-healing bone fracture * Participants with evidence or history of any bleeding diathesis, irrespective of severity * Any hemorrhage or bleeding event \>= NCI-CTCAE grade 3 within 28 days prior to the start of study medication * Significant acute gastrointestinal disorders with diarrhea as a major symptom e.g., Crohn's disease, malabsorption, or \>= NCI-CTCAE grade 2 diarrhea of any etiology * Participants with an active, known or suspected autoimmune disease. Participants with type I diabetes mellitus (T1DM), hypothyroidism only requiring hormone replacement, skin disorders (such as vitiligo, psoriasis, or alopecia) not requiring systemic treatment, or conditions not expected to recur in the absence of an external trigger are permitted to enroll * Participants with a condition requiring systemic treatment with either corticosteroids (\> 10 mg daily prednisone equivalent) or other immunosuppressive medications within 14 days of start of study treatment. Inhaled or topical steroids, and adrenal replacement steroid doses \> 10 mg daily prednisone equivalent, are permitted in the absence of active autoimmune disease * History of (non-infectious) pneumonitis that required steroids or current pneumonitis * History of interstitial lung disease * Subjects with previous malignancies (except non-melanoma skin cancers, and the following in situ cancers: bladder, gastric, colon, cervical/dysplasia, melanoma, or breast) are excluded unless a complete remission prior to study entry and no additional therapy is required or anticipated to be required during the study period * Diagnosis of immunodeficiency or is receiving chronic systemic steroid therapy (in dosing exceeding 10 mg daily of prednisone equivalent) or any other form of immunosuppressive therapy * Presence of symptomatic central nervous system (CNS) metastases, or CNS metastases that require local CNS-directed treatment (such as radiotherapy or surgery). Participants with stable CNS disease or previously treated lesions are eligible for study entry. In addition, subjects must be either off corticosteroids, or on a stable or decreasing dose of 10 mg daily prednisone (or equivalent) * Ongoing infection \> grade 2 NCI-CTCAE requiring systemic therapy * Known history of human immunodeficiency virus (HIV) infection (HIV 1/2 antibodies) * Any positive test result for hepatitis B virus (HBV) or hepatitis C virus (HCV) indicating presence of virus, e.g. hepatitis B surface antigen (HBsAg, Australia antigen) positive, or hepatitis C antibody (anti-HCV) positive (except if HCV-ribonucleic acid \[RNA\] negative) * Pregnancy or breast feeding * Psychological, familial, or sociological condition potentially hampering compliance with the study protocol and follow-up (FU) schedule * Previous treatment with live vaccine within 30 days of planned start of study drugs (seasonal flu vaccines that do not contain a live virus are permitted) * Known hypersensitivity to any of the study drugs, study drug classes, or excipients in the formulation