Regorafenib and Pembrolizumab in Treating Participants With Advanced or Metastatic Colorectal Cancer

Inclusion Criteria: * Patients who provided written informed consent to be subjects in this trial * Patients with histologically or cytologically confirmed advanced or metastatic colorectal cancer who had failed or are intolerant of oxaliplatin, irinotecan, and fluorouracil (5-FU) * Patients with an Eastern Cooperative Oncology Group (ECOG) performance status (PS) of 0 or 1 * Patients capable of taking oral medication * Patients with evaluable or measurable lesions as per Response Evaluation Criteria in Solid Tumors (RECIST) version 1.1 * Neutrophil count \>= 200/mm\^3 * Platelet count \>= 7.5 x 10\^4/mm\^3 (transfusion \> 2 weeks before testing permitted) * Aspartate transaminase (AST), alanine transaminase (ALT) =\< 2.5-times the upper limit of normal (=\< 5-times in patients with liver metastasis) * Total bilirubin =\< 1.5-times the upper limit of normal * Creatinine =\< 1.5-times the upper limit of normal * Lipase =\< 1.5 x the upper limit of normal (ULN) * International normalized ratio (INR) =\< 1.5 x ULN and partial thromboplastin time (PTT) or activated partial thromboplastin time (aPTT) =\< 1.5 x ULN unless receiving treatment with therapeutic anticoagulation. Patients being treated with anticoagulant, e.g. heparin, will be allowed to participate provided no prior evidence of an underlying abnormality in these parameters exists. Close monitoring of at least weekly evaluations will be performed until INR and PTT are stable based on a pre-dose measurement as defined by the local standard of care * In women with the potential for pregnancy (including patients with amenorrhea due to medical reasons, such as chemical menopause), after consenting to the study, the patient must agree to take contraception for at least 23 weeks after taking the final dose of the investigational drug (a period of 30 days \[ovulation cycle\] is added to five times the elimination half-time of I/O agent). Women with the potential for pregnancy include those who have begun menstruation, who have not undergone a hysterectomy, bilateral tubal ligation, or bilateral oophorectomy, and who have not gone through menopause. Menopause is defined as the consecutive absence of menstrual periods for \>= 12 months * In the case of men, the patient must agree after consenting to the study to take contraception for at least 31 weeks after taking the final dose of the investigational drug (a period of 90 days \[the spermatogenesis cycle\] is added to five times the elimination half-time of immuno-oncology (I/O) agent Exclusion Criteria: * Patients who have undergone systemic chemotherapy, radiotherapy, surgery, hormone therapy, or immunotherapy \< 2 weeks before enrollment. Immune checkpoint blockade as pretreatment is permitted * Patients with a history of taking regorafenib * Patients with hypertension that is difficult to control (systolic blood pressure \>= 150 mmHg and diastolic blood pressure \>= 90 mmHg) despite treatment with several hypotensive agents * Patients with acute coronary syndrome (including myocardial infarction and unstable angina), and with a history of coronary angioplasty or stent placement performed within 6 months before enrollment * Patients with a large amount of pleural effusion or ascites requiring more than weekly drainage * Patients with a \>= grade 3 active infection according to National Cancer Institute (NCI)-Common Terminology Criteria for Adverse Events (CTCAE) version 4.0 * Patients with symptomatic brain metastasis * Patients with partial or complete gastrointestinal obstruction * Patients with interstitial lung disease with symptoms or signs of activity * Patients who test positive for either anti-human immunodeficiency virus (HIV)-1 antibodies, anti-HIV-2 antibodies, anti-human T-lymphotropic virus (HTLV)-1 antibodies, hepatitis B surface antigen (HBsAg), or anti-hepatitis C virus (HCV) antibodies\* * Patients who test positive for either anti-hepatitis B surface antigen (HBs) or anti-hepatitis B core antigen (HBc) antibodies, and those who have hepatitis B virus (HBV)-deoxyribonucleic acid (DNA) measurements greater than the detection sensitivity will also be excluded * Patients with concurrent autoimmune disease, or a history of chronic or recurrent autoimmune disease * Patients who require systemic corticosteroids (excluding temporary usage for tests, prophylactic administration for allergic reactions, or to alleviate swelling associated with radiotherapy) or immunosuppressants, or who have received such a therapy \< 14 days before enrollment in the present study * Patients with a history or findings of \>= grade III congestive heart failure according to the New York Heart Association functional classification * Patients with a seizure disorder who require pharmacotherapy * Persistent proteinuria \> 3.5 g/24 hours measured by urine protein-creatinine ratio from a random urine sample (\>= grade 3, NCI-CTCAE version \[v\] 4.0) * Known hypersensitivity to any of the study drugs, study drug classes, or excipients in the formulation * Major surgical procedure or significant traumatic injury within 28 days before start of study medication * Non-healing wound, non-healing ulcer, or non-healing bone fracture * Patients with evidence or history of any bleeding diathesis, irrespective of severity * Any hemorrhage or bleeding event \>= CTCAE grade 3 within 4 weeks prior to the start of study medication * Women who are pregnant or breastfeeding, or with the potential for pregnancy * EXCLUDED THERAPIES AND MEDICATIONS, PREVIOUS AND CONCOMITANT * Concurrent anti-cancer therapy (chemotherapy, radiation therapy, surgery, immunotherapy, biologic therapy, or tumor embolization) other than study treatment (regorafenib and pembrolizumab) * Concurrent use of another investigational drug or device therapy (i.e., outside of study treatment) during, or within 2 weeks of trial entry (signing of the informed consent form is OK in the washout period) * Major surgical procedure, open biopsy, or significant traumatic injury within 28 days before start of study medication * Therapeutic anticoagulation with vitamin-K antagonists (e.g., warfarin) or with heparins and heparinoids. However, prophylactic anticoagulation as described below is allowed: * Low dose warfarin (1 mg orally, once daily) with PT-INR =\< 1.5 x ULN is permitted. Infrequent bleeding or elevations in PT-INR have been reported in some subjects taking warfarin while on regorafenib therapy. Therefore, subjects taking concomitant warfarin should be monitored regularly for changes in PT, PT-INR or clinical bleeding episodes * Low dose aspirin (=\< 100 mg daily) * Prophylactic doses of heparin * During the study, strong CYP3A4 inhibitors (eg, clarithromycin, grapefruit juice, indinavir, itraconazole, ketoconazole, nefazodone, nelfinavir, posaconazole, ritonavir, saquinavir, telithromycin, voriconazole) or strong CYP3A4 inducers (eg, carbamazepine, phenobarbital, phenytoin, rifampin, St. John?s wort) are not permitted * Live vaccines administered \< 30 days before the initiation of treatment with the investigational drug and during the trial period. Examples of live vaccines are as follows (however, the list is not exhaustive): measles, mumps, rubella, chicken pox/herpes zoster, yellow fever, rabies, BCG for tuberculosis, and typhoid vaccines. Inoculation with inactive vaccines (e.g., seasonal influenza vaccines) is permitted; however, the intranasal administration of attenuated influenza vaccines (e.g., Flu-Mist) is prohibited * Systemic glucocorticoids for purposes other than treating symptoms caused by notable events with a suspected immunological etiology. Upon deliberation with the trial coordinating committee, the use of corticosteroids may be permitted according to the physiological dose required to alleviate symptoms (e.g., to control symptoms of acute asthma)