Capecitabine and Bevacizumab With or Without Atezolizumab in Treating Patients With Refractory Metastatic Colorectal Cancer

Inclusion Criteria: * Histologically confirmed colorectal cancer that is either clinically or histologically proven to be metastatic and has progressed on regimens containing a fluoropyrimidine (e.g., 5-fluorouracil or capecitabine), oxaliplatin, irinotecan, bevacizumab and an anti-EGFR antibody (if tumor is RAS wild-type), or where the treatment was not tolerated or contraindicated * Measurable disease; Note: previously irradiated sites can be included if there is documented disease progression in that site * Capecitabine and bevacizumab considered appropriate treatment for the patient * Eastern Cooperative Oncology Group (ECOG) performance status (PS) 0-1 * Absolute neutrophil count \>= 1,500/uL (obtained =\< 7 days prior to randomization) * Platelets \>= 100,000/uL (obtained =\< 7 days prior to randomization) * Total bilirubin =\< 1.5 X upper limit of normal (ULN) (obtained =\< 7 days prior to randomization); patients with known Gilbert?s syndrome who have serum bilirubin =\< 3 X ULN may enroll * Aspartate aminotransferase (AST)/alanine aminotransferase (ALT) =\< 1.5 X ULN; \< 3 X ULN if known hepatic metastases (obtained =\< 7 days prior to randomization) * Hemoglobin \>= 9 g/dL continuation of erythropoietin products is permitted (obtained =\< 7 days prior to randomization); hemoglobin must be stable \>= 9 g/dL \>= 14 days without blood transfusion to maintain hemoglobin level * Calculated creatinine clearance must be \>= 50 ml/min using the Cockcroft-Gault formula or a 24 hour urine (obtained =\< 7 days prior to randomization) * The following laboratory values obtained =\< 14 days prior to randomization * Prothrombin time (PT)/partial thromboplastin time (PTT)/international normalized ratio (INR) =\< 1.5 X ULN if not anticoagulated; within local institutional guidelines per local physician if anticoagulated * Negative pregnancy test done =\< 7 days prior to randomization, for women of childbearing potential only * Provide informed written consent * Willingness to return to enrolling institution for follow-up (during the active monitoring phase of the study) * Willingness to provide tissue and blood samples for correlative research purposes * Life expectancy of \>= 3 months Exclusion Criteria: * Any of the following: * Pregnant women * Nursing women * Women of child-bearing potential must agree to use two forms of adequate contraception from time of initial consent, for the duration of study participation, and for \>= 6 months after the last dose of study drug; medically acceptable contraceptives include: (1) surgical sterilization (such as a tubal ligation or hysterectomy), (2) approved hormonal contraceptives (such as birth control pills, patches, implants or injections), (3) barrier methods (such as a condom or diaphragm) used with a spermicide, or (4) an intrauterine device (IUD); contraceptive measures such as Plan B, sold for emergency use after unprotected sex, are not acceptable methods for routine use; postmenopausal woman must have been amenorrheic for at least 2 years to be considered of non-childbearing potential; sexually active men must use at least one form of adequate contraception from time of initial consent, for the duration of study participation, and for \>= 6 months after the last dose of study drug * Chemotherapy, biologic anti-cancer therapy, or central field radiation therapy =\< 28 days prior to randomization; Note: local or stereotactic radiation =\< 14 days prior to randomization * Any investigational agent =\< 28 days or 5 half-lives prior to randomization (whichever is longer) * Prior treatment with atezolizumab or another PD-L1/PD-1 therapy * History of allergic reactions attributed to therapeutic antibodies; Note: patients with reactions to chimeric antibodies may be permitted on a case by case basis with approval by study chair by contacting the data manager * Known untreated central nervous system (CNS) metastases; Note: patients with radiated or resected lesions are permitted, provided the lesions are fully treated and inactive, patients are asymptomatic, and no steroids have been administered for this purpose =\< 30 days prior to randomization * Inadequately controlled hypertension (defined as average systolic blood pressure \> 150 mmHg and/or diastolic blood pressure \> 100 mmHg) * History of hypertensive crisis or hypertensive encephalopathy * New York Heart Association (NYHA) grade II or greater congestive heart failure * History of myocardial infarction, unstable angina, cardiac or other vascular stenting, angioplasty, or surgery =\< 12 months prior to randomization * Active coronary heart disease evidenced as angina or requiring medications to prevent angina * History of stroke or transient ischemic attack, or other arterial thrombosis =\< 12 months prior to randomization * Symptomatic peripheral vascular disease * Any other significant vascular disease (e.g., aortic aneurysm, aortic dissection, or carotid stenosis that requires medical or surgical intervention, including angioplasty or stenting) * Any previous National Cancer Institute (NCI) Common Terminology Criteria for Adverse Events (CTCAE) grade 4 venous thromboembolism * Clinically-significant evidence of bleeding diathesis or coagulopathy as so judged by the treating physician * History of active gastrointestinal (GI) bleeding or other major bleeding =\< 12 months prior to randomization; Note: patients who do not have resolution of the predisposing risk factor (e.g., resection of a bleeding tumor, treatment and endoscopic documentation of a resolved ulcer) will also be excluded * Major surgical procedure, open biopsy, or significant traumatic injury =\< 56 days prior to randomization * Anticipation of need for major surgical procedure =\< 6 months after randomization * Minor surgical procedure =\< 7 days prior to randomization; exception: insertion of an indwelling catheter or percutaneous needle biopsy =\< 48 hours prior to randomization * History of intra-abdominal abscess =\< 6 months prior to randomization; Note: if the affected area was surgically resected, and there is no further risk to the area, patients may enroll * History of abdominal or other significant fistula, gastrointestinal or other organ perforation; Note: if the affected area was surgically resected, and there is no further risk to the area, patients may enroll * Serious, non-healing wound, ulcer, or bone fracture as so judged by the treating physician * Known proteinuria defined by \>= 2+ protein by urinalysis (UA) or \>= 1 gram protein by 24 hour urine collection; Note: Subjects that are \>= 2+ or greater on dipstick but \< 1 g protein on 24 hour urine ARE eligible to participate * Intolerance to bevacizumab defined as any NCI CTCAE grade 3 or grade 4 toxicity attributed to this agent that required discontinuation of bevacizumab (e.g., arterial thromboembolism \[ATE\], perforation, wound healing difficulty, proteinuria, reversible posterior leukoencephalopathy syndrome \[RPLS\]); Note: patients with prior grade 3 bevacizumab-related hypertension may be permitted if hypertension was manageable with standard oral antihypertensives as so judged by the treating physician * Known dihydropyrimidine dehydrogenase (DPD) deficiency * Impairment of GI function or GI disease that may significantly alter capecitabine drug absorption * Active inflammatory bowel disease * History of diverticulitis, chronic ulcerative lower GI disease such as Crohn?s disease or ulcerative colitis, or other symptomatic lower GI conditions that might predispose to perforations * History of autoimmune disease including, but not limited to, systemic lupus erythematosus, rheumatoid arthritis, inflammatory bowel disease, vascular thrombosis associated with antiphospholipid syndrome, Wegener?s granulomatosis, Sjogren?s syndrome, Bell?s palsy, Guillain-Barre syndrome, multiple sclerosis, autoimmune thyroid disease, vasculitis, or glomerulonephritis; Note: patients with a history of autoimmune-related hypothyroidism on a stable dose of thyroid replacement hormone may be eligible for this study * Active current infection or history of recurrent bacterial, viral, fungal, mycobacterial or other infections, including but not limited to tuberculosis and atypical mycobacterial disease, hepatitis B and C, herpes zoster, and human immunodeficiency virus (HIV), but excluding fungal infections of nail beds * Vaccination with a live or attenuated vaccine =\< 28 days prior to randomization; Note: other types of vaccines, including inactivated/killed, toxoid (inactivated toxoid), and subunit/conjugate are all permitted at any time * Any reversible treatment-related toxicity that has not resolved to NCI CTCAE grade =\< 1 except neuropathy * Other concurrent severe and/or uncontrolled medical disease, psychiatric illness, or social situation, which could compromise safety of treatment as so judged by the treating physician; Note: this includes but is not limited to: severely impaired lung function, uncontrolled diabetes (history of consistent blood glucose readings above 300 mg/dL or less than 50 mg/dL), severe infection, severe malnutrition, ventricular arrhythmias, known active vasculitis of any cause, tumor invasion of any major blood vessel, chronic liver or renal disease, and active upper GI tract ulceration * Unwilling to or unable to comply with the protocol * Current or recent (=\< 10 days prior to randomization) use of aspirin (\> 325 mg/day), or clopidogrel (\> 75 mg/day) * Current or recent (=\< 10 days prior to randomization) use of therapeutic oral or parenteral anticoagulants or thrombolytic agents for therapeutic purposes, unless the patient has been on a stable dose of anticoagulants for at least 2 weeks at the time of randomization; Note: the use of full-dose oral or parenteral anticoagulants is permitted as long as the INR or activated partial thromboplastin time (aPTT) is within therapeutic limits (according to the medical standard of the institution) and the patient has been on a stable dose of anticoagulants \>= 14 days at the time of randomization; prophylactic use of anticoagulants is allowed * History or recent diagnosis of demyelinating disease * History of other carcinoma =\< 3 years; exception: if risk of recurrence is known to be under 5% at time of randomization * Current or recent (=\< 90 days prior to randomization) endoluminal stent in the stomach, bowel, colon or rectum * Colonoscopy, sigmoidoscopy, or proctoscopy =\< 7 days prior to randomization * Current or recent (=\< 28 days prior to randomization) use of sorivudine, brivudine, and St. John?s wort * Primary or secondary immunodeficiency (history of or currently active) unless related to primary disease under investigation * Prior allogeneic bone marrow transplantation or prior solid organ transplantation * Treatment with systemic immunosuppressive medications (including but not limited to prednisone, cyclophosphamide, azathioprine, methotrexate, thalidomide, and anti-tumor necrosis factor \[anti-TNF\] agents) =\< 14 days prior to randomization; exception: patients who have received acute, low-dose, systemic immunosuppressant medications (e.g. a one-time dose of dexamethasone for nausea) are eligible; the use of inhaled corticosteroids and mineral-corticoids (e.g. fludrocortisone) for patients with orthostatic hypotension or adrenocortical insufficiency is allowed