Durvalumab and Tremelimumab With or Without High or Low-Dose Radiation Therapy in Treating Patients With Metastatic Colorectal or Non-small Cell Lung Cancer

Inclusion Criteria: * Patients must have histologically or cytologically confirmed non-small cell lung cancer (cohort 1) or colorectal cancer (cohort 2) * Patients must have measurable disease, defined as at least one lesion that can be accurately measured in at least one dimension (longest diameter to be recorded for non-nodal lesions and short axis for nodal lesions) as \>= 20 mm (\>= 2 cm) with conventional techniques or as \>= 10 mm (\>= 1.5 cm) with spiral computed tomography (CT) scan, magnetic resonance imaging (MRI), or calipers by clinical exam * Patients in both cohorts must have progressive disease following prior therapy; specifically: * Cohort 1 (NSCLC): Patients must have evidence of radiologic or clinical disease progression during previous treatment with systemic PD-1 directed therapy and/or have been deemed not to derive clinical benefit from PD-1 directed treatment; this includes patients who demonstrated an initial response and subsequent progression; no prior treatment with chemotherapy or targeted agents are required; intervening therapy is allowed between previous PD-1 directed treatment and there is no required interval from prior PD-1 treatment required; PD-1 directed treatment includes treatment with antibodies targeting the PD-1 receptor such as pembrolizumab or nivolumab, as well as PD-L1 targeted antibodies such as MEDI4736 (durvalumab), atezolizumab and avelumab; these agents may have been administered as part of a clinical trial * Cohort 2 (colorectal cancer): Patients must have progressed on \>= one-line chemotherapy * At least 21 days must have elapsed from prior systemic therapy (chemotherapy or radiation) * Eastern Cooperative Oncology Group (ECOG) performance status =\< 1 (Karnofsky \>= 60%) and life expectancy greater than 6 months; furthermore, enrollment of patients with greater than 10 measurable lesions is discouraged * Patients must have normal organ and marrow function independent of transfusion for at least 7 days prior to screening and independent of growth factor support for at least 14 days prior to screening * Hemoglobin (Hgb) \>= 9 g/dl * Absolute neutrophil count \>= 1,500/mcL * Platelets \>= 100,000/mcL * Total bilirubin =\< 1.5 x normal institutional limits; this will not apply to patients with confirmed Gilbert's syndrome (persistent or recurrent hyperbilirubinemia \[predominantly unconjugated bilirubin\] in the absence of evidence of hemolysis or hepatic pathology), who will be allowed in consultation with their physician * Aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) = \< 2.5 x institutional upper limit of normal; for patients with hepatic metastases, ALT and AST =\< 5 x ULT * Measured creatinine clearance (CL) \> 40 mL/min OR calculated creatinine clearance (CL) \> 40 mL/min as determined by Cockcroft-Gault (using actual body weight) * Patients must have at least one lesion that has not previously been irradiated (and is not within a previously radiated field) and for which palliative radiation is potentially indicated and could be safely delivered at the radiation doses specified in this protocol; this lesion must not be the only measurable lesion so that it is still possible to determine the response rate outside of the radiation treatment field; this lesion must not be within the central nervous system (CNS) (brain or spinal cord) or requiring urgent or emergent palliative radiation given the timing of radiation specified on this protocol; furthermore, this lesion: * For cohort 1 (NSCLC cohort) - the lesion to be irradiated must be in the lung, lymph nodes, adrenal gland or liver * For cohort 2 (colorectal cohort) - the lesion to be irradiated must be in the liver * The effects of MEDI4736 and tremelimumab on the developing human fetus are unknown; for this reason and because radiation is known to be teratogenic, evidence of post-menopausal status or negative urinary or serum pregnancy test for female pre-menopausal patients is required; women will be considered post-menopausal if they have been amenorrheic for 12 months without an alternative medical cause; the following age-specific requirements apply: * Women \< 50 years of age would be considered post-menopausal if they have been amenorrheic for 12 months or more following cessation of exogenous hormonal treatments and if they have luteinizing hormone and follicle stimulating hormone levels in the post-menopausal range for the institution or underwent surgical sterilization (bilateral oophorectomy or hysterectomy) * Women \>= 50 years of age would be considered post-menopausal if they have been amenorrheic for 12 months or more following cessation of all exogenous hormonal treatments, had radiation-induced oophorectomy with last menses \> 1 year ago, had chemotherapy-induced menopause with \> 1 year interval since last menses, or underwent surgical sterilization (bilateral oophorectomy or hysterectomy) * Females of childbearing potential who are sexually active with a non sterilized male partner must use at least 1 highly effective method of contraception from the time of screening and must agree to continue using such precautions for 180 days after the last dose of durvalumab + tremelimumab combination therapy or 90 days after the last dose of durvalumab monotherapy; non-sterilized male partners of a female patient must use male condom plus spermicide throughout this period; cessation of birth control after this point should be discussed with a responsible physician; not engaging in sexual activity for the total duration of the drug treatment and the drug washout period is an acceptable practice; however, periodic abstinence, the rhythm method, and the withdrawal method are not acceptable methods of birth control; female patients should also refrain from breastfeeding throughout this period * Non-sterilized males who are sexually active with a female partner of childbearing potential must use a male condom plus spermicide from screening through 180 days after receipt of the final dose of durvalumab + tremelimumab combination therapy or 90 days after receipt of the final dose of durvalumab monotherapy; not engaging in sexual activity is an acceptable practice; however, occasional abstinence, the rhythm method, and the withdrawal method are not acceptable methods of contraception; male patients should refrain from sperm donation throughout this period * Female partners (of childbearing potential) of male patients must also use a highly effective method of contraception throughout this period * Highly effective methods of contraception, defined as one that results in a low failure rate (ie, less than 1% per year) when used consistently and correctly are described in the table below; note that some contraception methods are not considered highly effective (e.g. male or female condom with or without spermicide; female cap, diaphragm, or sponge with or without spermicide; non-copper containing intrauterine device; progestogen-only oral hormonal contraceptive pills where inhibition of ovulation is not the primary mode of action \[excluding Cerazette/desogestrel which is considered highly effective\]; and triphasic combined oral contraceptive pills) * Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately * Ability of a patient or a Legally Authorized Representative (LAR) to understand and the willingness to sign a written informed consent document * Body weight \> 30 kg * Must have a life expectancy of at least 12 weeks * Cohort 1 (NSCLC cohort) * Ability to undergo a fresh tumor biopsy for the purpose of screening for this clinical trial (including able and willing to give valid written consent) to ability or to provide an available archival tumor sample taken less than 3 months prior to study enrollment (and not obtained prior to progression on a PD-1/PD-L1 inhibitor) if a fresh tumor biopsy is not feasible with an acceptable clinical risk; tumor lesions used for fresh biopsies should be the same lesions to be irradiated when possible and should not be the same lesions used as Response Evaluation Criteria in Solid Tumors (RECIST) target lesions, unless there are no other lesions accessible; additional, optional archival tumor tissue is also requested from before the prior PD-1 directed therapy * Cohort 2 (colorectal cohort) * Ability to undergo a fresh tumor biopsy for the purpose of screening for this clinical trial (including able and willing to give valid written consent) to ability or to provide an available archival tumor sample taken less than 3 months prior to study enrollment if a fresh tumor biopsy is not feasible with an acceptable clinical risk; tumor lesions used for fresh biopsies should be the same lesions to be irradiated when possible and should not be the same lesions used as RECIST target lesions, unless there are no other lesions accessible * Microsatellite stable (MSS) tumor as documented by either: * Immunohistochemistry (IHC) testing that does not suggest loss of MLH-1, MSH-2, PMS2 or MSH6 * Polymerase chain reaction (PCR) testing that does not suggest microsatellite instability (MSI) Exclusion Criteria: * Patients who have had systemic (chemotherapy, biologic therapy or radiotherapy) within 3 weeks (6 weeks for nitrosoureas or mitomycin C) prior to entering the study * Receipt of prior radiotherapy or condition for any reason that would contribute radiation dose that would exceed tolerance of normal tissues, at the discretion of the treating physician * Patients who have not recovered from adverse events due to prior anti-cancer therapy (i.e., have residual toxicities \> grade 1) * Patients who are receiving any other investigational agents * Patients with untreated brain metastases, spinal cord compression, or leptomeningeal carcinomatosis should be excluded from this clinical trial; patients whose brain metastases have been treated may participate provided they show radiographic stability (defined as 2 brain images, both of which are obtained after treatment to the brain metastases; these imaging scans should both be obtained at least four weeks apart and show no evidence of intracranial progression); in addition, any neurologic symptoms that developed either as a result of the brain metastases or their treatment must have resolved or be stable either, without the use of steroids, or are stable on a steroid dose of =\< 10 mg/day of prednisone or its equivalent (and anti-convulsants) for at least 14 days prior to the start of treatment * History of allergic reactions attributed to compounds of similar chemical or biologic composition to tremelimumab and MEDI4736 or previous toxicity attributed to MEDI4736 or other PD-1 or PD-L1 directed therapy that led to drug discontinuation * Prior exposure to immune-mediated therapy, including durvalumab and tremelimumab, except for anti-PD-1 or anti-PD-L1 therapy (including durvalumab) in NSCLC patients; this includes anti-CTLA-4 agents (prior treatment with these agents is NOT allowed in either cohort) and, excludes therapeutic anticancer vaccines, excluding therapeutic anticancer vaccines; exposure to other investigational agents may be permitted after discussion with the study principal investigator (PI) * Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements * Pregnant women are excluded from this study because MEDI4736 (durvalumab), tremelimumab are immune checkpoint inhibitors with the potential for teratogenic or abortifacient effects, as is radiation therapy; because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with MEDI4736(durvalumab), tremelimumab and radiation, breastfeeding should be discontinued if the mother is treated with MEDI4736(durvalumab), tremelimumab and radiation * Female patients who are pregnant or breastfeeding or male or female patients of reproductive potential who are not willing to employ effective birth control from screening to 90 days after the last dose of durvalumab monotherapy or 180 days after the last dose of durvalumab + tremelimumab combination therapy, whichever is later * Human immunodeficiency virus (HIV)-positive patients are ineligible; appropriate studies will be undertaken in patients receiving combination antiretroviral therapy when indicated * Any concurrent chemotherapy, immune therapy, biologic, hormonal therapy for cancer treatment * Current or prior use of immunosuppressive medication within 14 days before the first dose of their assigned IP; the following are exceptions to this criterion: * Intranasal, inhaled, topical steroids, or local steroid injections (e.g., intra-articular injection) * Systemic corticosteroids at physiologic doses not to exceed 10 mg/day of prednisone or its equivalent * Steroids as pre-medication for hypersensitivity reactions (e.g., CT scan pre-medication) * Major surgical procedure (as defined by the investigator) within 28 days prior to the first dose of IP; Note: local surgery of isolated lesions for palliative intent is acceptable * History of allogeneic organ transplantation * Active or prior documented autoimmune or inflammatory disorders (including inflammatory bowel disease, diverticulitis \[with the exception of diverticulosis\]; sarcoidosis syndrome, or other serious gastrointestinal \[GI\] chronic conditions associated with diarrhea; systemic lupus erythematosus; Wegener syndrome \[granulomatosis with polyangiitis\]; myasthenia gravis; Graves disease; rheumatoid arthritis; hypophysitis; uveitis, sarcoidosis syndrome, etc.) within the past 3 years prior to the start of treatment; the following are exceptions to this criterion: * Patients with vitiligo or alopecia * Patients with hypothyroidism (e.g., following Hashimoto syndrome) stable on hormone replacement or psoriasis not requiring systemic treatment * Any chronic skin condition that does not require systemic therapy * Patients without active disease in the last 5 years may be included but only after consultation with the study physician * Patients with celiac disease controlled by diet alone * History of another primary malignancy except for * Malignancy treated with curative intent and with no known active disease \>= 5 years before the first dose of study drug and of low potential risk for recurrence * Adequately treated non-melanoma skin cancer or lentigo maligna without evidence of disease * Adequately treated carcinoma in situ without evidence of disease (e.g., cervical cancer in situ) * Mean QT interval corrected for heart rate (QTc) \>= 470 ms calculated from electrocardiograms (ECGs) using Fridericia's correction; abnormal ECGs should be repeated * History of active primary immunodeficiency * Known history of previous clinical diagnosis of tuberculosis * Active infection including hepatitis B (known positive hepatitis B virus \[HBV\] surface antigen \[HBsAg\]) result or, hepatitis C; patients with a past or resolved HBV infection (defined as the presence of hepatitis B core antibody \[anti-HBc\] and absence of HBsAg) are eligible; patients positive for hepatitis C (HCV) antibody are eligible only if polymerase chain reaction is negative for HCV ribonucleic acid (RNA); negative serologies documented over the past year are sufficient evidence of this * Receipt of live, attenuated vaccine within 30 days prior to the first dose of investigational treatment; Note: patients, if enrolled, should not receive live vaccine during the study and up to 30 days after the last dose of investigational treatment * Any condition that, in the opinion of the investigator, would interfere with evaluation of the investigational treatment or interpretation of patient safety or study results * Any unresolved toxicity National Cancer Institute (NCI) Common Terminology Criteria for Adverse Events (CTCAE) grade \>= 2 from previous anticancer therapy with the exception of alopecia, vitiligo, and the laboratory values defined in the inclusion criteria * Patients with grade \>= 2 neuropathy will be evaluated on a case-by-case basis after consultation with the study physician; patients with irreversible toxicity not reasonably expected to be exacerbated by treatment with durvalumab or tremelimumab may be included only after consultation with the study physician * Cohort 1 (NSCLC cohort) * In regards to administration of prior anti-PD-1 or anti PD-L1 antibodies, a patients: * Must not have experienced a toxicity that led to permanent discontinuation of prior immunotherapy * All adverse events (AEs) while receiving prior immunotherapy must have completely resolved or resolved to baseline prior to screening for this study * Must not have experienced a \>= grade 3 immune related AE or an immune related neurologic or ocular AE of any grade while receiving prior immunotherapy; NOTE: subjects with endocrine AE of =\< grade 2 are permitted to enroll if they are stably maintained on appropriate replacement therapy and are asymptomatic * Must not have required the use of additional immunosuppression other than corticosteroids for the management of an AE, not have experienced recurrence of an AE if re-challenged, and not currently require maintenance doses of \> 10 mg prednisone or equivalent per day * Eligibility for Food and Drug Administration (FDA)-approved agents targeting the EGFR, ROS1 or ALK pathway, which should be evaluated as per standard of care; exceptions to this requirement may be considered on a case-by-case basis by the principal investigator if the patient was previously treated with another targeted agent