Testing A New Combination of Anti-cancer Immune Therapies, Atezolizumab and CDX-1127 (Varlilumab) With or Without the Addition of a Third Anti-cancer Drug, Cobimetinib, for Advanced-Stage Biliary Tract Cancer

Inclusion Criteria: * Pathologically confirmed biliary tract cancer, having received at least 1 prior line of systemic therapy, and received no more than 2 prior lines of therapy in the metastatic setting (disease recurrence =\< 6 months from the last dose of perioperative therapy/day of surgery \[whichever is more recent\] in resected patients will be considered the first line of therapy) * Includes intrahepatic cholangiocarcinoma (IHC), extrahepatic cholangiocarcinoma (EHC), and gallbladder carcinoma (GBC), but not Ampulla of Vater cancers * Patients must have measurable disease by Response Evaluation Criteria in Solid Tumors (RECIST) version (v)1.1 * Age \>= 18 years. Because no dosing or adverse event data are currently available on the use of atezolizumab, cobimetinib, and CDX-1127 (varlilumab) in patients \< 18 years of age, children are excluded from this study * Eastern Cooperative Oncology Group (ECOG) performance status =\< 1 (Karnofsky \>= 80%) * Absolute neutrophil count \>= 1,500/mcL * Hemoglobin \>= 9.0 g/dl * Platelets \>= 100,000/mcL * Total bilirubin =\< 1.5 x institutional upper limit of normal (ULN) (Patients with known Gilbert disease who have serum bilirubin level =\< 3 x ULN may be enrolled) * Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamic pyruvic transaminase \[SGPT\]) =\< 3 x institutional ULN * Serum creatinine =\< 1.5 x institutional ULN OR * Creatinine clearance \> 30 mL/min/1.73 m\^2 (calculated by Cockcroft-Gault method) for patients with creatinine levels above institutional normal * Albumin \>= 3.0 g/dL * Prothrombin time (PT)/activated partial thromboplastin time (aPTT) =\< 1.5 x ULN (This applies only to patients who do not receive therapeutic anticoagulation; patients receiving therapeutic anticoagulation, such as low-molecular-weight heparin or warfarin, should be on a stable dose) * Creatine kinase (CK)/creatine phosphokinase (CPK) \< 5 x ULN * Oxygen saturation \>= 92% on room air * Left ventricular ejection fraction \> 50% * Human immunodeficiency virus (HIV)-infected patients on effective anti-retroviral therapy with undetectable viral load within 6 months are eligible for this trial * For patients with evidence of chronic hepatitis B virus (HBV) infection, the HBV viral load must be undetectable on suppressive therapy, if indicated * Patients with a history of hepatitis C virus (HCV) infection must have been treated and cured. For patients with HCV infection who are currently on treatment, they are eligible if they have an undetectable HCV viral load * Patients must be willing to undergo 2 sets of core needle biopsies. If possible, biopsied sites should be different than those used for measurable disease/RECIST measurements, but this is not mandatory * Patients must have an estimated life expectancy of greater than 3 months * Patients must be able to swallow pills * Patients should not have evidence of retinal pathology on ophthalmologic examination; or neurosensory retinal detachment, retinal vein occlusion (RVO), or neovascular macular degeneration * The effects of atezolizumab, cobimetinib, and CDX-1127 (varlilumab) on the developing human fetus are unknown. For this reason, women of child-bearing potential and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry, for the duration of study participation, and 5 months after the last dose of atezolizumab. Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately. Men treated or enrolled on this protocol must also agree to use adequate contraception prior to the study, for the duration of study participation, and 5 months (150 days) after completion of atezolizumab, cobimetinib, and CDX-1127 (varlilumab) administration * Ability to understand and the willingness to sign a written informed consent document Exclusion Criteria: * Patients with prior allogeneic bone marrow transplantation within the past 5 years or prior solid organ transplantation at any point * Patients who have had chemotherapy or radiotherapy within 4 weeks (6 weeks for nitrosoureas or mitomycin C) prior to entering the study or those who have not recovered from adverse events (other than alopecia or neuropathy) due to agents administered more than 4 weeks earlier. However, the following therapies are allowed: * Hormone-replacement therapy or oral contraceptives * Herbal therapy \> 1 week prior to randomization (herbal therapy intended as anticancer therapy must be discontinued at least 1 week prior to randomization) * Palliative radiotherapy for bone metastases \> 2 weeks prior to randomization * Prior treatment with anti-CTLA-4, anti-PD-1, or anti-PD-L1or other immune checkpoint inhibitor therapeutic antibodies or pathway-targeting agents with the following exceptions: * Patients who have only received previous durvalumab (anti-PD-L1) as part of first line in combination with gemcitabine and cisplatin (TOPAZ-1 regimen \[NCT03875235\]) are eligible. * Patients who have only received previous pembrolizumab (anti-PD-1) as part of first line in combination with gemcitabine and cisplatin (KEYNOTE-966 regimen \[NCT04003636\]) are eligible. * Prior treatment with MEK or ERK inhibitors * Treatment with any other investigational agent within 4 weeks prior to randomization * Treatment with systemic immunostimulatory agents (including, but not limited to, interferon \[IFN\]-alpha or interleukin \[IL\]-2) within 6 weeks prior to randomization * Treatment with systemic immunosuppressive medications (including, but not limited to, prednisone (\> 10 mg), cyclophosphamide, tacrolimus, sirolimus, azathioprine, methotrexate, thalidomide, and anti-tumor necrosis factor \[anti-TNF\] agents) within 2 weeks prior to randomization * Patients who have received acute, low dose, systemic immunosuppressant medications (e.g., a one-time dose of dexamethasone for nausea) may be enrolled * The use of physiologic doses of systemic corticosteroids and mineralocorticoids (e.g., fludrocortisone) for patients with orthostatic hypotension or adrenocortical insufficiency is allowed * The use of topical and inhaled corticosteroids are allowed due to low systemic absorption * Patients taking bisphosphonate therapy for symptomatic hypercalcemia. Use of bisphosphonate therapy for other reasons (e.g., bone metastasis or osteoporosis) is allowed * Presence of therapeutically actionable mutation with approved targeted therapy (e.g. FGFR fusion patients are eligible for study therapy in the 3rd line setting). Patient must have received somatic mutation testing (tissue or liquid) prior to enrollment * Clinically significant ascites (palpable on exam, paracentesis in last 3 months, and/or symptomatic) * Patients with known primary central nervous system (CNS) malignancy or symptomatic CNS metastases are excluded, with the following exceptions: * Patients with asymptomatic treated CNS metastases may be enrolled, provided all the criteria listed above are met as well as the following: * Radiographic demonstration of improvement upon the completion of CNS-directed therapy and no evidence of interim progression between the completion of CNS-directed therapy and the screening radiographic study * No stereotactic radiation or whole-brain radiation within 28 days prior to randomization * Screening CNS radiographic study \>= 4 weeks from completion of radiotherapy and \>= 2 weeks from discontinuation of corticosteroids * Follow-up brain imaging 3 months after central nervous system (CNS)-directed therapy shows no evidence of progression * History of malignant bowel obstruction * History of severe allergic, anaphylactic, or other hypersensitivity reactions to Chinese hamster ovary cell products, chimeric, humanized, or other recombinant human antibodies or fusion proteins * History of allergic reactions attributed to compounds of similar chemical or biologic composition to atezolizumab, cobimetinib, or CDX-1127 (varlilumab) * Patients receiving any medications or substances that are considered moderate to strong inhibitors or inducers of CYP3A and are not able to switch to an alternative that minimizes interaction potential will ineligible. Coadministration of cobimetinib with a strong CYP3A4 inhibitor can increase cobimetinib systemic exposure significantly (e.g. itraconazole increased serum systemic cobimetinib exposure by 6.7 fold). On the other end, coadministration of cobimetinib with a strong CYP3A inducer may decrease cobimetinib systemic exposure by more than 80% thus reducing its efficacy. Because the lists of these agents are constantly changing, it is important to regularly consult a frequently-updated list such as http://medicine.iupui.edu/clinpharm/ddis/; medical reference texts such as the Physicians' Desk Reference may also provide this information. As part of the enrollment/informed consent procedures, the patient will be counseled on the risk of interactions with other agents, and what to do if new medications need to be prescribed or if the patient is considering a new over-the-counter medicine or herbal product * Patients on mild inhibitors or inducers of CYP3A are allowed * Patients with a known clinically significant liver disease, including active viral, alcoholic, or other hepatitis; cirrhosis; fatty liver; and inherited liver disease. * Patients with past or resolved hepatitis B infection (defined as having a negative hepatitis B surface antigen \[HBsAg\] test and a positive anti-HBc \[antibody to hepatitis B core antigen\] antibody test) are eligible. * Patients positive for hepatitis C virus (HCV) antibody are eligible only if polymerase chain reaction (PCR) is negative for HCV ribonucleic acid (RNA) * Patients who have received immunosuppressive treatment for systemic autoimmune disease, including, but not limited to, systemic lupus erythematosus, rheumatoid arthritis, inflammatory bowel disease, vascular thrombosis associated with antiphospholipid syndrome, Wegener's granulomatosis, Sjogren's syndrome, multiple sclerosis, vasculitis, or glomerulonephritis within the last 2 years. * Patients with a history autoimmune endocrine disorders on stable doses of physiologic hormone replacement may be eligible. * Patients with controlled type 1 diabetes mellitus on a stable insulin regimen may be eligible. * Patients with eczema, psoriasis, lichen simplex chronicus of vitiligo with dermatologic manifestations only (e.g., patients with psoriatic arthritis would be excluded) are permitted provided that they meet the following conditions: * Patients with psoriasis must have a baseline ophthalmologic exam to rule out ocular manifestations * Rash must cover less than 10% of body surface area (BSA) * Disease is well controlled at baseline and only requiring low potency topical steroids (e.g., hydrocortisone 2.5%, hydrocortisone butyrate 0.1%, fluocinolone 0.01%, desonide 0.05%, alclometasone dipropionate 0.05%) * No acute exacerbations of underlying condition within the last 12 months (not requiring psoralen plus ultraviolet A radiation \[PUVA\], methotrexate, retinoids, biologic agents, oral calcineurin inhibitors; high potency or oral steroids) * Patients with history Guillain-Barre syndrome or myasthenia gravis at any point will not be eligible * History of idiopathic pulmonary fibrosis, pneumonitis (including drug induced), organizing pneumonia (i.e., bronchiolitis obliterans, cryptogenic organizing pneumonia, etc.), or evidence of active pneumonitis on screening chest computed tomography (CT) scan. History of radiation pneumonitis in the radiation field (fibrosis) is permitted * Patients with active tuberculosis (TB) are excluded * Severe infections within 4 weeks prior to randomization, including, but not limited to, hospitalization for complications of infection, bacteremia, or severe pneumonia * Signs or symptoms of infection within 2 weeks prior to randomization * Received oral or intravenous (IV) antibiotics within 2 weeks prior to randomization * Patients receiving prophylactic/suppressive antibiotics will not be eligible * Major surgical procedure within 28 days prior to randomization or anticipation of need for a major surgical procedure during the course of the study * Administration of a live, attenuated vaccine within 4 weeks before randomization or anticipation that such a live, attenuated vaccine will be required during the study and up to 5 months after the last dose of atezolizumab. * Influenza vaccination should be given during influenza season only (approximately October to March). Patients must not receive live, attenuated influenza vaccine within 4 weeks prior to Randomization or at any time during the study. * Coronavirus disease 2019 (COVID-19) vaccination is not exclusionary but should be administered at least 7 days before study start * Patients with psychiatric illness/social situations that would limit compliance with study requirements * Pregnant women are excluded from this study because one or more study agents have the potential for teratogenic or abortifacient effects. Because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with atezolizumab, cobimetinib, and CDX-1127 (varlilumab), breastfeeding should be discontinued if the mother is treated with atezolizumab, cobimetinib, and CDX-1127 (varlilumab) * Patients who are using ethinyl estradiol containing oral contraceptives when administered concomitantly with cobimetinib, are excluded due to increased risk of venous thromboembolism * Patients with a history of clinically significant cardiac dysfunction, including the following: * Left ventricular ejection fraction (LVEF) below institutional lower limit of normal (LLN) or below 50%, whichever is lower * Current unstable angina * Current symptomatic congestive heart failure (CHF) of New York Heart Association class 2 or higher * Uncontrolled hypertension \>= grade 2 (patients with a history hypertension controlled with anti-hypertensives to =\< grade 1 are eligible). * Uncontrolled arrhythmias * Myocardial infarction, severe/unstable angina, symptomatic chronic heart failure (CHF), cerebrovascular accident or transient ischemic attack within the previous 6 months * History of treatment with cardiotoxic agents