Durvalumab in Combination With Chemotherapy in Treating Patients With Advanced Solid Tumors, DURVA+ Trial

Inclusion Criteria: * Patients with histologically documented metastatic or locally advanced (not amenable to surgery) solid tumors whose disease has progressed following at least one line of standard therapy and/or no standard of treatment exists that has been shown to prolong survival. * If anti-PD-1 or one of the 6 chemotherapy agents is standard-of-care, prior therapy with the agent would not be required. * Patient must have tumor amenable to biopsy and be willing to undergo a tumor biopsy. * Flash frozen tissue collected as part of another study or from a procedure performed due to medical necessity may be acceptable as the baseline sample if the samples were collected within 3 months prior to registration and the patient has not received any investigational or targeted treatment since that time. * A patient who cannot be safely biopsied may be considered for the study upon discussion with Principal Investigator. * Patients must have measurable disease, defined as at least one lesion that can be accurately measured in at least one dimension (longest diameter to be recorded) as \>= 20 mm with conventional techniques or as \>= 10 mm with spiral computed tomography (CT) scan. * Patients with bone metastases or hypercalcemia on intravenous bisphosphonate treatment, denosumab, or similar agents are eligible to participate and may continue this treatment. Patients with prostate cancer may continue luteinizing hormone-releasing hormone (LHRH) agonists or antagonists. * Age ≥ 18 years. Children are excluded from this study, but may be eligible for future pediatric trials. * Eastern Cooperative Oncology Group (ECOG) performance status =\< 2. * Absolute neutrophil count \>= 1,000/uL (mcL). * Platelets \>= 100,000/uL (mcL). * Total bilirubin \< 1.5 x institutional upper limit of normal. * This will not apply to patients with confirmed Gilbert syndrome (persistent or recurrent hyperbilirubinemia that is predominantly unconjugated in the absence of hemolysis or hepatic pathology), who will be allowed only at the discretion of the principal investigator (PI), study chair or their designee. * Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 3 x institutional upper limit of normal, or up to 5 x upper limit of normal (ULN) if liver metastases are present. * Calculated creatinine clearance \> 40 mL/min by the Cockcroft-Gault formula * Any prior systemic therapy (including checkpoint inhibitors), or major surgery must have been completed \>= 3 weeks (\> 6 weeks for nitrosoureas or mitomycin C) or 5 half-lives of the agent, whichever is shorter, prior to enrollment on protocol, and toxicity from prior treatment must have recovered to eligibility levels. Radiation therapy must have been completed \>= 1 week prior to starting treatment. Radiofrequency ablation (RFA) of localized lesions should have been performed \>= 1 week prior to starting treatment. All radiation-related toxicity must have resolved to \< grade 2. * Palliative radiotherapy is permitted between disease progression on Arm 1 and crossover to a combination therapy arm (Arms 2-7), provided there is a washout period of \>= 1 week and any toxicity from radiation has resolved to \< grade 2 * Patients on any arm may receive palliative radiotherapy or loco-regional ablative therapy and remain on study, provided the radiation is not delivered to the target lesion and the patient does not have tumor progression by Response Evaluation Criteria in Solid Tumors (RECIST) * Treatment with systemic immunostimulatory agents (including, but not limited to, interferon-alpha or interleukin-2) must have been completed at least 3 weeks before the first dose of durvalumab. * Body weight \> 30 kg. * Human immunodeficiency virus (HIV)-infected (HIV1/2 antibody-positive) patients may participate if they meet all the following eligibility requirements: * They must be on an anti-retroviral regimen with evidence of at least two undetectable viral loads within the past 6 months on this same regimen; the most recent undetectable viral load must be within the past 12 weeks. * They must have a CD4 count \>= 250 cells/uL over the past 6 months on this same anti-retroviral regimen and must not have had a CD4 count \< 200 cells/uL over the past 2 years, unless it was deemed related to chemotherapy-induced bone marrow suppression. * For patients who have received chemotherapy in the past 6 months, a CD4 count \< 250 cells/uL during chemotherapy is permitted as long as viral loads were undetectable during this same chemotherapy. * They must have an undetectable viral load and a CD4 count \>= 250 cells/uL within 28 days of enrollment. * They must not be currently receiving prophylactic therapy for an opportunistic infection and must not have had an opportunistic infection within the past 6 months. * Monitoring for HIV-infected patients should include: * Viral load and CD4 count every 8-10 weeks. * The effects of targeted agents on the developing human fetus are unknown. The cytotoxic agents chosen for combination with durvalumab adversely affect human fertility and gestation. For these reasons, women of childbearing potential and men must agree to use highly effective contraception prior to study entry for the duration of study participation and for 6 months following the last dose of a study drug. * Because there may be a risk for adverse events in nursing infants secondary to treatment of the mother with these agents, breastfeeding should be discontinued while the patient is on this trial and for 6 months following the last dose of study drug. * Patients should be willing not to donate blood while participating in this study or for at least 90 days following the last dose of study drug. * Left ventricular ejection fraction greater than 50% or the institutional lower limit of normal by echocardiography (ECHO) at entry (patients enrolling on Arm 3 only). * Ability to understand and the willingness to sign a written informed consent document. Exclusion Criteria: * Patients who received prior therapy with a checkpoint inhibitor and were taken off drug for serious adverse events are excluded. Patients who had prior CTLA-4 inhibitor treatment and did not experience serious adverse events are eligible for all arms. Patients who had prior PD-L1/PD-1 inhibitor treatment and did not experience serious adverse events are excluded from the durvalumab monotherapy arm but are eligible for the chemotherapy combinations. * Patients with pancreatic cancer, prostate cancer, or microsatellite stable (MSS) colorectal cancer, or other histologies where clinical evidence exists that single-agent inhibition of PD-L1/PD-1 has minimal activity will not receive single-agent durvalumab but may be eligible to receive this agent with chemotherapy (Arms 2-7). * Women who are pregnant or breastfeeding. * Patients who are receiving any other investigational agents. Patients on other trials will be eligible as long as they are no longer receiving study treatment. * Active or prior documented autoimmune or inflammatory disorders (including inflammatory bowel disease \[e.g., colitis or Crohn's disease\], diverticulitis \[with the exception of diverticulosis\], systemic lupus erythematosus, sarcoidosis syndrome, or Wegener syndrome \[granulomatosis with polyangiitis, Graves' disease, rheumatoid arthritis, hypophysitis, uveitis, etc.\]). The following are exceptions: * Patients with vitiligo or alopecia * Patients with hypothyroidism (e.g., following Hashimoto syndrome) stable on hormone replacement * Any chronic skin condition that does not require systemic therapy * Patients without active disease in the last 5 years may be included but only after consultation with the study physician * Patients with celiac disease controlled by diet alone * History of idiopathic pulmonary fibrosis, pneumonitis (including drug induced), organizing pneumonia (e.g. bronchiolitis obliterans, cryptogenic organizing pneumonia, etc.), or evidence of active pneumonitis on screening chest computed tomography (CT) scan. Patients with active tuberculosis (TB) are also excluded. * Current or prior use of immunosuppressive medication within 14 days before the first dose of durvalumab. The following are exceptions: * Intranasal, inhaled, topical steroids, or local steroid injections (e.g., intra articular injection) * Systemic corticosteroids at physiologic doses not to exceed 10 mg/day of prednisone or glucocorticoid equivalent dose of another steroid * Steroids as premedication for hypersensitivity reactions (e.g., CT scan premedication) * Patients should not be vaccinated with live attenuated vaccines within 30 days before starting or after completing durvalumab treatment. * Patients who have a history of seizures will not be eligible, unless they have either not had seizures or have been on stable doses of anti-seizure medicine and had no seizures for 4 weeks, in which case they will be eligible. Patients taking enzyme-inducing anticonvulsants (i.e., carbamazepine, fosphenytoin, oxcarbazepine, phenobarbital, phenytoin, primidone) will only be eligible for Arm 1 (durvalumab monotherapy) and Arm 2 (durvalumab + gemcitabine). * Patients receiving warfarin are not eligible for Arm 4 (capecitabine) due to the potential for life-threatening interactions. Patients on warfarin are eligible to enroll in one of the other arms provided there is increased vigilance with respect to monitoring international normalized ratio (INR). * Patients with uncontrolled intercurrent illness including, but not limited to psychiatric illness/social situations that would limit compliance with study requirements, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, myocardial infarction in the past 6 months, invasive fungal infections, or active (acute or chronic) or uncontrolled severe infection, liver disease such as cirrhosis, decompensated liver disease, and active and chronic hepatitis (i.e., quantifiable hepatitis B virus \[HBV\]-deoxyribonucleic acid \[DNA\] and/or positive hepatitis B surface antigen \[HbsAg\], quantifiable HCV-ribonucleic acid \[RNA\]), are not eligible to participate. Testing for HBV-DNA and HCV-RNA will be mandatory for patients with hepatocellular carcinoma (HCC) only; testing for hepatitis B or other infections for eligibility will be performed only if clinically indicated. * History of grade \>= 2 infusion reactions or allergic reactions to humanized monoclonal antibodies. Exception: patients with a history of grade 2 infusion reactions to checkpoint inhibitors may be eligible if resumption of prior therapies with pre-medications has been documented without recurrence of infusion reactions of any grade; those patients should receive the same pre-medications with the first and subsequent doses of durvalumab. * History of primary immunodeficiency. * History of allogeneic organ transplant.